Measuring Retrieval Quality: Recall and Precision

Precision@k is the fraction of the top k results that are relevant. Recall@k is the fraction of all relevant items that appear in the top k (Evidently AI: Precision and recall at K). Both are easiest to reason about with binary relevance judgments (Unstructured: Evaluating Search Quality).

The formulas, stated plainly:

• precision@k = (number of relevant items in the top k) / k
• recall@k = (number of relevant items in the top k) / (total number of relevant items for that query)

A worked example

Suppose you have a query "how do I rotate an API key," and human judges mark four chunks as relevant: A, B, C and D. Your retriever returns ten results, and the relevant ones land at positions 1, 2 and 7 (A, B and C). D never appears.

• precision@10 = 3 / 10 = 0.30
• recall@10 = 3 / 4 = 0.75
• precision@3 = 2 / 3 = 0.67 (only A and B are in the top three)
• recall@3 = 2 / 4 = 0.50

This one query shows why you report both metrics at several k values. At k=10 the system finds most of the answer but pads the result list with seven irrelevant chunks. At k=3 it is cleaner but has already lost half the relevant material.

Now average across queries. If you have 100 queries, precision@10 is the mean of 100 per-query precision values, not the total relevant chunks divided by 1,000. Query-level averaging keeps one query with many relevant chunks from dominating the score.

Precision and recall move in opposite directions as you change retrieval breadth: adding candidates raises recall and lowers precision, while aggressive filtering does the reverse (Unstructured: Evaluating Search Quality). So a single number is never the whole story.

A compact implementation for computing both from a golden set is short enough to keep in your repository:

def precision_recall_at_k(retrieved_ids, relevant_ids, k):
    top_k = retrieved_ids[:k]
    hits = len(set(top_k) & set(relevant_ids))
    precision = hits / k
    recall = hits / len(relevant_ids) if relevant_ids else 0.0
    return precision, recall

results = []
for query in golden_set:
    retrieved = retriever.search(query["text"], limit=10)
    p, r = precision_recall_at_k(retrieved, query["labels"].keys(), k=10)
    results.append({"query_id": query["id"], "precision": p, "recall": r})

mean_precision = sum(x["precision"] for x in results) / len(results)
mean_recall = sum(x["recall"] for x in results) / len(results)

Libraries such as ranx cover recall@k, precision@k, MRR, MAP and NDCG through one consistent interface, which saves you from hand-rolling each metric (Qdrant: Measuring Retrieval Relevance).

Choosing k

Pick k to match what your product does with the results. If a chat assistant stuffs eight chunks into the prompt, measure recall@8, because that is your real ceiling on grounding. If a search UI shows ten results per page, measure precision@10, because that is what the user sees first.

A useful pattern is a two-number report: recall@k where k equals your generation context budget, and precision@k at the depth a human would scan. Watch the gap. A large gap means your ranker is pushing relevant material too far down and a reranking stage may be worth more than a better index.

Metrics are noisy. A jump from 0.71 to 0.74 recall@10 across 100 queries may be real or may be query mix. The decision rule is:

1. Change one thing at a time: chunking, embedding model, index type or reranking.
2. Run the full golden set against the unchanged control and the variant.
3. Compare per-query outcomes, not just the mean. Count how many queries improved and how many regressed.
4. Only ship when the mean improves and the regressions are explained. Regressions concentrated in one intent class are a warning sign, not noise.

To make step 3 concrete, store results as a small table keyed by query and configuration:

The chunk-size change earns its complexity: a broad improvement with comparatively few regressions. The index change adds a point of recall, which likely reflects that the approximate index finds slightly different neighbors than the exact scan, not a better representation. Approximate nearest neighbor search trades a little accuracy for speed at scale, so small recall swings between exact and approximate search are expected (Elastic: What is vector search?).

Why chunking changes move these numbers

Chunks that are too large mix several ideas into one embedding, producing a noisy averaged vector that matches nothing precisely. Chunks that are too small lose the context a human needs to interpret them (Weaviate: Chunking Strategies for RAG). Both failure modes show up as lost recall, but for opposite reasons: large chunks never score high enough, small chunks score high for the wrong query.

The practical test for a chunk is whether it makes sense to a human when read alone; if it does, it will make sense to the model (Pinecone: Chunking Strategies). Your labeled set lets you confirm that intuition with a number instead of an opinion.

Index choice is a separate axis from chunking, and your golden set can separate them. Vector indexes use approximate nearest neighbor algorithms that sacrifice perfect accuracy for efficiency in high-dimensional space (Elastic: What is vector search?). In pgvector, IVFFlat divides vectors into clusters and searches only the nearest clusters, while HNSW builds a navigable graph; both are approximations with tunable accuracy, and both can be measured against an exact scan (PGVector: HNSW vs IVFFlat).

The evaluation move is to run one configuration with exact search and the same configuration with the approximate index, holding chunking fixed. The recall difference is your index's accuracy cost, cleanly isolated from every other variable. Qdrant documents this as a distinct concern (ANN recall) from retrieval relevance, precisely because mixing them hides which one regressed (Qdrant: Measuring Retrieval Relevance).

Hybrid retrieval deserves its own row in your comparison table. Sparse keyword scoring such as BM25 excels on exact identifiers, and dense vector search excels on paraphrase, so combining them covers more query types than either alone (Hybrid Search: BM25, Vector and Reranking). Fusion is usually done with reciprocal rank fusion, which scores each document by 1/(k + rank) summed across result lists, so a document ranked highly by both methods wins without any score normalization (OpenSearch: Reciprocal rank fusion).

When you benchmark a hybrid configuration, split your golden set by query type first. Keyword-heavy needs (error codes, parameter names, SKUs) and conceptual needs behave differently, and a single aggregate recall number can hide the case where hybrid helped exact-match queries and hurt paraphrase. Slice-level reporting is how you catch that (Qdrant: Measuring Retrieval Relevance).

Tuning on the test set. It is wrong to report results obtained by tuning parameters to maximize performance on the same collection. Keep a development set for tuning and report on a held-out set for an unbiased estimate (Evaluation in information retrieval).

Unstable labels. If you re-chunk the corpus and your labels were attached to chunk IDs, every label is now invalid. Attach labels to stable source document IDs with a mapping to current chunks, or accept a re-labeling cost each time.

Synthetic query inflation. LLM-generated queries tend to be easier than real ones, so offline scores built only on synthetic data overstate production quality (Qdrant: Measuring Retrieval Relevance).

One-metric tunnel vision. A retriever can find one useful result, miss everything else, rank exact matches well and handle paraphrase poorly, all at once. Pick metrics that match the failure mode you care about rather than treating a single score as a scoreboard (How to Evaluate Information Retrieval Systems).

Ignoring the top of the list. Precision and recall count relevant items in the top k without rewarding where inside that window they appear (Evidently AI). If position within the prompt matters for generation quality, add mean reciprocal rank or NDCG alongside them.

A first evaluation pass, in order:

1. Export 200 production queries stratified across intent classes.
2. Define the retrieval unit and confirm a stable ID exists on every point.
3. Have two people label independently on 30 queries, compare, and write down tiebreak rules until agreement is high.
4. Label the remaining queries. Pool top results from the current system plus one alternative.
5. Compute precision@k and recall@k at two k values per query, then report means and per-intent breakdowns.
6. Commit the golden set and the scoring script to the repository. Treat the label file like test fixtures: reviewed changes only.
7. Re-run on every retrieval change and record the comparison table.

Step 6 is the one teams skip and then regret. A golden set that lives in a spreadsheet dies the moment its owner changes teams. A golden set in version control with a scoring script becomes a regression test suite for the entire retrieval layer.

If you are still assembling the pipeline itself, generating and comparing embeddings for many document types, the per-model API work adds up. Quanta Inference puts text, image, video, audio and embedding models behind one OpenAI-compatible API key, which shortens the loop between "run the golden set against a new embedding model" and seeing the recall number. When the measurement habit is in place, the rest of retrieval tuning becomes an ordinary, auditable engineering workflow.