Measuring Retrieval Recall After a Document Refresh

Recall@k answers one binary question per query: did at least one chunk I labeled relevant appear in the top k results? A practitioner-oriented tutorial computes it as the count of queries whose target appeared in the top k divided by total queries, and walks through a 100-query example where recall@1 came out at 0.52, recall@3 at 0.71, and recall@5 at 0.83 on the same system (Rajnish Khatri). Note what this implies: the same index can show a large drop at k=3 and a small one at k=10 because the metric only cares about presence inside the cutoff.

This is why recall@k is the right primary metric for a rebuild gate. It is set-level, so a reranker that reorders correct chunks without evicting them will not trip the gate. Rank-quality metrics move on every ordering change and generate noise you cannot act on. Use MRR or NDCG as secondary signals, not as the promotion criterion.

There is a second reason to keep the metric narrow. Retrieval quality is separable from generation quality, and many teams conflate the two and end up prompt-engineering a retrieval problem. A golden set of query-document pairs lets you gate every retrieval change on recall before you touch prompts or models (DataAIHub). Evidently's RAG evaluation guide draws the same line: evaluate retrieval with ground-truth ranking metrics like recall@k, and evaluate generation separately (Evidently AI).

Decision rule: if a query's labeled relevant set is empty or unlabeled, exclude it from the recall computation and report it as "unjudged" rather than counting it as a miss. Silent unjudged queries inflate drops and make the gate unreliable.

You cannot diff what you keep changing. Before any rebuild, snapshot three things: the query text, the labeled relevant chunk IDs per query, and the index parameters that produced the current baseline (splitter, chunk size, overlap, embedding request configuration, fusion weights, k values).

Label granularity matters. Anthropic's cookbook evaluation set contains 248 queries, each with a single golden chunk, and measures pass@k by checking whether the golden document appears in the first k results (Claude Cookbook). Single-golden-chunk labeling is cheap but brittle for a rebuild diff, because a re-chunking can split the golden chunk into two smaller pieces, neither of which matches the original ID. Where a document is genuinely relevant, label all passages that answer the question, not just the one your current chunker happens to produce. That is what makes the metric survive a boundary change.

If you have nothing labeled, you have three routes. Manual labeling of top-k results is what Google-style relevance rating programs do at scale, and LLM judges can stand in for a share of it (Evidently AI). Synthetic query-answer pairs bootstrapped from the corpus get you a first set faster, as does an LLM-judged relevance pass over retrieved chunks. Whatever route you take, freeze the labels before the rebuild so old and new indexes are scored against identical ground truth.

Rebuild into a new collection or namespace with a name that includes the parameters, for example docs_ctx_v2_k512_ov64. Keep the serving index untouched. Every retrieval call during the diff period takes a namespace parameter, so you can send the identical query to both.

def recall_at_k(hits, gold_ids, k):
    top_k = {h["chunk_id"] for h in hits[:k]}
    return 1.0 if top_k & set(gold_ids) else 0.0

def evaluate(index, queries, k=5):
    out = {}
    for q in queries:
        hits = index.search(q["namespace"], q["text"], top_k=k)
        out[q["id"]] = recall_at_k(hits, q["gold_ids"], k)
    return out

Run the evaluation on both indexes with the same k values. Pick your k from the pipeline, not from habit: Anthropic's pass@k evaluation used k equal to the number of chunks sent to the model, ten in that example. If your prompt budget carries five chunks, gate on recall@5 and report recall@3 and recall@10 as context.

Two operational notes. First, if you use a hosted inference layer for embeddings, keep the embedding request configuration in version control alongside the namespace name; a changed input type or truncation setting is a silent variable. Second, if your chunks embed external context, generate that context once and store it with the chunk record, for example the contextual prefix pattern described in the Anthropic guide. Regenerating context prefixes with a different prompt on the new pass is another uncontrolled variable.

Aggregate recall hides the thing you need to fix. A system that goes from 0.78 to 0.80 overall can still have thirty queries that fell off a cliff and forty that improved. Diff at query level.

A worked example with 200 labeled queries and k=5:

• Old index: 156 queries had a labeled chunk in the top 5. Recall@5 = 0.780.
• New index: 159 queries hit. Recall@5 = 0.795.
• Per-query diff: 41 queries dropped by one hit, 44 gained, and 115 were unchanged.
• With a 0.05 threshold applied to a per-query binary score, every change counts, so you flag all 85 changed queries. That is too many.

The threshold has to match the metric's granularity. For binary recall@k, use the threshold on the aggregate plus a second rule on the counts: block promotion if aggregate recall@k falls at all, and review every dropped query. For graded per-query scores (share of labeled chunks retrieved), a 0.05 drop threshold is sane. Two gates are worth running:

Report the diff with the queries sorted by largest drop, and attach the old top-k IDs, the new top-k IDs, and the labeled relevant IDs side by side. That table is the artifact you inspect in the next step.

Not every dropped query deserves equal attention. Use this order, and stop each branch when the cause is clear.

First, inspect the labeled relevant chunks for dropped queries. In a rebuild, these are the chunks that used to carry the answer. Check whether they still exist in the new index, whether their boundaries changed, and whether the new tokenizer truncated them. Contextual prefix changes, chunking parameter changes, and truncation are all silent and all show up here first. This is where the Contextual Retrieval numbers matter: a prefix change alone can move thousands of chunks' vectors at once.

Second, inspect chunks that changed rank across the k cutoff. These are the near-misses: relevant chunks that sat at position 3 in the old index and position 7 in the new one, or the reverse. Pull them by chunk ID from both namespaces and compare the new chunk text to the old. If the text is identical and only the vector moved, your embedding request or model changed. If the text differs, your splitter changed.

Third, inspect chunks with the highest query frequency. Count how many of your labeled queries point at the same chunk ID. A chunk cited by fifteen queries is worth more of your time than a chunk cited by one. Frequent chunks are also the best candidates for adding explicit context prefixes, because the fix compounds across every query that needs them.

Fourth, inspect chunks that grew or shrank past the k cutoff by more than one position. Log the chunk's token count in both indexes and its retrieval rank per query. A chunk that dropped from position 4 to position 9 usually means competition from neighboring chunks that split off in the new pass, which is a boundary problem rather than an embedding problem.

Fourth step, but worth stating as a separate check: review any query where the new index returned a labeled chunk the old one missed in a different chunk ID with the same source document. That is a boundary win, not a regression, and it should be logged as an improvement in the review queue so the next diff does not re-flag it.

Set the promotion rule before you look at results. A defensible one: promote the new index when aggregate recall@k is at or above the old index for every gate k, no query drops below the aggregate floor, and the unjudged count is zero. If you must trade, define which k value wins: for most RAG systems recall@k matters most at the k that matches your prompt budget, so weight recall@5 over recall@10.

A worked promotion decision. Old index recall@5 is 0.780. New index, first attempt, is 0.760 with 41 dropped queries and 9 of those dropping from hit to miss at k=5. Inspect the 9 first. Suppose seven of them involve a single source document whose sections were split at 512 tokens in the new pass instead of 1,024. Raise the chunk size for that document type, rebuild the namespace again, re-run, and you get 0.790 with 3 dropped queries. That clears the floor and passes, with the 3 drops logged as accepted trade-offs. Only after promotion do you consider re-embedding the full corpus through a new model or a new context-prefix prompt, and it goes through the same diff before it is promoted.

Re-embedding is not the first fix. The chunk-boundary and context-prefix fixes are cheaper, reversible and usually the actual cause.

If you are spending most of this cycle re-plumbing embedding calls across providers, that is a separate problem from retrieval quality. Quanta Inference gives you text, image, video, audio and embedding models through one OpenAI-compatible API key, which keeps the embedding request configuration consistent across rebuilds and makes the namespace-and-diff workflow simpler to run at scale: https://quantaintelligence.ai/inference