What Recall@k Misses: A Complementary Retrieval Check

Mean reciprocal rank (MRR) scores each query by the reciprocal of the rank of the first relevant result: 1 if the target is first, 0.5 if second, 0.33 if third, and so on. You then average across queries (the Evidently AI video on RAG evaluation). A worked example from a retrieval metrics explainer shows three queries where the top result lands at ranks 1, 2 and 4: the reciprocal scores are 1, 0.5 and 0.25, giving an MRR of 0.583 (the metrics explainer).

MRR is the right complement to recall when your queries need one specific chunk, for example "what is the refund window for enterprise plans" or "which error code does TS-999 refer to". It punishes the exact failure recall forgives: the correct chunk retrieved but ranked low.

Run the two metrics side by side on a small test set and the diagnosis writes itself:

That second row is the case recall cannot see. A system with recall@5 of 0.95 and MRR of 0.28 is retrieving the right material but ordering it badly, and the cheapest fix is usually a reranking step rather than new embeddings. Venelin Valkov's advanced retrieval pipeline tutorial walks through exactly this architecture: hybrid search over vectors and full-text, combined with reciprocal rank fusion, then a FlashRank reranker before the top chunks reach the model (the advanced retrieval pipeline tutorial).

When to use nDCG instead

MRR only credits the first relevant result, which is fine when one chunk answers the query. Many RAG queries need several chunks: a comparison, a policy plus an exception, a figure plus its footnote. In that case use normalized discounted cumulative gain (nDCG), which sums graded relevance across the ranked list while discounting each item by its position (the Toloka RAG evaluation guide). It rewards putting several useful chunks near the top, not just one. A medium tutorial on RAG evaluation lists nDCG alongside MRR and MAP as the standard offline position-aware metrics (the RAG evaluation walkthrough).

Decision rule: single-answer lookups → MRR. Multi-chunk answers with graded relevance → nDCG@k. Both → report recall@k plus one of them, never recall alone.

A minimal scoring helper looks like this:

def mrr(ranked_ids, relevant_ids):
    for position, doc_id in enumerate(ranked_ids, start=1):
        if doc_id in relevant_ids:
            return 1.0 / position
    return 0.0

def coverage(ranked_ids, fact_to_chunks):
    hit_facts = {f for f, chunks in fact_to_chunks.items()
                 if set(chunks) & set(ranked_ids)}
    return len(hit_facts) / len(fact_to_chunks)

The first function returns a position score. The second returns the fraction of required facts your retrieved set touches, which is the metric most teams are missing.

Here is the failure that both recall@k and MRR miss. A user asks a question that requires two facts: one from the pricing page and one from the contract terms. Your test set records one "golden chunk." The pipeline retrieves it at rank one. Recall@1 is 1.0, MRR is 1.0, and the answer is still wrong because the second required fact never made it into the context window.

Evidently AI's guide describes this as a context quality question: given the retrieved context as a whole, is there enough information to answer the user's question? (Evidently AI). The guide also notes that you can ask an LLM judge to produce that verdict at scale, which makes coverage practical to run in a regular evaluation loop (Evidently AI).

The coverage check turns that idea into a number. For each query in your test set, list the facts needed for a complete answer and the chunks or passages that supply each fact. Then, for the retrieved set, count how many of those facts are represented:

• Required facts: 3 (price, contract term, cancellation window).
• Retrieved chunks: 5, containing facts about price and contract term only.
• Coverage: 2 / 3 = 0.67.

Set a threshold, for example 0.9, and fail the query in CI when coverage drops below it. This catches the failure mode where the right chunk is present but the right set is not.

Coverage is also where freshness problems surface. A retrieval index built before a document changed will still return a chunk that matches the query, so recall looks healthy. The chunk is simply out of date, and the fact it supplies is stale. A coverage check tied to the current source of truth will flag the gap as soon as the fact is no longer present in any current chunk. The Galileo guide makes the related point with chunk attribution: a high-quality retrieval set that the generator ignores or that contains the wrong content still produces a weak answer, and measuring the connection between retrieved content and the final output is the only way to see it (Galileo).

Building the coverage test set

1. Sample 50 to 200 queries from real traffic logs. The DataAIHub guide recommends a golden set in that range built from your actual corpus and expected queries (the DataAIHub guide).
2. For each query, write the minimum set of facts a correct answer must contain. One to four facts per query is typical.
3. Map each fact to the chunks that contain it, and record the document version those chunks came from.
4. Score every retrieval change against recall@k, MRR (or nDCG) and coverage.

Evidently AI's video walks through the alternative path: rather than writing the dataset by hand, you can generate synthetic question and answer pairs from your corpus and use an LLM judge to label relevance and context sufficiency (the Evidently AI video). That is often the faster route to a usable coverage set for a large corpus.

Both hidden failure modes, buried chunks and missing facts, trace back to how the corpus was chunked and how it is searched.

Anthropic's post explains the core problem: traditional RAG removes context when encoding information, which causes the system to fail to retrieve relevant information from the knowledge base (Anthropic engineering). A chunk that reads "the company's revenue grew by 3% over the previous quarter" does not say which company or which quarter. A query about a specific company and period will not rank it well, so recall@5 drops or, worse, passes at rank five and fails at answer time (the Contextual Retrieval implementation writeup).

Exact-match failures cause the other side of the problem. Anthropic gives the example of the query "Error code TS-999": an embedding model may surface general error-code content while missing the exact string (Anthropic engineering). Venelin Valkov makes the same point: semantic embeddings miss specific IDs, acronyms and keywords, which is why hybrid search adds full-text search and combines the two result lists with reciprocal rank fusion (the retrieval pipeline tutorial).

Reranking then decides whether the right chunk actually lands at rank one. The Claude Cookbook's pipeline uses Cohere for reranking on top of hybrid retrieval as its final quality step (the Claude Cookbook). Once a reranker is in the pipeline, recall@k is effectively pinned: the same candidate set arrives, refined in order. Only a position metric can tell you whether the reranker is earning its latency.

Take a 100-query test set for a support assistant over product documentation and contracts. Before adding a reranker:

• Recall@5 = 0.90 (90 queries have the target chunk in the top five).
• MRR = 0.31 (targets are spread deep across the list).
• Coverage at threshold 0.9 = 0.68 (32 queries are missing at least one required fact).

The recall number looks fine, so a team relying on it alone would ship. Coverage reveals that a third of queries cannot be answered completely no matter where the chunks rank.

Now add a reranker over 20 candidates:

• Recall@5 = 0.90 (unchanged, as expected).
• MRR = 0.74.
• Coverage = 0.71.

The reranker fixed the ordering problem but only marginally improved coverage, because reordering cannot conjure a missing fact. That tells you the next investment is chunking or query expansion, not a bigger reranker. The Evidently AI guide's separate retrieval and generation framing supports this sequence: fix what the retriever finds before you tune what the model does with it (Evidently AI).

If you are generating synthetic queries or running an LLM judge across several models to score these metrics, Quanta Inference puts text and embedding models behind one OpenAI-compatible API, which keeps the evaluation harness simple when you compare backends.

The most common mistake is tuning prompts against a retrieval metric. If recall@k or coverage is low, prompt changes cannot help: generation quality cannot exceed retrieval quality (the DataAIHub guide). The second mistake is scoring retrieval and generation together, which makes it impossible to tell which stage failed.

A practical checklist before any retrieval change ships:

The only metric you should gate a deploy on is one that fails when a user would fail, and that is the combination, not any single number (the DataAIHub guide).