Measuring Retrieval Quality: Hit Rate, MRR and Recall@k for RAG

For a single query with a set of labeled relevant chunks, define the retrieved list as the top k results your system returns. Then:

• Hit rate@k (also called Pass@k): 1 if at least one relevant chunk is in the top k, else 0. Averaged over queries, it tells you how often the retriever gives the generator anything useful to work with.
• Recall@k: the number of relevant chunks in the top k divided by the total number of relevant chunks for that query. Averaged over queries.
• MRR: the mean of 1 divided by the rank of the first relevant chunk. A query whose first relevant chunk is at rank 1 contributes 1.0, rank 2 contributes 0.5, rank 3 contributes 0.33, and a query with nothing relevant in the top k contributes 0.

The definitions follow the standard information retrieval formulations described in Evidently's RAG evaluation guide and Anyscale's metrics reference. The same sources note that hit rate is a basic functioning signal, MRR suits factoid questions where the first correct result suffices, and precision@k matters when accuracy beats completeness.

A worked example makes the difference concrete. Take five queries, k=5, and one relevant chunk per query except query 3, which has three relevant chunks at ranks 1, 4, and 9.

Hit rate@5 is 4/5 = 0.80. Recall@5 is (1 + 1 + 0.67 + 0 + 1) / 5 = 0.734. MRR is (1 + 0.33 + 1 + 0 + 0.5) / 5 = 0.566. The hit rate looks healthy, but two of five queries put the answer below rank 2, and one has relevant content at rank 9 that k=5 will never see. If your generator needs three passages per answer, hit rate is flattering you.

This is why hit rate alone is a trap. It answers "is anything there" and stops. Recall@k answers "is everything there," and MRR answers "is it near the top." Report all three, plus the k you actually feed the model, and look at the histogram of first relevant ranks, not just the mean.

MRR has a hard ceiling worth knowing: with one relevant chunk per query, the best possible MRR is 1.0, but a system that is perfect at rank 1 for 80 percent of queries and rank 3 for the other 20 percent scores 0.8 + 0.2/3 = 0.867. If your MRR sits near 0.9, you have almost no room left and should be working on recall and multi passage coverage instead. The formulation of MRR and average precision in a RAG context is walked through in this evaluation write up.

Start with the queries users actually type. A retrieval test set of 200 queries with labeled relevant chunks is enough to see differences of a few points in recall@k; below about 100, one hard query moves the average enough to flip your decision. Anthropic's public contextual retrieval evaluation used 248 queries over nine codebases, each with a golden chunk, and reported Pass@10 rising from about 87 percent to about 95 percent after adding context to chunks (the Claude cookbook walkthrough).

That example is worth studying for method, not for the numbers. They scored pass@k, which is hit rate under another name, held the corpus fixed, changed one variable, and reported the same metric before and after. You can copy that discipline.

Practical steps:

1. Sample queries stratified by intent and by whether they contain identifiers (SKUs, error codes, ticket numbers, statute references). Lexical retrieval shines on identifiers, dense retrieval on paraphrase, and you need both represented in the same set.
2. Hide the query's own document from anyone labeling, so they cannot just point at the source. Force them to judge whether the chunk answers the question.
3. Record the chunk id, not a text snippet. Text drifts; ids stay stable and let you rerun the set after re chunking.
4. Freeze a held out slice, say 20 percent of queries, and only look at it when deciding to ship. Otherwise you will tune chunk size against the same 200 queries until they are memorized.

If hand labeling is too slow to start, generate candidate query answer pairs with a model and have a person verify and correct. Treat generated labels as drafts. Retrieval metrics are only as trustworthy as the labels underneath them, and synthetic sets miss the messy phrasing real users produce, a limitation noted in the RAG evaluation guide from Evidently.

In code, scoring is short. Keep it in version control next to the config it scores:

def score_queries(runs, k):
    # runs: list of (relevant_ids: set, ranked_ids: list)
    hits, recalls, rrs = [], [], []
    for relevant, ranked in runs:
        top = ranked[:k]
        found = relevant.intersection(top)
        hits.append(1.0 if found else 0.0)
        recalls.append(len(found) / len(relevant) if relevant else 0.0)
        rr = 0.0
        for position, doc_id in enumerate(top, start=1):
            if doc_id in relevant:
                rr = 1.0 / position
                break
        rrs.append(rr)
    n = len(runs)
    return sum(hits) / n, sum(recalls) / n, sum(rrs) / n

Report recall at two values of k: your generation cutoff (what the model sees) and a wider candidate cutoff (what a reranker could see). The gap between them is the reranker's opportunity.

To attribute failures cleanly, hold generation fixed and swap only the retrieval configuration. Concretely:

• Pin the embedding model, the chunk boundaries, the index, and the k. Change one thing per run.
• Log for every query: the ranked chunk ids, scores, and retrieval latency. Without the ranked list you cannot recompute a different metric later.
• Score retrieval first. Only then assemble prompts and score generation, using a separate judge or human review for faithfulness to the retrieved context. Anyscale's guide recommends exactly this order, because end to end scores alone cannot tell you which stage failed.
• Keep a no answer bucket. Some queries in production have no relevant chunk in the corpus. If your test set has none, you cannot see a retriever that confidently returns junk for unanswerable questions. Track precision for those separately.

One subtlety: chunking changes both what is retrievable and what "relevant" means. If you experiment with smaller chunks for factoid queries and larger or page level chunks for analytical ones, re label or map labels to the new units before comparing scores. NVIDIA's study found optimal chunk size varied by query type within similar document types, favoring 256 to 512 tokens for factoid queries and page level boundaries for complex analytical ones (NVIDIA). Comparing a 512 token index against a page level index on stale labels is a measurement error, not a model improvement.

Now the question practitioners actually ask: should I fuse a keyword retriever with the dense one, add a cross encoder reranker, or both?

Start from why fusion helps. BM25 and dense retrieval fail in complementary ways. BM25 is strong on exact strings like product codes and error identifiers and blind to paraphrase; dense retrieval is the reverse. Combining them by rank rather than raw score, using reciprocal rank fusion, avoids the problem that BM25 scores are unbounded positive numbers while cosine similarity is bounded between -1 and 1, so naively weighted blends let BM25 dominate (a reference on hybrid search and RRF). OpenSearch's write up makes the same point: rank based fusion is stable across differing score distributions and resistant to outliers, where min max and L2 normalization are not (OpenSearch on RRF). A PostgreSQL walk through shows the same architecture with BM25 and dense vectors fused by RRF inside the database (pgEdge).

Reranking is a different lever. It does not add candidates; it reorders them. A cross encoder scoring each (query, chunk) pair cannot scan a large corpus at query time, so the correct pattern is retrieve a wide candidate set of tens to hundreds, then rerank that shortlist (Qdrant's hybrid search with reranking tutorial). That means reranking can only fix what your first stage already fetched. This is the single most useful fact for the decision: if recall@50 is 0.70 and recall@5 is 0.65, a reranker can at best get you to 0.70. If recall@50 is 0.70 and recall@5 is 0.30, a reranker is the obvious next move.

Anthropic's numbers illustrate the composition. Contextual embeddings reduced failed retrievals by 49 percent, and adding contextual BM25 plus reranking reduced failures by 67 percent (Anthropic). Each layer contributed, and the combined effect exceeded either alone. That is a general pattern, but the size of each step is your own data's business.

So use this rule. Run your frozen test set through the current retriever and each candidate, and record recall@k at your generation cutoff, MRR, and p50 and p95 latency.

1. Compute the ceiling: recall at the wide candidate cutoff of the cheapest configuration. If your best first stage recalls 0.90 at the wide cutoff, no reranker can exceed 0.90 at the generation cutoff.
2. Estimate the reranker's realistic gain as recall@wide minus recall@cutoff, discounted. Ordering gains usually capture part of that gap, not all of it.
3. Do the same for fusion: rerun with the keyword retriever added and RRF applied, and measure the delta in recall@k and MRR over the single retriever, plus the extra latency of running two queries and merging.
4. Ship if the measured gain in recall@k, or in MRR where ordering is what your task needs, is larger than the latency you add, and if the gain survives on the held out slice. A change that moves recall by less than the noise on 200 queries, roughly a point or two, is not a result.
5. If neither fusion nor reranking moves recall@k, the problem is upstream: chunk boundaries, embedding quality, or missing documents. Change the index, not the ranking.

Worked example. Suppose recall@5 is 0.62, MRR is 0.55, recall@50 is 0.81, and p95 retrieval latency is 40 ms. Adding a cross encoder over the top 50 candidates costs, say, 120 ms at p95 and lifts recall@5 to 0.74 with MRR 0.71. Fusion alone costs 25 ms and lifts recall@5 to 0.69 with MRR 0.60. Under the rule, fusion is the cheaper win; the reranker is justified only if your users tolerate the latency and your product depends on ordering, as it does for single answer lookups. If recall@50 had been 0.63, no reranker would have been worth building, and the honest next step would be fixing chunking.

For the latency side, measure on your own hardware and traffic shape. Reranker cost spans orders of magnitude, from a few milliseconds for distilled cross encoders to well over a hundred for large ones on long candidate lists, per a survey of reranker patterns (a comparison of reranker types and costs). The abstraction in that piece also gives the right mental model: a first stage ANN retrieval of a few hundred candidates, then a reranker scoring each against the query, then the top passages to the model.

One more option to test before either: adding context to chunks. Prefixing each chunk with a short note about where it sits in the document, generated by a model, improved top 20 chunk retrieval failure rate by 35 percent averaged across data sources in Anthropic's testing (the Claude cookbook). It is an indexing change rather than a query time change, so it costs nothing in latency. If your chunks read as fragments, try it before adding infrastructure.

Offline metrics decide what to build; production signals tell you whether it worked. Capture the query, the ranked chunk ids, the retrieval latency, and whether the answer was accepted (thumbs up, citation clicked, follow up question, escalation). Do not compute hit rate in production, since you have no labels there. Instead watch the shape of the distribution: the share of queries where the model was given zero chunks below a relevance threshold, and retrieval latency at p95, both of which move before quality complaints arrive.

Log the ranked list itself, not only the final answer. When a user reports a bad answer, you can then check whether the needed chunk was in the list, which turns a vague complaint into either a labeling task or an engineering one, and you can add that query to the frozen set.

When you need to swap the embedding model or reranker without rewriting client code, running the models behind one endpoint keeps the evaluation harness unchanged while the backend changes. A single API in front of text, image and embedding models is the kind of setup that makes swapping one variable per run practical; Quanta Inference is that option if you would rather not wire a separate SDK per provider.