How to Evaluate a RAG Pipeline: Retrieval and Answer Metrics

Retrieval metrics come from decades of information retrieval work and do not require an LLM to compute, which makes them cheap, fast, and repeatable Evidently's RAG evaluation guide.

Recall@k asks a binary question per item: did at least one gold passage appear in the top k chunks the retriever returned? The fraction of items where the answer is yes is your recall@k. It is a pass/fail check per query, so it is simple to explain to stakeholders and hard to game Recall@k and MRR explained.

Mean reciprocal rank (MRR) asks a different question: how high did the first gold passage rank? Each item contributes 1 divided by the rank position of the first gold chunk, and you average across items. A gold chunk at position 1 scores 1.0, at position 3 it scores 0.33, and an item with no gold chunk retrieved scores zero Recall@k and MRR explained.

Here is a worked example with 5 items and k=5, so you can see how the two numbers diverge:

Recall@5 is 4/5 = 0.80. MRR is (1.00 + 0.33 + 0.20 + 0.00 + 0.50) / 5 = 0.41. A recall of 0.80 looks reassuring, but the MRR says the gold chunk typically sits in the middle of the pack, which matters because chunks placed low in the prompt are the ones the model is most likely to ignore.

That last point is not folklore. Retrieval research has documented a lost-in-the-middle effect, where relevant information buried inside long context is missed even when it was technically included Pinecone on chunking. MRR is the cheap proxy for that risk.

A small script computes both:

def retrieval_scores(results, k=5):
    # results: list of (retrieved_ids, gold_ids) per question
    hits, rr = 0, 0.0
    for retrieved, gold in results:
        top = retrieved[:k]
        ranks = [i + 1 for i, doc in enumerate(top) if doc in gold]
        if ranks:
            hits += 1
            rr += 1.0 / min(ranks)
    n = len(results)
    return hits / n, rr / n

Report recall@k for several values of k (1, 3, 5, 10) in one table. The gap between recall@3 and recall@10 tells you whether the right chunk is being found but ranked too low, or not found at all. The first problem is a ranking problem and a reranker or a larger k with reranking can fix it. The second is a coverage problem: chunking, embeddings, or a missing document.

Retrieval changes are where most of the wins are, and the metrics tell you which change to try.

If recall@10 is low, the chunk is not being retrieved at all. The usual causes are chunking and vocabulary. Chunks that are too small lose the context needed to be found, and chunks that are too large dilute the embedding so that the chunk matches everything weakly Pinecone on chunking. NVIDIA's chunking experiments found page-level chunking gave the best average end-to-end accuracy across their tested datasets, while optimal token sizes varied by document type, with some financial datasets favoring 512-token chunks and others 1,024 NVIDIA's chunking study. The transferable lesson is to measure chunk sizes on your own set rather than copying a default.

If recall@10 is fine but recall@1 or MRR is poor, the problem is ranking. Two fixes are well documented. The first is hybrid search: combine dense vector search with keyword search such as BM25, which catches exact identifiers, error codes, and rare terms that embeddings smooth away. Anthropic reports that combining embeddings with BM25 and adding reranking reduced failed retrievals by 67 percent in their tests Anthropic's contextual retrieval post. Google Cloud describes the same pattern, with hybrid search plus a re-ranker scoring results to push the most relevant to the top Google Cloud's RAG overview.

The second is contextual embeddings: prepend a short description of the source document to each chunk before embedding it, so a chunk that says "revenue grew by 3 percent" carries which company and which quarter it refers to. In Anthropic's cookbook this raised Pass@10 from about 87 percent to about 95 percent on their codebase dataset, and reduced the top-20 retrieval failure rate by 35 percent the Claude Cookbook.

Reranking fits well here because it is cheap: you run an expensive scoring model over a small candidate set already returned by a fast retriever, keeping latency low Qdrant's reranking tutorial. In metric terms, reranking cannot improve recall@10 (the candidates are already fixed) but it can lift recall@3 and MRR.

Use this checklist when reading your numbers:

With retrieval measured, turn to generation. Feed the pipeline the retrieved chunks, but for debugging also run a control condition where you force-feed the gold passages as context. If answers are good with gold context and bad with retrieved context, the problem is retrieval. If they are bad in both, the problem is the prompt or the model, and no amount of retrieval tuning will help.

Compare the generated answer against the reference answer with two classic text-overlap metrics. Exact match scores 1 if the normalized answer string equals the reference and 0 otherwise, which is appropriate for short factual answers such as a date, a number, or a name. Token-level F1 computes the overlap of tokens between the generated and reference answers, treating it as a precision and recall problem over the shared tokens, and is more forgiving of phrasing differences. Use exact match sparingly: if your reference answer is a full sentence, exact match will be near zero for reasons that have nothing to do with quality.

Overlap metrics have a blind spot. They cannot tell you whether the answer is supported by the evidence. That is what a faithfulness check does: take each factual claim in the generated answer and verify it against the retrieved passages. A simple version asks a judge model, given the answer and the retrieved chunks, to mark each claim as supported, contradicted, or not found. Google Cloud's evaluation tooling scores generated text and retrieved chunks on metrics that include groundedness and question answering quality, which is the same idea in product form Google Cloud's RAG overview. Open source tooling such as Ragas offers faithfulness and answer relevancy as ready-made checks the Label Studio walkthrough.

A practical scoring sheet keeps the four numbers side by side: recall@k, MRR, F1 against reference, and share of answers with all claims supported. Faithfulness is the one to watch in production, because an ungrounded answer is the failure users cannot detect for themselves.

Aggregate numbers tell you whether to keep a change. Failure analysis tells you what to change next. Export every item where recall@5 missed, where a gold chunk was retrieved, or where faithfulness dropped, and read them in batches of 20.

Sort failures into four buckets:

1. Not in the corpus. The answer does not exist in the knowledge base. Fix the knowledge base, not the pipeline.
2. In the corpus, not retrieved. A gold passage exists but never appears in the top k. Chunking, embeddings, or keyword coverage.
3. Retrieved, answer wrong. The evidence was there and the model still missed it. Prompt, context ordering, or too many distractor chunks.
4. Retrieved, answer unsupported. The model added claims not in the passages. Faithfulness problem, usually a prompt instruction problem.

Label found in the box.

The distribution across buckets is your roadmap, and it changes as you fix things. As Anthropic's numbers show, a retrieval change that looks small in aggregate (a 35 percent reduction in failure rate) can shift a large share of user-visible errors the Claude Cookbook.

Two habits keep analysis cheap. Keep the evaluation set in version control alongside the prompt, so every number is tied to a known configuration. And re-run the full set, not just the failing items, after each change, because fixes for ranking can quietly cost you recall.

If the pipeline improvement effort is competing with other work and you need a faster route to a working system, a managed inference layer lets you swap embedding, reranking, and generation models through one API without rewriting the harness. Quanta Inference (https://quantaintelligence.ai/inference) offers text, image, video, audio and embedding models through one OpenAI-compatible key, which makes the A/B comparisons in this article cheap to run.

Evaluation is not a one-time project. Use three tiers.

Per change. Run the full labeled set on every prompt, chunking, embedding, or model change. It takes minutes if you cache retrieved chunks and only regenerate answers.

Weekly. Sample 20 production questions, label them, and add any clear failures to the set. This keeps the set aligned with real usage.

Per release. Run a stress pass with adversarial inputs: questions about topics outside the corpus, ambiguous questions, and questions whose answer changed recently. The point is to confirm the system declines when it should.

For a stopping rule, pick thresholds before you tune, so you are not tempted to move the goalposts. A defensible starting point: recall@5 at 0.85 or higher, MRR at 0.60 or higher, and faithfulness at 0.95 or higher, with exact match or F1 set according to whether your answers are short facts or prose. Your domain will move these numbers, but writing them down converts an argument into a measurement.

Track the four headline numbers in a table over time so improvements and regressions are visible together:

The hybrid change lifted recall, the reranker lifted MRR without touching recall, exactly as the mechanism predicts. When your own table shows that pattern, you know both fixes worked for the reasons you expected.