Evaluating a RAG Pipeline End to End: Golden Sets, Judges, Drift

Retrieval metrics measure whether the right evidence reached the context window. Evidently's RAG evaluation guide separates retrieval from generation for exactly this reason: ranking metrics with ground truth tell you whether the retriever found useful information, independent of what the model did with it (Evidently's RAG evaluation guide).

The two metrics worth running on every commit:

• Recall@k: the share of questions where at least one required passage appears in the top k results. Recall@5 of 0.82 means 82 of 100 questions had a required passage in the top five.
• MRR (mean reciprocal rank): for each question, take 1 divided by the rank of the first required passage (1 if it is first, 0.5 if second, 0.333 if third), then average. MRR punishes putting the right passage at position 9 even when recall@10 is perfect.

A worked example. Take 10 questions, retrieve top 5 for each, and mark where the first required passage lands. The tutorial walkthrough of recall@k with a recipe corpus shows the per-question scoring: a target at position 3 counts as a failure for recall@1 and a success for recall@3 and recall@5, while a target at position 23 fails all three (recall@k and MRR explained).

Read the three numbers together. Recall@5 of 0.60 says four questions never saw their evidence in the top five, and no prompt change can fix those. MRR of 0.488 says that even among successes, a lot of evidence sits at rank 3 or 4, where a tight context budget may still cut it. If your generation step only feeds the top 3 passages to the model, q2 and q7 are effectively lost, and your real recall is closer to 0.30.

Log the full ranked list at a generous depth (top 50) so you can recompute recall at any k without re-running the pipeline. Store question_id, ranked_passage_ids, config_hash and corpus_version per row.

This is where the framework pays for itself. A reranker is a second-stage model that rescores a candidate list from first-stage retrieval, and cross-encoders read the query and document jointly, which is why they reorder well (cross-encoder mechanics and integration). Typical production shapes: retrieve 50 to 100 candidates, rerank, pass 3 to 10 chunks to the model, with reported precision gains of 15 to 35 percent and 20 to 150 milliseconds of added latency (the same integration guide).

A comparison of retrieval and rerank depths, with MRR@10 and latency reported per configuration, shows the shape you should expect: retrieving 20 and reranking to 5 gave MRR@10 of 0.612 at 80 milliseconds, 50 to 10 gave 0.683 at 105 milliseconds, 100 to 10 gave 0.695 at 125 milliseconds, and 200 to 10 gave 0.698 at 180 milliseconds, which is where the curve flattens (reranker benchmark summary). The decision rule: find the knee. If doubling the candidate pool buys less than the noise band of your golden set, stop at the smaller number.

Splunk's writeup on reranker selection adds two failure modes to test for specifically: phantom hits and context truncation, both often fixed by adjusting how text is fed to the reranker rather than by swapping the model (Splunk on selecting a reranking model). Add a handful of golden questions built around negation ("does not require") and rare exact terms (error codes, parameter names) so these cases are visible in the numbers.

Because retrieval is measured before generation, you can accept a reranker on evidence. Run the golden set three times: first-stage only, first-stage plus reranker A, first-stage plus reranker B. Compare recall@k and MRR, and log latency and token cost. A reranker that lifts MRR from 0.49 to 0.71 while recall@5 stays at 0.60 has improved precision, not coverage, and the four missed questions still need a different fix.

Answer scoring answers a different question: given whatever context was retrieved, did the pipeline produce a correct, grounded, complete response? Split this into two graders.

Deterministic checks first. Extract every number, date, currency amount, product code and named entity from the answer and compare against the reference answer and the retrieved passages. This catches the failure mode no judge needs to arbitrate: a model that says the SLA is 4 hours when the passage says 24. These checks are cheap, exact, and run on every commit. Keep a per-question list of tokens that must appear (or must not appear) so the checker has something to compare against.

A rubric judge second. Use a language model as a judge with three scores per answer, each 1 to 5:

LLM-judged evaluation is a standard part of RAG evaluation practice, alongside reference-based comparison to correct answers and reference-free checks for faithfulness and completeness (Evidently's guide). Redis frames the same three dimensions as context relevance, groundedness and answer relevance, and notes that missing any one of them means you are optimizing the wrong bottleneck (Redis on RAG evaluation).

Three rules make the judge trustworthy:

1. Freeze the prompt. Store the judge prompt in the repo with a version hash. Any run that mixes prompt versions is not comparable.
2. Grade blind. Do not pass the judge the config name or which variant produced the answer. Order the answers randomly within a batch.
3. Calibrate against humans once. Have a person score 30 answers with the same rubric, then measure agreement with the judge. If the judge and the person disagree on a category more than a quarter of the time, fix the rubric wording before trusting any pipeline comparison.

Keep the judge model fixed even when you upgrade the model under test. If the generator and the judge both change, you cannot tell which one moved the score.

Evaluation only pays off when it blocks a bad deploy. Treat the golden set like a test suite: run it in CI on every change to chunking, embedding model, index parameters, rerank depth, prompt or generator model.

Index parameters belong on that list because approximate nearest neighbor search trades recall for speed. IVFFlat partitions vectors into clusters and searches only nearby ones, and HNSW builds a graph that trades build time and memory for search quality; changing lists, probes or the HNSW m and ef_construction settings can quietly move recall (pgvector indexing deep dive). The explainer on IVFFlat in pgvector describes the same mechanism: queries touch only the closest clusters, so the index is approximate by design (IVFFlat in pgvector). A golden run after an index rebuild is the only cheap way to catch a silent recall drop.

Hybrid search changes need the same gate. Reciprocal rank fusion merges ranked lists by position rather than score, which sidesteps the problem that BM25 scores are unbounded while cosine similarity is bounded, and rank-based fusion is more resistant to outliers than min-max or L2 normalization (OpenSearch on RRF). Adding a keyword retriever to a dense pipeline changes rankings in ways reasoning will not predict, so measure it (MongoDB on RRF and hybrid search). PostgreSQL users can now generate sparse vectors alongside dense embeddings and fuse them with RRF inside the database (hybrid search in PostgreSQL).

A minimal gate:

THRESHOLDS = {"recall_at_5": 0.80, "mrr": 0.65, "faithfulness": 4.2}

def gate(baseline, candidate):
    for metric, floor in THRESHOLDS.items():
        if candidate[metric] < floor:
            fail(f"{metric} below floor: {candidate[metric]:.2f} < {floor}")
        if candidate[metric] < baseline[metric] - TOLERANCE[metric]:
            fail(f"{metric} regressed vs baseline: "
                 f"{baseline[metric]:.3f} -> {candidate[metric]:.3f}")

Set tolerances from observed variance: run the same unchanged pipeline three times, take the spread, and set the tolerance at roughly twice that spread. If repeated runs vary by two points, a one-point drop is noise and gating on it will make the suite untrustworthy and eventually ignored.

Comment on the PR with the metric table and the list of questions that flipped from pass to fail. That list is the most useful artifact in the whole system, because it usually points at one chunking decision or one query type.

Golden-set scores measure a frozen snapshot. Production changes underneath it: new documents land, old ones are edited, users ask questions nobody wrote down. Split monitoring into three streams.

Corpus drift. Track the share of golden-set required_passages that still resolve to an existing chunk. When documents are re-chunked or deleted, that number falls, and a falling number means your scores are partly stale rather than the pipeline getting worse.

Query drift. Cluster production questions and compare the distribution with the golden-set strata. When a new cluster grows past a set share of traffic, promote five to ten real examples into the golden set with hand-labelled required passages. Grow the set; never silently edit history.

Score drift. Run the golden set on a schedule against production configuration and alert on movement beyond the tolerance band. A slow slide is easy to miss in per-query logs and impossible to miss in a chart.

The discipline that ties all of it together: treat eval data, judge prompt, chunker config and index parameters as versioned artifacts shipped together. When someone asks why last month's answers were better, the answer is a diff, not a debate.

If your challenge is running the same golden set across several generators or rerankers without rewriting provider code for each one, Quanta Inference exposes text, image, video, audio and embedding models through one OpenAI-compatible API key with prepaid USD pricing: https://quantaintelligence.ai/inference.