Costing a Retrieval Eval: Golden Queries and Judge Calls

The honest answer is a range, because the number tracks the size of your decision. For a first eval, 50 to 200 golden queries is the right window. Fewer than 50 makes a single bad query move the aggregate by more than two percentage points, which is too noisy to act on. More than a few hundred before you have shipped a first pass means you are burning time on data you cannot yet interpret.

Use the margin of error to choose. If you have N queries, one failing query changes your recall@5 by 1/N. At 50 queries that is 2 points per query; at 200 it is 0.5 points. Write a table of that relationship before you write the queries:

The split within those queries matters as much as the total. A practical composition is 60 percent realistic user phrasings pulled from support tickets or search logs, 25 percent edge cases (ambiguous questions, questions that should return nothing, questions with a single correct document), and 15 percent multi-hop questions where the answer needs two retrieved documents joined together.

Coverage across the corpus is the second constraint. Retrieval evaluation depends on a ground truth set of queries paired with known relevant documents, often called qrels in information retrieval, and the metrics mean little if those pairs only touch a tenth of your corpus evaluating the retrieval pipeline. Count distinct source documents your queries touch and add queries until you cover the top of the access distribution.

If you draft queries with a model, treat the generated set as a first pass and edit it. A community thread on building a golden dataset with DeepEval's Synthesizer describes generating QA pairs from source documents and then building test cases from the real pipeline output golden dataset thread. That two-step flow (generate, then run the real system to capture what it actually returned) is the right shape, and it keeps the human review focused on the questions rather than on writing them from scratch.

Once the qrels exist, most of the eval is arithmetic. All of the following run offline and cost nothing beyond compute you already have.

Rank-unaware metrics count relevant items in the top k. Precision@k is the number of relevant items in the top k divided by k. Recall@k is the number of relevant items retrieved divided by the total number of relevant items for that query. Both fall between 0 and 1, and both are cheap to compute with a loop over your test cases retrieval evaluation metrics.

Rank-aware metrics care about position. MRR takes the reciprocal of the rank of the first relevant document and averages it across queries. NDCG discounts relevance by log of the position, so a relevant document at rank 1 counts far more than the same document at rank 9 retrieval evaluation metrics.

You do not need to implement these. The pytrec_eval package is a Python interface to TREC's trec_eval tool and computes precision, recall, MAP, MRR and NDCG from a qrels dictionary and a run dictionary retrieval evaluation metrics. A minimal offline check looks like this:

import pytrec_eval

qrels = {"return policy electronics": {"doc_return_301": 1, "doc_return_302": 1}}
run = {
    "return policy electronics": {
        "doc_return_301": 0.94,
        "doc_shipping_12": 0.88,
        "doc_return_302": 0.71,
    },
}

evaluator = pytrec_eval.RelevanceEvaluator(qrels, {"recall.5", "recip_rank", "ndcg_cut.5"})
print(evaluator.evaluate(run))

Alongside those, add deterministic checks that need no judgment at all:

• Exact match or normalized match against a short reference answer, for factual lookups where one string is right.
• Retrieval latency at p50 and p95, measured against an explicit budget.
• Payload and response shape checks: required fields present, no unexpected fields, response size under a cap. These are the same class of checks that catch latency creep and field bloat in an API contract performance contract tests.
• Whether retrieval returned zero results when the golden query was labeled "should return nothing", which is the check most teams forget and the one that catches an over-eager threshold.

One caveat worth knowing before you trust a single number: classic IR metrics assume a human reads results in order, but an LLM consumes all retrieved passages at once and can be distracted by irrelevant ones. A recent paper proposes a utility-based metric that accounts for this and reports better correlation with end-to-end answer accuracy retrieval evaluation for LLMs. For a costing exercise the takeaway is practical: NDCG and MRR are fine as regression signals, but do not assume a higher nDCG always means a better final answer. That is part of why the judge bucket exists.

Every judge-based check is one prompt that contains three growing pieces: the instructions and rubric, the question, and the retrieved context (plus the generated answer if you are grading faithfulness). Input tokens dominate, and they are easy to estimate before you spend anything.

The formula for one call:

input_tokens  = rubric_tokens + question_tokens + context_tokens + answer_tokens
output_tokens = expected_verdict_tokens
total_tokens  = input_tokens + output_tokens

Worked example with round numbers you can adjust to your own measurements. Suppose your rubric is a 250-token instruction block, the question averages 30 tokens, the retrieved context is 6 chunks of 400 tokens each (2,400 tokens), and the generated answer averages 300 tokens. That is 2,980 input tokens. If the judge returns a short verdict plus a one-sentence reason, budget about 80 output tokens. Call it 3,060 tokens per faithfulness judgment.

Now scale it. At 100 golden queries and two judge checks per query (faithfulness and answer relevance), you are at 200 calls, which is roughly 612,000 tokens per run if every call carries the same context. Drop the context to 4 chunks of 300 tokens and the same 200 calls come to about 400,000. The lever is context length, not the number of queries, which is why trimming retrieved chunks before judging saves more than shrinking the golden set.

This is the same measurement-before-enforcement discipline that token budgeting for AI products relies on: log prompt tokens, completion tokens, model, feature and environment for every request before you set hard limits, because the distribution tells you which workflow drives the spend setting token budgets. Instrument the judge calls the same way and you will learn within one run whether a full eval is a five-dollar exercise or a fifty-dollar one, without guessing.

Two practical ways to cut the bill without weakening the eval:

The last row is the most valuable. Order your pipeline so the offline checks run first and the judge calls only fire when the deterministic metrics pass. A run that breaks recall@5 should never spend a token on grading answers.

The comparison most teams skip is running the offline metrics and the judge scores side by side on the same queries. When they disagree, you learn something concrete: a query where recall@5 is 1.0 but the faithfulness judge flags an unsupported claim points at a generation prompt problem, not a retrieval one, and vice versa. Braintrust frames RAG evaluation as measuring retrieval quality, context utilization and answer grounding as separate components precisely because each fails independently RAG evaluation metrics.

You can get that side-by-side comparison cheaply by reusing one harness for both. The harness that runs the retrieval returns the top-k documents, the offline scorer computes recall and MRR against the qrels, and the judge scorer receives the same retrieved context. Keeping the inputs identical is what makes the two numbers comparable.

A parallel pattern from large refactors applies here too. When migrating a component, you can route a subset of traffic to the new implementation, compare outputs against the old one, and only cut over once accuracy crosses a threshold branch by abstraction walkthrough. Do the same across retriever versions: run both on the same golden set, diff the retrieved document IDs per query, and store the diff. The queries where the two versions disagree are the only ones worth judging with a model, which cuts judge spend to the disagreements rather than the whole set.

This is also where a single API for every model type helps. If your retriever uses an embedding model and your judge uses a chat model, running both through Quanta Inference removes the per-provider accounting and makes the per-call token logging above a single stream rather than a reconciliation job.

Use this as a gate. Each unchecked line is a reason not to run judge calls yet.

The discipline that keeps the suite honest over time is the same one that keeps a contract test suite from rotting: never add expectations you cannot justify, and fail the build when a new unlabeled query slips in without a qrels entry Pact consumer tests. A golden set that grows without labels drifts into an unmeasured set.

If writing the harness itself is the bottleneck rather than the budget, Quanta Skills lets you install a skill into your coding agent so the offline scorer, the token estimator and the judge wrapper arrive as working code you then adapt, rather than a week of scaffolding.