How to Write a Retrieval Test Set from Real Questions

Three sources cover most teams, and each has a distinct bias that you have to correct for during sampling.

Support tickets give you the questions users ask when something is broken or confusing. They are phrased in user vocabulary, often with typos and partial context, which is exactly what your retriever will face. They over-represent edge cases and angry customers, and under-represent the boring questions people answer themselves. Zendesk's guidance on writing knowledge content emphasizes using the wording your users use and repeating the topic from the title inside the body, which is easier to apply once you can see the actual ticket phrasing side by side with your article text (Zendesk help center).

Search logs give you volume and intent. A query typed into your site search with zero clicks is a strong candidate for a retrieval failure: the person asked, got nothing useful, and left. Search logs are short, sometimes truncated, and dominated by navigational queries (people typing a product name to find a page), which you should filter out.

Sales emails give you questions from a different population: prospects evaluating you, asking about limits, integrations, pricing structure and compliance. These are often multi-part ("do you support SSO, and what's the setup effort?") and they map to documents your support team never touches, such as security pages and contract FAQs. Sales emails over-represent pre-purchase concerns and can be oddly formal.

A workable extraction routine for each source:

The shares are a starting point, not a rule. The point is to write them down, because the mix is a claim about your traffic and you want it visible when someone asks why the test set scores 0.7 while customers are still unhappy.

Raw extraction will hand you thousands of near-duplicates. "Reset password" will appear 40 times in slightly different words. If you label all 40 you have a password-reset test set, not a retrieval test set.

Start with deduplication by meaning, not by string. Normalize to lowercase, strip punctuation, and group queries by an embedding similarity threshold or by a simple keyword rule set. Then sample one representative per group, keeping the shortest and clearest phrasing of each, because the goal is a query a human can grade consistently.

Next, size the set. Fifty to a hundred questions from real logs is a reasonable starting range that a team can grow (Olostep). For comparing chunking schemes, 100 to 200 is better: you want enough queries per intent bucket that a 5 point difference in recall@5 is visible above label noise. A practical split is roughly 60 percent common questions, 25 percent tail questions (low volume but real), and 15 percent questions you expect to be unanswerable from your corpus.

That last bucket matters more than people expect. A retrieval system that confidently returns a plausible but wrong passage is worse than one that returns nothing. Include questions your corpus genuinely cannot answer, label them with an empty expected-passage list, and score refusal separately. This is the same discipline Evidently describes when it recommends designing test cases that include unanswerable and adversarial inputs alongside the happy path (Evidently RAG evaluation guide).

One caution on synthetic questions: they are useful for widening topic coverage, but generated wording drifts from how users type. Compare synthetic questions against real logs by phrasing, length and topic before you keep any (Olostep). My recommendation is to keep the test set entirely real and use synthetic questions only for a separate stress set, so your headline number stays honest.

Classic precision and recall assume each record is either relevant or irrelevant, and that assumption is explicitly noted as a limitation: documents can be marginally relevant, very relevant or completely irrelevant, and what counts as relevant varies by reader (Creighton University search guide PDF). Retrieval evaluation inherits the same problem. A passage that mentions refunds is not the passage that states your refund window.

Use three grades:

• 2 (answers it): the passage contains the fact needed to answer the question completely, with no other passage required.
• 1 (supports it): the passage is on topic and contributes context, but cannot answer the question alone. A related policy, a prerequisite step, a definition used by the answer.
• 0 (not relevant): anything else, including passages that look similar by keyword but are about a different product, region or plan.

Two rules make this scale reliable across labelers. First, grade at the passage level you will actually store, and re-grade when chunking changes, because a grade 2 at 256 tokens may become two grade 1 chunks at 128 tokens. Second, write a short answer string for each query yourself. If you cannot write the answer from memory, you do not know which passage is grade 2 yet.

The payoff of the middle grade is that it gives you a ranking-quality signal. Binary labels make recall@k the only honest metric. With grades you can compute normalized discounted cumulative gain (nDCG@k) and mean reciprocal rank (MRR), which reward putting a grade 2 passage above a grade 1 passage. nDCG@k rewards highly relevant results appearing at the top and is the right metric when multiple levels of relevance matter; MRR@k focuses on the position of the first relevant result and suits tasks where finding the correct answer quickly is critical (Deepchecks RAG evaluation paper).

A worked comparison makes the difference concrete. Suppose ten queries, and for one configuration you retrieve top 5 chunks per query. Five of the fifty returned chunks are grade 2 and ten are grade 1. Binary recall@5 against "any grade 2 or 1 counts as relevant" is identical for a configuration that returns all ten grade 1 chunks first and one that puts five grade 2 chunks at positions 1 and 2. Precision at 5 also agrees, because it counts relevant items in the top k divided by k (Evidently, precision and recall at K). Only a graded metric separates the two, and the second configuration is obviously the one you want to ship.

Here is the labeling checklist I use before trusting a set:

Keep the format boring. One JSON Lines file, one judgment per line. The fields that earn their place: a stable query id, the exact query text, the source, an intent tag, the expected answer string, the passage id, the grade, and the corpus version the passage came from.

{"query_id":"q0042","query":"can I change my billing address after the order ships","source":"ticket","intent":"billing","answer":"Yes, until the carrier scans the parcel. Address changes after the first scan require a carrier intercept.","judgments":[{"passage_id":"kb-billing-03#p2","grade":2,"doc_version":"2026-03-11"},{"passage_id":"kb-shipping-11#p1","grade":1,"doc_version":"2026-03-11"}],"unanswerable":false}

Three design choices to make deliberately.

Store passage ids, not passage text. The chunk text lives in your index. If you store text in the test set you will eventually compare against a stale copy and get confusing scores. If you want the text for readability, store it in a separate build artifact keyed by passage id.

Store the document version. When a policy page changes, the version tells you which judgments need review, and you can compute coverage and freshness by joining the test set to the index.

Store the query as typed, and optionally a normalized copy. The typed version tests realistic spelling. The normalized version is useful when you debug embedding behavior. Do not replace the typed version: it is the thing your users actually did.

Then write the scorer once and keep it fixed:

def score(run, judgments, k=5):
    # run: {query_id: [passage_id, ...]} in rank order
    # judgments: {query_id: {passage_id: grade}}
    hits, ndcg = 0, 0.0
    for qid, ranked in run.items():
        grades = judgments[qid]
        top = [grades.get(pid, 0) for pid in ranked[:k]]
        hits += 1 if 2 in top else 0
        dcg = sum((2**g - 1) / math.log2(i + 2) for i, g in enumerate(top))
        ideal = sorted(grades.values(), reverse=True)[:k]
        idcg = sum((2**g - 1) / math.log2(i + 2) for i, g in enumerate(ideal)) or 1.0
        ndcg += dcg / idcg
    return {"answer_recall_at_k": hits / len(run), "ndcg_at_k": ndcg / len(run)}

Reporting answer recall at k (does a grade 2 passage appear in the top k) alongside nDCG@k gives you both a blunt "can it be answered" number and a ranking-quality number. When you swap chunking schemes, run both against the identical file and compare per intent bucket, not just overall. Chunk size effects are not uniform: smaller chunks of 64 to 128 tokens were optimal for datasets with concise fact-based answers, while chunks of 512 to 1,024 tokens improved retrieval where broader context was needed, and different embedding models showed distinct sensitivities to chunk size (arXiv chunk size study). Averages hide exactly this, which is why the intent tag belongs in the file.

A test set rots in three ways, and each has a cheap guard.

Stale answers. Every judgment carries a document version. When a source page changes, the join flags the affected queries into a review queue. Set a review cadence that matches how often your sources change, and re-label outside the schedule whenever a monitored page is edited (Olostep).

Test set overfitting. If you tune chunk size against the same 150 queries for a month, you will optimize to those 150. Hold back 20 percent of the queries, never look at their scores during tuning, and check them before you ship. When the held-back score diverges from the tuning score, your chunking choice has stopped generalizing.

Coverage gaps. A question whose answer exists nowhere in your index is a coverage failure, not a retrieval failure, and no ranking change can surface text that was never indexed (Olostep). Report coverage as a separate number: the share of queries with a grade 2 passage that resolves in the current corpus build. This one number prevents a whole class of wasted tuning.

One more discipline: version the test set itself. A file named retrieval-evals-v1.jsonl with a dated changelog lets you compare a score from three months ago to a score today without guessing what moved. When you add queries, add them to a new version and re-score the old version too, so you can see whether the new queries are harder.

Once the file exists, chunking becomes an experiment rather than an opinion. You swap the splitter, rebuild the index, run retrieval on the same queries, and read two numbers per intent bucket. The fixed judgments also let you check whether the generation step is the bottleneck: if a grade 2 passage is in the top 5 and the answer is still wrong, no amount of re-chunking will help.

If running the same query set across several embedding, reranking or generation models is the next step, Quanta Inference puts text, image, video, audio and embedding models behind one OpenAI-compatible API key, priced in US dollars, which keeps the comparison cheap to script (https://quantaintelligence.ai/inference).