A Retrieval Test Set for Embedding Model Swaps

Four artifacts, kept in version control:

1. A query list. Real queries, verbatim, with no cleanup that a user would not have typed.
2. A document store. The source documents, keyed by a stable document ID and character offsets, not by chunk ID.
3. Judgments. Three-level relevance scores for query and document pairs.
4. A runner. Code that chunkes the documents under a given scheme, embeds with a given model, retrieves top k, and maps each retrieved chunk back to its document ID before scoring.

That last step is the trick that makes the set reusable. Your index stores chunks, but your judgments reference documents. At scoring time you translate: a retrieved chunk resolves to its parent document, and the judgment for that document applies. Change the chunk size, change the overlap, change whether you split on headings, and the judgments still hold, because they never mentioned chunks in the first place.

The distinction matters because a generation set built for one prompt does not survive a change of prompt, while a retrieval set built for one chunking does survive a change of chunking. That is the property you are buying.

Pull queries from logs, not from imagination. Three sources, in descending order of value:

• Search and support logs. Verbatim strings users typed, including the awkward ones with no punctuation and the misspellings.
• Follow-up turns in conversations. Multi-turn dialogue changes what a query means, and later turns retrieve worse than early ones: rewriting the query with conversation context lifted one retriever's Recall@5 from 0.30 to 0.37 on a 110-conversation benchmark averaging 7.7 turns (Splunk, citing MTRAG).
• Synthetic queries, used sparingly. An LLM can draft a question from a passage, which is a useful way to cover thin areas of the corpus. Treat these as filler, and mark them so you can report scores with and without them.

Then, for each query, find the passage that should answer it. Do it by hand for the first few dozen. The person who picks the passage is also the person who assigns the relevance level, which keeps the two decisions coherent.

Size: 80 to 200 queries will separate models that differ meaningfully, and a 120-query set is a reasonable first target for a single product surface. Below about 50, a single query flipping its rank moves the metric by more than two points, which is noise pretending to be a decision.

A word on the boundary. Retrieval is not just semantic similarity: a query can be answered only by a document from the right period, and that constraint is better enforced with metadata filtering than with a better encoder (Splunk). If your test set is full of queries whose answers depend on a date, you are measuring your filter, not your embedding model. Decide which system you are testing before you label.

Two levels are too coarse and five levels are too fine for two raters to reproduce. Use three, and define them by what the passage lets the reader do.

Three rules keep the labels stable:

• Judge the passage alone. Do not reward a passage for the company it keeps in the ranked list.
• Judge against the query as written. If the query is ambiguous, that ambiguity is part of the test.
• Write the level and a one-sentence reason. The reason is what you review when the model disagrees with the rater.

Rubric-based relevance judgment done this way is what the TRUE framework formalizes, scoring passages on intent alignment, coverage, specificity, accuracy and usefulness rather than a single gut check, and it is explicitly built to be reproducible across prompts and datasets (Dewan et al. on arXiv, the WSDM paper). You do not need the full framework, but the principle carries: define the rubric before you see the model's output, or you will rationalize the output.

Check agreement before you trust the labels. Have a second person label a sample of 40 query and document pairs. Then compute Krippendorff's alpha, which handles any number of raters and missing labels and works on ordinal scales (Appen's explainer). Krippendorff's own convention treats 0.800 as the threshold for reliable conclusions and 0.667 as the floor for drawing tentative ones, and the ordinal variant is the right setting for a three-level ranked scale (Appen). If you land in the tentative band, the usual fix is adding boundary examples to the rubric, not adding raters (Appen). For a formal treatment of which agreement coefficients suit nominal and ordinal data and how to attach confidence intervals, see Zapf et al. in BMC Medical Research Methodology, and for the mechanics of building the agreement table by hand, Real Statistics' worked example.

One caution worth carrying into the set itself: an audit of ten widely used machine learning test sets found average label error rates of at least 3.3%, enough to reorder model rankings once corrected (Appen, citing Northcutt et al.). Your labels are not exempt.

With documents retrieved and mapped back to IDs, score every query against the judgments.

Recall at k is the fraction of relevant documents for that query that appear in the top k results. If a query has three documents at level 2 and two of them are in the top 10, recall at 10 for that query is 0.67. Average across queries for the set-level number. Recall is the metric that matters when missing a relevant item carries real cost (Openlayer).

Reciprocal rank is 1 divided by the rank of the first relevant result. If the first level-2 document sits at rank 3, the reciprocal rank is 0.33. Average across queries and you have mean reciprocal rank, which rewards models that put the most relevant result highest (Openlayer). NDCG handles graded relevance more fully; use it when you want partial credit to be visible in the headline number.

def recall_at_k(ranked_doc_ids, relevant_ids, k):
    top = set(ranked_doc_ids[:k])
    if not relevant_ids:
        return None
    return len(top & relevant_ids) / len(relevant_ids)

def reciprocal_rank(ranked_doc_ids, relevant_ids):
    for i, doc in enumerate(ranked_doc_ids, start=1):
        if doc in relevant_ids:
            return 1.0 / i
    return 0.0

Two implementation notes. Treat a query with no relevant documents as excluded, not as a zero, or you will drag the average down for a labeling gap. And pass the relevance threshold explicitly: counting level 1 as relevant inflates recall for every model equally, so pick one definition (level 2 only is the honest default) and state it alongside the number.

A worked comparison. Say you run 120 queries, and for simplicity each has exactly one level-2 answer document. Model A scores recall at 10 of 0.78 and MRR of 0.52. Model B scores recall at 10 of 0.81 and MRR of 0.61. Model B wins on both. Now look at model C: recall at 10 of 0.82, MRR of 0.44. Model C finds slightly more answers, but buries them deep in the list, which means larger context windows, more tokens per request and a higher chance that a distractor crowds out the answer. Model B is the safer swap. That comparison is invisible if you report one number.

Report the pair, always.

The judgments are the asset; the index is disposable. Run the swap as an expand and contract change, the same pattern used for risky schema migrations: add the new structure alongside the old, write to both, read from both, compare, then remove the old one once nothing depends on it (Prisma's data guide). The pattern exists because coupling two changes in one deploy doubles the risk and makes rollback hard (PlanetScale on backward compatible changes), and because at any intermediate step you can roll back without losing data (Tim Wellhausen's paper). Applied here: build the new index, evaluate it against the frozen judgments, keep serving from the old one, then cut over.

The mechanical steps:

1. Freeze the labeled set and tag it with a version.
2. Re-chunk the document store under the new scheme. Do not touch the judgments.
3. Embed with the candidate model and build a parallel index.
4. Retrieve top k for every query, map chunks to document IDs, score recall at k and MRR.
5. Compare against the baseline and against the rubric's known agreement level.
6. Cut reads over only if the numbers hold, and keep the old index warm until they do.

For gradual cuts, route a fraction of traffic to the new index and compare on live queries. Canary releases exist precisely so a small slice of traffic absorbs the risk before the whole population does, and the traffic split is adjustable when you want to widen it (AWS API Gateway canary deployment guide), with the model-serving version of the same idea letting you send a set percentage to a new version and the remainder to the previous one (Cloudera's canary deployment docs). Feature flags do the same job for the code path: flip the retrieval backend at runtime without a redeploy, and revert instantly if the swap misbehaves (Harness on feature flags). In LLM systems flags are better treated as configuration than as booleans, because a model choice carries cost, latency and safety settings together (MLflow on feature flags for LLM systems).

Three maintenance rules keep the set honest over time:

• Retire stale queries. Anything whose answer document no longer exists gets a note, not a silent deletion.
• Add queries from new failures. Every production miss you investigate becomes a candidate row, which is how the set stays representative.
• Re-check agreement when the rubric changes. A rubric edit invalidates the old alpha; re-sample before you compare new results to old ones.

For a broader map of evaluation approaches, including how graders, verifiers and leaderboards differ in what they can and cannot tell you, Sebastian Raschka's overview of LLM evaluation methods is a good companion read. And note that swapping encoders solves fewer problems than teams hope: temporal mismatches and constraint-ignoring usually need metadata filtering, reranking or multi-granularity chunking rather than a different model (Splunk). The test set is what tells you which of those you actually need.

When you are ready to run candidates side by side, Quanta Inference puts text, image, video, audio and embedding models behind one OpenAI-compatible API key with prepaid US dollar pricing, so the same runner can point at each model in turn.