Writing a Regression Query for Every Retrieval Bug

A judged query in this scheme is deliberately smaller than a formal information retrieval test collection. It has four required fields and three optional ones.

Two design decisions matter more than the rest.

First, store the user's exact query, not a cleaned-up version. Rewriting the query into something well-formed destroys the signal, because the failure often lives in the phrasing: missing punctuation, a plural where the index expects singular, a brand name spelled two ways. Google's guidance on RAG testing emphasizes that test sets should include variations in phrasing and question complexity that match real-world use cases (Google Cloud). A real complaint is the purest form of that.

Second, name the expected document by ID, not by a snippet. Snippets drift when documents are edited, and the assertion then fails for the wrong reason. If your chunker changes granularity, keep a stable parent document ID and let the assertion check parentage rather than an exact chunk. Record the chunk ID too, in a comment, so a future reader can see which passage the user should have seen.

For the failure reason, write the diagnosis, not the symptom. "Answer was wrong" is useless six months later. "Retrieved the unopened-item policy because the word return appears 11 times in that section and twice in the correct one" tells the next engineer what lever to pull: chunking, section-level metadata, reranking, or query expansion.

If you need a second opinion on whether two people would label the same case the same way, treat it as an annotation agreement problem. Krippendorff's alpha is a chance-corrected reliability coefficient that works with any number of raters and tolerates missing labels, and 1 means perfect agreement while 0 means agreement no better than chance (Appen). For a small incident suite you can skip formal alpha, but the underlying discipline (write the labeling rule down, apply it identically) is worth borrowing. The measurement literature is clear that vague category boundaries are the main driver of rater disagreement (BMC Medical Research Methodology).

Suppose a support assistant over a corpus of 4,200 chunks serves about 60,000 retrieval queries a month. A user reports that when they ask how long they have to return an opened item, the bot says 30 days. The correct policy says 14 days for opened items, 30 for unopened.

Walk the case through five steps.

1. Capture. The engineer pastes the exact query, the chunk the pipeline returned, the chunk that should have been returned, and the one-line diagnosis into a YAML file named after the incident.
2. Assert. Add a check that policy-returns-en-v3#section-2 appears in the top 3 results. Pick top 3 rather than top 1 if a reranker reorders aggressively; pick top 1 if the passage is unambiguous and you want to hold the line.
3. Fix. Suppose the fix is adding section headings to each chunk before embedding. The assertion now passes at rank 1.
4. Measure the cost. Run the full suite before and after the fix. If recall over the suite moves from 0.81 to 0.86, the fix helped six cases out of roughly thirty. That is a number you can show.
5. Ship guarded. The assertion runs in CI, so any future change that regresses it fails the build.

The arithmetic here is the insight most first-pass articles skip. If each incident costs about an hour to capture and assert, and the fix removes a class of failure that was hitting some fraction of those 60,000 monthly queries, the return depends entirely on how many queries share the failure. A failure affecting 0.5 percent of traffic is 300 queries a month. A failure affecting 0.01 percent is six. You do not need to know the exact rate to justify an hour of work, but you should record your estimate in the failure reason, because it tells you which suite entries deserve a hard top-1 assertion and which can tolerate top-3.

There is a second number worth tracking: the ratio of new incidents to queries served. If the suite is at 40 cases and you are adding 10 a month, you are in healthy territory. If you are adding 30 a month, the pipeline has a systemic problem and the suite is growing faster than you can fix root causes. That ratio is a better health signal than any single benchmark score, because it is measured on your own corpus. Splunk's write-up on embedding model choice makes the same argument: public benchmarks screen candidates, but only an evaluation on your corpus and queries finds the fit, and embedding failures are rarely solved by swapping encoders alone (Splunk).

A minimal file per incident keeps the cost of capture low enough that people actually do it:

# tests/retrieval/incidents/INC-4412.yaml
id: INC-4412
query: "how long do i have to send back a opened item"
expected_doc: "policy-returns-en-v3#section-2"
expected_answer: "14 days from delivery"
failure_reason: >
  Retrieved the unopened-item section (30 days) because "return" appears
  there far more often; the correct opened-item section is shorter and
  ranks 7th. Estimated affected share: ~0.4% of returns queries.
assert:
  must_contain_in_top_k: 3
tags: ["returns", "en", "chunking"]

One file per incident means the diff for a new regression test is reviewable in seconds, and git blame on a file tells you which incident it guards.

The argument for CI is straightforward regression logic: changes should not introduce anomalies in existing functionality, and the tests covering the modified and affected parts should run on every change (Digitate). Retrieval makes this more urgent than in ordinary software, because the behavior depends on artifacts you do not control and swap casually: the embedding model, the reranker, the chunker, the tokenizer, and the generation model.

Swapping any one of those is exactly the kind of change that reintroduces old failures, and the reintroduction is silent. The system still returns answers, still looks fluent, and still fails on the case someone reported four months ago. Nondeterminism makes this worse: the same prompt can produce different outputs across calls, so a manual spot check after a swap proves very little (MLflow).

There is a direct precedent for gating a rollout on a measured slice of traffic rather than a manual review. Canary deployments route a controlled percentage of requests to a new version so that the release is observed on real traffic before full promotion; Cloudera's inference documentation, for example, describes setting a traffic value such as 35 percent to divert a fraction of requests to a newly deployed model version while the remainder stays on the previous one, then promoting to 100 when satisfied (Cloudera, AWS API Gateway). Feature flags generalize the same idea to runtime behavior, letting you turn a code path on or off without redeploying (Harness).

Apply the pattern to retrieval in two stages:

• Pre-deploy gate. The incident suite must pass before the new embedding model or reranker is promoted past your staging environment. This is the cheap, high-signal check.
• Post-deploy canary. Route a small share of live traffic to the new configuration and compare retrieval metrics from production traces, with a one-flag rollback if the suite's failure classes reappear.

The suite is what makes the canary interpretable. Without a fixed set of judged queries, a dip in a live metric tells you something changed but not what broke; with the suite, you can name the failure class immediately.

For the comparison of what to assert on, the tradeoffs look like this:

The practical recipe is a per-incident assertion (document in top k) plus one suite-level floor (recall at k must not drop below the last accepted value). The per-incident check catches the specific old bug; the floor catches death by a thousand small regressions.

If you supplement with an LLM judge, do it carefully. Rubric-based relevance frameworks such as TRUE exist precisely because prompt-sensitive judging is hard to reproduce, and the authors describe evaluating documents against label-specific rubrics so that judgments become auditable and stable across prompts and datasets (arXiv, ACM DL). For a regression suite, the cheapest reliable judge is the document ID, not a model.

A regression suite degrades in three predictable ways, and each has a cheap countermeasure.

Documents change. The policy page gets rewritten and the expected chunk ID no longer exists. Countermeasure: assert on the parent document where possible, and run a suite linter in CI that fails when an expected_doc points at nothing. A broken reference is a bug in the test, and it should fail loudly rather than be skipped.

The corpus grows and old cases become trivially easy. Countermeasure: track the suite's pass rate by tag and watch for tags that always pass. Those cases have served their purpose; keep them (they cost little) but put your maintenance effort where failures still cluster.

Labels drift as the product definition changes. What counted as the correct answer six months ago may now be wrong because the policy changed. Countermeasure: the failure reason field is dated by the incident, so a reader can see the original reasoning and decide whether the expectation still holds. When you change an expectation, note why in the commit message, not by silently editing the YAML.

One more discipline pays off: only change one variable at a time between test runs, and do not change the questions, reference answers, or system-wide parameters while you are evaluating a fix (Google Cloud). The incident suite makes this easy, because the questions are frozen in files and a diff shows exactly which expectation moved.

If you find yourself disagreeing about what the correct document is, that is a labeling-rule problem, not a retrieval problem. Write the rule into the repository, next to the tests.

If you are one or two engineers maintaining a retrieval system, the incident suite is not overhead, it is your institutional memory. The person who fixes INC-4412 may be gone in a year; the file is not. The same logic applies to the tooling: keep the runner boring. A script that loads the YAML files, calls your retrieval function, and checks rank is a few dozen lines. You do not need a platform to start.

What you do need is a way to run the suite against multiple configurations without editing code. This is where a single interface for several model types helps: you can point the same suite at a candidate embedding model, a candidate reranker, and a candidate generator, and compare pass rates on identical judged queries. Quanta Inference exposes text, image, video, audio and embedding models through one OpenAI-compatible API key, which keeps the configuration matrix in one place. For teams that want a specialist to own the retrieval evaluation alongside the build, Quanta's build service pairs you with someone who ships the site or app with you.

Whichever tooling you pick, the decision rule stays the same. Capture the query, name the document, write the reason, assert the rank, and run it on every change. The suite grows from failures you actually had, which is the only sample you can trust.