Feeding a Stopped Review Back Into the Eval Set

A failing final answer is the least useful part of an incident. For agents and retrieval systems, the failure often lives in the execution path: the wrong tool was selected, an argument was malformed, the retrieved chunk was stale. Regression datasets built from production need the entire trajectory, including the user request and conversation history, the system prompt and its version, tool calls with inputs and outputs, retrieval results and which documents were not pulled, intermediate decision points, the final response, and metadata such as model version and session identifiers Arthur on regression datasets from production failures. If you store only input and output, the root cause disappears and the case becomes impossible to reproduce.

Before any of that becomes a fixture, it has to be sanitized. Strip names, emails, account numbers and any regulated data; remove credentials and keys; replace live database IDs and server URLs with deterministic placeholders or mocks; and capture whatever external state the failure depended on so it can be replayed. Where the original content is too sensitive to retain, rewrite it into a synthetic equivalent that preserves the failure pattern. A booking agent that confused Portland, Maine with Portland, Oregon does not need the real customer record; it needs a case asserting the agent asks a clarifying question when the destination is ambiguous Arthur.

Capture also needs a taxonomy, or the fixture set turns into an unlabeled pile. Tag each incident with a primary failure mode (wrong tool, bad retrieval, unsafe output, format violation, missing refusal) so similar issues cluster, priority becomes visible, and the dataset stays balanced across real weaknesses instead of being dominated by one loud category.

The labeling step is where most regression programs quietly fail, because "the answer should be better" is not a test. Behavior has to be specified in a form a machine can judge, or with a documented scoring rubric when only a human or a judge model can decide.

The decision rule that works: assert on what the system must do, not on how it says it.

Each row mixes deterministic assertions (tool name, schema, citation presence) with bounded checks where exact matching is impossible. The hybrid approach is standard advice: the scoring method matters less than the case being stable enough to fail again if the behavior returns NHI Mgmt Group.

Write the "reason it failed" field as prose aimed at a future maintainer. In six months someone will look at an odd-looking test and want to delete it. A one-paragraph note linking to the incident, naming the user harm, and stating the decision rule ("when a destination is ambiguous, ask; never guess") is what stops the guard from being removed because it looked annoying.

A regression case is a code artifact. Keep it in the repository next to the prompts, model configuration and retrieval settings that produced the failure, and review changes to it like any other diff. Anything that lives only in a chat thread or an issue tracker never becomes executable, and therefore never blocks anything NHI Mgmt Group.

A minimal, readable format goes a long way. The example below keeps the case self-contained and cites its origin; the fields are the four the workflow needs plus provenance.

# fixtures/regressions/case-0142-refund-ambiguity.yaml
id: case-0142
created: incident-link-in-ticket
failure_mode: wrong_tool_selection
severity: high
origin:
  source: production_sample
  reviewed_by: support_lead
  reason: >
    Agent issued a refund for order 88213 when the user had asked about
    a different order in the same thread. Root cause: stale conversation
    history was dropped before the tool call.
input:
  messages:
    - role: user
      content: "cancel the second one and refund me"
  history_retained: true
context:
  system_prompt_version: "prompt-7"
  model_config: "config-b"
  retrieved_chunks: []
  tool_mocks:
    order_lookup: {returns: ["88213", "88214"]}
expected:
  tool_calls: []
  behavior: ask_clarifying_question
  must_mention: ["88213", "88214"]
  must_not: ["refund initiated"]
assertions:
  - kind: no_tool_called
    tool: refund_create
  - kind: contains_all
    values: ["88213", "88214"]
  - kind: llm_judge
    rubric: "asks which order the user means, without guessing"
    threshold: 0.8

Two habits keep the set healthy. First, give each case a stable ID and never renumber; reports and dashboards will reference it for years. Second, version the fixture with the artifacts it depends on, so a prompt change that intentionally alters behavior carries the corresponding fixture update in the same pull request. Testing research makes the same argument for ordinary code: test cases must co-evolve with production code, and obsolete tests that no one updates stop revealing faults and start adding maintenance cost Identify and Update Test Cases When Production Code Changes.

That co-evolution rule has a sharp edge for AI systems. When the fix is a policy change, not just an output change, the expected behavior must be signed off by the control owner before it is encoded. Otherwise a "bug fix" case quietly encodes a behavior nobody approved.

The gate is the point of the whole exercise. A fixture that nobody runs is a document. The regression case should execute on every change to the prompt, the model, the retrieval index, the tool definitions, or the orchestration code, and it should fail the build when observed behavior departs from expected behavior.

The simplest wiring is a test that loads every fixture, replays it against the candidate configuration, and asserts:

# tests/test_regression_fixtures.py
import glob, yaml
from eval.harness import replay, assert_behavior

def test_all_regression_cases():
    cases = [yaml.safe_load(open(p)) for p in glob.glob("fixtures/regressions/*.yaml")]
    assert cases, "regression fixture directory is empty"
    for case in cases:
        result = replay(case)          # mocks tools, restores history, sets prompt version
        assert_behavior(case["assertions"], result)   # raises with case["id"] in message

Non-determinism complicates this, and the honest answer is that it is a statistical problem, not a testing-trick problem. An agent that passes today can fail the same case tomorrow. Three controls reduce flakiness to something manageable:

1. Run each case several times and require a pass rate, not a single pass. A case failing 3 of 5 runs is a real failure; one failing 1 of 5 is noise worth watching.
2. Separate deterministic assertions (schema, tool name, citation presence) from judged ones. Deterministic checks should pass every time; judged checks get a threshold.
3. Keep the fixture small and trusted. Regression cases protect specific failures; broad benchmark suites are slower-moving and should not gate individual merges.

Gating on a pass rate also gives you a way to prove the fix before drawing another sample. The workflow is: fix, run the specific case N times, observe N of N passes, merge. The case now stands as the evidence that the failure was addressed, and the gate as the guarantee it will not return. When one component of a composed system is swapped, regression testing selection research shows the same logic: you test the modules that matter and can do so at a fraction of the cost of full revalidation, which is exactly why recording the dependency in the fixture pays off Regression Test Selection for Updated Capability Modules.

Two operational details matter more than they seem. First, run the gate where a failure is cheap: inside the pull request, not after a deploy. Second, record which model, prompt, retrieval configuration and data-pipeline version produced the result, so a regression failure can be attributed to a specific change rather than a vague "it got worse" Bluepes on AI testing and release checks.

Once the gate is live, change how you sample. Draw batches to find new classes of failure, not to re-confirm known ones. Before drawing, filter out inputs that a fixture already covers (hash the normalized input plus the active prompt version) so review time goes to uncovered behavior. This is closer to how acceptance sampling treats a lot: if the sample result is inconclusive, a second stage is taken, and the plan is designed so the second stage only triggers when the first does not decide What is Double Sampling? (NIST/SEMATECH Engineering Statistics Handbook). The principle transfers: spend your review budget on decisions that are still open, not on lots you already decided.

The sample-size math also tells you where to spend effort. Estimating a rare failure rate is expensive: at 50% proportion, a 5-point margin costs 384 samples, and confidence climbs with every added decimal Sample Size Calculator. A 95% confidence interval for a proportion requires the standard error to be small relative to the estimate, and the width shrinks with the square root of n Confidence interval for a proportion. That means doubling sample size buys decreasing precision while doubling fixture count buys linear coverage of known failure modes at zero marginal variance. The fixtures are simply the better investment for defects you have already seen.

When you do draw, use a filter and a target. Keep the sample if you think a new failure class exists (new feature, new traffic source, model swap); skip routine sampling when only the fixture pass rate has moved. And keep a small holdout that fixtures cannot capture, because the long tail will always produce inputs no one wrote a test for.

Suppose a weekly review of 400 sampled support-agent outputs surfaces six failures. One is the refund-ambiguity case above. Here is the loop, with numbers.

1. Capture. Pull the full trace: two order IDs in history, the refund tool call fired against the wrong ID, the final message. Sanitize: no customer name, no payment details, mocked order lookup returning 88213 and 88214.
2. Label. Expected behavior: ask which order, mention both IDs, call no refund tool. Failure mode: wrong_tool_selection. Reason field: stale history dropped before tool call.
3. Store. Commit case-0142 to the fixture directory, linked to the incident. Reviewer signs off the requested behavior.
4. Gate. The test runs on every prompt and model change. It fails now (the bug is live) and on any reintroduction.
5. Prove. After the fix, run the case five times; require five passes. Merge. The case is now permanent.
6. Adjust sampling. The 400-item weekly draw now filters cases covered by fixtures. The next review finds new failure modes rather than rediscovering refund ambiguity.

Six new permanent cases a week over a quarter is 78 gates. Assume each fixture costs 20 seconds per full release run; 78 cases add about 26 minutes to a release, which is cheaper than a single incident review. The economics favor fixtures as soon as a defect is likely to recur.

None of this depends on which model you use, which is the point. The fixture suite should run against whatever candidate configuration is under review, and the models change constantly. That is a reason to standardize the access layer rather than the model: if every case replays through one endpoint, swapping the model under test is a configuration change, not a rewrite. Quanta Inference exposes text, image, video, audio and embedding models through a single OpenAI-compatible API key, so the harness holds one client and the regression run compares configurations cleanly.

The gate itself is a small piece of infrastructure. If you would rather have a specialist stand it up with you, Quanta takes a description of the site, app, document or plan and builds it with you.