Replaying Recorded Error Payloads Against Every Model

Most evaluation harnesses store an expected output string. That is the wrong unit here. Your model's job in an error path is a decision, so the fixture should name the decision space explicitly and let you compare decisions rather than prose.

Good fixture design keeps the schema small and typed. Langfuse's description of a golden dataset is a versioned set of cases where each case captures inputs, context, and a description of what a good response does, including behavioral expectations like "must refuse and explain why" rather than a rigid expected string the Langfuse golden dataset guide. For error handling, "a good response" is an action plus parameters.

Two rules keep this honest. First, redact at capture time, not at replay time, so a fixture can never leak a token into a prompt. The Microsoft API error guide lists avoiding exposure of database details, keys, and credentials in error messages as a core practice the Microsoft guide. Second, keep the schema flat. Agents parse flat JSON far more reliably than deeply nested objects, and that principle applies to your harness output as well as your tool responses the agent-friendly CLI guide.

An agent loop treats exit codes and error codes as control flow. The agent checks the outcome and decides what to do next, which is why tools that return a generic failure force the agent to parse free text and get it wrong sometimes the agent-friendly CLI guide. Your harness should invert that: define the control flow first, then measure whether each model produces the same one.

Three actions cover almost every error path:

• Retry: send the same request again, optionally after a delay derived from a header.
• Fix: change a named parameter, such as replacing an invalid field value or correcting a path.
• Escalate: stop and hand the case to a human or a different system.

Exit codes are the precedent. Linux reserves specific exit ranges, and the Advanced Bash Scripting Guide proposes keeping user-defined codes in a dedicated range so that a failure means one thing consistently the exit code table. Programmers still argue about whether sysexits is worth following, but the disagreement is about which codes, not whether a shared vocabulary makes automated handling easier the Hacker News thread. Your action set is that vocabulary for model routing.

A decision rule that holds up in practice: retry only when the error is transient and the response says how long to wait. HTTP clients are expected to retry 500 responses and to refresh credentials on 403, and 429 and 503 should always carry a Retry-After header Speakeasy's error design guide. 409 Conflict and 422 are fixable or terminal, never retryable the RFC 9457 walkthrough. Encode those as scoring rules, not vibes.

The harness has four parts: load fixtures, render prompts, call each model through one interface, and record structured decisions. When you route to multiple providers, a single OpenAI-compatible gateway removes per-provider client code from the test path; Quanta Inference exposes text, image, video, audio, and embedding models through one API key, which keeps the replay loop identical across models. Use that only if you are already routing through it.

import json, pathlib

ALLOWED = {"retry", "fix", "escalate"}

def load_fixtures(root):
    for path in sorted(pathlib.Path(root).glob("*.json")):
        fx = json.loads(path.read_text())
        assert set(fx["allowed_actions"]) <= ALLOWED
        assert "fixture_id" in fx and "error_class" in fx
        yield fx

def decide(client, model, fx, system_prompt, tools):
    user = (
        f"Operation: {fx['operation']}\n"
        f"Status: {fx['status']}\n"
        f"Headers: {json.dumps(fx['headers'])}\n"
        f"Payload: {json.dumps(fx['payload'])}\n"
        "Choose one action and return JSON with keys action and params."
    )
    resp = client.chat(model=model,
                       messages=[{"role": "system", "content": system_prompt},
                                 {"role": "user", "content": user}],
                       tools=tools,
                       response_format={"type": "json_object"})
    out = json.loads(resp.choices[0].message.content)
    if out.get("action") not in ALLOWED:
        return {"action": "invalid", "params": {}}
    return out

Three details decide whether results are trustworthy. Use the same system_prompt and the same tools list for every model; an extra tool or a reworded instruction changes the comparison. Parse failures as a distinct invalid action rather than dropping them, because a model that returns malformed JSON in an error path will do the same in production. And log the full decision trace, not just the action, so a disagreement can be read later.

This is the same discipline as contract testing. Pact's consumer guidance recommends making assertions only about things that would actually affect the consumer if they changed, and keeping tests as loose as possible while still catching breaking changes the Pact consumer docs. Apply it here: assert on the action and on the parameters your downstream code actually uses, not on the prose the model wraps around them.

Run the suite in CI on the same schedule as your other integration tests. ASP.NET Core's integration-test guidance recommends limiting heavy tests to the most important scenarios and preferring a unit test when the behavior can be covered that way the ASP.NET Core integration testing guide. Error replay is slow relative to a unit test, so keep the fixture set focused on classes that have actually caused an incident or sit in a high-volume path.

You now have a matrix: fixtures on one axis, models on the other, decisions in the cells. The headline number is agreement, but the useful number is agreement per error class. A candidate that matches on auth and diverges on validation is safe to route for one and unsafe for the other.

For a worked example, take 40 fixtures split across five error classes, so 8 per class. The incumbent and candidate agree on all 8 auth fixtures and all 8 conflict fixtures. On the 8 rate_limited fixtures with a Retry-After header, they agree on 6 and disagree on 2, where the candidate retries immediately and ignores the header value. On the 8 validation fixtures, they agree on 5, and the candidate's three misses all drop the field pointer and pick a plausible but wrong field. Aggregate agreement is 34 of 40, or 85%. That number is fine and misleading. The routing rule that follows from the per-class view is concrete: the candidate is eligible for auth and conflict, blocked from rate_limited until the header handling is fixed or a guard is added outside the model, and blocked from validation until fix precision improves. Reading the two rate_limited traces tells you which it is: the model never saw the header, or it saw it and discounted it.

Two things matter when you read a trace. First, confirm the hint was present and legible; models are routinely penalized for failing to use information that was truncated, reformatted, or never passed through. Second, check whether the disagreement is a real policy gap. A candidate that retries a 409 where the incumbent escalated is a correctness risk; a candidate that formats params differently but names the same field is a scoring artifact you should fix in the harness.

This connects to a broader finding in the structured-output literature: even with constrained decoding nearly removing syntax errors, structural and semantic errors persist, which is why structural fidelity alone is not enough and semantic correctness has to be checked separately the structured output study. Agreement on the action is structural. Agreement on which field to fix is semantic. Score both.

The output of a replay run should be a routing table, not a report. For each error class, record the decision: route to the candidate, route to the incumbent, or route to a deterministic guard. Keep the guards close to the code that already handles HTTP, so a model is never the sole line of defense for a class where it has shown it cannot read the hint.

A short checklist before you promote a candidate anywhere:

• Fixture set is versioned and the run records the set version alongside the model.
• The decision space is closed and validated; invalid outputs are counted, not discarded.
• Agreement is computed per error class, with a minimum number of fixtures per class.
• Every disagreement has a written disposition: fixed, guarded, or accepted.
• Headers and field pointers are confirmed present in the rendered prompt in at least one trace per class.
• The replay runs in CI and blocks promotion on critical disagreements.

The CI gate is what makes this stick. A recorded case that reproduces a past failure is the cheapest insurance you get, and the loop of capturing a bad outcome, labeling it, fixing the system, and confirming the fix keeps the set aligned with what production actually does the golden dataset guide. Braintrust makes the same point about treating a golden dataset as a living artifact: pull interesting traces, review them, and add the validated ones back, because a stale set gives false confidence the Braintrust definition.

If your platform team would rather buy than build the prompt-and-eval plumbing, Quanta experts read your context, do the work, and hand back finished deliverables, which can include a labeled replay set and a routing table.