Schema Changes That Break Retry Logic: A Compatibility Suite

Write these down before you write a single assertion. Everything in the suite is a mechanical check of one of them.

The fallback row is the one teams forget. When you introduce recovery, every code in the old enum needs a defined recovery value, and the default for unrecognized input needs to be the safe one. Look at the two common defaults: retrying an unknown code burns tokens on a permanent failure, while abandoning an unknown code drops a transient one. Pick the default deliberately and test it.

Backward compatibility testing exists precisely to answer whether an updated version keeps working for consumers built against earlier versions, and the catalogue of breaking patterns includes renaming a response field, changing a field type, and changing a field's meaning without renaming it backward compatibility testing for APIs. All three appear in error payloads, and all three are easy to ship by accident because error code paths get less review than success paths.

Start with the envelope. Every error must carry a stable top-level shape before you can assert anything about hints. Capture the previous version's serialized error for a handful of representative failures and store them as fixtures, the way golden files pin CLI output in end-to-end tests writing integration tests for a Go CLI.

Assert presence and type, not the entire serialized object. A test that compares a full error JSON to a stored snapshot will fail on harmless serialization changes and tempt someone to regenerate the fixture without reading the diff. Contract testing guidance is blunt about this: make assertions only about things that would affect the consumer if they changed, and use loose matching field by field Pact consumer tests.

def assert_error_shape_v1_compatible(err: dict) -> None:
    # Fields that existed at v1 must still exist with the same type.
    assert isinstance(err["code"], str)
    assert isinstance(err["message"], str)
    assert isinstance(err["status"], int)
    # New fields are allowed, but must be optional and nullable-safe.
    for optional in ("recovery", "retry_after", "field", "suggestion"):
        if optional in err:
            assert err[optional] is not None or optional == "retry_after"

The retry_after special case is worth spelling out. If the field is present with a null value, the agent has to decide what null means, so either forbid null and omit the field, or define null as "use default backoff" and assert it.

Error codes are the branch points of the retry loop, so treat the set of codes as an enumerated type with a compatibility rule. The Linux exit-code conventions are a useful mental model here: a small set of codes carries reserved meaning, and the documentation advises against reusing them for user-defined purposes because it makes troubleshooting ambiguous exit codes with special meanings. The same discipline applies to your application codes: reserved meanings, no reuse, documented.

Practice from CLI design aimed at agents is even more directly applicable. One practitioner's contract lists exit codes with fixed meanings, including a distinct code for conflict so a caller can skip creation rather than parse stderr CLI tools that agents want to use. The parallel for a JSON error payload is a stable map from code to action.

Generate the code inventory from the previous release tag and diff it:

# codes-v1.json and codes-v2.json are generated from the error registry
jq -r '.codes[] | "\(.code) \(.cause) \(.defaultAction)"' codes-v1.json | sort > v1.txt
jq -r '.codes[] | "\(.code) \(.cause) \(.defaultAction)"' codes-v2.json | sort > v2.txt
comm -23 v1.txt v2.txt   # must be empty: nothing removed or redefined
comm -13 v1.txt v2.txt   # additions: fine, but each needs a default action

Any line in the first diff is a release blocker. A code that vanished will land in an agent's unknown-code branch, and you had better have tested that branch. A code whose cause or default action changed is arguably worse, because the field-name diff is clean and only the meaning moved.

That last row deserves a note. Splitting TRANSIENT into RATE_LIMITED and UPSTREAM_TIMEOUT is a good design improvement, and it is still a breaking change for any agent that matched on TRANSIENT. The compatible path is to keep TRANSIENT with its old action and add the specific codes alongside it, then retire the broad code only at a major version.

This is the suite that catches the subtle break. Every hint field has three states: absent, present with a value, and present but unrecognized. Each needs an asserted outcome, and the outcome must match what the old version did.

Take recovery as the worked example. Suppose v1 had no hint and the agent's policy was "retry any error carrying a 5xx status, up to three attempts." v2 adds recovery with values like retry, correctable, and permanent. The compatibility requirement is that an error carrying the old shape (5xx, no recovery) produces the same retry decision in v2 as it did in v1.

Write the assertions against the agent's decision, not against the field. The decision is the thing the consumer depends on, and asserting it keeps the test honest when you add a fourth value to the hint vocabulary.

@pytest.mark.parametrize("fixture,expected", [
    ("fixtures/timeout_v1.json", "retry"),
    ("fixtures/timeout_v2_retry.json", "retry"),
    ("fixtures/validation_v2_correctable.json", "fix_request"),
    ("fixtures/auth_v2_permanent.json", "escalate"),
    ("fixtures/unknown_recovery_v2.json", "default"),
])
def test_retry_decision_unchanged(fixture, expected):
    err = json.loads(pathlib.Path(fixture).read_text())
    assert decide(err) == expected

One more rule for hints: a hint must never contradict a hard signal without an explainable reason. If recovery says retry but retry_after is absent and the status is 400, the agent has two conflicting inputs. Define precedence once (hint wins over status for the action, status wins for the category) and assert it, because "which wins" is the question every agent author will ask.

The fallback path is where new codes and new hint values land, and it is the least tested because it requires deliberately malformed input. Build three fixtures per version: an error shaped like the previous version, an error with an unknown code, and an error with an unknown hint value. Assert that the first behaves identically to the previous release and the other two both land on the defined safe default.

The case for this is strongest in multi-agent systems, where failures cluster at the handoff and the fix is validation at every edge between agents error handling and retry logic between agents. An unknown error code is a handoff where one side's vocabulary ran out. A schema-level validator at the edge turns that into an immediate, visible failure instead of a corruption that cascades three steps downstream.

Record-and-replay tooling gives you the cheap version of this suite. Capture real error traffic from staging, replay it against the new version, and diff the decisions. Tools in this space describe the approach as capturing actual requests and responses and replaying them as deterministic tests, with statistical detection of fields that vary run to run, such as timestamps and identifiers, excluded from strict comparison record and replay testing. Point that at your error fixtures and you get the "did any retry decision change" diff for free.

A worked number makes the payoff concrete. Suppose an agent makes 10,000 tool calls a day on one path and 2 percent hit a transient error, so 200 retries a day. A schema change that makes those retries fail deterministically replaces 200 recoverable failures a day with 200 user-visible degradations, and the retry budget shows at most 600 attempts (three each), still under a budget sized for a heavy day. Nothing pages. The only metric that moves is "retries that recovered," which most dashboards do not chart. Adding that metric and the fixture suite together turns a two-week silent break into a red build.

Run the compatibility suite as a required check on every pull request that touches the error serializer, the code registry, or the hint vocabulary. Keep three collections of fixtures: v1 errors produced by the previous release, v2 errors produced by the current branch, and synthetic errors that are deliberately wrong (unknown code, unknown hint, missing optional field, wrong type in an optional field). The first two prove the contract holds; the third proves the fallback holds.

Structure the tests the way contract tests are structured: unit tests for the decision function, contract tests for the serialized shape, and integration tests for the round trip through the real transport integration tests in ASP.NET Core. Each layer answers a different question, and the contract layer is the one that catches hint drift because it asserts against a fixed, versioned expectation rather than whatever the current code produces.

Two operational habits make the suite durable. First, a named owner for the error contract, meaning a person whose job includes watching the code registry and hint vocabulary for changes, because an agent does not receive the deprecation email the tool schema migration write-up. Second, a rule that regenerating a fixture requires a written sentence in the pull request explaining which consumer behavior changed and why. The second rule is what prevents a green build from meaning nothing.

Recording the fixtures is mostly a matter of capturing real traffic. Most integration suites already run against a test host, so the capture step is one middleware that writes the error body to a file when a flag is set integration tests in ASP.NET Core. Keep the capture limited to errors, because that is the surface the suite asserts on.

If you run agents across several model providers, the same fixtures are worth running through each provider's tool-calling path, since a hint that parses correctly in one tool schema may arrive wrapped differently in another. Quanta Inference puts text, image, and embedding models behind one OpenAI-compatible API key, which makes the multi-provider fixture run a config change rather than a rewrite, and the Quanta Inference page documents the endpoint.

Sometimes the right move is a deliberate break. A hint vocabulary that turned out to be wrong, a code whose meaning drifted internally, or a retry default that caused an incident: those justify a major version. The rule is not "never break," it is "break loudly and only on purpose." A deliberate break means a new version field or endpoint, a migration note that names the affected codes and hints, and a compatibility suite entry that asserts the old behavior is gone, so the change cannot be mistaken for an accident.

For agent-building teams, the payoff of the suite compounds with scale. Every agent, every skill installed into a coding agent, and every workflow that consumes your errors is a consumer you cannot see. The compatibility suite is the only way to enumerate them, because it encodes the behaviors instead of the callers. The assertions you write about shape, codes, hints, and fallback are the same assertions every downstream agent team would write if they had the time. Writing them once, at the boundary, is the cheapest place to put them.