Schema Tests for Tools: Fixtures That Assert Every Error Code

A fixture is a recorded request plus the exact response it produced, stored as data. The test replays the request and checks three things.

1. Envelope shape. The response must have the documented fields with the documented types. Keep this separate from business rules: schema validation proves consumers can safely parse the body, it does not prove the values are correct the JSON Schema testing guide. Use a strict validator, set additionalProperties: false in the error schema, and declare the dialect in the root schema so a keyword that means one thing in one draft is not silently reinterpreted in another.

2. Hint identity. The machine-readable hint must equal the expected error code exactly. Equality, not a substring match. Models use structured errors to self-correct: an LLM told "invalid date format, expected ISO 8601" can fix its next call, while a raw stack trace usually cannot error handling and recovery patterns. Precise hints are the input to that loop, so an inexact hint is a functional regression.

3. Retryability. The flag must equal the documented boolean. Classify at the tool-definition level into transient, terminal and needs-clarification so the control loop branches correctly instead of guessing error handling and recovery patterns. Practical classification rules exist: 429, 500, 502, 503 and 504 are transient, while malformed prompts, exceeded context windows and invalid keys are permanent agent error handling patterns. For 5xx codes on non-idempotent operations, the safer rule is to require an idempotency key before any retry is allowed at all whether to retry 500 errors.

Keep fixtures in the tool's own package: tools/<tool-name>/fixtures/errors/<code>.json plus schemas/error-envelope.json. The schema describes the public representation of the error, never a database row or ORM type, so an accidental data exposure does not get normalized into the expected output the JSON Schema testing guide.

There is a useful shortcut for building the schema. Record one or two real responses, convert them to a schema, and clean up the result: a single response rarely infers optional fields correctly, and you should add formats for emails, URLs and identifiers using JSON Schema for third-party response validation. Then test the schema itself, because a schema that validates everything is worse than no schema. Keep a known-bad fixture per error class in a negative/ folder and assert that the validator rejects it.

Two more rules keep the set honest. First, a fixture belongs to a contract version: when you change an error code on purpose, update the fixture in the same pull request, so the fixture diff is the contract diff and reviewers see it. Second, do not try to do this with end-to-end tests against a third-party API. That is slow, depends on outside servers and is close to impossible to set up with useful fixtures the same writeup.

The harness needs two layers. Unit tests replay fixtures against the tool's error constructor in-process, which is fast and runs on every commit. Integration tests call the live tool behind a feature flag, which is slower but catches wiring problems.

Start with the classifier, because every fixture depends on it. A minimal contract:

TRANSIENT = {429, 500, 502, 503, 504}

def classify(status=None, exc=None):
    if status in TRANSIENT:
        return {"hint": "TRANSIENT", "retryable": True}
    if exc and "timeout" in str(exc).lower():
        return {"hint": "TIMEOUT", "retryable": True}
    return {"hint": "PERMANENT", "retryable": False}

The test replays each fixture's input through the tool, then asserts the three properties. Run schema conformance with a strict validator and fail on the first violation, printing the violated path and the actual response, because that turns an opaque "must have required property" message into a fast diagnosis the JSON Schema testing guide. Zod and native matchers such as expect.any cover simpler cases, but for a multi-field envelope a real JSON Schema validator is less typing and stricter Playwright schema validation walkthrough.

Run the unit layer on every pull request, and the live layer on a schedule plus before deploy. The important part is the gate: the build fails when the fixtures fail, not when a human notices in a dashboard.

Practical CI steps:

1. Lint the schemas. Vendor the validator and run it with a fixed configuration so a validator upgrade cannot silently change dialect behavior.
2. Replay every fixture. Assert shape, hint and retryability, and treat a missing fixture for a known error class as a failure.
3. Run the negative fixtures. If a known-bad envelope passes, the schema is permissive and the build stops.
4. Report drift. Print expected versus actual for all three assertions in one message, grouped by error class.

Contract testing tools follow the same shape at the deployment boundary. can-i-deploy blocks a deployment when verification between the version you are deploying and its counterparts has not succeeded, and it can poll for results that are still unknown rather than failing immediately resilient builds with can-i-deploy. Your error fixtures are the same idea one level down: they are the check that says the failure contract still holds before the tool ships.

If your tools call several providers, point them at one endpoint through Quanta Inference (https://quantaintelligence.ai/inference), so a single key covers text, image, audio and embedding models and the failure contract has one place to live.

Take one error class, PAYMENT_DECLINED. The fixture records a request, the envelope and the three expectations:

{
  "fixture": "payment_declined",
  "request": {"tool": "charge_card", "args": {"amount": 42, "currency": "USD"}},
  "response": {
    "error": {
      "code": "PAYMENT_DECLINED",
      "hint": "PAYMENT_DECLINED",
      "message": "Card issuer declined",
      "retryable": false
    }
  }
}

The test asserts that retryable is exactly false, that hint equals PAYMENT_DECLINED, and that no additional properties appear. Now suppose a refactor accidentally routes this code through the transient branch. The tool will retry a declined charge, and because a timeout after dispatch does not establish whether the side effect committed, repeated attempts can create duplicate effects unless the guaranteeing service deduplicates them retry safety and idempotency. A single boolean assertion catches it before it costs anything.

Assume the tool retries three times with a base delay of one second and exponential backoff, capped at three attempts agent error handling patterns. With the correct flag, one declined charge produces one dispatch and about zero seconds of waiting. With the wrong flag, you get three dispatches and roughly three seconds of backoff, plus cumulatively rising load on the payment provider. Multiply by the volume of declined cards in an hour and the cost of the bug is measured in duplicate attempts, not minutes.

For timeout classes, the numbers work the other way. Suppose a network blip produces one timeout. If the fixture wrongly forces retryable: false, one recoverable failure becomes one failed run plus a support ticket. If it correctly says retryable: true, the orchestration layer absorbs it silently and the model never learns a transport error occurred LLM tool calling error handling. The fixture is what keeps that routing decision from regressing.

A failing fixture should tell you which of four things happened: the tool genuinely changed, the fixture is stale, the schema is too loose, or the environment differs. Work through them in that order.

If the tool changed on purpose, update the fixture in the same pull request and say why in the commit message. If it changed by accident, you have caught real drift. If the schema is too loose, tighten the validator settings and re-run the negative fixtures. If the environment differs (a sandbox returns a different provider error code), split the fixture by environment rather than loosening the assertion.

One practical check: replay the fixtures against the last release as well as the current build. A failure in both means the fixture was stale before your change, so you do not chase a ghost. Retry policy itself should be service-aware and configurable rather than one hardcoded timeout applied to every call commercetools error handling guidance, and the fixture set should mirror that configuration so per-service differences are asserted, not assumed.