Contract Tests for the Tools Your Agent Calls

The mistake most teams make is building one test suite and expecting it to do everything. Separate the work into three layers, each with a distinct failure mode it catches.

The rule: fixtures catch drift you caused (a code change that no longer matches the contract); live probes catch drift the vendor caused; runtime validation stops the blast radius when neither of the first two ran in time. All three matter, and they are cheap to run together.

This maps to the distinction between schema testing and contract testing. A schema test asserts that one system is compatible with a schema at a point in time; a contract test asserts that two systems can communicate by agreeing on interactions and testing concrete examples of agreed behavior (Pactflow). Fixtures give you the concrete examples; the agent and the vendor's real behavior are the two systems.

Start by capturing the responses your agent actually consumes. Do not hand-write fixtures from memory; hand-written fixtures encode what you think the API returns, which is the thing you are trying to verify.

Three practical techniques, in order of preference:

1. Record in a test environment against a sandbox or test account, with real request parameters the agent uses.
2. Record in production behind a sampling proxy that stores responses for a small percentage of calls, triggered by the specific endpoints the agent depends on.
3. If you cannot record live traffic, replay a recent trace from your observability stack and save the tool results.

Whichever route you take, strip sensitive values before committing. Laravel's HTTP fixture package exists precisely because saved JSON responses can contain sensitive data and filtering it by hand is tedious (Gromatics/httpfixtures); the general lesson applies to any language. Two rules make this safe:

• Replace identifiers, emails, tokens, and free-text fields with deterministic values, not random ones. A fixture that regenerates on each run tells you nothing.
• Keep the fixture's structure exactly as recorded, including nulls. Nullability is one of the highest-value things you are testing.

Store fixtures under version control with a name that encodes the endpoint and the condition: fixtures/vendor/orders_get/success.json, fixtures/vendor/orders_get/not_found.json, fixtures/vendor/orders_get/rate_limited.json. The raw-HTTP approach, where the fixture is literally the output of a curl -i call including status line and headers, has a strong argument behind it: it works across languages and preserves what the server actually sent, which is how DNSimple built client tests in Ruby, Go, Elixir and Node from the same fixture files (DNSimple). If your agent cares about status codes or pagination headers, fixtures should include them.

The fixture is the example; the schema is the rule. Keep both. The fixture proves the rule was true at least once; the schema is what you enforce on every response afterward.

In Python, Pydantic gives you schema and validation in one object, which is why it is the pragmatic default for agent tool adapters. Keep the model strict: forbid unknown fields so a silently added field is a signal rather than noise, and make optional fields explicitly optional so a null does not slip through as a missing key.

from datetime import datetime
from typing import Literal, Optional
from pydantic import BaseModel, Field, ConfigDict, field_validator

class Order(BaseModel):
    model_config = ConfigDict(extra="forbid")

    id: str = Field(min_length=1, max_length=64)
    status: Literal["pending", "paid", "shipped", "cancelled", "refunded"]
    total_cents: int = Field(ge=0)
    currency: Literal["USD", "EUR", "GBP"]
    created_at: datetime
    tracking_number: Optional[str] = None

    @field_validator("created_at")
    @classmethod
    def not_in_future(cls, v: datetime) -> datetime:
        if v.date() > datetime.utcnow().date():
            raise ValueError("created_at is in the future")
        return v

If you prefer language-neutral contracts, JSON Schema is the other sensible choice, and it composes with OpenAPI so the same schema can validate requests at a gateway. Schema validation at the edge catches malformed data before it reaches critical systems, and it enforces one shared interpretation across teams instead of each adapter reading the response its own way (Zuplo).

A decision rule for which to pick: if your agent is Python and the tool adapter is Python, use Pydantic and skip the extra file. If multiple services consume the same vendor, or you want the schema to double as gateway validation, write JSON Schema and generate the validators.

Schema checks catch type and shape errors. They do not catch the changes that actually break agents: a status value the enum did not anticipate, a currency amount that arrives in dollars instead of cents, a pagination cursor that stops advancing, an error body that returns HTTP 200.

Write one assertion module per tool and test the fixture with it on every run. The checks that pay for themselves:

Pagination deserves its own paragraph because agents loop. An agent that sees has_more: true with an unchanged cursor will keep calling the tool until a budget or timeout stops it. A fixture pair (page_1.json, page_2.json) plus an assertion that cursors differ and result IDs are disjoint catches this class of change before it becomes a production loop. The same discipline applies to retries: stateful mock behavior that returns a deterministic sequence, such as two failures followed by a success, is how you test that your client retries correctly and stops when it should (Beeceptor). Freeze those sequences as fixtures too, and assert that your retry wrapper gives up after the right number of attempts.

Error shapes are the other high-value target. A vendor that changes from {"error": {"code": "..."}} to {"errors": [{"code": "..."}]} will not break a typed parser if the field is optional, but it will break your agent's error handling logic, which now reads a null. Record at least one fixture per error class the agent handles, and assert on the exact shape each time.

The contract test belongs in the same run as your unit tests, offline and fast. The pattern that makes it enforceable is version locking: freeze the resolved set, commit the lock file, and make CI fail rather than silently re-resolve. NuGet's --locked-mode does exactly this for package restores, and the same principle applies to fixtures, where a contract that drifts without a visible diff is a contract nobody reviewed (vensas.de). Treat a fixture change like a lock file change: it must appear as a reviewable diff in the pull request.

A minimal CI job:

# fail if a fixture changed without the contract being re-approved
python -m pytest tests/contracts -q --strict-fixtures
# validate every fixture against its schema and semantics
python tools/validate_fixtures.py fixtures/ --schema contracts/ --strict

Two habits make this sustainable. First, fail closed: if the validator cannot load a fixture or its schema, exit non-zero. A skipped contract test is worse than no contract test, because it gives false confidence. Second, keep the fixture refresh a deliberate, visible act. Nightly, probe the live vendor, diff the new response against the frozen fixture, and open a pull request when they differ rather than overwriting the fixture silently. That pull request is your early warning, and it gives the team a place to decide whether the change is cosmetic or breaking.

Request interception tooling gives you a rehearsal path before you commit a schema change, since request hooks can log real requests and responses or respond with dummy data so tests do not depend on a third party's uptime (TestCafe). Use it to reproduce a vendor's reported change locally, then run the contract suite against the captured response before touching production code.

CI protects you between deploys; runtime validation protects you between the deploy and the vendor's next change. The adapter that calls the vendor should be thin and boring: call, validate, normalize, return. If validation fails, return a structured tool error to the agent instead of the raw payload, so the model gets a clean signal ("this tool returned an unexpected shape") rather than garbage it will try to reason around.

Two implementation notes that matter more than they look:

• Validate against the same schema the tests use. A runtime validator that drifts from the CI validator is a second contract, and two contracts is zero contracts.
• Log the validation failure with the vendor name, endpoint, and the diff between expected and received, minus secrets. That log is what you hand the vendor when you report the change, and what you use to write the new fixture.

Runtime validation is not a substitute for the fixtures. It catches shape and value violations you anticipated; it cannot catch a semantically wrong value that still satisfies the enum and the range. That is why the fixtures and semantic assertions exist, and why the two layers should share one source of truth.

Not every vendor warrants the same contract depth. Use impact to decide.

If a tool is read-only and low stakes (a time lookup, a public status endpoint), a schema check with permissive settings is enough. If a tool writes (payments, refunds, ticket closure, account changes), the contract should be strict, extra fields forbidden, error shapes pinned, and a human approval gate for high-impact actions, consistent with guidance that human-in-the-loop approval should gate high-impact calls even when a component was trusted at deployment (Sweet Security).

If a tool is a foundation model endpoint rather than a data API, the same pattern applies with different assertions: pin the response envelope, assert on finish reasons and usage fields, and freeze one fixture per error class. When you consume several model providers through one interface, the contract work multiplies, which is the argument for putting that layer behind a single API rather than re-validating each vendor's envelope by hand. Quanta Inference exposes text, image, video, audio, and embedding models through one OpenAI-compatible API key, which lets your validator target one response contract instead of one per provider (https://quantaintelligence.ai/inference).

Teams that spread this discipline across services usually end up needing shared ownership too: one schema registry, one fixture directory convention, one CI gate. That is the point where a shared credit pool and per-person keys start to matter for running the probes and validation jobs without one team's key becoming the bottleneck (https://quantaintelligence.ai/for-teams).