Contract Tests Between Your App and an Inference API

The trap is confusing a contract test with an output test. Contract tests are deliberately narrow: they do not check whether the order was actually created or whether the summary is accurate, only whether the interface structurally conforms (Total Shift Left). At a model boundary, that narrowness is a feature, because output text is nondeterministic by design and asserting on it produces a flaky suite nobody reads.

So the contract is a checklist, not a document. Six things belong in it.

Schema-based approaches are the cheapest way to encode most of that table. Matt Fellows describes three common strategies and the trade-offs between them: generating a schema from code and diffing it, and code-based schema tests that invoke your real handler inside your unit test framework and check the output against a shared schema (PactFlow). His code-based example is worth copying almost verbatim: call the route function directly without starting a server, then assert the response matches the schema and the status code matches the expectation (PactFlow). That is milliseconds per test, deterministic, and dependency free.

One caveat on written specifications. A community thread on OpenAPI 3.0 contract testing shows practitioners wrestling with how to point a test run at a schema and validate a real response against it, and the discussion is instructive precisely because it shows the specification file and the runtime response drifting apart unless something checks them together (Postman Community). If your provider publishes a specification, treat it as a second contract to test, not as the test itself.

Fixtures are the whole game. Hand-written sample payloads encode what you think the provider sends. Recorded payloads encode what it actually sent, including the fields you did not know about, the nulls where you expected strings, and the stray wrapper.

The capture loop has four steps.

1. Turn on request and response logging in your adapter for one run in a non-production environment.
2. Redact secrets (API keys, user identifiers, any raw text you cannot keep) before the file touches the repository.
3. Save each pair as a named fixture next to the test that uses it, with a name that says what it represents: success, truncated stream, rate limited, invalid request.
4. Assert against the fixture in the stub tests, and assert against the live provider only with the shape assertions from the previous section.

The reason redaction comes second and not fourth is that a committed fixture with a live credential is an incident, and rotation is slower than writing a scrub function. Keep fixtures in the repository as plain files, then load and parse them in the test. A minimal pattern looks like this:

import json, pathlib

FIXTURES = pathlib.Path(__file__).parent / "fixtures"

def load(name: str) -> dict:
    return json.loads((FIXTURES / f"{name}.json").read_text())

def test_success_response_shape():
    body = load("completion_success")
    assert isinstance(body["choices"], list) and body["choices"]
    first = body["choices"][0]
    assert isinstance(first["index"], int)
    assert first["message"]["role"] == "assistant"
    assert isinstance(first["finish_reason"], (str, type(None)))
    usage = body["usage"]
    assert all(isinstance(usage[k], int) for k in ("prompt_tokens", "completion_tokens"))

Notice what is absent: no assertion on the content string beyond its type. That is the difference between a contract test and a regression test on model quality, and it is the reason this suite stays green through a provider upgrade that changes wording but not shape.

The stub side of the loop is where mocking discipline matters. LangWatch's testing guide sorts mocks into levels, from tool function mocks up to dependency injection, and points out that mocking at the right boundary is what makes scenarios deterministic and offline friendly (LangWatch). The inference adapter is that boundary. Mock it, not the HTTP library underneath it, or you are testing your mock.

Three surfaces deserve their own section because they are the ones that break silently.

Error shapes. Record at least one failure fixture per class you care about: invalid request, authentication failure, rate limit, and server error. Then assert that your adapter converts each into your own error type with a stable code. The point is not the provider's message text, which will change, but the mapping. When your retry logic branches on an error code, that code is part of the contract, and it needs a test that fails loudly if the provider reshapes it.

Timeouts. A timeout test has two assertions: that the client gives up inside a bounded window, and that the failure surfaces as your error type rather than as a raw socket exception. Keep the window generous, since the test is about behavior, not speed. If your budget is a 30 second request timeout with two retries, the assertion should be that your adapter raises your Timeout error after its own retry policy has run, not that the underlying transport behaved in any particular way.

Streaming chunks. Here the fixture is a captured sequence rather than a single object, and the assertions are structural. Each line parses on its own. Each chunk carries a recognizable type marker. Exactly one chunk carries the terminal signal, and it is the last one. Reassembling all content deltas yields the same string the non-streaming call produced for the same fixture. That last assertion is a shape assertion in disguise and it catches the most common streaming bug, which is a dropped final delta.

A worked example makes the economics concrete. Suppose an adapter test suite has 40 tests. Thirty of them are fixture-driven stub tests at roughly 2 milliseconds each, six are error and timeout tests at 50 milliseconds each because they involve short sleeps, and four touch a local stub server at 30 milliseconds each. That is under a second for the whole suite, which means it runs on every commit without anyone complaining. The live provider run is a separate job with ten shape assertions and a cost budget, and it runs on a schedule and before every release, not on every push.

There is a second lever here that teams often miss. Because the adapter is the only thing under contract, you can run the same suite against a second provider. Fowler's account of incremental replacement is explicit that new features get built on top of, yet separate from, the old code, and behavior moves across bit by bit (martinfowler.com). If your shape tests are written against your own domain types rather than a provider's JSON, adding a candidate provider means writing a new adapter and reusing the suite. If they are written against provider JSON, you will rewrite the tests. The AWS guidance on the same pattern frames the payoff as reduced transformation risk and less disruption, which is precisely what a shared suite across two adapters buys you (AWS Prescriptive Guidance).

One more decision rule. If a surface has never changed in your recorded history and is unlikely to, do not write a test for it. Contract suites rot when they assert on everything; they stay useful when every assertion corresponds to a failure you would actually want a red build for. Feathers' change algorithm is a good filter: identify the change point, find the test point where the behavior can be sensed, and test that (BSSw).

Two jobs, two purposes.

The commit job runs the stub suite. It is fast, it needs no network, no key and no budget, and it fails for exactly one reason: somebody changed the adapter's shape handling without updating the fixtures or the assertions. That is a signal worth blocking a merge on.

The scheduled job runs the live shape assertions. It is slow, it costs money, and it fails for a different reason: the provider changed something. Because it is not on the critical path for every pull request, a flake in it does not train your team to ignore red builds. Treat its result as a maintenance ticket with a deadline, not an outage.

Keep the live job's scope tight. Ten to fifteen shape assertions on real requests is enough to detect a renamed field, a changed error envelope, or a new chunk type. Anything more and you are paying to rediscover your own stub suite.

If your stack already calls several model kinds, the adapter is also where portability pays off. Pointing that adapter at one API that serves text, image, video, audio and embedding models, rather than wiring a new client per modality, keeps the number of contracts you maintain small and makes the live job's coverage meaningful, which is the practical argument for Quanta Inference.

A contract suite is a living artifact, and it decays in three predictable ways.

It drifts from the fixtures. Someone hand-edits a JSON fixture to make a failing test pass. Guard against this by regenerating fixtures with a script that hits the provider and writes the file, so the only legitimate edit is a rerun of the script, and the diff is reviewable.

It grows assertions nobody needs. Every quarter, delete assertions for surfaces you no longer read. The signal-to-noise ratio is the whole value of the suite.

It loses its red-build meaning. When the live job has been yellow for three weeks, the suite is dead. Assign an owner and a response window, even if the window is generous.

There is a useful habit from gradual replacement work: never add to the old path. In the branch by abstraction write-up, the discipline is that developers stop adding queries in the old style, and one way to enforce it is a build that fails when a new one appears, so the count can only go down (continuousdelivery.com). Translate that to this boundary: new code calls the adapter, never the provider client directly, and a lint rule or a simple grep in CI enforces it. The contract suite only protects code that goes through the contracted path.

If your team writes a lot of integration code with coding agents, the same discipline carries over: the agent should be extending the adapter and the fixture set, not reaching around them, which is a good case for installing a shared skill so agent-written changes land inside the seam instead of beside it, as described in Quanta Skills.