Mocking a Model Endpoint Without Losing Fidelity

The fix begins before anyone writes a fixture. A mock should be an executable view of a versioned contract, and that contract should be verified against the real provider in CI; a contract without provider verification can drift, and provider verification without a consumer-owned expectation can miss what consumers actually use stop mock APIs from drifting.

That gives you two sources of truth, and you need both. The provider's published schema (an OpenAPI document, or an equivalent machine-readable description) tells you which fields are required, their types and their enums. Recorded traffic tells you what the provider actually does in a given scenario, including the parts a schema cannot express. Schemas are not contracts: they validate structure at a point in time but cannot verify that a running service honors its own spec, and they cannot state preconditions, optionality semantics or ordering guarantees schemas can be contracts. Contract testing as a discipline exists precisely to check endpoints, request and response schemas, status codes, headers, auth and data types on every commit what API contract testing is.

A practical recipe:

1. Pin the spec. Download the provider's schema artifact once and commit it with a version label. The mock server reads that file, not a hand-maintained JSON file.
2. Record a small set of exchanges. Ten to twenty representative calls are enough to start: a happy path, a long response, a tool call, an embedding, a 401, a 429 and one 5xx. Capture both request and response, including headers.
3. Normalize at record time. Replace request IDs, trace IDs and timestamps with typed placeholders. Keep token counts in the captured file (they are load-bearing for cost code) but have the recorder mark them as matching by type rather than by literal value.
4. Validate scenarios against the spec at mock startup. A representative example is not automatically a valid instance merely because it sits beside a schema; fail fast on unknown operations and unexpected request bodies stop mock APIs from drifting.

The benefit of this approach is proportion. If you write fixtures by hand, you write the cases you can imagine. Traffic-driven capture finds the cases you cannot: recording live API requests and responses (read-only) and replaying them in staging is documented to reach roughly 95 percent coverage of production behavior against about 40 percent for simplified synthetic data Keploy environment replay. Synthetic load misses real-world patterns, integration complexity and data-handling edge cases precisely because it lacks real payloads Speedscale on traffic replay, and a replay pipeline can capture production traffic without adding latency and anonymize it before storing traffic replay in Kubernetes.

The mock is only half the system. The other half is a test that runs on a schedule (nightly is typical), sends the recorded exchanges to a real sandbox, and compares provider behavior to the fixture. When they disagree, it fails and tells you which field or frame changed.

Keep the comparison narrow and explicit. Checking everything produces noise from volatile fields; checking nothing produces false confidence.

ALLOWED_VOLATILE = {"id", "created", "system_fingerprint"}

def diff(recorded, live, path=""):
    problems = []
    for key in recorded:
        if key in ALLOWED_VOLATILE:
            continue
        if key not in live:
            problems.append(f"missing field: {path}{key}")
        elif type(recorded[key]) is not type(live[key]):
            problems.append(f"type change: {path}{key}")
        elif isinstance(recorded[key], dict):
            problems.extend(diff(recorded[key], live[key], f"{path}{key}."))
    return problems

Run it against the happy path and against every error case. An error-path test asserts the exact status code, the error type string and the required fields in the error body. Those are the cases most likely to be skipped and most likely to break a client: a timeout is a transport failure, a 429 needs a retry-after header, and a 401 needs a distinguishable error code so the client can refresh credentials rather than retry.

The error-path coverage is not optional polish. Testing agents is hard because they are non-deterministic and can reach a correct answer by an unsafe path, which is why evaluators look at trajectories and tool calls, not just final output Splunk on agent testing. A mock that never emits a malformed response or a rate-limit rejection leaves that entire surface untested. Teams that adopt contract testing in the pull request get feedback in milliseconds per test because the check touches only the interface layer what API contract testing is.

The single cheapest decision is to make the schema version part of the mock's identity. Name the artifact, commit it, and have the mock refuse to start if the artifact is missing. Then a provider change becomes a visible diff in a pull request rather than a surprise in production.

The second decision is what to do when a model ID is retired. Providers publish deprecation dates and recommended replacements, and the notice window varies by tier: generally available models get the longest window, specialized variants shorter, preview models the shortest OpenAI deprecations. Snapshot behavior matters here too. A pinned dated snapshot is a promise that the underlying model stays constant for the life of that ID, while a convenience alias may resolve to the newest dated snapshot Claude model IDs and versioning. Multi-platform models diverge further, since each partner platform sets its own retirement schedule; the same model can have different dates on different platforms the lifecycle calendar.

The decision rule: if your mock embeds a model ID, treat that ID as a dependency with an expiry date. Record the deprecation date next to the artifact, and when the retirement date approaches, create a new fixture set for the replacement model rather than editing the old one. Editing in place destroys the ability to diff old and new behavior, which is exactly what you need during a migration.

A worked example makes the cost concrete. Suppose the recorded set has 18 exchanges: 11 happy path, 3 streaming, 2 tool call and 2 error, and the nightly contract run costs 18 live calls. At a few hundred input and output tokens per call, that is a few thousand tokens per night, a trivial spend at any current per-million rate. The alternative, a manual check before each release, is one engineer-hour and catches less. The automated check wins on both axes, which is the decision rule: automate the contract check when the live-call cost per run is below the cost of one manual verification.

When your fixtures need to cover several providers at once, routing them through one OpenAI-compatible key reduces the number of endpoints and error body formats your mock layer has to model. Quanta Inference (https://quantaintelligence.ai/inference) exposes text, image, video, audio and embedding models behind one API, which also means one schema surface and one set of error shapes to verify.

Use this before the mock goes into CI.

Two operational notes. First, keep the recorder separate from the mock server, so a change to the mock cannot change the recordings. Second, store recordings as plain files in the repository, not in a database. A fixture is data, and data in a repository shows up in code review, which is where drift gets caught.

If the consumer of the mock is an agent rather than a plain client, drift shows up as wrong tool selection rather than a parse error. The agent reads tool documentation, and that documentation is often under-specified, over-specified or misaligned with the implementation, which is exactly the failure mode automated tool testing targets: ToolFuzz generates natural queries that cause runtime errors or incorrect agent responses and reports that many community and production tools are erroneous, with fixes raising GitHub task accuracy from 22.9 percent to 35.4 percent and file management from 29 percent to 39 percent ToolFuzz on arXiv and the ICLR submission.

The practical consequence for mocks is that the schema is not enough for agent consumers. The mock must also exercise the documented contract of each tool: which arguments are required, which values are valid, and what the error looks like when the argument is wrong. Mock at the right level deliberately. Mocking a tool function tests agent reasoning and parameter passing without testing the tool implementation; mocking the API boundary tests the tool's HTTP calls; dependency injection tests the whole path but requires designing for it up front mocking external APIs in agent tests. Pick one level per test and be explicit about what it covers and what it does not.

The parallel with general API testing is direct: the well-known advice for external integrations is to mock the boundary, unit test request construction and response parsing, and keep a small number of real integration checks for side effects integration tests for external APIs. Mock APIs exist precisely so you can test the AI-specific parts, such as whether the model decides to call a tool, without the noise of a live system, and so you can simulate timeouts and rate limiting on demand WireMock on testing agents. Start with deterministic tooling you already run to establish a baseline before layering on agent-specific evaluation Tricentis on deterministic tooling. A mock generated from the provider's own framing, including streaming SSE, removes one whole class of false failures MockServer LLM mocking.

If building and maintaining that verification layer is where your team stalls, a specialist can take it on end to end. Quanta (https://quantaintelligence.ai/) picks up a described job, such as a contract-test harness for a model endpoint, and builds it with you.