Stubbing Model APIs So CI Quickstarts Stay Deterministic

A recorded fixture is a saved request and response pair taken from a real call. You match an incoming request and return the recorded body. This proves your client parses the real response shape. It does not prove the server is reachable today, and it does not prove the provider still returns that shape unless you refresh recordings on a schedule.

A local mock server speaks the provider's wire format and returns responses you write yourself. This proves your integration layer handles streaming, tool calls, retries, malformed JSON, 429s and 500s, and it is the only one of the three that exercises error paths cheaply. CopilotKit's LLMock, for example, advertises authentic server-sent events in real provider formats, fixture routing by substring, regex or predicate, and tool call injection for agent workflows (the CopilotKit announcement). It proves nothing about the provider.

A cheap real model closes the loop. It proves credentials work, the endpoint URL is current, the SDK matches the provider, and the response shape has not drifted. It is the only approach that catches a provider-side breaking change, and the only one that can fail for a reason outside your control.

The key point first search results often miss: the real-model suite is not a better version of the fixture suite. It tests a different property, namely that the outside world has not moved. Running it on every pull request gives you the worst of both, because you pay the flakiness and gain nothing the offline tests do not already give you.

Recording is a one-off, run by a developer with a real key, and its output is committed like any other test asset.

There are two ways to capture. The first is a proxy in front of the real API that forwards each request and writes the request and response to a file. Mokapi documents exactly this: during recording it forwards requests through to the real backend and appends each pair to a JSON file, then in replay swaps the forwarding handler for a matcher that returns the captured response without the backend being reachable (Mokapi's record and replay guide). The second is a thin wrapper in your own harness that logs the raw HTTP exchange whenever it runs against the real endpoint.

Three properties make fixtures durable. They must be plain JSON in the repository so they diff in review. They must be keyed by something stable, normally a hash of the normalized request. And they must exclude volatile fields, or normalize them at replay time, so a timestamp or request ID does not cause a spurious miss. Keploy's write-up describes automatic noise detection that excludes timestamps, UUIDs and session tokens from strict assertions and freezes the clock to the recording time (Keploy's record and replay guide). You do not need their instrumentation to borrow the idea; a normalization function inside your test code is enough.

import hashlib, json, pathlib

FIXTURES = pathlib.Path("tests/fixtures/model")

def key_for(request: dict) -> str:
    body = {k: v for k, v in request.items() if k not in ("request_id", "timestamp")}
    normalized = json.dumps(body, sort_keys=True, separators=(",", ":"))
    return hashlib.sha256(normalized.encode()).hexdigest()[:16]

def replay(request: dict) -> dict:
    path = FIXTURES / f"{key_for(request)}.json"
    if not path.exists():
        raise FileNotFoundError(
            f"No fixture for request. Re-record with a real key and commit {path.name}"
        )
    return json.loads(path.read_text())["response"]

The miss behavior matters more than the hash. A fixture suite that returns a generic 404 for an unrecognized request hides new call sites forever. One that fails with the instruction to re-record surfaces them in the pull request that introduced them, which is the moment someone can decide whether the new call is intended.

Fixtures excel at the happy path and are awkward everywhere else. You cannot easily record a 429 with a specific retry-after value, a stream that drops mid-response, or a tool call the model never made because the prompt was ambiguous. Write those by hand.

The mock should be a real HTTP server on localhost, not an in-process stub, because a real server exercises the same transport and connection handling your client will use. The Free-Way issue is explicit: the mock server should start and stop cleanly inside the test script, and the whole run should avoid real provider credentials (the Free-Way issue). A server also lets other processes in the same CI job reach it, which matters when the quickstart spawns a CLI or a container.

Keep the list short. The three smoke test filter questions apply: if this fails, can the user do anything at all; does every user hit this path; can this break because of a build or deployment issue (Drizz). A missing key and a streamed response pass all three. An obscure tool-call edge case belongs in the regression suite.

One more benefit: the mock makes the quickstart's error messages testable. A quickstart that prints "401 Unauthorized" when the key is missing is a bad quickstart. Postman's guide to API documentation makes the point that error documentation listing each code with troubleshooting guidance is part of what good docs contain, and the same reasoning applies to your failure output (Postman). If the smoke test asserts on the wording of the missing-key message, that message will not rot.

The core CI advice is unglamorous and applies directly. GitLab's CI best practices recommend keeping builds fast and simple, and cite Martin Fowler's ten-minute build guideline as achievable for most modern projects (GitLab). A stub-based quickstart smoke test runs in seconds. A real-model version runs in seconds too, until it does not, and then it costs someone half a morning.

The real-model suite needs guardrails. Cap it at a handful of requests so a runaway retry loop cannot generate a surprise bill, and treat its budget as a fixed line item rather than something that scales with commit volume. Keep the key in the CI provider's secret store, never in the repository, and run it from a workflow that forks cannot trigger.

A fixture refresh belongs in the same plan. Pick a cadence, run the recorder against the real API, and open a pull request with the diff. Reviewing that diff is where you catch a field that quietly changed type or an example that no longer matches the README. Fern's guide notes that treating the API specification as the single source of truth prevents drift between implementation, SDKs and documentation, with generation wired into CI/CD (Fern's best practices guide). Refreshed fixtures are the model-API version of that discipline: a machine-readable record of what the outside world actually said, checked by a human when it changes.

Suppose your quickstart repo gets 300 pull requests a month across a team of eight. Your real-model smoke test makes three calls per run, each with a median latency of 1.5 seconds.

Stubbed, the whole job is roughly 12 seconds: checkout, install dependencies, start the mock, run four scenarios, stop the server. Real-model, the same checks take about 5 seconds of provider latency in the good case and, based on typical provider behavior, between 1 and 3 percent of runs fail for reasons unrelated to the change: a 429, a timeout, a transient 5xx. At 300 runs that is roughly 3 to 9 red builds a month that someone must investigate. At 10 minutes each for investigation and re-run, that is 30 to 90 minutes of developer time per month burned on noise, plus API spend, plus slow erosion of trust in a red build.

Move those calls to a nightly job. You get 30 real-provider checks a month instead of 300, still more than enough to catch a schema change within a day, and the pull request path becomes deterministic. The nightly failure is a signal, not a blocker, so it goes to a channel someone watches rather than sitting in front of a merge.

The rule that falls out is simple: run the check that can only fail for external reasons on a schedule, and run the checks that can only fail for internal reasons on every commit.

The failure mode to avoid is a fixture suite that passes while the README is wrong. This happens when the quickstart drifts, someone updates the example code, and nobody updates the recorded request.

The fix is to make the smoke test execute the README's own steps rather than a parallel copy. If the README shows a shell command, the test runs that command. UC Berkeley's documentation guide recommends a README containing a brief description, installation instructions and a short example or tutorial, which is exactly the surface a smoke test should cover (UC Berkeley Library). Stoplight's guide makes the related point that a reference, a guide and a concrete example are three distinct jobs, and the concrete example is the one people copy (Stoplight). If the copied example is wrong, everything downstream is wrong, so it is worth a test.

Two habits keep the suite honest. Make the fixture miss loud, as in the harness above, so a new call site fails in review. And when the nightly real-model run fails, treat it as a fixture refresh plus a README check, not a flaky job to re-run. The refresh opens a pull request, the diff shows what changed, and the reviewer asks whether the quickstart still reads correctly.

If your quickstart reaches several providers or model types, the number of wire formats to mock grows quickly, and maintaining a mock per provider is the kind of work that quietly stops happening. Routing everything through one OpenAI-compatible endpoint reduces the surface your stubs must cover, which is what Quanta Inference provides: text, image, video, audio and embedding models behind a single key.