Recording Vendor Responses for Replay in CI

There are three practical shapes for a fixture, and the choice affects portability and review noise.

Raw HTTP text. The fixture stores the status line, headers and body exactly as they came off the wire. This is the approach DNSimple took for multi-language client testing, using curl output so the same fixture works in Ruby, Go, Elixir and Node: "the fixture can be obtained by using the API itself and then manipulated to clean data, and includes the HTTP response line, all of the response headers, as well as the response body" 4. The advantage is fidelity; the cost is that volatile headers (dates, request IDs, rate-limit counters) create diffs on every re-record.

Structured JSON with normalized metadata. The fixture is a JSON object holding request method, path, query, selected request headers, response status, curated response headers and the parsed body. Normalizing at record time removes the noise before it reaches a diff.

Framework cassette. A library-specific recording, such as the cassettes used by VCR-style tools, where an adapter patches the HTTP library and the matcher decides which recorded response answers which request 5. Fast to adopt, but the format is tied to the tool.

A structured JSON fixture is the most useful default for CI, because the diff between two recordings reads like a code review. A minimal example:

{
  "name": "create_order_success",
  "request": {
    "method": "POST",
    "path": "/v2/orders",
    "query": "",
    "body_sha256": "9f2c...",
    "headers": {"content-type": "application/json"}
  },
  "response": {
    "status": 201,
    "headers": {"content-type": "application/json", "x-request-id": "<redacted>"},
    "body": {"id": "ord_123", "state": "open", "amount_cents": 4200, "currency": "USD"}
  },
  "recorded_at": "2026-01-14T09:12:00Z",
  "vendor_api_version": "pinned-header-value"
}

Note what is deliberately absent: the date header, the rate-limit counters and the request ID. Those change constantly and belong in a normalization list, not in a fixture. What stays is everything a consumer could break on.

A recorded vendor response is a copy of real data, and often real customer data. Redaction has to happen at record time, in the recording code, not by hand afterwards, because the one time someone forgets is the time it lands in repository history. Git history is effectively permanent, and removing a committed token is an incident, not a cleanup.

Use three categories:

Credentials and identifiers. Authorization headers, API keys, session cookies, account IDs, email addresses, phone numbers. Replace with stable placeholders. If a fixture needs an account ID to match a request, use a fixed fake value and map the real one away at record time.

Volatile values. Timestamps, request IDs, trace IDs, rate-limit remaining counts. Either drop them or replace them with a placeholder string so the matcher and diff ignore them. Request IDs in particular are the classic example of a field that changes frequently and is not meaningful to surface 6.

Everything else. Keep it, because unreduced real payloads are the whole point of recording over hand-written mocks.

Two enforcement steps worth adding: a pre-commit hook or CI check that greps fixtures for patterns matching the vendor's key format and your own credential shapes, and a scanner that parses every fixture as JSON and rejects any key named like a secret (authorization, token, secret, password, api_key). A fixture that fails either check fails the build. Recording tools should refuse to write a fixture until redaction rules have run, so the unsafe version never exists on disk in a committed path.

One more subtlety: redaction rules are themselves a test. If the vendor renames a field from account_id to accountId, your redaction rule stops matching and real IDs start flowing into fixtures. The re-record job described later should also run the redaction audit, so a rename breaks a check rather than a data boundary.

There are two ways to replay, and the choice depends on whether you control the client library.

A client interceptor wraps the HTTP client so outbound requests are matched against fixtures and answered in-process, with no server listening. The recording library's adapter patches the HTTP functions your code already calls, so record mode performs the real request and playback mode returns the stored pair instead 5. This is the least invasive option for a single-language service and the fastest to run in CI.

A local stub server binds to a port, serves fixtures by matching method and path, and the application points at it through a base URL. This is the approach behind most fixture-based client testing, where a stub library matches a GET against a path regex and returns the stored response 4. It is the only option when the HTTP call happens in a language or process you cannot patch, and it doubles as a way to let a coding agent or a designer hit the fake vendor locally.

Whichever you pick, the matcher is where replay lives or dies. Order-based matching pairs the first request with the first response, and that breaks the moment a test adds a request or runs in parallel. HTTP-based matching compares request properties against the recording 5. For vendor contracts, match on method plus path plus a hash of the request body, and ignore headers except the ones that change the response (content type, API version header, an idempotency key if the vendor uses one for dedupe).

Make unmatchibg requests fail loudly. A replay layer that returns a default 200 for an unknown request will hide a real code path that calls a new vendor endpoint, and the first time anyone notices is production. Fail with the request method, path and body hash so the fix is to record a new fixture.

Also decide where the vendor boundary sits in your test suite. Recorded fixtures are best treated as an integration-layer check that runs in CI on every pull request, distinct from end-to-end tests that verify the whole system. The release-gate mechanics are the same either way: the test command must return a nonzero exit code on failure so the CI provider can turn it into a required check 7. A wrapper that swallows the exit status, such as appending || true, turns a failing contract test into a passing job.

Fixtures are only a contract while they match reality. The re-record job is the mechanism that keeps them honest: on a schedule, run the same recorded requests against the live vendor, redact with the same rules, and diff the new response against the stored fixture. Fail when the difference is structural.

The diff rule matters more than the tool. Compare field presence, JSON types and enum-like values, and ignore values that are documented as volatile. A field that disappears or changes type is a breaking change. A new field appearing is usually safe for consumers but worth surfacing, because it may signal a new required request parameter. A changed value in a field nobody reads is noise.

Where the re-record job runs is a practical decision. It needs network access and a vendor credential, so it should not be the same job as the CI test suite, which should run with no network. Run re-record in a separate scheduled workflow, commit the resulting fixtures on a branch, and let a human review the diff. Auto-merging re-recorded fixtures defeats the point, because it converts an observable breaking change into silent approval.

The deeper reason to do this is the failure mode contract testing exists to catch: a provider adds or renames something, nothing is removed so nobody flags it, and downstream jobs start failing quietly hours later 3. A scheduled structural diff turns "we found out from finance" into "a job failed on Tuesday." Keep the fixtures in the repo, keep the request definitions in code, and let the system tell you when the vendor moved.

If the vendor publishes an OpenAPI spec or JSON schema, you can add a second layer that compares the fresh response against that schema, not only against the old fixture. Schema comparison against a published, golden schema is a documented way to catch documentation drift for public APIs 8. The fixture diff catches change; the schema check catches a fixture that was wrong from the start.

Stability without a sandbox comes from removing every source of nondeterminism except the fixtures themselves. Four practices do most of the work.

Pin the clock and the IDs. Any request that carries a timestamp, nonce or client-generated ID will fail matching on the second run. Generate them from a seeded fake in test mode, and record fixtures using the same seed.

Separate the layers. Live re-record is a scheduled job, fixtures are committed artifacts, and CI runs offline. Mixing them means every flaky network moment looks like a contract failure, which trains engineers to ignore the suite.

Budget the runtime. A contract suite built on local fixtures should finish fast, and speed is the property that keeps it in the required-check path. Smoke-style suites are kept narrow by design so they stay trustworthy as a gate 9. Contract suites should follow the same instinct: a handful of endpoints that carry real risk beat a broad, slow sweep.

Fail on unevaluated cases. A gate that only fails on explicit failures can pass a run where half the fixtures never executed, because unexecuted cases are not failures and drop out of pass-rate math 10. Configure the contract job to require that every fixture was exercised, so a matcher regression that silently skips half the suite turns red instead of green.

A worked example of the budget. Suppose a vendor has 6 endpoints your service consumes, each with 2 recorded cases (success and a documented error), for 12 fixtures. Replay through an interceptor runs in well under a second per fixture locally; the whole job is dominated by process startup. The scheduled re-record hits 12 live requests plus whatever auth handshake the vendor requires, so it is the job that must tolerate latency and rate limits, with a defined retry policy. The cost split is intentional: the frequent job is cheap and deterministic, the infrequent job is slow and touches the network.

When your own service composes several vendors, or when the thing you are testing is a model-backed endpoint rather than a REST API, the same record and replay discipline applies, and the natural next step is routing those calls through one place. Quanta Inference exposes text, image, video, audio and embedding models behind a single OpenAI-compatible API key, which makes the request set small and the fixtures easy to keep stable.