Field Order as a Contract: Testing the First Keys

Start from the rendering path, not the model output. Walk your component or client handler and ask a blunt question for each field: if this key is missing at chunk 3, does the user see a broken or empty state? If the answer is no, it does not belong in the contract.

A workable derivation:

1. List every field the client reads before the user's first meaningful interaction.
2. For each, mark the earliest chunk index at which it is genuinely needed.
3. Keep only the fields needed at index 1 and 2 for the strict set.
4. Move everything else to a soft set that you record but do not fail on.

A typical result for a chat-style object looks like this.

Two strict-set fields arrive in chunk 1 and two in chunk 2, so your contract is "these four names appear within the first two chunks." That is a test you can read in five seconds.

Be honest about the trap of over-specifying. Asserting that content appears early feels safe, but streaming text often arrives as a delta that grows, and a partial parser may or may not yield the field before the closing quote depending on configuration (the typed-object gap write-up). Keeping growing strings out of the strict set removes an entire class of false alarms.

Store the contract as a small JSON file with an explicit version, a chunk budget, and the required names. Include a canonical hash of a real captured stream so reviewers can see what the contract was derived from, without committing a giant transcript.

{
  "contract": "first_keys",
  "version": 1,
  "chunks": 2,
  "required": ["id", "type", "status", "role"],
  "optional": ["content"],
  "forbidden_early": [],
  "capture": {
    "source": "chat_stream",
    "chunk_bytes": 8,
    "canonical_sha256": "bdb1c2c2dc7a2b3a4b8a4a3ba1e9e6c3bd45c59da5c6e0b8d8b0b6e0a4a5b1f2"
  }
}

Three design choices matter here.

The name is contract_v1.json, not snapshot.json. The numeric suffix is the contract version, and it appears in the filename so a diff in the pull request is unmistakable. Pact users get the same signal from publishing versioned consumer contracts (the Pact consumer guide).

The unit is a chunk budget, not a byte offset. Model providers change chunk sizes when they change transport or tokenization, and a byte offset test would fail for reasons unrelated to your consumer. "Within the first two chunks" survives that change.

forbidden_early is worth having even when empty. When you move a sensitive field such as a raw internal error or a billing detail out of the early stream to avoid showing it mid-render, that decision belongs in the file. Snapshot testing of API responses has caught accidental exposure of fields such as a hashed password that a manual assertion missed (the API snapshot testing write-up).

Automate capture, never automate approval. The script below records the key set per chunk and writes the proposed fixture, which a human then edits down to the strict set.

import json
from json_repair import repair_json  # tolerant of truncated prefixes

def keys_per_chunk(chunks, limit):
    seen, out = set(), []
    buf = ""
    for i, chunk in enumerate(chunks):
        buf += chunk
        partial = repair_json(buf, return_objects=True)
        if isinstance(partial, dict):
            seen |= set(partial.keys())
        out.append(sorted(seen))
        if i + 1 >= limit:
            break
    return out

The repair step exists because a truncated prefix is not valid JSON, and json_repair exists precisely to recover objects from malformed or cut-off text (the json_repair repository). Equivalent libraries in JavaScript do the same job, returning best-effort partial data from incomplete input (best-effort-json-parser, jsonchunk). Pick one and pin it, since repair behavior differs between libraries and a repair change can move your observed key set.

Review the output by hand against the derivation table. Delete every name you cannot justify, add anything you missed, and only then commit contract_v1.json. Resist the urge to regenerate it from production later without a review step; snapshot tests degrade into rubber stamps when nobody looks at the diff (the API snapshot testing write-up).

The check compares two sets: what the fixture requires and what the observed early chunks actually contain. It should be pure and fast, with no network call on the critical path.

import json

def check(observed_keys_by_chunk, fixture):
    budget = fixture["chunks"]
    window = observed_keys_by_chunk[:budget]
    seen = set().union(*window) if window else set()
    missing = [k for k in fixture["required"] if k not in seen]
    leaked = [k for k in fixture["forbidden_early"] if k in seen]
    return {"ok": not missing and not leaked, "missing": missing, "leaked": leaked}

fixture = json.load(open("contracts/contract_v1.json"))
result = check([["id"], ["id", "type"], ["id", "type", "status", "role"]], fixture)
assert result["ok"], result

Recording a new baseline should be a separate mode, run on purpose, that writes a candidate file next to the committed one rather than overwriting it. That keeps the failure message useful: the test says which required name was missing from which chunk window, and the reviewer sees the proposed change as a diff.

Canonicalize before comparing. Streams contain timestamps, request IDs, random UUIDs and unstable key ordering, and snapshotting them produces flaky tests that engineers learn to ignore (the HTTP/3 snapshot testing article). The telemetry snapshot pattern of replacing UUIDs and timestamps with placeholders and sorting keys before serialization transfers directly here (the OpenTelemetry snapshot guide). In practice you rarely need value canonicalization at all if you compare only key names, which is one more reason to scope the assertion to the prefix set.

Conditional fields are where prefix contracts usually rot. Three patterns cover most cases, and they differ in cost.

The first is a required set plus an allowed set. Any key in the allowed set may appear early; any key outside it is a warning, not a failure. This is loose enough to survive provider additions and tight enough to catch removals.

The second is a variant dimension in the fixture, keyed by a scenario name. A tool-call response and a text response have different natural orders, so give each its own entry inside the same version.

The third is numeric scoring instead of binary pass or fail. The practical version is a small score: start at the size of the required set, subtract one per missing name, add a warning for each unexpected early key, and fail below a threshold. This buys tolerance for one ambiguous field at the cost of a less obvious failure message.

Whichever you choose, keep nullability out of the prefix test. Whether id is null or absent is a schema question, and mixing it into the ordering test makes the failure hard to attribute. Contract tests verify the interface boundary and explicitly define nullability and required fields in the schema, not in an ad hoc assertion (the MinimumCD contract guide).

Bump the contract version when the strict set changes, the chunk budget changes, or a field moves between strict and soft. Do not bump it for a capture hash refresh caused by a provider changing tokenization, as long as the observed key set is unchanged; that is noise, and version churn erodes trust in the filename.

A decision rule that holds up in review:

Keep both fixtures for one release so you can run the old contract against the new provider and prove nothing broke before you delete it. That is the same compatibility discipline contract testing applies at the schema level, where breaking-change detection runs on every pull request (the MinimumCD contract guide).

When a real regression lands, the error-handling instinct is to retry. Resist that in the contract test. Retrying transient HTTP failures at runtime is correct, since rate limits and 5xx errors resolve on their own, and backoff with jitter is the standard remedy (the retries write-up). A deterministic ordering test that flakes under retries hides the drift you built it to catch.

Speed comes from separating capture from assertion. Do the network call once, store the chunk sequence as a small text file, and let CI read it. A hundred assertions against a committed 12 KB capture run in milliseconds; a hundred live model calls do not.

Two more levers:

Run the prefix assertion against a recorded capture, and run the live integration test on a schedule rather than on every commit. Contract tests are meant to be deterministic and fast, executed in pipelines against test doubles, with separate integration tests closing the gap to the real system (the MinimumCD contract guide).

Cap the work. Parse only the chunks inside the budget, and stop reading once the required set is satisfied. If your parser re-parses the whole prefix on every chunk, you are paying quadratic cost for no benefit; benchmarked numbers put a stateful incremental parse at 43 ms against 16.7 seconds for the naive re-parse on a 12 KB document (the typed-object gap write-up). A prefix contract naturally bounded at two or three chunks cannot drift into that territory.

Finally, treat the assertion as a guard, not a benchmark. If you want to know that progressive rendering actually helps, measure first meaningful paint separately. Community demand for partial structured output in agent tooling, including requests to stream JSON tokens rather than only the final object, is a sign that consumers increasingly depend on arrival shape and that prefix behavior deserves an explicit test rather than an implicit assumption (the Claude Code feature request).

If you are wiring several providers behind one client, testing the prefix contract against each model endpoint individually gets tedious fast. A single OpenAI-compatible endpoint that fronts text, image, audio and embedding models, such as Quanta Inference, lets you keep one capture harness and one fixture family rather than one per vendor.