What a Gateway Should Assert Before Forwarding a Model Reply

The order matters because each check gets cheaper to reject than the next, and because later checks depend on earlier ones passing.

Run them in that order. Content type is a header check that costs nothing. First-key presence needs only the opening bytes, which arrive before the rest of the reply. The size cap can often be evaluated from the declared length before any body is read, and falls back to a running count when the upstream uses chunked encoding without a length. The schema hash is the only check that needs the whole object, so it goes last and only runs on replies that survived everything else.

A worked example with numbers. Suppose the contract is:

{
  "required_first_key": "answer",
  "allowed_keys": ["answer", "confidence", "citations"],
  "max_bytes": 8192,
  "schema_hash": "sha256:9f2c..."
}

A 6 KB reply that starts with {"answer": "The cap is 8192 bytes", "confidence": 0.81} passes all four. A 40 KB reply with the correct first key fails at check 3, and the gateway never buffers the remaining 34 KB. A reply that starts with {"confidence": 0.81, "answer": ...} fails at check 2 even though every key is present, because the contract pins the first key, not just the set. That pinning is deliberate: JSON objects are unordered in the specification, so a stable first key only means something if the contract requires it and the upstream actually emits it.

The schema hash is where most teams get it wrong. Hashing the raw bytes of the reply is useless, because the model produces different text every time. Hash a canonical projection instead: sort the top-level keys, record each key's type, and recurse one level into nested objects. Then compare the hash of that projection against the pinned value. That catches an upstream that added a key, dropped one, or changed confidence from a number to a string, without failing on every new sentence.

Streaming replies make the size cap load-bearing. A consumer that parses incrementally and renders as tokens arrive has no natural stopping point until the stream closes. If the upstream misbehaves and keeps emitting, the consumer keeps allocating.

The performance literature on partial parsing shows how fast that goes wrong. A naive implementation that re-parses the whole buffer on every chunk is O(n²): one write-up measured 16.7 seconds for a 12 KB document, with individual late chunks taking 19 to 20 milliseconds each, against 43 milliseconds for a stateful incremental parser in this analysis of streaming JSON parsers. The naive version became unusable around the 5 KB mark.

That number gives you a default. If your incremental consumers degrade near 5 KB of accumulated buffer, a cap of a few times that leaves headroom without inviting pathological replies. The cap belongs at the gateway because it is the only party that can stop the bytes before they reach every consumer at once. It also bounds the refusal case: when generation fails and an upstream retries and re-streams, the cap limits how much reaches the caller before the gateway cuts it off.

The first-key check is a contract test in miniature. Contract testing literature is clear about scope: consumer tests should assert only on the subset of the response the consumer actually uses, and providers ask whether their changes break any consumer in the Pact consumer documentation. A gateway enforcing a first-key contract does the consumer-side job for every consumer at once.

The MinimumCD contract testing guide draws the same line. Contract tests target only the boundary interface: payload shapes, headers, status codes, message formats. They do not test business logic or deep end-to-end behavior in the MinimumCD practice guide. The first key, the allowed key set, and the size cap are boundary facts. They are exactly the things a gateway can check without knowing what the reply means.

There is a second reason. Contract drift is silent until it is expensive. The Kreya snapshot testing writeup describes the failure mode directly: a hashed password field added to a user object response, passing review because no test asserted the field should not appear in the Kreya snapshot testing article. A schema hash catches that class of change at the gateway, on the first reply, for every caller.

Gateways already do this on the request side. KrakenD's response schema validator runs a JSON schema check before the gateway returns a response to the end user, supporting several JSON Schema drafts in the KrakenD documentation. The first-key contract is the cheap, streaming-compatible subset of the same idea: it does not need a full schema validator, which matters because a full validator needs the whole document and a streaming reply might not close for seconds.

When any assertion fails, the gateway must not forward the upstream body. It should return its own response with a stable shape, so consumers can handle it without guessing.

{
  "error": "upstream_contract_violation",
  "check": "first_key",
  "expected": { "first_key": "answer" },
  "observed": { "first_key": "error" },
  "reply_bytes_read": 212,
  "upstream_request_id": "req_8f21c"
}

Five rules make that refusal useful:

1. Use a distinct HTTP status or error code that consumers can branch on without parsing prose. A 502 or 424 signals a gateway-side failure, not a caller mistake.
2. Name the check that failed with one of four fixed values: content_type, first_key, size_cap, schema_hash. Fixed values make dashboards trivially groupable.
3. Echo the expected and observed values, but truncate them. Never echo more than a few hundred bytes of the upstream body, because it may contain sensitive prompt content.
4. Record how many bytes were read before the failure. This tells you whether the gateway caught the violation early or late.
5. Pass through the upstream request ID if the provider supplied one, so the failure can be correlated on the provider side.

For streaming consumers, the refusal cannot arrive as a JSON body if the response has already started with text/event-stream. Send it as a terminal event in the same stream format, using the same error object, then close the stream. Consumers that parse incrementally will see the error as the last event instead of a truncated object, which is the failure mode you want.

Most gateways give you a hook that runs before the response is forwarded. Caddy's reverse proxy documentation describes response interception through handle_response with a matcher on status or header, which is where these checks attach in the Caddy reverse_proxy reference. Whatever gateway you run, the shape of the code is the same:

def check_first_bytes(status, headers, first_chunk, contract):
    ctype = headers.get("content-type", "")
    if not ctype.startswith("application/json"):
        return refusal("content_type", {"expected": "application/json"}, {"observed": ctype})

    text = first_chunk.lstrip(b"\xef\xbb\xbf \t\r\n").decode("utf-8", "replace")
    if contract["fence_ok"]:
        text = strip_markdown_fence(text)
    if not text.startswith("{"):
        return refusal("first_key", {"first_key": contract["required_first_key"]}, {"observed": text[:1]})

    first_key = read_first_key(text)
    if first_key != contract["required_first_key"]:
        return refusal("first_key", {"first_key": contract["required_first_key"]}, {"observed": first_key})

    declared = headers.get("content-length")
    if declared and int(declared) > contract["max_bytes"]:
        return refusal("size_cap", {"max_bytes": contract["max_bytes"]}, {"declared": int(declared)})

    return None  # pass, continue to schema hash check on the full object

Three implementation details decide whether this works in practice.

First, read_first_key must handle partial input. You cannot call a full JSON parser on the opening bytes, because they are almost never valid JSON on their own. Write a small tokenizer that walks the opening {, skips whitespace, reads a quoted string, and stops. That is the same incremental-tokenization problem the streaming parsers solve, and it is a few dozen lines for the first-key case.

Second, decide fenced replies explicitly. best-effort-json-parser extracts JSON from fenced blocks automatically and leaves the remaining text in lastParseReminding in its npm documentation. If your prompts can produce fenced output, the gateway must strip the fence before the first-key check, and the contract should record that fenced replies are allowed. If they are not allowed, failing at check 2 with the backtick as the observed byte is correct, and it shows you which prompt paths need tightening.

Third, handle top-level arrays. A reply that opens with [ fails a first-key check by design. That is usually right, because a top-level array gives you no named field to pin, and the parsing confusion around top-level arrays is well documented in this Stack Overflow question. If you need arrays, add a separate contract variant that allows [ and pins the element key set instead.

Turn the checks on in report-only mode first. Everything passes through, but every violation is logged with the check name and the upstream that produced it. Run that for a week of normal traffic and group the logs by check. The distribution tells you which check is safe to enforce first, and it usually is content type and size cap, because those failures are almost always genuine upstream faults rather than contract mismatches.

Then enforce in that order, and only after the report-only logs for that check show zero violations from healthy upstreams. A first-key check enforced before the upstream is pinned to emit that key first will reject everything, which is a fast way to lose trust in the whole mechanism.

Snapshot tests belong alongside this. A canonical snapshot of a passing reply, with volatile fields replaced by placeholders, gives you a regression test for the gateway's own checks, and stripping non-deterministic fields before comparison is standard practice in telemetry snapshot testing in this OneUptime guide. Run the gateway checks against that snapshot in CI so a change to the canonicalization, and therefore to the schema hash, fails the build before it fails production.

If you run several models behind the gateway, centralize the contract per model. Quanta Inference puts text, image, video, audio and embedding models behind one OpenAI-compatible API key at Quanta Inference, which keeps the reply envelope stable while the model underneath changes. That is the point of the gateway contract: the checks describe the envelope, not the model, so swapping providers does not mean rewriting every consumer.