Payload Shapes Across Model Families: An Acceptance Table

The table has two axes. On one axis, model families (not individual model IDs, unless two models in the same family differ). On the other, payload features that routing can break. The features worth a column are the ones where a body that passes on one family returns a 400, gets silently rewritten, or produces a response your parser cannot read.

Google's own function-calling reference documents the declaration and FunctionCallingConfig parameters as its contract, which is a good reminder that the tool schema is an API contract, not a helper type (Gemini function calling reference). OpenAI's function-calling guide likewise states that some model generations require one API surface for tool calling and that the other examples use a specific model with reasoning disabled, which means "tool call works on this family" is itself a conditional claim you have to encode (OpenAI function calling).

Each cell of the table is a fixture with four parts: the exact request body that was sent, the response status, the response body as received, and the minimum assertion that must hold. The first two are facts you record. The third becomes the golden file. The fourth is the decision you are actually testing.

The golden-file pattern is well established: store the expected output on disk, compare actual output against it, and provide a flag to regenerate the file when behavior legitimately changes (writing integration tests for a Go CLI). Table-driven tests make the feature list cheap to extend, which is exactly what you want when a new payload feature or a new family arrives.

The important discipline is what you compare. Pact's consumer-testing guidance is blunt about this: assert only on things that would affect the consumer if they changed, and choose exact or loose matching field by field. A full-schema snapshot breaks on harmless serialization changes, while targeted assertions survive them (Pact consumer tests). For an acceptance table that means asserting the tool name, the required fields, the field types and your own routing invariant, not the entire serialized provider schema.

A second technique worth combining in: record real traffic against each family, then replay it against the candidate after a routing change. Record-replay tooling captures actual requests and responses, auto-generates mocks from dependency responses, and stores cases as version-controlled YAML that can be diffed in review (Keploy record and replay guide). The matrix tells you what to test. Replay tells you whether the new route reproduces the recorded behavior.

Suppose your router sends a support-triage workload to three families: family A (your current default), family B (a cheaper family you want to shift batch traffic to), and family C (a fallback).

You build the matrix as 3 families by 8 payload features, which is 24 cells. Filling it honestly, you find that family B rejects a top-level $ref in the tool schema, so 2 of the 8 features (tool schema and tool choice) fail on B for the affected tools. Family C accepts the streaming request but emits a different event name for the terminal event, so your reassembler times out on 1 feature.

Without the matrix, the cutover for B and C looks fine on the happy-path smoke test. With it, the failure rate is visible before shipping. A simple way to quantify the decision: if B carries 70 percent of batch traffic and 2 of 8 features are broken for the tool-calling subset, then surface the break against the share of requests that actually use tools, not against total traffic. A defect rate expressed per feature hides how much of the workload it touches; a defect rate per request-hour does not.

The cost side is equally concrete. A full 24-cell run against live endpoints is a small number of calls, so it belongs in the pipeline that gates a routing change rather than in a nightly job. The pattern to copy is the ASP.NET Core test host: a test project references the system under test, spins up a host with test configuration, sends real requests through the client, and asserts on the actual response (ASP.NET Core integration tests). Swap the HTTP client for your router and the assertions become your minimum assertions.

Three rules keep the table from becoming unmaintainable.

Rule one: block on broken cells, warn on unverified cells. A cell with a recorded failure for a feature the workload uses must fail the build. A cell nobody has recorded yet must produce a warning, not a pass, because an unverified cell is exactly where a cutover surprise hides. Keploy's comparison of record-replay against manual authoring makes the same point from the maintenance side: manual tests drift, while re-recording refreshes coverage cheaply (record and replay).

Rule two: classify errors by transience, not by exception type. Retry on rate limits, 5xx and timeouts. Do not retry on 400, 401, 404 or a capability refusal, because a schema mismatch will fail identically on every attempt (error handling between agents). This rule changes what your fixture must assert: it must assert the error class your classifier produces, not only that the call succeeded.

Rule three: assert the error envelope, not just the happy path. Providers return different error shapes, and the mature ones follow a consistent structure with a code, a message, and enough context for a machine to act. The advertising protocol spec formalizes this with a two-layer model, where a fatal failure populates both the payload errors[] array and a typed transport signal, and warns that populating only one of the two is the single most common source of interop bugs (AdCP error handling). That is a good default for your own error mapping: one typed code your router can switch on, plus a human-readable message in the body.

{
  "family": "B",
  "feature": "tool_schema",
  "request": { "method": "POST", "path": "/v1/messages", "body": { "tools": [ "..." ], "messages": [ "..." ] } },
  "observed": { "status": 400, "body": { "error": { "code": "invalid_schema", "message": "items must declare a type" } } },
  "assert": { "status_is": 200, "tool_call_arguments": ["order_id"] },
  "verdict": "block"
}

The verdict field is what makes the fixture a decision. Copy the observed response verbatim, then write down the assertion you require. When the provider fixes the bug or your adapter adds a rewrite, you re-record and the diff shows exactly what changed.

The integration test sequence maps neatly onto this: arrange the request for the target family, act by sending it through the router, assert against the minimum assertion, and report per cell (ASP.NET Core integration tests). Make the test runner emit one result per (family, feature) pair so a failing cell names both coordinates in the report.

Gate cutover on the table, not on a smoke test. A backward-compatibility program treats schema diff, contract testing, recorded replay and end-to-end journey checks as complementary rather than substitutes, because each catches a different category of break (backward compatibility testing). Your matrix is the contract-testing layer for provider payloads specifically. Replay covers the response-shape drift. End-to-end checks cover the agent workflow on top.

Name an owner for each cell. The schema-drift incident above turned on a boring administrative fact: the agent was a consumer nobody had on the deprecation list, so the migration window protected every human consumer and none of the automated ones (tool schema migration). If a row in your matrix has no named owner, treat its cells as unverified by default.

One practical shortlist before you write any code: pick the five features that cause the most breaks in your own logs (system placement, tool schema, tool choice, response envelope, streaming termination are the usual suspects), record a passing and a failing example for each family, and only then automate.

Teams increasingly route through a single gateway rather than calling each family's native endpoint. That removes some shape differences in the request, but it does not remove them from the contract: a translating gateway carries, converts, drops or rejects different fields per route, and thinking blocks, cache breakpoints and tool call IDs are common casualties (what a translating gateway drops). Add a row to the matrix for every route you expose, even if the body is nominally identical.

If you would rather spend engineering effort on the workload than on building and maintaining routing adapters per provider, a single OpenAI-compatible endpoint for text, image, video, audio and embedding models through one prepaid API key is the natural next step: Quanta Inference lets you keep one request shape and test against the families you actually route to.