Contract Tests for Tool Schemas When a Field Changes

Not every change to a tool schema is dangerous. Some are safe expansions that your code should tolerate, and some will break parsing or routing the first time the model exercises the new shape. Postel's law applies: be liberal in what you accept, conservative in what you send the Pact documentation on writing consumer tests. The trick is knowing which changes fall on which side of that line.

Two rows deserve a warning. Enum narrowing is the sneakiest, because a missing value can make a model pick the nearest legal option and produce plausible but wrong argument values, and nothing in your logs says "the enum changed." Description edits are invisible to structural assertions; a rewritten description can move tool selection quality without touching a single type, so snapshot descriptions and review diffs rather than asserting byte equality the Pact documentation on writing consumer tests.

The contract file should be generated from the exact object your code hands to the model vendor, not from a separate document someone maintains by hand. This is the branch by abstraction idea applied to model providers: put an abstraction in front of the provider so the rest of the system deals with a stable interface, and change one implementation at a time Martin Fowler's description of branch by abstraction. Your abstraction is a single module that builds tool definitions, so the snapshot has one source.

A minimal pattern: a function that returns the tool definitions, a JSON snapshot committed to the repo, and a test that compares the live definition against the snapshot.

# tools/definitions.py
def build_tools():
    return [{
        "type": "function",
        "function": {
            "name": "get_order",
            "description": "Fetch one order by its identifier.",
            "parameters": {
                "type": "object",
                "properties": {
                    "order_id": {"type": "string"},
                    "currency": {"type": "string", "enum": ["USD", "EUR", "GBP"]},
                },
                "required": ["order_id"],
            },
        },
    }]

The comparison test uses the plain assert style that pytest rewrites for readable failure output the pytest API reference. Keep the assertion set narrow: only fields your code actually reads. The Pact guidance is that you should only assert on things whose change would affect the consumer, so the provider can evolve without false alarms the Pact documentation on writing consumer tests.

from tools.definitions import build_tools

READ_FIELDS = {"get_order": {"order_id": "string", "currency": "string"}}
REQUIRED_ENUMS = {"get_order": {"currency": {"USD", "EUR", "GBP"}}}

def test_no_rename_or_type_change():
    for tool in build_tools():
        name = tool["function"]["name"]
        props = tool["function"]["parameters"]["properties"]
        for field, expected in READ_FIELDS[name].items():
            assert field in props, f"{name}.{field} disappeared"
            assert props[field]["type"] == expected, f"{name}.{field} type moved"

def test_enums_still_cover_our_mapping():
    for tool in build_tools():
        name = tool["function"]["name"]
        props = tool["function"]["parameters"]["properties"]
        for field, needed in REQUIRED_ENUMS[name].items():
            assert needed <= set(props[field]["enum"]), f"{name}.{field} enum narrowed"

def test_no_new_required_property():
    for tool in build_tools():
        params = tool["function"]["parameters"]
        assert set(params.get("required", [])) <= {"order_id", "currency"}, \
            f"{params['required']} added a required argument"

Run this in CI on every commit; it is pure JSON comparison, milliseconds per test, and needs no network. The contract-testing literature is consistent on the economics: because contract tests only touch the interface, they are fast, deterministic and trivially parallel the totalshiftleft guide on API contract testing.

A structural contract is blind to three things, and a good test suite names them so nobody assumes coverage it does not have.

First, it cannot see behavior. Whether the model actually calls get_order with a sensible order_id is a functional question, not a contract question; Pact's own docs draw this line sharply, noting that contract tests should not be aimed at exposing bugs in the provider the Pact documentation on writing consumer tests. Keep a separate evaluation suite for tool-selection quality.

Second, it cannot see semantics. A status field that changes from "shipped" to "in_transit" while both remain legal strings passes every structural check a discussion on detecting API contract drift. If your prompts depend on the meaning of a value, add an explicit allowlist assertion on the values you map, separate from the type check.

Third, it cannot see environment-specific differences. Vendors occasionally expose slightly different tool catalogs across endpoints or regions. The contract discussion on drift notes that response shape can change "only in specific environments" a discussion on detecting API contract drift. Run the snapshot test against the exact endpoint you deploy to, and record which environment produced the snapshot.

One more structural check is worth adding: tool count. If a provider splits get_order into two tools, a per-tool assertion loop still passes for the missing name only if the name itself is gone, but a model that once had one option now has two and selection behavior changes. Assert the set of tool names, not just their contents.

Spec-driven tooling gets you charted, executable contracts from a specification. Specmatic runs an OpenAPI document as a contract test against a live service and reports pass counts per endpoint the Specmatic contract testing documentation, and the Microcks approach treats the spec as the source of truth for both mocks and tests the Microcks and Pact comparison. The equivalent for tools is to keep a JSON Schema file for each tool's parameters object and validate your builder output against it, which also gives you a shareable artifact for prompt and description review.

Snapshots prove your code is stable. They do not prove the vendor's live behavior. Once per pipeline (not per commit, not per test case), make a single live call per tool with a canonical input and verify the argument names and types that come back. This is the consumer-driven half: the recorded expectation is yours, the verification runs against the real provider the PactFlow guide to user-driven contract workflow.

A worked example with numbers. Suppose you have 6 tools, a CI pipeline that runs 60 times a day, and one live call per tool per pipeline. That is 360 metered requests a day just for contract verification. If each verification request carries roughly 1,200 input tokens (schema plus a short prompt) and about 80 output tokens (one tool call), you are paying for roughly 430,000 input tokens and 29,000 output tokens daily. Multiply by your vendor's per-token rate and you have the exact cost of the safety net. The token budget guidance is to log prompt tokens, completion tokens, model, feature and estimated cost for every request so this number stays visible rather than surprising the Yotta Labs guide to token budgets.

Three rules keep live calls cheap and meaningful:

1. Set tool_choice to force the specific tool, so the test measures argument shape rather than selection quality.
2. Use a deterministic canonical input (a fixed order ID, fixed currency) so failures are attributable to the contract, not to sampling.
3. Fail the pipeline on mismatch, but call a separate alert channel so a vendor change does not block unrelated merges; run the check on a schedule if your pipeline volume is high.

If you use one API surface for several model families, the router is also the natural place to centralize the snapshot and the verifier, so each vendor's tools get the same assertion set. Quanta Inference exposes text, image, video, audio and embedding models behind one OpenAI-compatible key, which means one contract harness can cover every model you route to (https://quantaintelligence.ai/inference).

The order of operations matters. Structural assertions first, because they are free and catch your own regressions. Then a schema-validation pass against the committed JSON Schema. Then the single live call. Then, only on failure, a recorded diff artifact attached to the build.

Pact-style tooling adds a deployment gate on top: the broker answers whether a version is safe to deploy given verification results the PactFlow guide to user-driven contract workflow. You can approximate this without a broker: a small JSON file mapping tool name to last verified provider version, checked at release time. If the provider moved since the last verification, the release waits for one live call. Feature-flagged rollout patterns apply here too, since they let you ship the new prompt or tool wiring behind a switch and roll back without redeploying the Harness guide to feature flags.

Review discipline is the part teams skip. A schema diff that only adds an optional property should be merged without ceremony. A diff that touches required, an enum, or a type needs a named reviewer who owns the tool. Swagger and OpenAPI validation tooling will flag required fields, types, enums and unexpected fields automatically a discussion on detecting API contract drift, and you can port the same checklist onto your tool snapshot. The test that fails on a new required property is the highest-value single assertion in the file: it catches the change that would otherwise surface as a production validation error.

For code-generation approaches, generate the schema from typed code and diff it against the approved schema, failing the build when the full schema does not match the Pactflow guide to schema-based contract testing. Typed tool builders in TypeScript, Pydantic models in Python, or Go structs all give you this for free.