A Compatibility Test for Tool Schemas Before Agents Switch

The mechanical part is simple: keep both schemas registered, give each a distinct name or a required version field, and route on that field rather than inferring from the presence of arguments. Inference is where most dual-schema setups break, because an argument that happens to exist in both shapes makes the router's guess arbitrary.

Whichever you pick, make the version explicit and required in the new shape. If the old shape has no version field, treat "absent" as version 1 for the duration of the window, and log it. Then record every call with its version, its raw arguments and the parsed internal request. That record is what makes the retirement decision auditable rather than a judgement call, which is the same reasoning behind authorization decision records that bind identity, tool, resource and policy version per Permit.io's audit trail guidance. For a schema migration the analogue is: version, raw arguments, parsed request, outcome.

One cost to budget for: tool definitions are loaded into context, and a server exposing dozens of tools burns tokens before any call is made as this comparison of MCP and CLI notes. Two versions of the same tool is the cheapest window you will ever run, so accept the duplication for the observation period and do not extend it out of habit.

The test you want is not "does the new parser accept the old payload." It is "do both parsers, given the same logical call, produce byte-identical internal requests." Write it as a table of cases, parse each case through both code paths, and compare the normalized internal request.

import json

CASES = [
    # (label, old_arguments, new_arguments)
    ("minimal", {"title": "Crash on save"}, {"title": "Crash on save"}),
    ("same logical call", {"path": "/tmp/a.txt"}, {"file_path": "/tmp/a.txt"}),
    ("explicit null", {"title": "x", "body": None}, {"title": "x", "body": None}),
    ("number as string", {"timeout": "30"}, {"timeout": 30}),
    ("extra field", {"title": "x", "legacy_flag": True}, {"title": "x"}),
]

def assert_equivalent(old_args, new_args):
    old_req = parse_old(old_args)   # raises on invalid input
    new_req = parse_new(new_args)
    assert normalize(old_req) == normalize(new_req), (old_req, new_req)

for label, old_args, new_args in CASES:
    assert_equivalent(old_args, new_args)

normalize should sort keys, drop fields the internal request does not own, and render missing values as an explicit sentinel so that "absent" and "null" never compare equal by accident. Run this as a gate that executes after the agent's last edit and before merge, the same position a quality gate occupies in a definition-of-done pipeline described here.

Assert on the exit code of the test command, not on the agent's summary. Exit status 0 means success, and the reserved codes above it carry meaning worth using: 1 for general failure, 2 for usage errors, 126 for a command that cannot execute and 127 for a command that was not found, with 128+n for termination by signal n per the exit code table. A schema compatibility gate that returns 2 when the fixture file is malformed saves you an hour of debugging a red build.

Most regressions land in a short list. Cover all six deliberately.

Required fields. Moving a field into or out of the required set is a breaking change in both directions. Make one case per required field where it is missing, and assert the exact error text, not just that an error occurred. Agents read error strings and retry against them, which is why actionable error messages are treated as documentation in CLI design per OpenStatus.

Extra fields. Decide explicitly whether unknown properties are rejected or ignored, and hold that decision across both versions. OpenAI-compatible clients and wrapper layers often add fields; silently dropping an unknown field that a caller believed was meaningful is the quiet failure mode.

Type coercion. "30" and 30 are not the same argument. Assert both that the intended coercion works and that the unintended one fails. This is the category where a rename plus a retype can pass review and still be wrong, because the value arrives, parses and looks plausible.

Nulls and defaults. Three distinct states exist: field absent, field present and null, field present with a value. Assert all three for every optional field, because null and "use the default" are frequently conflated by JSON schema handling in different stacks.

Nested objects. Deep structures hide breakage, because a top-level comparison passes while an inner field changed meaning. Compare the normalized nested request, and add at least one case with a nested list of two or more items so ordering and merging rules are exercised.

Error text. Pin the exact messages for a small set of cases, and accept that they will change, which is the point: a changed message should require a deliberate diff. The CLI guidance is blunt about this: an error should name what went wrong and what to do next, and missing-argument errors should list exactly which flags to supply per this CLI guide. The same applies to tool arguments.

A checklist you can paste into the pull request template:

Take a tool that reads a file. The old schema takes path (string) and max_bytes (string, because the first version passed it through a shell). The new schema takes file_path (string) and max_bytes (integer). Three separate changes ride together: a rename, a retype, and a constraint.

Parse both into the same internal request:

{"file": "/tmp/a.txt", "limit": 4096}

The cases that matter, with what each must prove:

The fifth row is the one people forget. Retyping from string to integer opens a new range of values the old schema could not express, and negative sizes or zero limits are exactly where the old "4096" path never went. The sixth row is the decision you must write down: strict rejection is safer, lenient coercion is friendlier, and either is defensible as long as the test asserts which one you chose.

Then run the pair in production for a fixed window, and watch two numbers: mismatches between old and new parsed requests for the same logical call, and the count of calls carrying the old version. Retire when both are zero across the whole window.

Retirement is a policy, not an event. Write it down before the window opens so nobody negotiates with it at the end.

1. Fixed observation window. Pick a duration long enough to cover the slowest caller you know about, for example one full billing cycle for a scheduled job. Do not extend it because of a hunch, and do not shorten it because a dashboard is green.
2. Zero mismatches. Every recorded call parsed through both paths and compared. Mismatches, not errors, are the metric; a silent coercion never shows up as an error.
3. Deprecation notice. Announce the removal date and the new shape, and put a warning in the old parser's response so an agent reading the result sees the deadline. Machine-readable warnings are the same pattern as structured errors on stdout with everything else on stderr per this CLI guide.
4. Working fallback. Keep the old parser reachable behind a flag for a short grace period, and make the fallback emit a warning rather than going silent. A fallback that hides itself is indistinguishable from a caller that never migrated.

Then delete the old schema in a separate change from the new one, so a rollback restores exactly one thing. Record the retirement with the same envelope you used during the window: version, window dates, mismatch count, and the final old-version call count. That record answers a future "why did we think this was safe" question without a memory of the incident.

If the agent that reads your tool schemas runs against several model providers, the compatibility surface multiplies: one model may emit "4096" where another emits 4096 for the same field. Testing the parsers against recorded transcripts from each provider you actually use is cheaper than discovering the difference in production. A single API across text, image and other model types reduces how many transcripts you have to collect; Quanta Inference (https://quantaintelligence.ai/inference) exposes them through one OpenAI-compatible endpoint, which keeps the recorded-call set small.