Cutting Live Agents to a New Tool Schema

The protocol has one write path and one observe path. The write path is the production tool call through the schema the agent is currently using. The observe path is the same call replayed against the candidate schema with side effects suppressed.

Side-effect suppression is the hard part and it is where naive implementations break. A beginner's guide to shadow testing from F22 Labs lists the failure modes directly: mirrored requests can duplicate database writes, trigger emails, expose personal information and double infrastructure load unless the shadow environment is isolated. It also stresses that if the shadow call is slow or fails, the production response must be unaffected.

Apply the same isolation rules to tool mirrors:

• Read-only tools replay against a replica or a snapshot with real data, because the worst case is a wasted query.
• Write tools replay against a dry-run mode, a sandbox tenant, or a stub that validates arguments and returns a synthetic result. The candidate schema is validated on argument shape and return shape, not on the real write.
• Tools that send (email, SMS, push) never replay at all. Capture the arguments, skip execution, and score the call on arguments and schema validity alone.
• Latency budget: the mirror runs asynchronously off the response path, queued and drained by a worker, so a slow replay never adds milliseconds to the user's call.

A minimal interceptor in the tool layer records both argument sets and statuses under a shared correlation ID, which is the same idea as the trace ID that Permit.io's audit trail guidance puts at the center of each agent action record.

import uuid
from datetime import datetime, timezone

def invoke_tool(tool_id, args, ctx, store):
    trace_id = str(uuid.uuid4())
    # 1. Real call, current schema, real side effect.
    current = ctx.registry.current(tool_id)
    outcome = current.run(args, ctx)

    # 2. Replay, candidate schema, side effects suppressed.
    candidate = ctx.registry.candidate(tool_id)
    if candidate and ctx.mirror_enabled(tool_id):
        record = {
            "trace_id": trace_id,
            "tool_id": tool_id,
            "current_version": current.version,
            "candidate_version": candidate.version,
            "ts": datetime.now(timezone.utc).isoformat(),
            "current_args": current.normalize(args),
            "outcome": outcome.status,
        }
        try:
            shadow_args, shadow_result = candidate.validate_dry_run(args)
            record["candidate_args"] = shadow_args
            record["candidate_status"] = shadow_result.status
            record["discrepancies"] = compare(
                record["current_args"], shadow_args
            )
        except Exception as exc:
            record["candidate_error"] = type(exc).__name__ + ": " + str(exc)
        store.enqueue(record)  # async, off the response path
    return outcome

Two details matter more than they look. First, the replay reads the same raw arguments the model produced, not the normalized ones, so the candidate validator sees what the old validator actually accepted. Second, the error path records the exception class, because a candidate schema that rejects a call the old schema accepted is the single most valuable disagreement you will find.

If the tool layer fronts several vendors' models, the interceptor belongs behind one gateway rather than inside each integration. Quanta Inference at https://quantaintelligence.ai/inference gives you text, image, video, audio and embedding models through one OpenAI-compatible key, which keeps the tool-calling surface uniform enough that one interceptor covers every caller.

Argument comparison is where shadowing earns its keep or turns into noise. Raw JSON differences are useless: {"amount": 100} and {"amount": 100.0} are the same intent, and {"qty": "3"} versus {"qty": 3} may or may not be.

Normalize first, then classify. A workable order:

1. Coerce and canonicalize. Trim strings, collapse whitespace, lowercase enums if the old schema was case-insensitive, convert numeric strings to numbers where the old schema accepted them, and sort keys. Apply the same date and currency formatting on both sides, which is the same principle behind the additive total and totalMoney example in Milan Jovanovic's versioning piece.
2. Required fields. Any argument the current schema requires and the candidate no longer accepts is a hard fail. Any newly required argument the model did not supply is also a hard fail, because the candidate call failed validation.
3. Optional fields. A dropped optional field is usually benign; a renamed optional field is a silent data loss risk and deserves a warning class.
4. Types. String to integer, number to string, object to scalar. Signadot's account of shadow-based contract testing describes a fintech case where a string-to-integer field conversion slipped past OpenAPI validation but showed up immediately in response comparison. Expect the same class of miss in tool arguments.
5. Enums. Removed enum values block. Added values are safe only if the downstream service tolerates unknown values, and you should verify that rather than assume it.
6. Ranges and formats. Minimum and maximum bounds, string length caps, regex patterns, and units. A candidate that narrows a range will reject live calls the old schema accepted, so score those as warnings against real traffic.
7. Outcome shape. Compare status codes and top-level response keys, not full bodies. Full-body diffs on generative outputs will drown you in noise.

Emit one record per mirror with a discrepancies array of {field, category, severity} objects. The categories are the point: they let you see that most disagreements on one tool are a cosmetic enum casing change while a handful are type breaks.

A compact comparator that does steps 3 through 6:

SEVERITY = {
    "required_removed": "block",
    "required_added": "block",
    "type_changed": "block",
    "enum_value_removed": "block",
    "range_tightened": "warn",
    "optional_renamed": "warn",
    "optional_removed": "info",
}

def compare(old, new):
    out = []
    for field, spec in old["required"].items():
        if field not in new["properties"]:
            out.append({"field": field, "category": "required_removed"})
            continue
        n = new["properties"][field]
        if n.get("type") != spec.get("type"):
            out.append({"field": field, "category": "type_changed"})
        old_enum = set(spec.get("enum", []))
        new_enum = set(n.get("enum", []))
        if old_enum - new_enum:
            out.append({"field": field, "category": "enum_value_removed"})
        if tighter(spec, n):
            out.append({"field": field, "category": "range_tightened"})
    for field in old["properties"]:
        if field not in new["properties"] and field not in old["required"]:
            out.append({"field": field, "category": "optional_removed"})
    return [
        {"field": d["field"], "category": d["category"],
         "severity": SEVERITY[d["category"]]}
        for d in out
    ]

Run the schema diff once at deploy time and the argument comparator on every mirrored call. The first tells you what changed; the second tells you which changes actually bite under live traffic.

Here is the decision rule that separates a calm cutover from a gamble. For each tool, define:

• D = disagreements / mirrored calls, where a disagreement is any warning or block record, computed daily and across the whole burn-in.
• B = the ceiling you wrote down in advance.
• W = the burn-in window, expressed in days and a minimum call count.
• Z = zero block-severity discrepancies on the final day of burn-in.

Then the tool retires when D stays below B for every counted day of W, the total mirrored call count meets the minimum, and Z holds. That last condition matters because rates hide absolute counts: at 1,000 calls a day, a 1 percent ceiling permits ten disagreements, and if all ten are required-field breaks, the rate is irrelevant.

A worked example. Suppose one tool sees 40,000 live calls over a 14-day burn-in and the mirror captures all of them. You set B at 1 percent, and you declare that only the second half of the window counts, because the first half is a fixing period. Daily disagreement counts run 410, 380, 350, 300, 220, 180, 140, 110, 90, 70, 60, 50, 40, 30. The counted half runs from 90 down to 30, and every counted day sits under the 400-call ceiling. Total disagreements are about 2,430, roughly 6 percent of all calls across the window, but only the trailing stretch decides the rule.

Add a decay condition so a late spike still blocks you: require each of the final three days to stay below half of B, which in this case means under 200 calls. With days of 50, 40 and 30, the tool passes on the numbers and the decision falls through to the Z check on block-severity records.

If instead you segment by tool, the decision becomes per-tool and migration stops being all-or-nothing. A common shape:

Those numbers are starting points you set yourself, not industry standards. The discipline is that you choose them before burn-in and then do not move them, the same principle as fixing a statistical test before collecting data.

Cutover itself is dull by design. You flip routing so new calls use the candidate schema as current, keep the old schema loaded in the registry, and keep the mirror running for a short confirmation window, this time comparing the new current against the retired version for any call that would have failed there.

Rollback stays one step: a feature flag or routing rule that points the tool back at the old schema. Keep it active until the confirmation window closes, then keep it documented and tested but off. Portainer's comparison of shadow, canary and blue-green makes the trade-off plain: shadow exposes no users to the new version, but blue-green gives you a fast full rollback by flipping traffic back. The one-step flag is how you get both properties.

Evidence is the part teams skip and later regret. Archive, per tool and per day, the mirror volume, the disagreement counts by category and severity, the version pair compared, the comparator version, and the exact threshold you applied. This is not bureaucracy. Permit.io's audit trail post is blunt that tool traces are operational telemetry rather than control evidence, and lists what an audit-ready record needs: agent identity, delegation chain, tool and parameters, policy version, decision and reason, timestamp, trace ID, and the outcome of the side effect. Your shadow records should carry the same envelope, because they will be read the same way.

Two further practices matter. Runlayer's audit guide warns against shared service accounts that make it impossible to say which agent did what, and recommends sequence-based alerting rather than single-event alerts, because anomalies live in patterns. A schema retirement produces exactly that kind of pattern: a spike in a new disagreement category mid-burn-in is the alert you want, and a single odd call is not. The same guide recommends storing a cryptographic hash of each log entry to detect tampering, which is cheap and appropriate for shadow archives you intend to keep.

JSON Lines with a per-batch hash is enough:

{"trace_id":"...","tool_id":"orders.create","current_version":"v1","candidate_version":"v2","discrepancies":[{"field":"amount","category":"type_changed","severity":"block"}],"day":"2026-06-01","hash":"sha256:..."}

Store at rest under the same retention policy as your other audit data and keep the manifest of thresholds alongside it, so a reader can reconstruct why the cutover was justified.

Shadow mirroring assumes the candidate schema sees representative traffic. When it does not, the protocol lies to you.

Three cases need separate handling:

• Low-volume or seasonal tools. A tool called twice a day will not clear a minimum call count in any reasonable window. Fall back to a contract test suite plus a schema diff, and accept that the disagreement rate is unknown.
• Tools whose inputs are correlated. If the model only calls a tool with one shape because the description nudges it that way, the mirror never exercises the untested optional fields. Add synthetic calls that cover every required and optional field, enums at their boundaries, and range edges. Baseten's function-calling documentation is a useful reminder of the real contract here: a tool is a name, a description and a JSON-schema definition, and models fill it from docstrings, so a schema change that also edits the description changes model behavior, not just validation.
• Multi-version routing where you already segment by caller. If your gateway already routes by version, which the Lunar.dev analysis of CLI versus MCP frames as a governance boundary rather than a protocol argument, then shadowing becomes a per-cohort measurement: mirror within the cohort that moves next, not globally.

One more property is worth knowing. Schema disagreement is a weaker signal than behavior disagreement, and behavior is what users notice. If a candidate schema accepts every argument but the tool behind it returns a different shape, your comparator must score the outcome too, at least on status and top-level keys, or you will retire a version that breaks callers downstream. Comparing candidate output to baseline output on live traffic is why the shadow pattern exists at all, and the same caution applies here.