Setting the Tolerance Band for an Agent Equivalence Rule

Write the policy as a table before you write any code. Here is a starting structure, with the rationale in plain words.

The text band deserves the most care, because 0.92 is a starting number, not a law. Calibrate it: take 200 past edits that reviewers already judged equivalent, score them with your embedding model, and set the threshold below the lowest score in that set but above the scores of edits you judged non-equivalent. If your sample clusters at 0.95 and up, 0.92 gives you headroom. If your sample has pairs at 0.88 that humans still called equivalent, 0.92 will generate false alarms every week and your team will learn to ignore them.

Structure is the opposite: no tolerance at all for required keys. Downstream code that reads a field does not care that the sentence above it changed by three words, and it cares very much when the key disappears.

Tool calls need their own rule because exact-match assertions stop working the moment a model chooses what to do next. Running the same refund request twice can yield two valid routes, with the agent looking up an order first in one run and checking the refund policy first in another, and both routes are correct when the required checks happen before the refund (Braintrust). That is the decision rule for order tolerance: allow reordering only when no call depends on another's output and every tool involved is safe to repeat.

Two extra checks earn their keep here. Treat each connected agent as an agentic tool with a description from the parent's point of view, and log when it was invoked and what it did, since the parent and the child keep separate transcripts that you will need to correlate when something breaks (Microsoft Learn). And keep the fixture's tool grants minimal, because an overprivileged agent widens the blast radius of any mistake far faster than a human user would (ThreatLocker). A fixture that can delete records is not the one to run on every pull request.

Arguments get a split rule. Identifiers, amounts and enumerated values compare exactly. Free-text fields such as a note or a summary get a similarity band like the response text. The split exists because the cost of the two errors is asymmetric: a reworded note costs a reviewer a glance, and a transposed order ID costs a customer money.

Latency and token counts belong in the band, but they are early-warning signals rather than definitions of behavior change. A session that suddenly calls far more tools than its baseline is the case worth catching, and one public drift monitor handles that with simple statistics: after the first 10 completed sessions it computes a mean and standard deviation for input tokens, output tokens, session duration and tool call count, then flags a candidate when any z-score exceeds 3, plus a Jaccard comparison of the tool set that fires when similarity drops below 0.5 (tokenjam.dev).

That approach has a useful property: the alert is interpretable. "This session called a shell tool 14 times more than baseline" tells a reviewer where to look, and the same data already sits in your traces without an embedding pass over every span (tokenjam.dev). For the equivalence rule, borrow the shape rather than the constants: a latency warning at 1.5x baseline and a failure at 3x will suit most interactive agents, and a token ratio outside 0.7x to 1.4x usually means a prompt leak, a retry loop or a changed context window rather than a genuinely different answer.

Keep these two fields graded as warnings by default. A change that makes the agent 40 percent slower but leaves every answer identical is still a change you want to see before release, but it is not the same class of event as a dropped required key.

Suppose an engineer tightens a system prompt so the agent cites its sources more consistently. The fixture set has 300 cases. Here is what the rule returns.

Text: 284 cases score above 0.92 similarity, 16 score between 0.85 and 0.92, and 4 of those 16 are new citations the reviewers wanted. Verdict: the band fails on 12 cases, but a reviewer marks all 12 as improvements and re-baselines them.

Required structure: zero failures on required keys, 30 cases gain an optional sources array. Verdict: pass, optional keys allowed.

Tool calls: two cases reorder a search and a fetch. Both tools are read-only and idempotent, so the reorder passes. One case calls a write tool before its prerequisite read. Verdict: fail on sequence.

Latency: p50 rises from 1.8s to 2.1s, which is 1.17x. Verdict: pass with a note. Token counts rise 1.6x on 40 cases because of the added citations. Verdict: fail on ratio, but the reviewer accepts a waiver because the growth is explained by the new field.

The insight is that three of four fields failed and the change still shipped. The band was doing its job: it forced a human to look at exactly the twelve text cases and forty token cases that moved, instead of either blocking the change or waving it through. Without the fixture, the same edit ships on a Friday and the first signal is a support ticket.

Assign three roles and keep them distinct.

The agent owner can tighten any band in the table without further approval, because tightening only makes the rule stricter and cannot let a regression through. The platform owner approves any loosening of the tool-call, latency or token bands, since those touch infrastructure behavior across agents. The reviewer approves re-baselining text cases, and needs to see the before and after text plus a one-line reason for each case.

Waivers need a minimum evidence pack: the fixture ID, the field that failed, the measured value against the band, the reason the difference is intended, and a link to the change that caused it. And the waiver should expire, so it gets revisited rather than becoming invisible policy.

The governance literature is blunt about why this matters. Most enterprise AI programs have data governance and almost none have prompt governance, and ungoverned system prompt changes with no audit trail are the most common and least visible governance failure, with prompts as standing policy documents rather than configuration (Solytics Partners). Prompt management makes the complementary point: saved versions should be immutable, with any modification creating a new version rather than overwriting, so rollback stays reliable (Braintrust).

Record the band itself the same way. The rule provenance draft describes recording each version of each rule, who set and approved it, and which rules were in force when a decision was made, binding a decision to the exact request rather than to a later approval (IETF datatracker). Applying that to your tolerance table costs one YAML file in version control and saves an argument six months later.

band_version: 7
approved_by: platform-owner
approved_on: 2026-03-02
fields:
  text_similarity:
    threshold: 0.92
    calibrated_on: 200 judged-equivalent edits
  required_keys: exact
  optional_keys: allowed
  tool_order: same_unless_idempotent
  latency_ms_p50:
    warn: 1.5x
    fail: 3.0x
  token_ratio:
    min: 0.7
    max: 1.4
waivers:
  required_evidence: [fixture_id, field, measured, band, reason, change_link]

A band set once and never reviewed becomes a workaround. Schedule three reviews: a weekly skim of failures that were waived, so recurring waivers get promoted to band changes; a monthly recalibration of the text threshold against newly judged edits; and a quarterly look at sample sizes, because a band validated on 300 fixtures is not validated on 3,000 after the suite grows.

Watch for drift that arrives without any edit at all. Long-running agents can degrade as context gets compacted, and one widely cited report describes instruction compliance starting at 100 percent in the first 30 minutes, dropping to 50 to 60 percent after the first compaction and reaching 20 percent after the second (tokenjam.dev). A fixture that only runs on fresh sessions will never see that, so include at least a few long-context cases in the set. The academic framing of the same problem names three manifestations (semantic, coordination and behavioral drift) and proposes a composite stability index across 12 dimensions including response consistency and tool usage patterns, which is a useful checklist even if you measure only four of them (arXiv).

If your fixtures run across several model providers, the plumbing is the boring part: one OpenAI-compatible key and one pricing model removes a whole class of per-provider fixture variants, which is what Quanta Inference is for. When the policy work itself needs a second pair of hands, a scoping call with an expert who has written these bands before shortens the calibration loop considerably.