Diffing Two Subagent Instructions by Behavior

A fixture is a frozen task plus a frozen environment. Both versions of the instruction set must see exactly the same input or the comparison is meaningless. You need four ingredients.

Tasks. Twenty to forty prompts that cover the agent's real job, not its happy path. Include the ordinary request, the ambiguous request, the request with a missing field, the request that touches a sensitive tool, and at least one task the agent should refuse. Single responsibility is the stated best practice for subagents, so your task list can be narrow, but it must include the edge cases the prompt is supposed to police (Microsoft Learn's multi-agent guidance).

Tool mocks. Never run the comparison against live services. Stub each tool with recorded state so a refund is not actually issued and an email is not actually sent (Braintrust). The mock should return a fixed payload for a given argument set, which means you can compare arguments exactly.

Seeds and settings. Fix the model, temperature and any sampling seed, and record them in the run manifest. If the two runs use different models or settings, you are comparing models, not instructions.

A place for failures. When production produces a bad trajectory, add it to the suite as a new fixture. The test case list should grow from real failures rather than from imagination.

Free text is the wrong comparison surface, because two correct answers rarely share wording. Force both versions to emit a schema: the ordered list of tool calls, each with its name and a normalized argument object, plus the final answer and any explicit refusal. Structured output features exist precisely so the response adheres to a supplied schema instead of drifting in format (OpenAI's structured outputs guide), and agent SDKs expose the same pattern through a validated model on the result object (the Strands SDK docs).

A run manifest plus per-fixture record can be as small as this:

{
  "run_id": "instructions-b",
  "instructions_ref": "reviewer-agent@9f3c",
  "model": "<fixed model id>",
  "temperature": 0,
  "seed": 42,
  "fixtures": [
    {
      "task_id": "refund-missing-order-id",
      "tool_calls": [
        {"name": "lookup_order", "args": {"order_id": null}},
        {"name": "ask_user", "args": {"field": "order_id"}}
      ],
      "final_answer": "I need the order number before I can check the refund policy.",
      "refused": false
    }
  ]
}

Then compute three deltas per fixture. The sequence delta is an edit distance over tool names. The argument delta is a field-by-field comparison after normalization (lowercase enums, sort list arguments, trim whitespace, round floats to a stated precision). The answer delta is a similarity score against the reference answer, plus a hard check on required content such as a citation or a refusal. Keep the raw records so a reviewer can see exactly which step moved.

A tolerance band is a per-fixture, per-dimension limit that says how much drift counts as noise. Writing it down is the step teams skip, and it is the step that makes the gate defensible in a review. The band should be set once, reviewed like code, and changed only with an explicit decision.

Three practical rules for setting it:

1. Sequence is strict. If the agent calls a tool the reference run never called, or drops a required check such as reading policy before issuing a refund, that crosses the band regardless of the final answer. Ordering flexibility is only allowed where the reference itself shows several valid routes.
2. Arguments are exact on identity fields and tolerant on formatting. A different order ID is a crossing. A timestamp formatted differently, or an argument that differs only in case, is not.
3. Answers use a similarity threshold plus hard gates. Set a threshold per fixture class, and add non-negotiable checks (contains a source, declines on the sensitive task, does not invent a field) that override the score.

If you are unsure where to place a threshold, run the same instructions against themselves ten times. The spread you see is your noise floor, and no band should be tighter than that.

The rule, stated so it can be quoted in a review: a pair of instruction sets is behaviorally equivalent when every fixture in the suite stays inside its preset band on tool sequence, argument values and answer content, across the stated number of repeated runs. Everything else is a behavior change.

Turn that into a routable decision:

• Zero crossings, across all fixtures and all repeats: record the change as a no-op, mark it equivalent, and let it skip the full gate with a logged reference to the run manifest. This is the payoff. Prompt edits are frequent, and a "be concise" rewrite that changes nothing should not burn reviewer time (Braintrust on prompt management).
• Any crossing: the edit is a behavior change. Score it, route it, gate it.

The risk score should absorb the result rather than sit beside it. A simple additive model works: start from a base score, add a weight per crossed fixture scaled by fixture criticality (a sensitive-tool fixture weighs more than a formatting fixture), add a term for how far the answer similarity fell below the threshold, and add a penalty if any run showed non-determinism above the noise floor. Two numbers matter for calibration: the crossing rate (crossed fixtures divided by total fixtures) and the worst single crossing. A change that crosses one low-criticality fixture should route to a fast review; a change that crosses three fixtures including a sensitive one should route to the full gate.

This is the same logic change management has used for years. The 7 R's ask who raised the change, why, what return is expected, what the risks are, and what happens if you reject it (SolarWinds' checklist). Behavioral equivalence answers the risk question with evidence instead of opinion, and the run manifest becomes the audit trail: what changed, which fixtures ran, what the deltas were, who approved the equivalence call (the prompt governance overview).

A worked example. Take 30 fixtures, with 4 marked critical because they touch a write tool. Version B crosses 3 fixtures: two formatting-only answer changes and one refusal on an ambiguous task that is not critical. Sequence and arguments match everywhere. If your band classifies formatting as equivalent, you have one real crossing out of 30, a crossing rate of about 3%, on a non-critical fixture. Route to fast review. Now flip one detail: one of the crossed fixtures is the refund task that no longer reads policy first. That single critical crossing should outweigh the other 29 passes, and the model should escalate the score to the full gate. The lesson is that the crossing rate alone is the wrong trigger; criticality has to multiply it.

A second worked example on scaling. Suppose the team ships 40 prompt edits a quarter and each full gate review costs a reviewer half a day. If equivalence lets 25 of those skip the gate, you recover roughly 12 reviewer-days a quarter. That is the business case for building the suite, and it is worth stating in the design doc so the work is not treated as optional quality overhead.

The suite decays if nobody feeds it. Three maintenance habits keep it sharp.

Feed production failures back in. Every bad trajectory becomes a fixture, so the suite tracks real behavior rather than the team's first guesses (Braintrust).

Keep grounding and tools constant between runs. If the tools themselves changed, you are testing the tools, not the instructions, and attribution collapses (the A/B guide).

Re-baseline deliberately. When a behavior change is accepted, the new outputs become the reference for the next comparison. That re-baseline is itself a reviewable event, so record it with the same manifest.

One more safeguard worth adopting: because subagents are granted narrow tool sets by design, apply least privilege when you author the fixtures too. A comparison run should never hold credentials it does not need, since an agent with unnecessary access expands the blast radius of a test that goes wrong (the least privilege guide). Mocks make this easy; live tools make it a risk.

If your fixtures span several model providers and you want one run to compare instruction sets across them, Quanta Inference exposes text, image, video, audio and embedding models through a single OpenAI-compatible API key, which keeps the model setting in your manifest a one-line change.