When a Model Upgrade Forces a Tool Re-eval: A Checklist

Teams that run tool evals usually classify changes into a few buckets: prompt edits, tool schema edits, tool implementation edits, and retrieval or data changes. Model changes sit awkwardly outside that list because they can produce symptoms that look like every other bucket. Adding a model type fixes the ambiguity, provided you classify by the first point of divergence rather than by the symptom.

The classification matters because the remediation differs. A model-caused failure gets fixed with prompt adjustments, tighter schemas, a fallback model or a change in the loop's control logic. A tool-caused failure gets fixed in the tool. Making that call from a final-answer score is guesswork, which is why the next section starts with traces.

Agent evaluation practice already separates these layers. Anthropic's guidance defines a transcript (also called a trace or trajectory) as the complete record of a trial, including tool calls, reasoning and intermediate results, and distinguishes the transcript from the outcome, which is the final state in the environment (Anthropic on evals for AI agents). That transcript-versus-outcome split is the backbone of the triage method below.

Before touching the model, you need a frozen record of how the current model behaves on the full tool suite. Without it, every comparison after the swap is anecdotal.

Treat the baseline as an immutable artifact. Prompt versioning practice holds that each version gets a unique identifier, includes metadata such as the model, parameters and tool configuration, and remains unchanged once created so that logs and evaluations can reference it reliably (Braintrust on prompt versioning). Apply the same rule to the eval run: record the model identifier, the parameter set, the tool definitions, the dataset version and the date, and never overwrite the record.

For agent work, run multiple trials per task, because model outputs vary between runs and repeated trials produce more consistent results (Anthropic on evals for AI agents). A single run per task will show you noise and call it a regression.

A minimal trace record that supports later diffing looks like this:

{
  "task_id": "refund-flow-014",
  "model": "<recorded model identifier>",
  "params": {"temperature": 0, "tool_choice": "auto"},
  "trial": 2,
  "steps": [
    {"step": 1, "tool": "get_order", "args": {"order_id": "A-1042"}, "result_status": "ok"},
    {"step": 2, "tool": "issue_refund", "args": {"order_id": "A-1042", "amount": 42.0}, "result_status": "ok"}
  ],
  "stop_reason": "end_turn",
  "final_answer": "...",
  "outcome_state": {"refund_created": true}
}

Record the stop reason explicitly. In the small-model benchmark, every failing task log carried the same signature, stop=end_turn, tools=none, which is what made the failure mode visible at a glance (the Llama 3B benchmark write-up). A stop reason and a tool count per step cost almost nothing to store and save hours during triage.

Finally, make sure the suite covers the behaviors that decide whether a configuration is safe to release, including task correctness, format compliance, refusal and escalation boundaries, tool and retrieval behavior, and operational limits such as latency and token cost (Braintrust on prompt testing). If your baseline suite only covers correctness, add the tool-behavior and boundary cases before the switch, not after.

A paired run means the same tasks, the same dataset, the same prompt version, the same tool definitions, the same parameters and the same environment, with the model as the only variable changed. Anything else that differs between the two runs becomes a confounder you cannot remove later.

Version control for the prompt unit is what makes this possible. A central prompt registry with immutable versions and explicit environment assignments lets you trace every output back to the exact prompt that produced it, and promote a version to production only after validation (LangWatch on prompt management). If the prompt is hardcoded across repositories, the paired run quietly tests two variables at once.

Set the sampling parameters to fixed values for the comparison. Temperature above zero adds variance that competes with the effect you are measuring, and with a small task set the model effect will be buried. Run the same number of trials per task in both arms.

Score at two layers. Step-level process scoring identifies where a chain breaks, while end-to-end outcome scoring verifies whether the final environment state matches the goal (NVIDIA on evaluating agents from tool calls to task completion). Report both, because a model can reach the right outcome through a wrong path (which is a latent failure) or take the right path and still miss the outcome (which is a tool or environment problem).

A worked example of reading paired results:

The numbers above are illustrative, but the pattern is the useful part. Nine tasks lost overall, yet the tool error count did not move, so the tools themselves behaved identically under both models. The regression sits in tool selection and argument construction, which is where the remediation effort belongs.

If you are running many model candidates at once, keeping the comparison honest across providers is its own problem. A single API that exposes several model families behind one key removes a class of configuration drift from the experiment; Quanta Inference covers text, image, video, audio and embedding models through one OpenAI-compatible endpoint (Quanta Inference). Whatever platform you use, the rule stands: one variable per comparison.

Diffing final answers tells you that something changed. Diffing traces tells you what. Walk each failing task from step one forward and stop at the first place the two arms differ. That step is the failure's origin, and everything after it is downstream noise.

Sort the divergences into four kinds, in this order:

1. Tool selection divergence. The two arms call different tools at the same step. This is model-caused unless the tool descriptions are ambiguous, which you can test by checking whether the old model also hesitates on that pair.
2. Argument divergence. The same tool is called with different arguments, wrong types, missing required fields, or values that violate the schema. Model-caused if the schema is clear; schema-caused if the description left room for interpretation.
3. Tool response divergence. The tool returned an error, a timeout or a different response shape. Tool-caused, unless the arguments from step 2 caused it, in which case it is model-caused upstream.
4. Next-step divergence. The tool returned the same correct response in both arms, and the models did different things with it. Model-caused, and usually the hardest to fix, because it means the model's interpretation of tool output changed.

That four-way split maps onto how evaluation tooling already separates layers of an agent workflow: output, retrieval, trajectory (tool selection, arguments, plan adherence) and task outcome are distinct layers, and a single compound score hides problems that live in one layer only (Cognee on open-source agent evaluation tools). Frameworks expose this directly; DeepEval, for example, includes Tool Correctness and Argument Correctness metrics for individual tool decisions alongside trajectory metrics such as Task Completion and Step Efficiency (Cognee on open-source agent evaluation tools).

Two practical rules shorten triage. First, triage tool-caused failures before model-caused ones, because a broken tool contaminates every task that touches it and will inflate the apparent model regression. Second, when a failure looks like a tool-selection error, check whether an error was attributed to the model or to a tool. Research on the self-correction blind spot found that models reliably fix an error presented as coming from the user or a tool, but fail on the byte-identical error presented as their own reasoning, with a blind-spot rate of 64.5% across fourteen non-reasoning models (Splunk on agent self-correction). If your loop feeds the model a summary of its own prior step rather than the raw tool response, a fixable error can become invisible to it after a model swap.

The triage is only useful if it produces a decision. Use a rule that pairs the failure origin with the blast radius.

The last row deserves its own gate. Prompt edits that add examples, context or reasoning instructions can improve accuracy while also raising response time or spend, so cost and latency belong in the same test run as quality (Braintrust on prompt testing). A model that passes the outcome gate but triples steps per task may still be the wrong swap for a high-volume workflow.

Keep the failures you accept as known risks, with an owner. A risk register is the standard artifact for this: entries carry a description, potential impact, example controls, and fields for likelihood, residual risk and a named risk owner (the AI risk register template). An agent that occasionally selects an unnecessary but permitted tool is a residual risk with a mitigation, not a mystery.

Record the outcome of every model change in the same place you record model versions. Model registry governance exists for exactly this: version control, metadata management, approval workflows, access controls, audit trails and monitoring integration as the components of a registry that is a control rather than a database (VerifyWise on model registry governance). The same source reports, citing an IBM report, that 85% of companies adopting AI say they struggle with managing model risks while only 20% feel confident in their model management practices (VerifyWise on model registry governance). A registry entry that links the model to its eval results, its prompt version and its known risky task families is the difference between a record and a control.

Close the loop by feeding production failures back into the suite. Turning real failed examples into future test data is the standard way to keep regression coverage aligned with the failures users actually encounter (Braintrust on prompt testing). For each model swap, add the failures you found to the stored baseline dataset so the next swap starts with a sharper suite.

One governance habit from agent security work transfers directly: record the agent's identity, access, authority and evidence, so that you can reconstruct what it saw, why it acted and what it changed (Living Security on third-party agent risk). After a model swap, "why did it act" changes meaning, and the trace plus the registry entry are what make that question answerable.

If your team needs a second opinion on how to structure the suite, the swap gate or the fallbacks, a free 30-minute AI discovery call can turn these steps into a plan for your systems (Quanta Consulting).