When to Re-run Evals for a Tool Change, Not a Prompt Change

Run through this list on every tool diff. Each row is a yes or no. Two or more yes answers means targeted evals at minimum, and a contract class change means the full suite.

The side effects row deserves emphasis. Research on tool-using agents found that sensitive information can be exposed during entirely benign requests, and that successful task completion often coincided with data-handling failures the SG AISI and KR AISI evaluation. A tool that gains a write or send capability is a new security surface, not a plumbing tweak, which is the same reasoning behind least agency: an agent that needs to recommend an action does not automatically need permission to apply it guidance on third-party agent risk.

Targeted evals mean one narrow suite per changed tool, not the whole regression set. The rule that keeps this honest: targeted evals must include (a) the tasks that previously called this tool, (b) the tasks that should call it, and (c) a handful of tasks that should not call it. The third group is the one teams skip and the one that catches new-tool hijacking.

Anthropic's writeup on agent evals makes the mechanics concrete: a task has defined inputs and success criteria, each attempt is a trial, and a grader scores aspects of performance, with the outcome being the final environment state rather than what the agent said Anthropic's agent evals guide. That distinction matters for tool changes, because a renamed parameter may produce a fluent confirmation while the environment never changed. Grade the environment, not the transcript.

Prompt testing frameworks already carry tool configuration as part of the versioned unit, so a tool diff is testable with the same harness as a prompt diff Braintrust on prompt testing. The practical version number should bump on tool schema changes too, not just prompt text, or you lose the link between a bad run and what actually caused it.

A worked example with numbers, using an invented but representative setup: an agent has 40 tools and a suite of 300 eval tasks, of which 120 touch the tool being changed. A full suite run costs 300 tasks times 3 trials, so 900 graded trials. A targeted run covers 120 callers, plus 15 should-call and 30 should-not-call cases, which is 165 tasks times 3 trials, so 495 graded trials. That is roughly a 45 percent cut for a change that touches 40 percent of the suite. Now the decision rule: if the change is contract class, run full anyway. If behavioral class, run targeted and only escalate to full if the regression rate on the targeted set exceeds your threshold, for example more than 3 of 165 tasks failing where they passed on the baseline. Cosmetic class gets zero eval trials and a monitoring alert instead.

The interaction is where most teams get burned. A prompt edit that adds a sentence about when to use a tool is a tool change in disguise, because it changes routing. Conversely, a tool description rewrite changes the model's selection behavior as surely as a prompt rewrite, since the description is part of the context the model reads.

There is direct evidence that API behavior shifts can flip which prompt is better: a case study found that after one API update, one prompt outperformed another by 8.7 percent accuracy, reversing the earlier ranking research on regression testing for evolving LLM APIs. The lesson transfers to tools: selection rankings between overlapping tools are not stable across configuration changes, so a routing eval that passed last week is not evidence it passes now.

The practical policy is a combined change log. One entry per release, listing prompt diffs, tool schema diffs, model settings, and retriever changes, because a score difference between two runs can be traced to any of them only if they travel together Braintrust on prompt versioning. Immutability helps here: once a version is created it should not be edited, so every change is a new record rather than an overwrite LangWatch on prompt management.

If a release contains both a tool change and a prompt change, test them separately first. A combined run tells you pass or fail but not which diff caused it, and separating them costs one extra targeted run.

Adding a tool is a contract change even when nothing existing was edited, because the callable surface grew. Two questions decide the scope. First, does the new tool overlap in purpose with an existing one? If yes, run routing evals across the full set, since the model may now prefer the newcomer on tasks that used to route correctly. Second, does the new tool have side effects or wider data scope? If yes, add a safety review and a data-handling eval suite alongside the functional one.

Retiring a tool is riskier than adding one, and it is under-tested because the deletion looks clean. Before removing, grep prompts, code, and other tools for references. A prompt that names the tool by name will now instruct the model to call something that no longer exists. A second tool that called the first internally will fail at runtime. Nested or chained tools are the common blind spot.

The same governance body that tracks models and their lifecycle can hold the tool inventory. Model registry practice includes audit trails of deployments, rollbacks, and retirements, plus monitoring integration to catch drift model registry governance. A tool registry with the same fields (owner, purpose, permitted actions, retirement trigger) makes the "who depends on this" question answerable in seconds instead of a codebase search. Organizations that skip this end up with a production blind spot: models or tools that were deprecated in name but never actually switched off Collibra on model governance.

Pass/fail per case against a stored baseline is the signal you want from prompt and tool tests, and prompt evaluation scores are a separate, slower signal about quality Braintrust's comparison of testing and evaluation. Keep them apart. Failing a regression check should block the release; a dip in a quality score should trigger a look, not a rollback.

Regression testing for LLMs has to handle non-determinism and many acceptable answers, which is why sample inputs, reference answers, and code-based plus judge-based checks are combined rather than relying on exact matches Evidently's regression testing tutorial. For tool changes, add deterministic assertions where possible: did the tool get called with a schema-valid argument set, did the environment state change as intended, did a prohibited tool get called. Those are cheap and unambiguous.

Finally, treat a production failure as a test case, not an incident. When a tool change causes a bad run in the wild, add that task to the suite and to the relevant risk register entry with its score and owner, which is the pattern behind structured AI risk registers an AI risk register template and general risk matrix practice of ranking by likelihood and impact risk assessment matrix overview. The register then tells you which tool changes deserve the full suite permanently, rather than re-litigating the scope call every sprint.

If you are setting up this kind of pipeline across several model providers, running the same eval harness against text, tool-calling, and embedding models through one key removes a variable from every comparison, which is what Quanta Inference is built for.