When to Re-run Evals After a Prompt Edit

The decision rule is: re-run the smallest eval set that covers every risk the diff can touch. Scope tiers, from cheapest to most expensive:

• Tier 0, spot check: 10 to 30 hand-picked cases plus deterministic format assertions. Minutes, near-zero cost.
• Tier 1, affected slice: the full case set for the task or intent the edit targets, typically 50 to 300 cases, plus format checks.
• Tier 2, full suite: every regression case you maintain, including historical failures.
• Tier 3, full suite plus probes: everything in Tier 2, plus adversarial injection cases, data-boundary probes and safety/policy scenarios.

Here is the mapping. Treat the "Minimum scope" column as a floor, not a ceiling.

Two refinements matter in practice. First, structural and format edits should always ship with deterministic assertions, not LLM-judge scores alone, because schema violations are checkable in code and should not consume judge budget. Second, safety edits should be paired with adversarial probes drawn from a stable attack library so results are comparable run to run.

Teams routinely underestimate the tool/format row. A schema change is a contract change. If a downstream service parses the model's JSON, the prompt's output constraints are part of your API surface, and improper output handling is a recognized risk category precisely because it is where generated text meets code that trusts it OWASP Top 10 for LLM applications.

Certain changes should skip the classification argument entirely. If any of the following four statements is true, go to Tier 3. This is the decision rule most teams are missing, and it is the one that prevents the incidents that make headlines.

1. The change alters tool access. Adding a tool, widening a tool's scope, or changing a tool description changes what the model can do, not just what it says. Over-permissioned tools fall under excessive agency in the OWASP taxonomy, where functions, permissions and autonomy combine with untrusted input to produce damage Promptfoo's OWASP TLDR. Tool descriptions are also an injection surface: a persuasive description can nudge the model toward a tool it should not call for a given input.

2. The change alters data boundaries. Anything touching which documents are retrieved, which tenant's vectors are searched, what the model may quote, or whether it may send data to an external recipient is a data-boundary change. In multi-tenant retrieval systems, inadequate partitioning causes cross-context leakage where one user's query returns another's embeddings, and embeddings can be inverted to recover source text OWASP LLM08 on vector and embedding weaknesses. A prompt edit that changes retrieval phrasing or tool arguments can move that boundary without anyone noticing.

3. The change alters output format consumed by code. If the output is parsed, stored, rendered or acted on, format compliance must be verified deterministically on the full suite. Format compliance means valid JSON, schema adherence, required fields and length limits, all checkable without a judge Braintrust on prompt testing.

4. The change alters safety behavior. New refusal rules, softened refusals, new escalation instructions, or anything that changes how the model handles sensitive requests. Safety edits deserve adversarial probes because polite inputs rarely reveal a broken guardrail. OWASP's prompt injection entry exists because user-supplied text can override system instructions, and only hostile test cases find this OWASP Gen AI risk archive.

The hardest cases are combinations. A wording edit to a sentence like "share relevant account details with the customer" is a wording change on the surface and a data-boundary change underneath. When two classes apply, take the higher scope.

Suppose your support assistant has a 600-case regression suite with a judge-based scorer, and each case costs about two cents in model calls. A full suite pass is roughly $12 plus judge runs and engineering review time. A targeted 120-case slice is about $2.40. These numbers are illustrative, but the ratio is what matters: widening scope from a slice to the full suite costs about five times more, and adding adversarial probes can double or triple judge work because adversarial cases often need human review.

Edit A (wording). You change "Answer politely" to "Answer in two short sentences and never speculate about refunds you cannot see." This touches tone and a refusal-adjacent boundary. Class: wording, but trigger 4 applies because "never speculate about refunds" is a policy statement. Minimum scope: Tier 3. That feels expensive for a one-line edit, and it is the correct answer. A broken refund guardrail costs far more than a suite run.

Edit B (wording). You change "Summarize the customer's issue" to "Summarize the customer's issue in one sentence." No tools, no data boundaries, no policy. Class: wording. Minimum scope: Tier 1, the summarization slice, plus a length assertion. Run 120 cases, check the sentence-length assertion fires, ship.

Edit C (structural). You move the tool-use instructions above the tone instructions and add two examples. Class: structural. Minimum scope: Tier 2. The reordering can change which instruction wins when they conflict, so every regression case matters.

Edit D (format). You rename a JSON key from priority to severity. Class: format. Minimum scope: Tier 2 plus a schema test suite in CI. The schema test is cheap and deterministic, and it should run on every commit, not only on prompt releases.

The insight: the cost of evals is driven by the trigger, not the size of the diff. A two-word safety edit can legitimately cost more than a rewrite of a tone paragraph. Teams that budget by character count get this backwards.

A "wording only" classification assumes the rest of the system is unchanged. That assumption fails often, because LLM APIs change behavior without changing their signature. A documented case study showed that an API update caused a major performance downgrade on a toxicity classification task and also flipped which of two prompts performed better, by a margin the authors reported as 8.7 percentage points the regression testing paper on evolving LLM APIs. If that can happen with no prompt edit at all, then a prompt edit made in the same week as a silent model change is not a controlled experiment.

The practical mitigation is to pin the model identifier and record it with the prompt version, then compare against a stored baseline on every run. Experiment comparison against a stored baseline is what turns a score into a regression signal Braintrust on prompt testing. Keep a small "canary" suite, 30 to 50 cases spanning each task, that runs on a schedule independent of prompt releases. When the canary moves without a prompt edit, you have found drift, and every pending prompt edit should be re-scoped to at least Tier 2.

This is where centralizing model access helps. If every model call goes through one endpoint, logging the model identifier, parameters and prompt version becomes a single integration rather than five. Quanta Inference puts text, image, video, audio and embedding models behind one OpenAI-compatible API key with prepaid US-dollar pricing, which makes it straightforward to keep that metadata in one place: Quanta Inference.

The mapping only works if your suites are structured so a slice can be selected. Three habits make that possible.

Tag every case. At minimum: task (summarization, routing, extraction), risk category (correctness, format, safety, tool use, data handling), and origin (synthetic, production failure, adversarial library). Then "Tier 1 for summarization" is a query, not a manual export.

Keep a golden dataset with explicit expectations. Regression testing for LLM outputs starts from reference inputs and outputs, often called a golden dataset, and the same inputs must be rerun to detect drift Evidently's regression testing tutorial. Treat production failures as new cases: every incident should end with a case added to the suite, otherwise your Tier 2 grows stale.

Separate deterministic checks from judged checks. Schema validity, length limits, forbidden substrings and refusal tokens are code assertions. Correctness, tone and helpfulness need a judge or human review. Mixing them means format failures consume judge budget and slow the loop.

A checklist for the release decision:

• Diff classified into one of the five classes, with combination of classes noted.
• Checked against the four triggers (tool access, data boundaries, code-consumed format, safety behavior).
• Minimum scope tier selected, and recorded in the prompt version metadata.
• Deterministic assertions pass on the full suite regardless of tier.
• Judge-based scores compared against a stored baseline, not against memory.
• Canary suite green, confirming no concurrent model-side drift.
• Any failing case either fixed or logged as a known limitation with an owner.

That last item connects evals to governance. A risk register works because it names a risk, an owner and a response plan, and it is reviewed on a cadence, not once Drata's guide to risk registers. Evals are the technical control behind the register entry called "prompt regression." If a change ships without a recorded scope decision, the register entry has no evidence behind it.

Some of the highest-stakes prompt edits are the ones that change how an agent behaves across a sequence of steps, not how it answers a single question. Recent joint testing by the Singapore and Korea AI Safety Institutes found that across three tested agents, none achieved fully correct and fully safe execution across 12 routine, non-adversarial tasks, and that successful task completion often coincided with data-handling failures such as accessing unnecessary information or disclosing it to the wrong recipient the data leakage evaluation. The report's key methodological point is that capability and data-handling safety should be evaluated separately.

That has a direct consequence for scope. If your prompt edit touches a multi-step agent, a single-turn eval suite will not see the failure. You need task-level scenarios with tool environments, a simulated user, and granular per-step criteria, which is exactly the kind of harness those institutes built. For most teams, that means a small number of rich agent scenarios (10 to 20) rather than hundreds of single-turn cases. Third-party agent risk adds a governance layer on top: identity, access, authority and evidence all need a named owner before an agent acts in your environment Living Security's third-party agent guide.

If you are standing up this kind of harness for a regulated line of business and want a second opinion on scope and triggers, a free 30-minute discovery call is a reasonable next step: Quanta Consulting.