A Regression Suite for Prompt Changes That Won't Break Refunds

The test set is the asset that outlasts every prompt and every model. Not the vendor dashboard, and not a folder of questions your team typed from memory.

Start with your own history. A practical method used by an AI-native helpdesk that pre-launch-tests every deployment: sample two sets from 90 days of your own resolved tickets, a volume set of about 200 cases drawn in proportion to intent frequency, and a risk set of 60 to 100 cases deliberately over-weighted toward money, policy exceptions, and anger the Richpanel evaluation method. That split matters because a volume set tells you how the prompt behaves on a normal Tuesday, while a risk set tells you how it behaves on the day that costs you money.

Write every case as an expected outcome, never an expected wording: the decision, the action to take or refuse, and whether it should escalate the Richpanel evaluation method. This is the difference between a test suite that survives a rewrite and one that fails on a synonym.

Broaden coverage beyond history, because history rarely contains enough adversarial or handoff cases. A seven-part test plan covers golden-set accuracy, unanswerable questions, multi-turn context, action and tool calls, policy and adversarial attempts, handoffs, and channel or language variants, with a pass criterion for each the IrisAgent testing plan. For refunds, the action and adversarial rows carry the most weight.

Here is a checklist you can adapt:

Cresta's discovery phase closes with a structured test plan that separates happy-path scenarios from critical deviations and high-stakes edge cases, and converts post-launch issues into regression tests the Cresta test coverage article. Borrow that structure: every production incident becomes a permanent test case the same week.

For refunds, the number that matters is not how many conversations the agent handled. It is how many decisions it got right and how much money it exposed.

Deflection rate is a tempting but misleading headline. It measures whether a contact stayed off an agent's queue and says nothing about whether the problem was solved; a customer pushed into a bot that cannot help registers as deflected and as a repeat contact on a different report the Teneo analysis. The same trap applies to "assumed resolution," where silence after a final answer counts the satisfied customer and the customer who gave up identically the Richpanel evaluation method.

Score five things instead, mapped to refunds:

1. Correct resolution rate. Cases where the agent's decision (approve, partial, deny, escalate) matched the expected outcome.
2. False approval rate. Refunds approved that policy says should have been denied. This is your most expensive error class.
3. Dollar exposure. Total refund value approved on the test set; compare candidates on this before and after approval rate, because two prompts can approve the same count at very different average values.
4. Escalation precision and recall. Precision: of cases escalated, how many truly needed a human. Recall: of cases that should have escalated, how many did. Missing a required escalation (a disputed charge, a fraud signal) is worse than an unnecessary one.
5. Re-contact rate inside a fixed window. Did the customer return with the same issue, the metric that separates real resolution from a closed ticket the swept analysis.

Add a cost per genuinely resolved conversation so cost savings cannot quietly fund an accuracy loss.

Two guardrail rules. Score a human baseline by running your own agents through a subset of the same cases, so "the AI is good" has a reference point the Richpanel evaluation method. And keep the metric count small and fixed across runs; changing what you measure mid-experiment makes every comparison meaningless, a discipline also recommended in general LLM evaluation guides the Confident AI metrics guide.

Run the candidate prompt through the identical test set, same model config, same context revision, same number of repeats. Score it blind where you can, meaning the person grading cannot see which prompt produced which output, because refund decisions invite motivated reasoning in both directions.

Then resist the urge to declare a winner from one or two flipped cases. With roughly 300 cases, a few flips are noise. The practical move is to report differences with a simple confidence interval on the proportion and to pre-commit a minimum effect size before the run. A decision rule that works without heavy statistics:

• Any false approval on a risk-set case blocks the candidate outright, regardless of the average.
• A drop in escalation recall on the risk set blocks the candidate.
• Dollar exposure may move up only if correct resolution and escalation recall also improve.
• Everything else is a judgment call and gets a written rationale in the change record.

Automated judges help at scale but need calibration. Common approaches combine statistical scorers with model-based scorers such as G-Eval and faithfulness-style checks, and pairing them usually beats either alone the Confident AI metrics guide. A cheap and effective pattern: a deterministic policy engine checks the hard constraints (date windows, refund caps, duplicate claims) while an LLM judge scores the softer parts (tone, policy explanation quality). Judge output never overrides a hard-constraint failure.

This is also the moment to widen the failure lens. Small changes have large blast radii: a reworded prompt can shift date formatting in a way that breaks a downstream parser, or start refusing an edge case the old version handled the gradual rollout guide. Include a structured-output compliance check in your harness so a schema violation fails the build rather than reaching a customer.

If your team runs models from several providers and switching between them is part of the experiment, an OpenAI-compatible gateway keeps the harness stable across providers, which is the problem Quanta Inference (https://quantaintelligence.ai/inference) is built to solve with one API key.

Passing the test set earns a candidate a promotion interview, not the job. Two stages stand between a green suite and live refunds.

Shadow first. Send production requests to both the current prompt, which serves the customer, and the candidate, which only logs its output. Compare the two offline against your criteria. Replaying historical traffic through the candidate and having a judge diff it against the baseline is a fast way to find regression areas before touching production infrastructure the gradual rollout guide. Shadow mode roughly doubles inference spend during the evaluation window, so scope it to a day or two of traffic rather than leaving it running.

Then canary. Route a small share of traffic to the candidate, often starting at 1 percent and lower for high-stakes work, while monitoring both cohorts on the same metrics the gradual rollout guide. Write the rollback trigger before you flip the switch. A workable trigger set for refunds:

• False approvals in the canary cohort exceed the baseline by more than the noise threshold you pre-committed to.
• Dollar exposure per approved refund exceeds the baseline by a set percentage.
• Escalation recall on flagged high-risk cases drops.
• Re-contact within the window rises above baseline.
• Any structured-output parse failure rate above zero.

Rollback must be a configuration change, not a redeploy. If reverting takes an engineer forty minutes of git archaeology, the gate is theater.

Shadow and canary also serve as your live monitor after full rollout. Keep running the suite on every change that could move an answer: prompt edits, knowledge base updates, model swaps, and tool schema changes. A knowledge edit on a Tuesday can change answers nobody touched, which is exactly why re-testing after any change is standard practice the IrisAgent testing plan.

Say you hold 300 labeled cases: 200 volume, 100 risk. Each case carries the expected decision and, for approved refunds, the order value. Your baseline prompt scores as follows.

The headline looks like a two-point accuracy gain. The false approval count more than doubled, dollar exposure rose 35 percent, and escalation recall fell 9 points on the risk set. Every one of those is a blocking signal under the decision rules above, even though the average improved. The candidate is rejected and the reason is recorded against the prompt version. This is the scenario the regression suite exists to catch: an average that hides a concentrated, expensive failure.

Now imagine a different candidate: same accuracy, false approvals flat at 4, dollar exposure steady, escalation recall up to 91 percent. That one earns shadow, then a 1 percent canary with the rollback triggers armed. The numbers give you a conversation with finance, not a gut feeling.

A minimal harness skeleton, deliberately provider-neutral:

{
  "suite_version": "refunds-v3",
  "baseline_prompt_sha": "a1b2c3",
  "model_config": {"model": "current", "temperature": 0.0},
  "cases": [
    {
      "id": "risk-014",
      "turns": ["My order from six weeks ago never arrived, refund me now."],
      "expected_decision": "escalate_or_partial",
      "order_value": 189.00,
      "must_escalate": false,
      "hard_constraint": "refund_window_check"
    }
  ]
}

Store the harness beside the prompt, and store every run's scores beside the harness. The change record for any prompt edit becomes a diff of scores, not a debate.

Five failure modes recur.

Test sets built from invented questions. They are too clean. Real customers misspell product names, bury the question in paragraph three, and paste error logs the IrisAgent testing plan. Sample from tickets.

Grading on wording. Rewording the expected answer turns a legitimate improvement into a failure and vice versa. Grade outcomes.

No sealed holdout. If you tune the prompt against the whole test set, you overfit it and the score inflates. Seal 20 percent of each set and never use it for tuning the Richpanel evaluation method.

One model config for baseline, another for candidate. A model bump smuggled into a prompt experiment makes every result unattributable. Change one thing.

Optimizing a proxy. Deflection, containment, and "assumed resolution" all reward silence the Teneo analysis. Resolution and false approval are the honest metrics for money-moving decisions.

One cultural point worth stating: the suite is expensive to build and cheap to run. Roughly a day of sampling and labeling per few hundred cases buys you the ability to change a prompt on a Tuesday afternoon without wondering, on Thursday, why support tickets moved. If you would rather have experts set this up for your line of business than build it from scratch, a free consulting call is the shortest path.