How to Test a Refund Agent's Decisions Against Real Tickets

Pull 3 to 6 months of resolved refund and return tickets from your help desk, group them by intent, and sample the top intents in proportion to real volume so the set mirrors your queue IrisAgent. A second sample, deliberately over-weighted toward money, policy exceptions, and angry customers, is what actually finds the expensive bugs. The Richpanel method samples roughly 200 cases by volume plus 60 to 100 risk cases, and seals 20% of each as a holdout never used for tuning Richpanel.

Returns-related inquiries often make up 30 to 50% of support tickets, and return rates average around 25% for ecommerce, so refund history is usually large enough to sample from without inventing anything fin.ai on returns automation.

For each ticket, record the facts the agent would see and the decision a good human reached:

Two labeling rules save weeks later. First, write the expected decision, never the expected wording: the action to take or refuse, and whether it should escalate Richpanel. Second, label ambiguous tickets honestly. If two experienced agents would disagree, mark the case ambiguous and score it as "escalate or explain, never guess." Ambiguity is a property of your policy, and hiding it in the labels inflates accuracy without improving the agent.

Do not skip the unanswerable bucket. Add out-of-scope requests and questions your docs do not cover on purpose, because the failure mode that costs trust is the confident fabrication rather than the honest refusal IrisAgent.

Treat the export as a data-handling task, not just a copy and paste. Refund tickets carry payment details, addresses and, in some categories, medical-adjacent notes, so strip or tokenize anything the agent does not need for the decision before the set leaves your help desk, and keep the mapping in your own stores Petronella. A test set is a durable asset; it should not become a compliance liability.

Deflection rate, the share of conversations that never reach a human, is the wrong headline for a refund agent because it counts the call, not the resolution. A customer pushed into a bot that cannot help registers as a deflection and a repeat contact on a different report Teneo on resolution vs deflection. A refund agent's version of that trap is an auto-denial that looks like automation and turns into a chargeback.

Score these five instead:

1. Decision accuracy. Share of cases where approve, partial, deny or escalate matches the label. Report routine, edge and ambiguous separately.
2. Dollar error. Sum of |refunded minus expected| across the set, reported as total and as a mean per case. This is the metric executives understand.
3. Policy-violation rate. Cases where the agent approved something the policy forbids, or denied something it requires. This is a correctness gate, not an average: a single class of violation (refunding a final-sale item, say) should block a rollout.
4. Escalation precision and recall. Of the cases the agent escalated, how many truly needed a human; of the cases that truly needed a human, how many were escalated. The Richpanel scoring list pairs these with re-contact rate and cost per genuinely resolved conversation Richpanel.
5. Argument correctness on tool calls. When the agent calls your refund tool, were the order id and amount right? Task completion and tool-call correctness are scored separately in agent evaluation practice for exactly this reason Confident AI on LLM evaluation metrics.

A worked example with numbers makes the trade-off visible. Take 120 tune cases with a total of $9,400 in expected refunds:

• Agent A gets 104 decisions right (86.7% accuracy), refunds $8,910, a dollar error of $490 in total, about $4.08 per case. It also escalates 9 cases, and 8 of those truly needed a human.
• Agent B gets 110 right (91.7%), but refunds $10,650, a dollar error of $1,250, and commits 4 policy violations on final-sale items.

Agent B looks better on the headline number and is clearly worse for the business: more overpayment and a compliance problem. That inversion is the decision rule: gate on policy violations first, then compare dollar error, then decision accuracy, then escalation quality. Choose the agent with zero violations even when its accuracy is lower.

One caution on averages. A mean dollar error of $4 hides a distribution, and the cases that matter are the tail: the $400 overpayment on one disputed order. Report the maximum error and count the cases above a threshold you set (say, anything over $50) alongside the mean, so a cluster of small misses cannot mask a single expensive one.

A score you cannot reproduce is a rumor. Three things must be frozen for every run: the agent configuration (prompt, tool rules, sampling settings), the inputs (verbatim tickets plus fixed order facts, never live API lookups), and the grader. If any of the three drifts, the score means nothing.

The harness itself is small: load the case set, replay each ticket through the agent, log the decision and tool calls, score, and write a run record keyed by a hash of the configuration. A sketch in Python:

import hashlib, json

def config_hash(cfg: dict) -> str:
    blob = json.dumps(cfg, sort_keys=True).encode()
    return hashlib.sha256(blob).hexdigest()[:12]

def score(case, out):
    results = {
        "decision_ok": out["decision"] == case["expected_decision"],
        "dollar_err": abs(out.get("amount", 0) - case["expected_amount"]),
        "violation": out["decision"] == "approve" and case["policy_rule"] == "final_sale",
        "escalated": out["decision"] == "escalate",
    }
    return results

def run_suite(cases, agent, cfg, run_id):
    cfg_hash = config_hash(cfg)
    rows = []
    for case in cases:
        out = agent.decide(case["customer_message"], case["order_facts"])
        rows.append({"case": case["ticket_id"], "cfg": cfg_hash, **score(case, out)})
    save_run(rows, run_id=run_id, cfg=cfg, cfg_hash=cfg_hash)
    return rows

Two habits keep it honest. Pin the model and sampling settings in the config rather than in your head, and store the raw agent output for every case, not just the pass/fail flag, because root-cause grouping needs the reasoning trace. Helpdesk evaluation tooling works the same way: a dataset goes in, the agent responds, and each item is marked resolved or not, with the result reviewed next to real conversations Atlassian's evaluation docs. Running an agent against a dataset and reviewing outcomes is the workflow; a spreadsheet is a fine first version.

Establish a human baseline before you judge the machine. Run 50 of the same cases through your own agents to see what an honest human score looks like on these tickets Richpanel, then treat that number as the bar for autonomous handling. Humans will not score 100% either, and knowing where they miss (usually the same ambiguous cases) tells you which misses are policy problems rather than agent problems.

For grading, combine deterministic checks (decision mismatch, dollar math, tool arguments) with model-based scoring only where judgment is unavoidable, such as "did the reply correctly cite the return window." Model-based scorers are useful and imperfect, which is why they belong beside exact checks rather than instead of them Confident AI. A mixed approach keeps the scoreboard cheap to explain.

Keep every run's raw output. When a number moves, you want to open three failing cases and read them, not rerun the suite to find out what happened.

The value of the harness is the failure list, not the score. After a run, sort every miss into a small, fixed taxonomy:

Real deployments show this is a system problem more often than a model problem: one team cut incorrect responses by 60% after switching from pure generative replies to a retrieval-first design that referenced real resolution patterns from past tickets Bland on agent challenges. Your taxonomy will show the same pattern. Most refund-agent misses are missing knowledge or rule ordering, not reasoning.

Escalation deserves its own category because it carries both risk and load. Good trigger design distinguishes what the agent may do, what it may recommend, and what needs human approval, and it escalates on low-confidence intent, missing identifiers, and requests that require financial or policy authority Petronella. Refunds beyond standard limits, disputed fees and overrides sit in that second bucket, so a refund agent that approves them autonomously is failing even when the decision happens to be generous.

Then iterate the disciplined way. Change one root cause per iteration, re-run the full tune set plus the holdout, and compare like with like using the gate order from the metrics section. A prompt edit that fixes the partial-refund arithmetic should not also quietly loosen the final-sale rule; the policy-violation count will tell you if it did. Prompts in this work are best treated as behavioral specifications written separately from the knowledge base, which is one of the setup distinctions that separates agents that resolve from chatbots that frustrate Crisp's build playbook.

If you want a second pair of eyes on the taxonomy before you start tuning, that is the sort of thing an expert with your ticket data can produce as a finished deliverable rather than a slide deck: Quanta Experts read your context and hand back the work.

Ambiguity is where refund agents earn or lose trust, and it is where test sets are most often wrong. A customer asks for a refund 34 days after purchase on an item your written policy covers for 30 days but your team routinely stretches for defective goods. If you label that case deny and expect the agent to match the letter, you will tune the agent into a rules lawyer. If you label it approve, you will tune it into a liability.

Label it ambiguous, with the expected behavior stated as "explain the window, ask about the defect, route to a human if the defect is confirmed, never promise a refund." Both humans and agents "pass" by not guessing. Research on evaluation design warns that a test suite built against an imagined version of user behavior tests the world the team imagined, not the one customers inhabit Cresta on test coverage from real conversations. Ambiguous cases are the densest part of that gap.

The holdout protects you from yourself. Every change you make after seeing a failure is a small overfit to that failure, and 20% of cases never used for tuning tell you whether improvements generalize Richpanel. The rule: if a change lifts the tune set and drops the holdout, you have taught the agent to pass your test rather than to follow your policy. Revert and rethink.

Rollout follows the same logic as the score gate. Start in shadow mode, where the agent decides and a human still acts; then draft-for-approval, then autonomous on narrow intents, then widen. Re-run the whole suite on every change that could move an answer, and convert post-launch surprises into regression cases so the same failure cannot return silently Cresta.

One practical add-on: measure repeat contact inside a fixed window after a refund decision. A denial that generates three follow-up tickets is a cost even when it was policy-correct, and re-contact rate is one of the paired quality metrics that make automation claims verifiable Swept on verifying agent metrics.