Auditing an Agent's Stated Reason for a Handoff

Before you can score anything, you need a ground truth. The ticket history is usually enough, but only if you define which fields are authoritative. Pick the fields once, write them down, and use the same definition for every probe.

Two rules keep the ground truth honest. First, record only what a field actually says, quoted, not your interpretation of it. Second, when the fields conflict, mark the case ambiguous and keep it out of the headline score; you can still use ambiguous cases to see how often the agent picks one side of a contradiction.

A worked example. Ticket 4471 has three messages. The customer first asks where an order is. The agent supplies tracking. The customer returns and writes "the box arrived empty, I want my money back." The account shows the order was delivered four days ago and the refund window is 30 days. The true cause here is a refund request following a damaged or incomplete delivery. A stated reason of "customer has a billing dispute" is not the same cause, even though both funnel to a money team.

You do not need thousands of cases. You need coverage of the trigger types your agent actually uses, plus the near-misses where a plausible wrong reason is most likely.

Start by listing the escalation triggers your system is supposed to fire on. Practitioner guides converge on a small set: explicit request for a human, issue outside the knowledge base, repeated failed attempts, frustration or complaint, sensitive actions such as refunds or account recovery, and tool or integration failure (Denser). Human-in-the-loop design guidance adds risk level, model confidence, complexity, regulatory mandate and domain boundaries as separate signals (LiveKit). Your probe set should sample each of these.

Then add the hard cases deliberately:

• Mixed intent. The customer asks about a delivery and mentions a charge in the same message. Which cause does the reason name, and does it name both?
• Cause drift. The reason matches an early message but not the latest one.
• Explicit request buried mid-conversation. The customer asks for a person, then continues describing the issue; the reason should still name the human request.
• Policy near-miss. The case sits just inside or just outside a rule (a refund one day before or after the window closes), so you can see whether the reason cites the rule or invents a billing error.
• Tool failure. A backend call errored. The reason should say so rather than convert it into a customer-side cause.

Aim for a first pass of roughly 60 to 100 cases, weighted toward the categories that route to expensive or regulated queues. Store each case with the ticket ID, the quoted source fields, your ground-truth cause, and the agent's stated reason copied verbatim. Keep the agent's stated reason out of the file the scorer reads as ground truth, so nobody grades by vibes.

A probe set that lives in a spreadsheet decays. Put it in version control as structured records so you can rerun the audit after every prompt or model change.

{
  "probe_id": "P-0142",
  "ticket_id": "4471",
  "channel": "chat",
  "true_cause": "refund_request_damaged_delivery",
  "trigger_type": "sensitive_action",
  "source_fields": {
    "latest_customer_message": "the box arrived empty, I want my money back",
    "first_message": "where is my order",
    "ticket_state": "reopened",
    "order_status": "delivered",
    "refund_window_days_remaining": 26
  },
  "stated_reason": "Customer has a billing dispute about charges.",
  "expected_queue": "refunds_damaged_goods"
}

Two fields do the heavy lifting later. trigger_type lets you report scores per trigger, which is far more useful than one blended number. expected_queue lets you check the operational cost of a wrong reason: if the reason misclassifies the cause, does the case land in a queue that cannot fix it?

Keep the source fields verbatim. If you paraphrase the customer's message when you build the probe set, you are grading the agent against your summary rather than against what the agent could read.

Score each probe on three separate scales, from 0 to 2. Separating them is the whole point: a reason can be specific and well supported yet name the wrong cause, or correct but generic.

Cause correctness. Does the named cause match the ground-truth cause?

• 2: matches the ground truth and, where there are multiple intents, names the operative one.
• 1: same family but wrong branch (refund cited as a billing dispute) or correct but incomplete in a way that changes routing.
• 0: a different cause, or a cause the ticket does not support at all.

Evidence support. Can every claim in the reason be traced to a field?

• 2: every claim maps to a quoted field.
• 1: the main claim is supported, a secondary claim is not.
• 0: the central claim has no support in the ticket.

Specificity. Does the reason name the trigger rather than a category?

• 2: names the concrete trigger ("customer asked for a refund on an order delivered with missing contents").
• 1: names a category ("customer wants money back").
• 0: names nothing actionable ("escalating for further assistance").

The scoring table below is the checklist a reviewer applies, in order, for each probe: find the central claim, map it to a field, then judge the cause.

A worked scoring pass: stated reason "Customer is frustrated with a billing error." Ground truth: refund request for an empty box, no billing error anywhere in the ticket. Cause correctness is 0. Evidence support is 0, because the billing claim has no field behind it even though frustration may be visible in tone. Specificity is 1, because "billing error" is a category. Total 1 out of 6, and the single most important fact, that this is an evidence support failure, is visible in the scores rather than buried in prose.

Contrast with a second reason on the same ticket: "Customer wants a refund because the delivered order arrived empty (message 3); refund window open 26 days." Cause correctness 2, evidence support 2, specificity 2. Total 6 of 6.

Report the three dimensions separately, then a single pass rate. A reason counts as passing only if cause correctness is 2 and evidence support is at least 1. Specificity stays out of the pass rule, because a generic but accurate reason is a quality problem, while an unsupported cause is a correctness problem. Different owners, different fixes.

Take a 100-probe set: 68 score 2 on cause correctness, 19 score 1, 13 score 0. On evidence support, 61 score 2, 24 score 1, 15 score 0. On specificity, 44 score 2, 33 score 1, 23 score 0. Under the pass rule (cause 2 plus evidence 1 or 2), suppose 60 pass. That headline number understates the risk, because 23 reasons are too generic to route reliably even when they are true. The specificity column is your routing queue problem, and it is the easiest to fix with prompt changes that demand a quoted trigger.

Then break the numbers down by trigger type. If explicit human requests pass at a high rate but tool failures pass poorly, the agent is converting system errors into customer-side stories, which is precisely the failure LiveKit's pattern warns about: escalation should carry the actual state of the attempted action, not a narrative about it (LiveKit). Denser's trigger table makes the same point, that tool errors should transfer with the error preserved (Denser).

The dangerous class is not the obviously wrong reason. It is the fluent one. Judge models are known to prefer long and confident answers, so an LLM grader alone will not catch this class; you need the trace check (Medium, agent evaluation). Use the judge for cause correctness if you like, but make evidence support mechanical: extract each claim as a sentence and require a field match, whether by exact substring, an embedding lookup over the source fields, or a short human read. Mechanical checks do not flatter confident prose.

Watch for three recurring patterns, which together account for most of the 0s in a first audit:

1. Symptom-to-cause substitution. The customer describes a symptom ("I was charged twice"); the reason invents the cause ("pricing error"). The cause is a hypothesis the ticket does not confirm.
2. Category inflation. A routine question becomes a "complaint" because the tone was terse. Frustration is a legitimate trigger, but the reason should then say the customer was frustrated, not that a complaint was filed.
3. Policy hallucination. The reason cites a rule ("outside the return window") when no policy field was read or the window is open. This is the highest-cost error, because it changes eligibility, not just routing.

Regulatory pressure makes the third pattern more than an internal quality issue. Consumer protection authorities have taken the position that inaccurate chatbot statements can be unfair or deceptive, and enforcement bodies have said they will use existing authority against harmful AI outcomes (Debevoise). A reason that states a policy condition the customer does not meet is exactly the kind of statement that gets quoted back at you.

Run the probe set like a regression suite. Every prompt change, model change or policy update gets scored against the same probes, and any drop in cause correctness or evidence support blocks the release. Agent evaluation frameworks treat this gating as standard practice for multi-step systems (Braintrust, Algolia).

Two cheap additions pay for themselves. First, sample live production handoffs monthly and add the worst ones to the probe set, so the set grows toward your real failure surface. Second, store the stated reason alongside the receiving agent's disposition. If humans routinely reclassify the cause when they open the case, you have found misclassification the probe set has not yet covered. Closed-loop tagging of escalation outcomes is the mechanism that lets routing and prompts improve from every handoff (Pedowitz Group).

For teams running many models or channels, the plumbing cost of the audit often exceeds the analysis cost, because each model behind the agent needs the same trace capture. Quanta Inference gives you text, image and embedding models through one OpenAI-compatible key, which keeps the probe harness stable when you swap the model that writes the stated reason (Quanta Inference). Keep the scoring itself deterministic wherever you can.

One governance note. Storing full tickets in a probe set means storing customer data. If your support operation falls under disclosure rules, the handoff behavior and the audit trail around it may be reviewed, and regulators are moving toward explicit requirements on what customers are told and when they can reach a human (Quarles). Redact and retain the probe set under the same policy as the tickets it came from.