A Reason Code Taxonomy for Agent Escalations

Write each definition as one sentence that names the condition, not the symptom. The sentence has to be specific enough that two reviewers reading the same transcript land on the same code. Here are working definitions.

Missing data. The agent could not act because a required input was absent from the request, the record, or the tool response. Examples: an order lookup returns no shipping address, so the agent cannot quote a delivery date; the user asks to change a booking but never supplies the booking reference; a CRM record has an empty contract tier field that the refund rule depends on.

Conflicting sources. The agent retrieved two or more sources that disagree on a fact the answer depends on. Examples: the help center says returns are accepted within 30 days while the contract PDF says 14; two order records show different ship dates; a policy page and a tool response disagree on whether an item is in stock.

Out of policy. The requested action is outside the rules the agent is permitted to apply, even with complete and consistent information. Examples: a customer asks for a refund outside the stated window; a user asks the agent to delete an account it has no delete permission for; the request needs an approval the agent cannot obtain. Note that the telecom example in escalation design guidance is exactly this shape: the agent must hold the line on credits outside policy while still acknowledging the frustration escalation rule design.

Low confidence. The agent can act, the inputs are complete and consistent, and its own confidence score falls below the threshold you set. Examples: the classifier scores intent at 0.41 against a 0.70 threshold; the drafted answer scores below the faithfulness bar; retrieval returns weak matches for a question that is technically in scope.

Here is how the set looks as a routing table.

The fifth row matters. You need a place for "none of the above," but it should be a log entry reviewed weekly, not a queue. If taxonomy gaps exceed a small share of escalations, your list is wrong and you should rewrite a definition rather than add a code.

A code alone is not enough to review or count. Require four fields on every escalation, and reject the handoff at the tool boundary if any is missing.

1. The code, from the closed list only.
2. The evidence pointer: the specific record, document ID, or tool call that triggered the code. For conflicting sources, both source IDs. For missing data, the field name and the call that returned null. For out of policy, the rule ID. For low confidence, the score and threshold.
3. The attempted action: what the agent was about to do, in one line, so a reviewer can judge whether the escalation was correct.
4. The trace ID: a link into the full transcript, so the reviewer can see the sequence of tool calls and reasoning, not just the last message.

Field four is where most schemes quietly fail. Output only evaluation misses hidden failures, because a correct final answer can hide a wrong tool call or a hallucinated retrieval that got lucky LangChain's trajectory evaluation guide. Google's framing of the same problem calls it a silent failure: the agent reports the right number but cites last year's report the agent evaluation guide. Without the trace, a reviewer codes the outcome and misses the cause.

A minimal schema you can paste into an escalation tool looks like this:

{
  "reason_code": "conflicting_sources",
  "evidence": {
    "source_ids": ["kb-returns-001", "contract-8821-v3"],
    "conflict_field": "return_window_days",
    "values": [30, 14]
  },
  "attempted_action": "quote refund eligibility to customer",
  "confidence": 0.62,
  "threshold": 0.70,
  "trace_id": "tr_8f21c0",
  "escalated_at": "2026-02-11T09:14:02Z"
}

Validate the code against an enum and reject unknown values at write time. A free text field will drift back to prose within weeks.

The reason each code maps to exactly one queue is that the fix differs by code, and the fix owner should see the case first. Routing on the code rather than on who is free follows the same logic as skill based ticket routing, where the queue that can resolve the issue is chosen before the individual who is available Tray's ticket routing model.

• Missing data goes to data and integrations. The reviewer checks whether the field exists, whether the tool returned it, and whether it was requested. Fixes are schema changes, retries, or a follow-up question the agent should have asked.
• Conflicting sources goes to knowledge and content. The reviewer decides which source is authoritative, updates or retires the stale one, and records the precedence rule. These are usually content bugs wearing an agent costume.
• Out of policy goes to policy exceptions, not to engineering. The case needs a human decision on the customer's situation, and the outcome feeds a log of requests that policy should probably cover. Track how many of these are approved; a high approval rate is a signal that the rule is too tight.
• Low confidence goes to model quality. The reviewer asks whether the threshold was right, whether retrieval was weak, or whether the prompt lacks the guidance that would raise the score legitimately. Raising the score by lowering the threshold is not a fix.

Two routing rules keep the system honest. First, the agent may not choose a queue directly; it picks the code and the router maps it, so a prompt change cannot quietly reroute traffic. Second, every queue has a named reviewer and a weekly count. A queue without an owner is just a longer escalation.

One caution on volume. Teams usually write their first trigger set too cautiously and end up escalating cases the agent could handle, with refund, complaint, or scheduling keywords sending more than half of conversations to humans in one reported case the escalation rules guide. When that happens, the counts by code tell you which trigger to narrow: a large low confidence bucket often means the threshold is set above what the agent reliably handles, not that the agent is weak.

The reason codes should not stay in the escalation queue. They belong in the evaluation set, because an escalation is a labeled failure and labeled failures are the raw material for regression tests.

The working pattern is a loop: production failures flow into annotation queues for expert review, then become regression tests that stop the same bug reaching users again LangChain's evaluation framework. Assign each escalation a code and you have a pre-sorted annotation queue. The reviewer does not start from scratch; they start from a hypothesis and confirm or correct it.

To build the graded set, sample escalations per code rather than at random. If you sample uniformly, a code that fires ten times a month gets almost no coverage while the largest bucket dominates. Sample a fixed number per code, keep the human reviewer's final label as ground truth, and grow the graded set from there. This mirrors the way custom evaluators are built: annotate a benchmark set, run the judge template against it, compare the judge's label with your human label, and iterate until the agreement is good enough Arize's evaluator build.

Two numbers make this measurable. First, judge agreement per code: the share of graded cases where the automated judge assigns the same code as the human reviewer. Second, escalation precision per code: the share of escalated cases a reviewer agrees should have escalated. A precision of 0.3 on low confidence means seven in ten of those escalations were the agent being needlessly timid, and the threshold, not the model, is the thing to change.

LLM judges are a reasonable instrument for scaling this. A judge has been reported to agree with human reviewers about 85 percent of the time, which is higher than two humans agree with each other on the same task Confident AI's LLM-as-a-judge guide. Treat that as a reason to calibrate, not to skip humans: check the judge against your graded set before trusting its counts, and re-check whenever you change a code definition.

Suppose a month of escalations gives 800 cases:

Read that table as four decisions, not one. Missing data at 0.95 precision is working: the agent correctly stops when a field is absent. Out of policy at 220 cases is a product question, since each approval is a customer asking for something the rules do not cover; if most approvals are granted, the rule needs rewriting, not the agent. Conflicting sources is small but expensive, because each case is a content owner deciding precedence between two sources; the thirty reviewed cases should produce a short list of document pairs to fix.

Low confidence is the problem. At 0.35 precision, roughly two-thirds of those 400 escalations were avoidable, which means the threshold is misjudged or retrieval is weaker than the agent's guidance assumes. The action is to pull the 250 plus cases where the agent could have acted, run them through the graded set, and tune the threshold and retrieval rather than the prompt's tone. Then re-measure. If precision rises toward 0.7 and volume falls, the change worked.

The trap to avoid is optimizing the wrong number. Total escalations going down is not progress if precision drops, and precision rising while volume holds is not progress either. Track both, per code.

You do not need a platform migration to start. Three steps get a working taxonomy running on existing traces.

First, backfill. Take your last few hundred escalations, read the traces, and apply the four definitions by hand. The share you cannot classify is your taxonomy gap rate, and reading those cases is how you sharpen definitions before the agent ever writes a code.

Second, enforce the schema. Add the enum and required fields to the escalation tool call, and make the agent's prompt state each definition verbatim. Definitions in the prompt should match the definitions in the router, word for word; two slightly different versions of "out of policy" is how reviewers end up disagreeing with the agent.

Third, wire the loop. Sample per code, grade with a judge calibrated against human labels, publish precision per code weekly, and feed corrected cases back into the graded set. Human review of ambiguous cases belongs in this loop, since the point of human in the loop is to place judgment where the cost of an error is highest rather than on every decision Plane's human-in-the-loop overview.

If you are running several models across these agents, Quanta Inference puts text, image, video, audio and embedding models behind one OpenAI-compatible key with prepaid dollar pricing, which keeps the reason code judge on the same footing as the agents it scores.