Teaching an Agent to Pick Its Own Reason Code

Treat the code list as data, not prose. Each code needs three things: the code itself (a stable string your database and reports already use), a one-line definition, and one example of an item that belongs there plus one that does not.

Here is a system instruction block for a support triage agent. The structure matters more than the wording.

You assign exactly one reason code to each support conversation.

Available codes (use the code string exactly as written):

- BILLING_DUPLICATE: The customer was charged twice for the same period.
  Belongs: "I see two charges of $49 on the same day."
  Does not belong: "I want to downgrade to save money."

- BILLING_REFUND_STATUS: The customer asks where a promised refund is.
  Belongs: "You said the refund would arrive in 5 days, that was 9 days ago."
  Does not belong: "Please refund this order."

- ACCESS_LOCKED: The customer cannot sign in and has already tried a reset.
  Belongs: "I reset my password twice and still get 'account locked'."
  Does not belong: "How do I change my email address?"

Selection rules:
1. Return exactly one code from the list. Never invent a code.
2. If two codes could apply, choose the one that describes the customer's
   primary request, not the background detail.
3. If no code fits, return NO_MATCH and quote the sentence that made you
   reject every code.
4. Justify in one sentence, quoting at most 15 words from the conversation.

Return JSON only, in this shape:
{"code": "<CODE>", "justification": "<one sentence>"}

Four details do most of the work. First, the codes appear in a fixed order and exactly as they appear in the database, so there is no normalization work later. Second, each definition describes the customer's situation, not your internal process. Third, the negative example ("does not belong") often matters more than the positive one, because it draws the boundary the agent would otherwise cross. Fourth, the fallback code NO_MATCH is explicit. Without it, an agent that has no good option will silently pick the closest wrong one.

The output shape matters too. Asking for the code plus a short justification gives you a cheap review signal: when a reviewer disagrees with the code, the justification usually shows why in one line. Do not ask for a confidence score. Models are poorly calibrated on self-reported confidence, and a justification can be checked directly against the transcript.

An agent that returns free text will eventually wrap the code in a sentence, add punctuation, or return two codes. Parse strictly on your side and fail loudly rather than guessing.

import json

def parse_code(raw: str, valid: set[str]) -> dict:
    try:
        obj = json.loads(raw)
    except json.JSONDecodeError as e:
        raise ValueError(f"not JSON: {raw[:120]}") from e
    code = obj.get("code")
    if code not in valid:
        raise ValueError(f"code not in list: {code!r}")
    return obj

Two rules on top of the parser. Reject anything that is not exactly one value from the valid set, and count every rejection. A rejection rate above a couple of percent usually means the output shape is under-specified, not that the model is bad. If your inference layer supports constrained generation, use it: the Label Studio comparison of automated error-detection methods shows constrained decoding against a fixed list of choices as a way to stop the model drifting outside the label set (Label Studio).

The same discipline applies when the agent is picking fields for a routing workflow rather than a single code. Tray's routing guide is blunt that classification should be read from the ticket content rather than a dropdown the requester guessed at, and that the priority a requester selects carries almost no information because everyone picks urgent (Tray). Your agent is that content-based classifier, so the taxonomy it uses has to be built from real conversations and defined precisely enough that a machine can apply it.

If you route to several models (a cheap one for volume, a stronger one for appeals), Quanta Inference gives you one OpenAI-compatible endpoint for text models with a single key and prepaid USD pricing, which keeps the code list in one place instead of per vendor: https://quantaintelligence.ai/inference

Ambiguity in reason codes has a small number of distinct causes, and each has a different fix. Diagnosing which one you have is the part most teams skip.

The overlap case is the most common. If BILLING_DUPLICATE and BILLING_REFUND_STATUS keep trading places, add one line: "If the customer mentions two identical charges, use BILLING_DUPLICATE even if they also ask for a refund." Naming both codes in the tie-break is what makes it usable. A generic instruction to "prefer the primary request" is not enough when both look primary.

For genuinely multi-label work, do not force a single field. Change the output shape to an ordered list and define the order.

{"codes": ["BILLING_DUPLICATE", "ACCESS_LOCKED"], "primary": "BILLING_DUPLICATE",
 "justification": "Two charges on the same day; the lockout is a side effect."}

Then score two things separately: set match (all codes correct, nothing extra) and primary-code match. Teams routinely find that primary accuracy is high while set accuracy is much lower, because the agent finds a second real issue that the human missed. That is not a model failure, it is a labeling policy gap, and it is worth knowing before you penalize the agent for it.

The customer target that makes this worth doing is large. Crisp's playbook cites a Gartner prediction that agentic AI will autonomously resolve 80 percent of common customer service issues without human intervention, cutting operational costs by 30 percent along the way. A code list the agent cannot apply consistently is the main thing standing between you and that number.

The test set is the whole project. Everything else is maintenance.

Start from real items, not synthetic ones. Pull a sample of recent conversations or tickets that have already been labeled by a person who read them, and have a second person independently re-label a subsampled share to measure human agreement. If two humans disagree on a code, that code is not ready for automation.

Fifty is a floor, not a target. A useful rule of thumb for a set of N codes is at least 15 items per code for the codes you care about most, plus a slice of NO_MATCH cases, plus the 10 to 15 items you already know are hard. A 50-item set across 12 codes means roughly four items per code, which will not tell you whether a rare code works.

Keep the set in a spreadsheet or a table like this, one row per item.

Freeze the set. Every time you "fix" a test item to match the agent's output, you are training on your test set. Keep a separate small dev set for prompt iteration if you find yourself needing one. Lyzr's guide to golden datasets for agent evaluation treats the labeled set as ground truth rather than a scratchpad, and it stresses that the labels are what make measurement meaningful at all (Lyzr).

Human labeling is not overhead to be skipped; it is the mechanism that makes the whole system trustworthy. Google's overview of human-in-the-loop ML describes the standard ways people contribute to a model's lifecycle, from providing labels for training data to evaluating a model's predictions and feeding corrections back in (Google Cloud). Your test set is the second of those: humans evaluating predictions against a reference only they can supply.

Exact-match accuracy on the code field is the headline number, but on its own it hides the failure that matters most: one code that never gets chosen correctly.

Use per-code precision and recall, and report the confusion pairs. Evidently's guide to LLM evaluation metrics lays out the classification metric set (accuracy, precision, recall and per-class breakdowns) as the reference-based approach for exactly this kind of task (Evidently AI).

A worked example. Suppose you have 60 test items across 6 codes, 10 each, and the agent matches 51 of them.

• Overall match rate: 51/60 = 85 percent.
• One code, BILLING_REFUND_STATUS, accounts for 5 of the 9 misses.
• Excluding those 5, the agent is right on 51/55 = 93 percent of the rest.

That breakdown changes the decision. An 85 percent headline might tempt you to rewrite the whole prompt. The breakdown says the problem is one definition and probably one missing example. Fix that code, re-run, and expect the overall number to move toward 93 percent if the diagnosis was right.

Report stability as well. Run the same test set three times and look at the spread. An agent whose score moves several points between identical runs is not ready for unattended use, even if the average looks good. Anthropic's guidance on agent evals makes the point directly: outputs vary between runs, so multiple trials are needed to produce consistent results, and a task can carry several graders each with several checks (Anthropic).

One more metric worth tracking: justification agreement. Sample 20 correct matches and check whether the quoted evidence actually supports the code. A model that gets the right code for the wrong reason will break the first time the conversation format changes.

The loop is short and should be boring.

1. Run the agent on the frozen set, three trials.
2. Sort mismatches by code, then by confusion pair.
3. For each cluster, read the justification and decide: overlap, too-broad definition, missing code, or a genuinely multi-issue item needing a policy decision.
4. Make one change at a time: a tie-break line, a definition edit, a new example, or a new code.
5. Re-run the full set. Compare overall match rate, per-code recall and stability.

Stop when the number stops moving for two consecutive changes, and when the leftover mismatches are items your human reviewers also disagree on. At that point more prompt text is not the lever.

Where the loop does not converge, the honest options are to route the uncertain codes to a person, or to reduce the taxonomy. A short, well-defined list of codes that the agent gets right is worth more than a comprehensive list it gets right 70 percent of the time. If uncertainty clusters around high-stakes cases, keep a person in the loop for those specific ones: human-in-the-loop design is about placing checkpoints where the cost of an error is high, not about reviewing everything (Plane). IBM's treatment of the same concept describes the loop as a two-way interaction in which human input actively shapes what the system does next, which is exactly what a reviewer's code correction should do to your agent (IBM).

A practical production guardrail: let the agent assign codes automatically where per-code recall on the test set is high, and send the rest to a review queue. Track how often reviewers change the agent's code. If that rate climbs, the conversation mix has shifted and the test set needs new hard cases.

If the reason codes are one part of a larger agent you need built end to end, Quanta pairs you with a specialist who builds it with you: https://quantaintelligence.ai/