Golden Sets for Support Agents: Intents and Escalation Paths

Pull 300 to 500 resolved tickets from the last two quarters and read them before you build any tooling. You are looking for verbs, not product areas. "Where is my order" and "my order arrived broken" are different intents even though both are about orders, because the first is a lookup and the second is a claims process.

Intent classification is the standard first step for support automation, and public tutorials label a fixed set of intents from real queries to train the classifier the Orq intent classification cookbook. Do the same, but keep the list short. Twelve to twenty intents is plenty for a first golden set. If a candidate intent has fewer than ten real tickets behind it in your sample, fold it into a neighboring intent and note it.

Be careful about how the intent list interacts with the action field. Some intents are almost always "answer" (reset password, change shipping address). Others are almost always "hand off" (legal threat, safety incident, data deletion request). The interesting cases are the ones in the middle, where the same intent can go either way depending on facts in the ticket. Those middle cases are where your agent will fail, and they should be over-represented in the golden set on purpose.

• Start from resolved tickets only, so you know the outcome the customer got.
• Exclude anything a human resolved by accident or by breaking policy.
• Write the intent list in plain language your new hires already use.
• Keep a parking lot file for intents you decided not to include yet.

A useful check: if two experienced agents can read the intent list and do not argue about which label applies, the list is ready. If they argue, the definition is too vague, and the agent will fail for the same reason.

Labeling alone produces a set that measures one person's taste. Label in pairs and have a third person, the gold owner, break ties. This mirrors how content moderation teams build golden data: one designated owner has final authority, and the team documents reasoning, not just labels the Musubi moderation dataset guide. Support is a policy domain in the same way. Who counts as authorized to request a refund? What counts as a cancellation threat?

The disagreement log is the most valuable artifact of the labeling week. Every time two labelers disagree, you have found either an ambiguous ticket or an ambiguous policy, and both are worth a line in the log. If more than roughly one in ten tickets ends in disagreement, stop labeling and fix the policy or the intent list first. Labeling through ambiguity just bakes it into the gold labels.

For the expected action field, the handoff boundary deserves special attention. Write down the trigger, not the theme. "Hand off when the customer mentions a chargeback, a regulator, or a lawyer" is a trigger. "Hand off when the customer is angry" is a theme, and angry customers often just need a fast answer. For the escalate path, write the queue and the two facts the receiving team needs, because an escalation that arrives without context gets bounced back and doubles the cost.

The practical answer to "how many examples do I need" is smaller than most teams expect. A focused set of 50 to 100 well-chosen cases is enough to start, and sourcing from real failures beats inventing examples the QASkills golden dataset guide. For a support agent, allocate by expected loss rather than by traffic share.

Here is a worked allocation for a 240-example set, modeled on a subscription business with a billing team, a technical support team, and a policy queue.

The shares are deliberately lopsided. Account takeover, data deletion, and legal threats together are 60 of 240 examples, a quarter of the set, even though they may be a small slice of ticket volume. That is the point. A golden set should be intentionally difficult and weighted toward what is hardest to get right, not what is most common the Musubi moderation dataset guide.

Do a quick arithmetic check before you commit. Suppose the account takeover examples carry a bold-error cost of 5 and the password reset examples carry a cost of 1. If your agent answers 10% of the takeover cases when it should hand off, that is 3 failures worth 15 units. If it over-escalates 30% of password resets, that is 6 failures worth 6 units. The smaller, rarer failure does more damage, and a frequency-weighted set would have hidden it.

A pass rate hides the cost structure. Score each run with expected loss instead: for every example, add the cost of the mistake the agent actually made, then sum. An over-cautious agent that hands off everything scores low on bold errors and high on cautious ones, and the totals tell you which failure mode you are shipping.

The rule that falls out of this table is simple. Pick the run with the lowest total expected loss, not the highest accuracy. Moving from v2 to v3 barely changes the total, but it moves risk from the expensive column to the cheap one, and for a support agent that is usually the right trade. For a limited set of categories, you can also go stricter and require zero bold errors on the highest-cost intents, the way high-stakes moderation policies get a larger minimum sample than routine ones the Musubi moderation dataset guide.

Beware the reason escalations hurt so much. Tickets that leave the first tier can cost two to three times more than those resolved there, according to benchmarks cited in a support-operations analysis the HDI Conference article on escalations. Another analysis puts a first-tier resolution around $22 against $104 or more at tier three, citing HDI and MetricNet data the IrisAgent escalation guide. Treat those as directional, but the direction is consistent: a swallowed escalation is expensive, and a needless one is merely annoying. Set your cost tags so one swallowed account takeover outweighs several dozen unneeded clarifications, then let the arithmetic decide.

Three habits keep a golden set trustworthy. First, freeze it. Once a version is labeled and agreed, do not edit it; create a new version when policy changes so you can compare across versions the Langfuse engineering guide. Second, never train on it. Golden data is an independent test, and leakage quietly inflates your scores until the set stops predicting production behavior the QASkills golden dataset guide. Third, treat it as a living artifact that evolves with new failure modes and changing requirements, which is also what governance frameworks like NIST's AI Risk Management Framework expect from evaluation evidence the Maxim golden dataset guide.

The growth loop is straightforward. Every week, pull the tickets where the agent got a thumbs down, a human took over mid-conversation, or the case was reopened. For each, write the four fields, add it to the next version, and confirm the fix. Over a quarter the set becomes a catalog of every way your agent has actually failed, which is far more useful than a synthetic set that encodes your assumptions.

Before you scale, run one last realism check. If you test only against mocked routing tables and canned ticket text, you will not see how the agent behaves when the ticketing API is slow or returns a renamed field, the same trap teams hit when every service is mocked and the first real integration fails badly the dev/prod parity discussion. Golden examples should include a few live integrations, real order IDs, and stripped but realistic customer text, which means you need a safe way to use production-shaped data. Anonymization that preserves format and referential integrity is how teams get production-realistic test data without exposing customer records the Xata anonymization guide, and format-preserving encryption keeps identifiers valid while hiding their content so validation logic still works the Tonic FPE guide.

When you are ready to run the golden set against more than one model, sending the same 240 cases through a text model and a cheaper classifier is the usual next step, and an OpenAI-compatible gateway that takes one key for every model family keeps that comparison cheap and consistent; Quanta Inference is built for exactly that comparison. If building the agent itself is the bottleneck, Quanta's specialists can pick up the site, app, or internal tool and build it with you.