How to Build an Eval Set from Your Support Tickets

Your helpdesk already has a taxonomy. Tags, custom fields or ticket forms usually encode something close to "billing", "password reset", "shipping delay" or "integration error". Don't invent categories from scratch: start from the tags, then merge them until each category is big enough to sample.

A practical filter is to keep categories that account for at least 5% of ticket volume, and fold the rest into an "other" bucket you sample by risk rather than volume. Then check the tags against a sample of 30 tickets by hand, because tag hygiene is usually worse than the tag list suggests.

The categories you keep should satisfy three tests:

• Distinct failure modes. Billing disputes fail differently from shipping questions. If two categories always fail for the same reason, merge them.
• Enough volume. A category with 20 tickets in the window cannot support 200 eval cases. Either widen the window or merge.
• Stable over time. A category that only appears during a launch or a seasonal peak will make your pass rate swing for reasons unrelated to quality.

This mirrors the Jobs-to-be-Done habit of defining the job in terms of what the customer is trying to accomplish, not in terms of your product's internal structure dscout.com. "Get a refund for a duplicate charge" is a job. "Billing" is an org chart.

Two sampling strategies, used together, give you a set that is both representative and useful.

Proportional sampling keeps the eval set's category mix matched to real ticket volume. If billing is 40% of tickets and shipping is 10%, your eval set should be roughly 40% and 10%. This matters because a balanced set overstates performance on rare categories and understates it on the ones that dominate your queue ertas.ai.

Risk sampling adds extra cases from categories where a wrong answer is expensive: refunds, account access, safety or legal questions. You are deliberately over-weighting the failures you cannot afford, and you should say so when you report the number.

The rule of thumb: report the proportional number, act on the risk-weighted one. If you only have one set, report both cut from the same labeled cases.

Target a first batch of 100 to 200 tickets per category for a robust set, with 50 as the bare minimum for a meaningful accuracy estimate ertas.ai. That is a sensible starting point, but it is not derived from anything about the decision you need to make. The next section fixes that.

A pass or fail label is only useful if a second person, reading the same rubric, would agree. Vague rubrics produce disagreement, and disagreement inflates or deflates your pass rate depending on who labeled which half.

Write the rubric as a short checklist per category, not a paragraph of values. Each item should be answerable yes or no from the transcript alone.

A workable structure for a support reply:

1. Correct action. Does the reply take the action the resolution required (refund issued, reset link sent, order reshipped)?
2. Correct facts. Are all specific claims (amounts, dates, policy) supported by the ticket and the knowledge base?
3. No invention. Does the reply avoid mentioning steps, tools or policies that do not exist?
4. Tone. Is the reply free of blame and free of apologies for things that did not happen?

A case passes only if items 1, 2 and 3 are all yes. Item 4 is a separate score, not a pass condition.

Two habits make the rubric stick:

• Label blind. The labeler should not see the agent's original reply before judging, or they will anchor on it. Hide the resolution until after the verdict.
• Reconcile out loud. Have both labelers score the same 30 tickets, then discuss every disagreement and rewrite the clause that caused it. This is the same review habit experienced interviewers use, where the value comes from the after-interview review, not the interview itself commoncog.com.

Record agreement as a percentage of tickets where both labelers gave the same verdict. Below about 80% agreement, fix the rubric before scaling up. No confidence interval can rescue labels that two people cannot reproduce.

Here is the decision rule most eval guides skip. Your observed pass rate on the eval set is an estimate, and estimates from samples carry error. If 82 of 100 cases pass, you should not tell leadership that quality is 82%. You should say the true rate is probably somewhere around it.

The standard error of a proportion is:

SE = sqrt( p * (1 - p) / n )

and the 95% confidence interval is the pass rate plus or minus 1.96 times the standard error statology.org. The 1.96 comes from the 95% confidence level, and the calculator tables show the same z-values for 90%, 95% and 99% calculator.net. The formula is only trustworthy when the sample is not tiny and the rate is not near 0% or 100% sjsu.edu.

Worked example. Suppose you label 100 cases in the billing category and 82 pass.

• Pass rate: 0.82
• SE = sqrt(0.82 × 0.18 / 100) = sqrt(0.001476) = 0.0384
• 95% interval: 0.82 ± 1.96 × 0.0384 = 0.82 ± 0.0753, so roughly 74.5% to 89.5%

That is a 15 point wide interval. If your release gate says "ship if quality is at least 80%", this set cannot answer the question, because the interval straddles 80%.

Now label 400 cases and find 328 pass (same 82%).

• SE = sqrt(0.82 × 0.18 / 400) = 0.0192
• 95% interval: 0.82 ± 0.0376, so roughly 78.2% to 85.8%

The interval narrows to under 8 points. You can now say the true rate is above 78% with 95% confidence, and the release decision has a defensible basis.

If you have no prior estimate, assume a 50% pass rate, which gives the widest possible interval for a given n. That produces a simple planning table, using the same formula:

Two things follow. First, if you need to detect a 5 point improvement between two model versions, you need roughly 400 cases per category at a 50% rate, and more if the rate is far from 50%. Second, categories with a low failure rate need larger samples, not smaller ones, to put a useful bound on the failure rate. This inverse-square relationship between sample size and interval width is standard sample size practice, and the same calculators show it for proportions with a known or assumed rate select-statistics.co.uk.

A useful shortcut when you have no prior: a margin of error of about 5 points at 95% confidence needs roughly 400 cases, and about 7 points needs roughly 200 select-statistics.co.uk. Anchoring your first batch at 200 to 400 cases per category, as the short answer suggests, lands you in the range where the interval is narrow enough for most release decisions.

One caveat: that interval assumes each case is an independent random draw from the same population. Sampling by category breaks that, so compute the interval within a category rather than across a mixed set, or accept that the pooled number is a weighted average with a wider real interval. Proportional sampling restores some of the independence, which is another reason to keep the proportional set as your reporting set.

An eval set rots. Ticket categories change, policies change, and the model itself changes behavior in response to prompts you did not touch. The set needs a maintenance loop:

• Refresh quarterly. Add cases from the newest resolved tickets and retire cases whose resolution is now obsolete. Keep the retired cases in an archive so old model comparisons stay reproducible.
• Add every production failure. When a real customer hits a failure, add the ticket to the set before fixing the bug. This is the edge case discovery that synthetic data cannot give you ertas.ai.
• Anonymize. Replace names with placeholders, redact account numbers and swap real email domains for example.com before anything enters the set ertas.ai.
• Re-run agreement. Every time the rubric changes, run the 30-ticket calibration again. Rubric drift is the quiet killer of an eval set.
• Store the verdict and the reasoning. A pass label with no note is worthless six months later when someone asks why a case changed.

If you are evaluating more than one model or migrating between providers, keep the eval harness provider-agnostic. Quanta Inference gives you text, image, video, audio and embedding models through one OpenAI-compatible API key, which means the same eval script can run against several models without a rewrite, at https://quantaintelligence.ai/inference.

Week one: export the last 90 days of resolved tickets, hand-check 30 tags, and settle on four to six categories.

Week two: write the rubric and run the 30-ticket calibration with two labelers. Rewrite the clauses that caused disagreement.

Week three: sample 200 cases proportionally and add 50 risk-weighted cases from the costliest category. Label them across two people, with the same cases split so each labeler sees a mix.

Week four: compute pass rates and intervals per category. If any interval is wider than your decision can tolerate, extend that category to 400 cases before you report a number.

That is the whole method. The eval set stops being a guessing game and becomes a measurement with a stated error bar, which is the only thing that lets a pass rate on old tickets say anything credible about new ones.