A Confidence Threshold That Sends Weak Tickets to People

You cannot pick a floor without a reference. A golden dataset is a fixed, versioned set of test cases you run against before every meaningful change the Langfuse guide. It is the answer key: labels are fixed, so you can measure how well the system matches your standards and how that changes over time the Musubi guide.

Three practical rules shape a set you can threshold against.

First, size and composition. A single policy area needs 30 to 50 examples at minimum, and high-stakes categories need at least 100 the Musubi guide. For per-intent thresholds you need enough examples per intent to estimate precision with a usable confidence interval, so aim for at least 50 labeled tickets per intent and expect to grow the set from real failures. Sourcing cases from production failures rather than invented examples is the difference between a test that predicts field behavior and one that flatters your assumptions the qaskills.sh guide.

Second, difficulty. Deliberately hard cases are the point. If your golden set accuracy is consistently 100 percent, it is too easy the Musubi guide. Include tickets that sit near the boundary between billing and cancellation, because those are the ones your threshold will actually decide.

Third, versioning and leakage. Lock a version, keep old versions for trend tracking, and keep evaluation data completely separate from anything used for training or prompt tuning the Musubi guide. Decontamination from training data prevents inflated metrics the Maxim guide. A minimal case in JSONL, the format most eval tools accept, looks like this:

{"ticket_id": "t-1042", "text": "charged twice for august, need one refund", "expected_intent": "billing", "tags": ["billing", "duplicate-charge"], "notes": "near boundary with cancellation"}

Once the set exists, run the classifier over it and record, for every ticket, the predicted intent and the model's confidence for that intent. Then compute precision per intent at candidate thresholds. Precision here answers a narrow question: of the tickets the system labeled billing with a score at or above some floor, what share were actually billing according to the golden set?

Accuracy alone misleads when classes are imbalanced. If 70 percent of your tickets are billing, a classifier that always says billing scores 70 percent accuracy while doing nothing useful, and a judge that always predicts the majority class can score high while agreeing with humans no better than chance the Cohen's kappa guide for LLM judges. Cohen's kappa corrects for that by subtracting expected chance agreement, computed as (Po minus Pe) divided by (1 minus Pe), where Po is observed agreement and Pe is agreement expected from the two label distributions the Galileo explainer. Kappa runs from minus 1 to 1, with 1 as complete agreement and 0 as no agreement beyond chance the OneUptime calibration walkthrough.

For a routing gate, treat kappa as a precondition, not a target. If agreement between your classifier and the golden labels sits below roughly 0.6, no threshold will save you, because the errors are not concentrated in low-confidence outputs. The standard interpretation bands put 0.61 to 0.79 at substantial and 0.80 and above at almost perfect, with the 0.80 line originating in medical and social science research and widely adopted by the NLP evaluation community the Cohen's kappa guide for LLM judges. Use 0.80 as the bar for fully autonomous routing on an intent, and require human review on anything below.

A caution on small samples. With fewer than 10 annotations per category, the 95 percent confidence interval on kappa is wider than plus or minus 0.30, which makes the number statistically meaningless the Cohen's kappa guide for LLM judges. Fifty tickets per intent is a floor, not a luxury.

Here is the rule, stated precisely enough to implement.

For each intent, sort the golden-set tickets predicted as that intent by descending confidence. Walk down the list and compute cumulative precision at each score. The floor is the lowest confidence score at which cumulative precision at or above that score still meets or exceeds your target (95 percent in the worked example below), subject to a minimum volume so you are not setting a floor based on three tickets.

Three guards keep the rule honest:

Notice what the rule does not do. It does not aim for uniform precision across intents in a way that pushes a floor to 0.99 and turns three-quarters of a healthy queue into human work. Target precision is a business input. Pick it per intent from the cost of a wrong routing, which in support is asymmetric: the HDI and MetricNet benchmarks cited in industry writeups put a Tier 1 resolution near 22 dollars against 104 dollars or more once a ticket reaches Tier 3, and escalated requests are reported at three times the cost of first-level resolution the IrisAgent escalation guide and the Maven AGI cost-per-ticket roundup. That asymmetry argues for a high floor on intents whose misroutes tend to escalate, and a lower one on intents whose mistakes are cheap and correctable.

A threshold that lives only in a config file is unauditable. Six months later nobody can explain why ticket 4471 was auto-routed while its near-twin went to a human. Store the decision inputs and the decision in one record per output. A shape that works:

{
  "ticket_id": "t-4471",
  "intent": "billing",
  "confidence": 0.83,
  "floor": 0.86,
  "gate": "route_to_human",
  "model": "classifier-prod",
  "floor_version": "floors-2026-03-billing-v2",
  "golden_set_version": "golden-v3",
  "kappa_at_floor": 0.81,
  "decided_at": "2026-04-02T09:14:22Z"
}

Three fields carry the weight. floor lets you recompute the decision without re-running the model. floor_version and golden_set_version let you reproduce which evidence justified the number. kappa_at_floor records the agreement measured when the floor was set, which is the earliest signal that the floor has gone stale.

Stale floors are a real risk, and not only because models change. Golden datasets capture a snapshot while production distributions keep moving, which is why calibration has to be continuous rather than one-time the Galileo calibration guide. Provider-side model updates can shift scoring distributions silently, and re-judging with an updated model version has been observed to produce uniformly more lenient scores the Galileo calibration guide. Label your outputs with the floor and dataset version from day one; retrofitting that provenance is miserable.

The same discipline applies to how you measure drift in the underlying data. Choosing monitoring thresholds well requires looking at how your data has actually changed over past periods, not guessing at sensitivity the Evidently historical drift tutorial. Dataset stability can be benchmarked against drift rather than assumed the dataset stability benchmark.

Now the part teams skip. Suppose your support desk handles 2,000 tickets a day. You have measured the golden set and arrived at these floors:

Apply the floors to a recent traffic sample and count what falls below. Weight each intent's share by its below-floor rate:

• Billing: 2,000 times 0.55 times 0.18 equals 198 handoffs per day.
• Login: 2,000 times 0.30 times 0.31 equals 186 handoffs per day.
• Cancellation: 2,000 times 0.15 times 0.24 equals 72 handoffs per day.

Total: 456 handoffs per day, or 22.8 percent of traffic.

That number is the staffing question. If a reviewer handles 60 tickets per eight-hour shift, 456 handoffs need about 7.6 reviewer shifts per day, so plan 8, plus cover for absence. Two things about the arithmetic matter more than the answer.

First, the weighted average hides the intent story. A naive blend of the three floors would have suggested roughly 20 percent of traffic, but the actual mix puts login second by volume of handoffs despite being the smallest intent except cancellation, because its floor catches almost a third of its tickets. If login review is expensive, that is where to invest in better training data, not in raising the billing floor.

Second, run a single global floor of 0.80 through the same sample and compare:

The global gate is not cheaper in any meaningful sense. It moves 84 tickets a day from human review to automated routing, and the billing tickets among them are the ones with confidence between 0.80 and 0.86, which is precisely the band where billing precision falls below your target. You have traded a known number of misroutes for a smaller queue. Whether that trade is good depends on what a misroute costs, which is why the precision target has to come from the business before you compute anything.

A practical sequencing note: forecast on a sample that is genuinely recent and stratified, not on the top of your queue. Random sampling surfaces tail failures that unstratified sampling misses, and stratified sampling of high-confidence extractions is the only way to catch a novel error pattern at high confidence, because nobody looks at auto-routed items the human review calibration guide.

Treat the gate as a staged rollout with measurable exit criteria rather than a toggle.

1. Replay, do not route. Run the floors against two weeks of historical tickets and compare the routing decision to what actually happened. Disagreements where the gate auto-routed something a human later re-routed are your false accepts.
2. Shadow on live traffic. Let the gate compute and log decisions while humans still see everything. Compare daily handoff volume to the forecast; a gap above roughly 15 percent means your sample was unrepresentative or the live distribution shifted.
3. Measure routed precision directly. Weekly, sample a stratified slice of auto-routed tickets (including high-confidence ones) and label them. If measured precision on an intent falls below target for two consecutive weeks, raise that floor.
4. Watch kappa on the routed slice. A drop in agreement on auto-routed traffic is the signal that the model, the floor, or the golden set has gone stale, and the fix differs in each case the OneUptime calibration walkthrough.
5. Then reduce human review, intent by intent, on the segments that have demonstrated sustained precision the human review calibration guide.

If you are building the classifier itself and want to swap models or compare a hosted model against your current one without rewriting the client, one API key across text, image, video, audio and embedding models is what Quanta Inference provides; it is OpenAI-compatible and prepaid, so the threshold work above stays the same regardless of which model sits behind the scores.