How to Set a Minimum Accuracy per Ticket Category

Accuracy as a raw number hides the failures you care about. A system can be 99 percent accurate and still produce 100 wrong actions a day at volume, which is the point made in a survey of accuracy metrics for AI evaluation Galileo. The fix is to convert errors into service outcomes.

Do this per category, with three inputs:

1. The volume. How many tickets per period does this category handle? Use a real export, not an estimate.
2. The service-criticality. What fraction of errors in this category actually causes a breach of the promise? A misrouted billing ticket is usually caught by the customer or the agent; a misrouted safety report is not.
3. The recovery path. If an error is caught, how much time does recovery add? A re-route costs minutes. A wrong auto-reply costs a correction plus trust.

The second input is the one people skip, and it is the one that does the work. Not every classification error breaches the service level. If a ticket lands in the wrong queue but the queue is still a customer-service queue and an agent sees it within the first-response window, the promise holds. The error matters only when it pushes the ticket past the target. Call that fraction the breach rate, and estimate it from your own history by sampling 100 to 200 resolved tickets per category, as recommended when building an evaluation set from real conversations Ertas.

That source also warns that models evaluated on clean synthetic test sets score 8 to 15 percentage points higher than the same models evaluated on real production data. Use real tickets for the breach-rate estimate, or your thresholds will be built on a number that does not exist in production.

Now the formula. For a single category:

allowed_breaches   = category_volume * (1 - service_target)
max_error_rate     = allowed_breaches / (category_volume * breach_rate)
min_accuracy       = 1 - max_error_rate

Which simplifies to:

max_error_rate = (1 - service_target) / breach_rate
min_accuracy   = 1 - (1 - service_target) / breach_rate

This is the decision rule in one line. The category's volume cancels out, which surprises people. It cancels because both sides of the ratio scale with volume. Volume only matters when you combine categories into an overall figure.

A worked example. Suppose you promise that 95 percent of tickets in a period are resolved within 24 hours, so the target is 0.95 and the allowed breach share is 0.05.

Read the second row out loud. A billing dispute that goes to the wrong queue breaches the 24-hour promise half the time, so you can afford a 10 percent error rate, which means 90 percent accuracy is enough. A feature request routed to the wrong team still gets an answer inside 24 hours four times in five, so you can tolerate 25 percent error, and 75 percent accuracy meets the promise.

Now check the aggregate. The allowed breaches across all categories are 8,000 times 0.05, which is 400 tickets. Each category's error budget is its volume times its maximum error rate: 4,000 times 0.0625 equals 250; 2,000 times 0.10 equals 200; 1,500 times 0.25 equals 375; and 500 times 0.0526 equals about 26. That sums to 851 allowed errors. But only 851 times the weighted breach rate breaches the promise, and since the breach rates differ by category, the aggregate has to be computed error by error, not as a single average accuracy.

This is the part first search results miss. The weighted average accuracy across the four categories is about 88.9 percent, comfortably below the 95 percent service target, and yet the promise still holds, because most errors land in categories where errors are cheap. If you had set a flat 95 percent accuracy floor on every category, you would have spent review capacity on feature requests that did not need it and starved the outage queue. Set the floor from the promise, per category, and the aggregate takes care of itself.

For a stricter promise, the same arithmetic tightens fast. At a 99 percent resolution target, the allowed breach share is 0.01, so the outage category with a breach rate of 0.95 needs 98.95 percent accuracy, while the feature request category needs 95 percent. Note what happened: the cheap-error categories moved least. That is the decision rule working as intended.

A threshold is only useful if the measurement can tell you whether you crossed it. Accuracy is a proportion, so it carries sampling noise. The standard error of a proportion is the square root of p times (1 minus p), divided by n Statology. The normal approximation behind it is adequate when n times p times (1 minus p) is at least 3 San Jose State, and a confidence interval around the estimate is the sample proportion plus or minus z times the standard error Statology.

Use it as a sizing tool. For the outage category, suppose you sample 200 tickets and the model is right on 186 of them. Accuracy is 0.93, the standard error is the square root of (0.93 times 0.07) divided by 200, which is 0.018, and the 95 percent interval, using a z of 1.96 Statology, runs from 0.895 to 0.965. Your threshold is 0.9474. The interval straddles it, so you cannot yet tell whether the system clears the bar. At 500 tickets the half-width shrinks to roughly 0.012, and the question becomes answerable. The arithmetic for the interval is the same one a basic proportion confidence interval calculator performs Sample Size Calculators.

The mirror image is a sample size question. If you want a margin of error of plus or minus 0.05 at 95 percent confidence, the planning formula gives roughly 400 observations when you assume maximum uncertainty, and about 300 if you assume the true proportion is 0.25 San Jose State. Survey calculators make the same point: a 5 percent margin at 95 percent confidence needs around 383 respondents when the population is large, and a 3 percent margin needs about 2,345 Select Statistical Services. Decide the margin you need to distinguish your threshold from the true rate, then size the review sample accordingly, and remember that when the population is small relative to the sample the requirement shrinks Survey Calculator.

One more check specific to classification. Accuracy can be misleading when classes are imbalanced: a naive classifier can score 95 percent on a dataset that is 5 percent positive simply by labeling everything negative Galileo. Ticket categories are exactly this shape. Outage reports are rare, so a model that never predicts an outage can still look accurate overall. That is why the outage category gets a recall-first threshold on the urgent class specifically, as the same guide recommends, and why raw accuracy is a poor gate for the rare categories Galileo. For a rare category, set the threshold on recall for that class and accept a higher false-positive rate, because over-flagging is far cheaper than under-flagging Precision Federal.

Here is the full calculation as a small script you can port to a spreadsheet:

import math

def min_accuracy(service_target, breach_rate):
    return 1 - (1 - service_target) / breach_rate

def interval(correct, n, z=1.96):
    p = correct / n
    se = math.sqrt(p * (1 - p) / n)
    return p, (p - z * se, p + z * se)

for name, vol, breach in [
    ("access", 4000, 0.80),
    ("billing", 2000, 0.50),
    ("feature", 1500, 0.20),
    ("outage", 500, 0.95),
]:
    print(name, round(min_accuracy(0.95, breach) * 100, 2))

print(interval(186, 200))

Run it and you get the table above plus an interval of roughly 0.895 to 0.965, which is the same conclusion the arithmetic gave by hand.

Thresholds should not be frozen, and they should not drift with every complaint. Use a fixed decision rule with three triggers.

Trigger one: standing service data. Compare the measured breach share for a category over a rolling period against the allowed share. If the measured breach share exceeds the allowed share for two consecutive periods, tighten the category's minimum accuracy. If it is below the allowed share for two consecutive periods and the interval clears the threshold, relax it by one step. Two periods, not one, because a single window is noisy, as the standard error already showed you.

Trigger two: the interval over the threshold. Before you tighten, check that the measured accuracy is statistically below the floor. The interval formula above is the test. If the interval contains the threshold, the honest answer is that you do not know, and the correct action is a larger review sample, not a model change.

Trigger three: taxonomy change. Any change to the category list invalidates the accuracy test, because a category that was merged or split now measures something different Ploch. Treat a taxonomy edit as a reset, re-run the human agreement check, and recompute the breach rates before the next measurement.

That last row deserves a note, because it is where the decision rule earns its keep. If the outage category grows from 500 to 1,000 tickets a month, its per-category minimum accuracy does not move, since volume cancelled out of the formula. The overall promise gets harder to keep, though, because cheap-error categories now make up a smaller share of the mix. The right response is to check capacity in the expensive categories, not to raise their accuracy floors. Raising the floor spends effort on the part of the system that was already working.

If you are still deciding where classification sits in the workflow, or you want a review-set built from your own ticket export before you commit to thresholds, a short scoping conversation is often faster than a spreadsheet you build alone. You can book a free 30-minute discovery call to map your service targets to per-category thresholds with a specialist: Quanta Consulting. And when the thresholds exist and you need one endpoint to run the classifier across chat, email and voice transcripts without stitching providers together, Quanta Inference offers text, image, video, audio and embedding models through one API key.

The most expensive trap is setting one accuracy number for the whole system and reporting it weekly. It satisfies a dashboard and tells you nothing, because it averages a cheap error with an expensive one. The second trap is measuring on a clean test set. A synthetic set reflects what you imagine the distribution looks like, not what it actually is, and it overstates performance on the categories that matter Ertas. Pull the measurement from production tickets, and keep the edge cases in a separate set rather than mixing them into the core score.

The third trap is evaluation that stops at launch. An offline score is a statement about a fixed dataset at a moment in time, and it says nothing about live traffic after a prompt or model change LaunchDarkly. If the judge or scoring method has its own error, the threshold is measured through a blurry lens; some studies of automated judges found the weakest of four frontier judges matched an expected score only 40 percent of the time on paraphrased responses, as reported in that same review LaunchDarkly. A practical habit is to keep a human-reviewed sample in the loop so the threshold is anchored to reality rather than to another model's opinion Confident AI.

The fourth trap is measuring one thing. A combination of code checks, model scoring, human review and production sampling covers failure modes that no single method catches Arize. For ticket classification the working set is small: an exact code check for the fields the form provides, a sampled human label for the ambiguous cases, and a live sample that confirms the queue routing actually happened.

Finally, tie the threshold to a check that is cheap to run every period. The formula is stable, the breach rates are not, so refresh them quarterly from a fresh sample of resolved tickets using the 100 to 200 ticket guidance Ertas. If you want to see how a real classifier is judged before you commit, the same sampling discipline applies to any classification task with an operational consequence Precision Federal.