How to Choose a Model for Ticket Triage on Your Own Cases

The label is the category the ticket should have been routed to, which is usually the category it ended in after resolution. That is not always the same as the category it was opened under. Pull the final assignment, then spot check.

Labeling discipline is where most test sets quietly break. Three rules keep it honest:

1. One label per ticket. If a ticket is genuinely two categories, either drop it or create a combined label and count it separately. Multi-label tickets inflate the difficulty in a way that is hard to compare across models.
2. Label blind to model output. Have someone label before seeing any model predictions, then freeze the file. If you relabel after seeing results, you are grading to the answer key.
3. Record the labeling date and the ticket date. Ticket language drifts. A set built from tickets older than your product has existed tests the wrong things.

For subjective categories, the human agreement rate sets the ceiling. If two experienced agents agree on the category only 85 percent of the time, a model at 88 percent is effectively at the top of the useful range, and the remaining errors may not be errors at all. This is the same saturation problem qualitative researchers hit: at some point additional data stops changing the picture, and the useful question becomes whether your labels are consistent, not whether the model is perfect a systematic review of saturation in sample sizes. Qualitative sampling guidance makes the same point in product terms: homogeneity of the population drives how many examples you need, and diverse populations need more a guide to sample sizes in qualitative research.

Practical labeling target: aim for 30 to 50 examples per category you care about, so a category with a 10 percent share of traffic needs about 300 to 500 tickets pulled with stratification. That is a few hours of an agent's time and is the single highest-return investment in the whole project. If you would rather not staff that, an expert who has run this labeling loop before will get you through it faster than starting from a blank sheet: Quanta Consulting offers a free half hour to scope it.

"Accuracy" as a single number hides exactly the failures that hurt. Define three metrics, each with a written definition, before any model runs.

Accuracy per category. For each category c, the count of tickets whose true label is c and whose predicted label is also c, divided by the count of tickets whose true label is c. This is recall for that category, and it answers "how often does the model find this kind of ticket." You should also track precision per category (of the tickets predicted as c, how many really are c), because a model that over-predicts a category will flood that queue. The distinction matters: recall tells you how many real cases you miss, precision tells you how much you can trust the model's positive calls precision and recall in classification metrics. Raw accuracy on an imbalanced set is misleading for the same reason: a model that predicts the majority category every time can look decent and be useless classification metrics and imbalanced data.

Latency, median and 95th percentile. Record both. Triage sits in a request path, and the slow tail is what users notice. Report time to first token and end-to-end latency separately, and note that the distribution matters more than the mean reproducible performance metrics for LLM inference. Measure under realistic concurrency, not one request at a time, because shared endpoints throttle and a single-threaded benchmark will flatter every provider.

Cost per thousand tickets. Compute it as (input tokens per ticket times input price) plus (output tokens per ticket times output price), all times 1,000. Use your measured token counts, not the vendor's example. Total cost of ownership methods that fold in hardware, hosting and licensing for self-hosted options follow the same structure NVIDIA's cost per prompt methodology. A short output matters more than a short input: the same benchmarking work found that 100 input tokens have roughly the same latency impact as one output token, so constraining the model to a bare category label is a cheap win reproducible performance metrics.

Here is the scorecard to keep for each candidate:

The whole point of a fixed set is that you can hand the same file to Model A, Model B and Model C and compare like with like. Freeze the file, hash it, and run all candidates on that exact version. Any model that sees a different set does not get compared.

Keep the prompt as close to identical as the model allows. If one candidate needs a longer instruction to perform, that is a finding, but log the difference so you know the comparison is not purely apples to apples. Ask for a single category label and nothing else, then parse it. Free-form explanations cost tokens and add parsing failures that will show up as accuracy loss.

A minimal harness, enough to run the comparison honestly:

import json, time, hashlib

def run(model, tickets, ask, client):
    results = []
    for t in tickets:
        start = time.perf_counter()
        resp = client.complete(model=model, prompt=ask + t["text"], max_tokens=8)
        elapsed = time.perf_counter() - start
        results.append({
            "id": t["id"], "truth": t["label"], "pred": resp.text.strip().lower(),
            "latency_s": elapsed,
            "in_tokens": resp.usage.input_tokens, "out_tokens": resp.usage.output_tokens,
        })
    return results

Run each candidate two or three times and record the variance. Outputs are probabilistic, so a single pass can flatter or penalize a model by chance. Where a scoring step itself uses a model (an LLM judge for near-duplicate labels), sanity check the judge against human labels first, since judges are not reliable out of the box LLM evaluation guidance on judge reliability. For category labels you usually do not need a judge at all: string matching or a small normalization step is cheaper and more reproducible. When you do need mixed scoring, combine code checks, model judges and human review rather than relying on any one method a practical guide to LLM evaluations.

Because you are comparing several models across text, embeddings and possibly reranking, running the harness once per provider means three SDKs and three billing accounts. One key for every model type removes that friction, which matters when you will rerun this set every time a candidate changes: Quanta Inference exposes text and embedding models through a single OpenAI-compatible endpoint with prepaid, dollar-denominated pricing, so the harness stays the same as candidates rotate.

Now you have a table. The temptation is to sort by the macro average and pick the top row. Resist it. The right reading depends on what your service targets actually are, and that is a business question, not a benchmark question.

Start by writing down your targets before you look at results. A realistic set looks like this:

• Refunds and disputes: recall at least 95 percent, because a missed dispute breaches a policy clock.
• Security or account access: recall at least 95 percent, same reason.
• Feature questions: recall at least 80 percent, because a misroute costs one extra handoff.
• Everything else: recall at least 85 percent.

Then apply a hard filter, not a weighted score. A model that misses the 95 percent floor on disputes is out, regardless of how cheap or fast it is. Filtering on floors prevents the classic failure where a blended score hides a single catastrophic category.

Here is a worked example. Suppose you process 40,000 tickets a month and your categories break down as 55 percent feature questions, 25 percent billing, 12 percent refunds and disputes, and 8 percent account access.

Model A has the second-highest macro recall but falls below the 95 percent floor on refunds and disputes, where 12 percent of 40,000 tickets is 4,800 tickets a month, so a 11 percent miss rate means about 530 disputes routed wrong monthly. Model B clears the refund floor and is the cheapest and fastest, but its feature-question recall of 79 percent is under your 80 percent floor and matters at scale: 22,000 feature tickets a month times 21 percent is roughly 4,600 misroutes. Model C clears every floor, but at 3,800 ms in the tail it will breach a two-second triage SLA, and it costs more than four times Model B.

The decision rule that falls out of this: route by category. Use Model B as the fast, cheap default, and send only the categories where it misses the floor to Model C. Billing and feature questions go to B; refunds and disputes go to C. The blended cost lands far below all-C and the tail latency only affects the small dispute volume. This is the insight a public leaderboard cannot give you: the winner is a routing policy, not a model.

Two more reading habits separate a useful result from a decorative one. First, plot the confusion matrix, not just the diagonal. If most of Model B's feature-question errors are landing in billing, that is a prompt or taxonomy problem you can fix before changing models. Second, check whether your errors are near-misses. "Refund not received" landing in disputes rather than billing is a harmless label boundary; the same ticket landing in feature questions is a policy failure. Error cost is not uniform, and a weighted accuracy that reflects your cost per misroute is often the most honest single number you can report.

Finally, do not treat the result as permanent. Models change, your product changes, and ticket language changes. Schedule a rerun of the frozen set on a fixed cadence, and keep the set versioned so an old result can be reproduced. That is the difference between a one-off bake-off and an evaluation practice, and the practice is what keeps triage accurate as the queue evolves offline versus online evaluation and versioned configuration.

Sometimes the honest answer is that your set cannot separate the candidates, and the right move is not to pick one anyway. If two models are within a few points and both clear your floors, choose on cost and latency, and say so. That is a defensible decision with a stated basis.

If neither clears a floor, the problem is usually upstream of model choice. Common causes: categories that overlap so much that humans disagree, a prompt that does not enumerate the categories, or a taxonomy with 40 labels where 8 would do. Fix the taxonomy, relabel a fresh slice, and rerun. No model will rescue an ambiguous label set.

There is also a middle path worth testing: a two-stage design. First ask a cheap model to choose among the top-level categories, then ask a second, more careful call only for the categories that matter most. This is the same routing idea as the worked example, expressed inside one pipeline, and it usually beats swapping models because it spends expensive inference only where the cost of error is high. Benchmark the pipeline as a unit, using the same set and the same metric definitions.

Whichever route you take, the deliverable is the same: one frozen set, one scorecard with per-category accuracy, p50 and p95 latency, and cost per thousand tickets, and one written decision rule that says which categories go to which model and why. That document is worth more than any leaderboard, because it is the only comparison that uses your cases.