Confidence Thresholds for Support Answers

Write the rule down before you tune anything. For a given confidence score c, the assistant should answer when:

P(wrong | c) * C_wrong  <  C_escalation

where C_wrong is the full expected cost of a wrong answer and C_escalation is the cost of handing the ticket to a human. Rearranged, the break-even probability of being wrong is:

P(wrong | c) < C_escalation / C_wrong

That second form is the useful one. If an escalation costs $8 and a wrong answer costs $40, you should answer whenever the probability of being wrong is below 0.20. If wrong answers cost $120, the bar tightens to 0.067. The threshold moves with the ratio, not with the absolute numbers, which is why two teams with identical budgets can end up with very different fallback points.

To turn that into a confidence threshold, sort your labeled sample by confidence score and measure the empirical error rate in each bucket. The lowest score at which the observed error rate falls below C_escalation / C_wrong is your threshold. Anything below it escalates.

C_wrong is the number teams get wrong most often, because they count only the agent minutes to fix it. The full figure has at least four parts.

Rework time. The customer contacts you again, an agent reads the whole thread, apologizes, and re-resolves. Even modest rework compounds: a 2.3 contact-per-issue average means the real cost per issue is 2.3 times the cost per contact according to Lorikeet's cost-per-ticket benchmarks. A wrong answer is one of the main ways an issue acquires that second contact.

Escalation that happens anyway, later. A wrong answer rarely ends the conversation. The customer comes back angry and the ticket escalates with more history to read. Benchmarks cited by Maven AGI put escalated requests at roughly three times the cost of first-level resolution in its cost-per-ticket statistics roundup.

Refunds, credits and policy giveaways. If the assistant promises something the policy does not allow, someone either honors it or spends time unwinding it. Track these from your billing or credit system, not from agent memory.

Churn risk. This is the hardest to measure and the easiest to ignore. Customers are 2.4 times more likely to stay loyal when problems are resolved quickly, and a confidently wrong answer is the opposite of fast resolution per Crisp's analysis of chatbot economics. You do not need a perfect churn model. A rough expected loss, applied only to the share of customers who are at renewal risk, is enough to keep the number honest.

There is also a trust cost that shows up in surveys rather than ledgers. In a survey of 1,000 US consumers reported by CX Dive, half said they blame business leadership when AI support fails to answer their query per the CX Dive summary of AI failure costs. That is a brand cost that lands on the same ticket as the rework.

C_escalation is simpler, but it is still more than the hourly rate divided by tickets per hour. Build it from:

• Human handle time for that intent, including reading the AI transcript and any prior attempts.
• Channel cost, if escalation moves the customer to phone or a scheduled callback.
• Queue cost, the delay the customer experiences, which feeds satisfaction even when the eventual answer is right.
• Opportunity cost, the complexity your agents did not get to work on because they were handling a ticket the assistant could have closed.

Benchmarks are a starting point, not an answer. Published ranges run from a few dollars for simple retail interactions to tens of dollars for complex B2B cases per Lorikeet's benchmark summary, and a service desk benchmark cited by Maven AGI reports an average of $22.50 per resolved incident in its statistics roundup. Use your own payroll and volume data for the number you actually tune against.

Take a subscription software company. Suppose the fully loaded cost of an agent hour is $45, an agent handles 7.5 tickets per hour on average, so a simple escalation is $6.00 in handle time. Add $1.50 for the extra context reading that an AI handoff requires, and $0.50 of queue and tooling overhead. Call it $8.00 per escalation.

Now price a wrong answer on the billing intent:

Break-even probability of being wrong is $8.00 / $40.00 = 0.20. Now bucket your labeled sample by confidence:

The error rate first drops below 0.20 in the 0.70 to 0.79 bucket, so the billing threshold is 0.70. Every ticket scoring below that goes to a human.

Run the same exercise on the password-reset intent. There, a wrong answer mostly means one extra contact with no refund and negligible churn: C_wrong might be $10.00 against the same $8.00 escalation. Break-even is 0.80, so the threshold is far higher, and the assistant answers a much larger share of those tickets. This is exactly why a single global number loses money in both directions.

You need a sample of real tickets with a ground-truth label: would a correct answer have resolved this, and did the assistant's answer match policy. Two or three hundred tickets per intent, labeled by someone who knows the product, gets you bucket-level error rates with enough resolution to pick a threshold. A smaller sample gives noisy buckets at exactly the scores where the decision is close.

Three practical rules make the sample trustworthy:

1. Label against policy, not against plausibility. An answer that sounds right but contradicts the refund policy counts as wrong.
2. Re-label after any knowledge base change. Retrieval quality shifts when content changes; chunking strategy affects retrieval precision directly, and page-level chunking produced the highest average end-to-end accuracy with the lowest variance across the datasets NVIDIA tested per NVIDIA's chunking study.
3. Keep a holdout set. Pick the threshold on one sample and confirm the cost outcome on a second before you ship it.

If your assistant exposes a per-response confidence or a retrieval similarity score, check that the score is calibrated before you trust the buckets. A model that says 0.9 and is wrong 30 percent of the time at that score will send your threshold in the wrong direction.

A threshold only pays off if the handoff is cheap. If the human agent has to re-read the whole conversation and re-ask every question, your C_escalation is much higher than $8 and the correct threshold moves up accordingly.

Pass the transcript, the retrieved articles the assistant considered, the confidence score, and the reason for escalation. OpenAI's guidance on building agents makes the same point in its own terms: on failure, the system should halt and transfer control back to the user rather than improvise per OpenAI's practical guide to building agents. A clean transfer is a design decision, and it changes the economics of the threshold you set.

Knowledge quality is the other lever. An AI agent can only resolve what it can retrieve, so gaps in the knowledge base show up as low confidence and, if you answered anyway, as errors per Fin's guide to AI knowledge bases. Fixing content moves scores up and lets you lower the threshold without raising the error rate. Zendesk's guidance on writing for generative AI points the same way: focused, self-contained content in plain language produces better retrieval per Zendesk's content optimization guide.

Two more operating notes. First, log the confidence score on every answered ticket alongside the downstream outcome, so you can audit the error rate continuously rather than in quarterly projects. Second, treat the threshold as a cost control, not a quality score. A high threshold is not automatically safe: it may push volume to humans that could have been handled, raising total spend without improving satisfaction.

When you want a single endpoint for the text, embedding and audio models that sit under a support assistant like this, Quanta Inference puts them behind one OpenAI-compatible key with prepaid dollar pricing, at Quanta Inference. If your team would rather have the threshold model designed against your own ticket data, a free call through Quanta Consulting walks through your goals and returns a tailored recommendation.