What a Support Agent's Confidence Score Should Trigger

Every threshold recommendation in this article is beaten by one thing: your own data. Pull a sample of recent turns and join each one to an outcome signal. The two cheapest outcome signals are a post-interaction satisfaction response and a re-contact on the same issue within 24 to 72 hours.

Start with a sample of at least a few hundred turns across intents, not just the easy ones. For each turn record:

• the logged confidence score
• the intent or queue
• whether the customer confirmed resolution
• whether they re-contacted on the same topic within the window
• whether a human later reversed or corrected the agent's answer

Then bin the confidence values and compute accuracy per bin. Four buckets (0 to 0.5, 0.5 to 0.7, 0.7 to 0.85, 0.85 to 1.0) are enough for a first pass. What you are looking for is the point where accuracy climbs steeply. That inflection is where the score starts carrying information.

The measurement trap is confusing a closed conversation with a solved one. A contact counts as deflected only when the underlying issue is actually resolved, not when the customer abandons the interaction before reaching a human Decagon's glossary on deflection rate. A post-interaction survey suffers from low response rates, so many teams use a behavioral proxy: no human ticket and no repeat contact within 24 to 72 hours Decagon. That proxy is the right one for setting thresholds, because it is the same signal you will use to test the rule later.

Two more sanity checks before you move on. First, look at how confidence is distributed. If it clusters at a handful of round values like 0.85 and 0.90, the score has coarse resolution and your cuts need to sit between clusters rather than inside them. Second, split the sample by intent. A cut calibrated on order-status queries will overcook refund disputes, and the fix is per-intent cuts or a risk tier multiplier, not one global number.

A single threshold forces every borderline turn into the same bucket as the clearly hopeless ones. Three actions solve that: answer, verify, hand off. Here is the decision table a routing layer should implement, with the actions written out.

The verify band is where most of the engineering value lives, because it is where borderline cases get a second chance before a customer sees them. The cheapest verification is a grounding check: confirm that the claim in the draft answer is entailed by a retrieved span, and abstain if nothing supports it, a technique that outperformed temperature scaling and "rate your confidence" prompting in the calibration work above USQRD. An ensemble disagreement check, where you sample the answer three to five times at moderate temperature and measure agreement, is a good second option USQRD. The support-operations lesson is the same: one team cut incorrect responses by 60 percent after switching from pure generative responses to a retrieval-first design that referenced real resolution patterns from past tickets Bland's write-up of AI agent challenges in support.

For the hand-off band, what travels with the ticket matters as much as the routing itself. A good hand-off carries customer identity signals, timestamps, the user's exact request, prior attempts, and extracted constraints, and a clean handoff that keeps the customer from repeating themselves is one of the quality signals that separates real resolution from quiet abandonment Thinklytics on measuring AI support deflection. Escalation quality is what makes the low band defensible.

Setting the numbers is arithmetic, not taste. Pick a target error rate for auto-answered turns, say 2 percent. From your binned accuracy, find the lowest confidence where the bin error is at or under that target, and set that as the high cut. Then pick a tolerance for the verify band's cost (retrievals are cheap, human minutes are not) and find the confidence below which verification rarely recovers a good answer. That is the low cut. Where the low cut sits is also where escalation cost starts to dominate, so weigh it explicitly.

Worked example. Suppose you log 10,000 turns a month. Your calibration says turns above 0.88 are correct about 96 percent of the time, turns between 0.62 and 0.88 are correct about 80 percent of the time, and turns below 0.62 are correct about 45 percent of the time. Your sample says 55 percent of turns land above 0.88, 30 percent in the middle band, and 15 percent below. Auto-answering the top band gives you 5,500 answers at roughly 4 percent error, or about 220 wrong answers. Verifying the middle band with a grounding check that catches, say, 60 percent of the unsupported claims leaves roughly 240 wrong answers from 3,000 turns, and the rest either pass cleanly or route to a human. Handing off the bottom band protects 1,500 customers who were otherwise a coin flip. Now you can price the rule: compare the cost of 3,000 grounding checks plus the human minutes saved from rerouting wrong answers against the cost of 220 customer-facing errors, taking your own per-error cost from your refund and churn data. For context, human-handled inquiries are commonly priced at $4.13 to $6.00 each against $0.50 to $0.70 for AI interactions, so the arithmetic hinges on your own numbers rather than the averages Supportbench's review of deflection expectations.

The rule should not be static across all traffic. Some requests need a human regardless of score: refunds beyond standard limits, disputed fees, account recovery when verification fails, and anything that could bind the company. Low-confidence intent is another hard route, including competing intents, missing required parameters, and fallback spirals where the agent keeps asking clarifying questions without progress. Encode those as hard routes above the confidence rule, and make the reason codes specific enough that you can count them later.

The verify band has to do real work, so define its checks before trusting its results. A useful verify step has four properties:

1. It runs on the same retrieved context as the draft answer.
2. It checks a claim against evidence rather than re-asking the model how sure it feels.
3. It returns a crisp pass, fail, or retry, so the route decision stays deterministic.
4. It has its own escalation path: if verification fails twice, go to hand-off rather than looping.

A minimal implementation in Python-flavored pseudocode:

def route(turn, cfg):
    score = turn.confidence
    if turn.risk_tier == "high":
        return handoff(turn, reason="risk_tier")
    if score >= cfg.high_cut:
        return answer(turn)
    if score >= cfg.low_cut:
        check = verify(turn, mode="entailment")
        if check.result == "pass":
            return answer(turn, verified=True)
        return handoff(turn, reason="verify_failed")
    return handoff(turn, reason="low_confidence")

def verify(turn, mode):
    spans = turn.retrieved_spans
    for claim in turn.draft_claims:
        if not entails(spans, claim):
            return Check("fail")
    return Check("pass")

Keep the checks cheap and the pass criterion strict. A verify step that waves everything through is worse than no step, because it adds latency and gives false assurance. Track the verify band's pass rate by intent; a band with a very high pass rate is probably not checking anything, and a band with a very low pass rate is really a hand-off queue in disguise.

The telemetry you need for this is standard agent observability. Traces of each run, structured logs of tool calls and decision records such as selected routes and scores, usage metrics, and evaluations tied back to the run that produced the output are the building blocks that let you tell whether a route decision was correct groundcover's guide to AI agent observability. Microsoft's guidance frames it the same way: observability adds evaluations and governance to metrics, logs, and traces, and lets you ask why and how an agent behaved, not just what happened Azure's agent observability best practices. Store the cut values in the log record itself so you can reconstruct which rule version handled each turn when you audit later.

One more design choice: log the verify outcome, not just the final route. When a turn passes verification, that is evidence the middle band is doing its job. When it fails and hands off, that is a candidate for a knowledge-base gap. Over time the failed verifications point at the exact policy or product content that needs fixing. Most AI support failures are not model failures but data failures, where the agent acted on stale history or an out-of-date knowledge base, so the verify backlog is also a data maintenance queue Thinklytics.

Split your logged turns into a tuning set and a held-out set before you touch any threshold. Tune the two cuts on the first set; evaluate on the second, once. If you keep adjusting after seeing held-out results, it stops being held-out.

For the held-out evaluation, report at least five numbers per intent band:

• auto-answer rate and error rate on auto-answered turns
• verify pass rate and the error rate on verified-and-answered turns
• hand-off rate and share of hand-offs the human judged necessary
• repeat-contact rate for auto-answered turns within the window
• satisfaction on deflected contacts versus human-handled ones

A high deflection or containment rate paired with a high repeat-contact rate is a warning, not a win, because abandonment looks identical to resolution in a raw count Thinklytics on measuring AI support deflection. Containment measurement has the same trap: the 24-hour re-contact adjustment can lower reported containment by 5 to 15 percentage points relative to naive counting Decagon's containment rate glossary. Run your held-out evaluation with that adjustment applied, or you will ship a rule that looks better than it is.

The step-level escalation research offers a useful benchmark for how much the middle band is worth. On a held-out test split, a routing approach that escalated selectively hit 82.7 percent task success with 30.8 percent escalation, against 67.3 percent local-only and 75.4 percent for random escalation at a similar escalation rate the StepGate paper on uncertainty-aware handoff. Selective escalation beat random escalation at matched cost, which is exactly the claim your verify band is making. If your band's results look like the random-escalation row, tighten the cuts or improve the check.

Also measure the rule against the quality signal your stakeholders already trust. Deflection figures get cited in leadership reviews, and pairing the volume number with verified resolution, satisfaction on deflected contacts, and escalation quality is what turns a claimed deflection rate from a claim into evidence Swept's guide to verifying deflection metrics. Bring your held-out table to that conversation instead of the headline rate alone.

Finally, decide in advance what will trigger a re-tune. New intents, a policy change, a model change, or a jump in repeat contacts should all send you back to the reliability diagram. Calibrate on your own data, per task, and treat the cuts as configuration with an owner, not a constant buried in code.

The rule above assumes one agent and one model. Teams often end up with several models across channels and intents, and the routing layer then has to hold confidence cuts per model rather than per conversation. Quanta Inference exposes text, image, audio, and embedding models through one OpenAI-compatible API with prepaid US dollar pricing, which keeps the per-turn cost of the verify band easy to read from a single invoice. If you want a second set of eyes on the calibration work itself, a free consulting call is a reasonable place to pressure-test your reliability diagram before you change production cuts.

The organizational side is just as real: whoever owns escalation policy should agree the risk tiers, and whoever owns the knowledge base should own the failed-verification backlog. Those two conversations determine whether the middle band improves or quietly fills with noise. Then write the rule down in one place, with the cuts, their owners, the review cadence, and the held-out evaluation results attached, so nobody re-litigates the thresholds every quarter.