A Confidence Score the Reviewer Can Interpret

Two models can have identical accuracy and report completely different confidence, and one of them will be actively misleading. Accuracy asks "how often is the model right overall?" Calibration asks "when the model says 80%, is it right 80% of the time?" These are independent properties, and a reliability diagram exposes the systematic over- or under-confidence that ranking metrics like ROC AUC cannot see the MetricGate reliability diagram guide.

The Brier score is worth calling out because it looks like a calibration measure and is often used as one. It is a strictly proper scoring rule that combines reliability, resolution and the inherent randomness of the data, so a lower Brier score does not necessarily mean better calibration the scikit-learn calibration guide. That is why the bin table, not a single number, is the artifact you put in front of reviewers.

The cleanest framing for a practitioner: discrimination is the model's ability to sort hard cases from easy ones; calibration is the model's ability to be honest about the sort. You want both, and you can only inspect the second one by grouping predictions and comparing scores to outcomes.

The procedure is short enough to write on a card. It takes an afternoon the first time and minutes after that.

1. Freeze a labeled evaluation set. Pull tickets that have a verified resolution label (the queue that handled them, the intent an agent confirmed, the extracted fields a human corrected). Keep it separate from anything used to build prompts or train anything. A few hundred per confidence range is workable; a few thousand across the set is better.
2. Run the current model on it, unchanged. Same prompt, same parameters, same post-processing. Record the confidence score the model actually produces in production, not a re-derived one.
3. Define correctness. For classification this is exact match of the chosen label to the verified label. For multi-field extraction, decide in advance whether "correct" means all fields correct or each field scored separately. Write the rule down; changing it later invalidates the numbers.
4. Bin the cases by confidence. Ten equal-width bins of 0.1 is the common default and the right starting point: 0.0 to 0.1, 0.1 to 0.2, and so on the Arize calibration overview. Put a 1.0 in the top bin.
5. Count and compute per bin. For each bin, record how many tickets landed in it, the mean confidence, and the share the model got wrong. That share is the number the reviewer needs.
6. Widen any bin with too few examples. A bin with nine tickets produces an error rate with a huge margin of error. Merge it with its neighbor and say so.
7. Publish the table and wire the lookup into the note. The same script that produces the table writes the annotation.

Here is a compact Python sketch of steps 4 through 6. It assumes an array of scores and a parallel array of booleans for whether each case was correct.

import numpy as np

def reliability_table(scores, correct, width=0.1):
    rows = []
    edges = np.arange(0.0, 1.0 + width, width)
    for lo, hi in zip(edges[:-1], edges[1:]):
        upper_ok = (scores <= hi) if hi >= 1.0 else (scores < hi)
        mask = (scores >= lo) & upper_ok
        n = int(mask.sum())
        if n == 0:
            rows.append((lo, hi, 0, None, None))
            continue
        mean_score = float(np.mean(scores[mask]))
        error_rate = float(np.mean(~correct[mask]))
        rows.append((lo, hi, n, mean_score, error_rate))
    return rows

Convert error rates to "wrong per 100" only after the bin has enough tickets to justify a whole number. Rounding 0.174 to "about 17" is honest; rounding a nine-ticket bin to "about 22" is not.

A filled-in table looks like this. The numbers are illustrative but shaped like a real run: the model is conservative in the middle and overconfident at the top.

Read the table as a risk map, not a grade. The shape matters more than any single cell. Here the low bins are wildly pessimistic (92 wrong in 100 at a mean score of 0.06), the middle is roughly honest, and the top bin is excellent. A reviewer working from this table learns three things: low scores are near-useless, mid scores are coin-flip-ish with a bias toward correct, and 0.9 plus is genuinely safe.

Two patterns you will hit and how to interpret them:

• Everything is overconfident. The 0.8 bin is wrong 40 times per 100. The model is reassuring but not calibrated. Either recalibrate the scores or abandon scores as a gate and route on other criteria.
• The table is flat. Every bin is wrong about a third of the time. The score carries no information at all. That is not a calibration problem, it is a signal problem, and you should stop showing the score.

The binning scheme itself affects the picture. Equal-width bins are easy to interpret but can leave the extremes sparse; equal-frequency (quantile) bins guarantee each bin has cases but produce bins with odd edges the MetricGate reliability diagram guide. Use equal-width for communicating to humans, because "the 0.8 bin" is a phrase a reviewer can hold in their head. Use equal-frequency when you are diagnosing the model with statisticians.

One more caution: binning is unstable when data is scarce, because each bin's error rate is estimated from a small sample. A paper on stable reliability diagrams argues that binned estimates can wobble and proposes smoother estimators for that reason stable reliability diagrams. For a reviewer-facing note the approximation is fine as long as you show sample counts and say how old the data is.

The default of ten bins is a convention, not a law. What matters is that each bin contains enough labeled cases for its error rate to mean something. Here is a decision rule you can apply mechanically.

Width also depends on where decisions happen. If your automation threshold sits at 0.85, you want a narrow band around 0.85, for example 0.8 to 0.85 and 0.85 to 0.9, because that is the part of the curve the gate actually reads. Precision where the knife edge is, coarser bins elsewhere.

A worked sizing example: suppose you have 2,000 labeled tickets and expect most scores to land between 0.6 and 1.0. Ten equal-width bins gives 800 in the middle region and almost nothing below 0.2. In that case, drop to five bins (0.0 to 0.2, 0.2 to 0.4, and so on) or switch to quantile bins, and spend your labeling budget on tickets near the threshold rather than on tickets the model already handles confidently or refuses entirely.

There is a second sizing problem that gets missed: the labels must be right. A calibration run on tickets labeled by a queue assignment is measuring agreement with the router, not with the truth. If your ground truth is "which team an agent eventually sent it to," say so on the note. Reviewers can handle an imperfect reference; they cannot handle an unstated one. The same applies when you use a trust score on structured output: the score tells you which fields warrant human review, and the measured error rate tells the reviewer how much a given score should worry them Cleanlab's structured output benchmark.

The annotation should replace a vague number with a measured statement. Three formats, in order of usefulness:

• Plain statement. "Confidence 0.84. On 200 labeled tickets that scored between 0.8 and 0.9, the suggestion was wrong 17 times (about 17 per 100). Check the label before routing."
• Threshold-aware statement. "Confidence 0.84. Recommendations at this level were wrong about 17 per 100 in the last calibration run. Review anything above 0.9 and anything below 0.5 carefully; the middle range has the least evidence."
• Action statement. "Confidence 0.84 (wrong on about 17 of 100 in this range). Verify the intent and add a short note if you change it."

Keep the sentence short and put it where the reviewer already looks. The handoff literature is blunt about this: the failure is almost never the transfer, it is that the sender's working knowledge was never written down, and a note written while the context is live beats one reconstructed later the CaseMargin handoff guide. A tired agent at the end of a shift writes the same thin note as everyone else, which is an argument for a fixed template and for letting software draft the first version the eesel shift handoff guide. Apply the same logic to your annotation: generate it automatically so no tired human has to remember the calibration table.

Do not put the raw score alone on the note if you have not run the calibration. A bare 0.84 invites overtrust. Research on human oversight keeps making the same point: the gate belongs where a mistake is expensive and hard to undo, and the reviewer's attention is the scarce resource the human in the loop guide. Give that attention a real number to spend.

Calibration drifts. New products, new support macros, seasonal volume, and a changed prompt all move the mapping between score and error rate. Three habits keep the table current:

• Re-run on a fixed cadence and on every model or prompt change. Store the table with the date, the sample size and the labeling rule. A note on a ticket should be able to say which run produced the number.
• Watch the top bin hardest. If the 0.9 to 1.0 bin's error rate moves from 6 to 15 per 100, your automation threshold just became unsafe. That is the alert worth paging on.
• Track the decision, not just the model. If reviewers change the label on 30 percent of tickets that scored above 0.9, the table is wrong or the reference labels are. Either way, the note is misleading until you find out which.

Decide in advance what the reviewer is expected to do with the number, then check whether they did it. The distinction between the three oversight postures matters here: in a human-in-the-loop design the AI proposes and a person disposes; on-the-loop means the AI acts and a person monitors; fully autonomous means no human at the decision point the human in the loop guide. Calibration is what makes the first posture tractable, because it tells the reviewer how much to check and when to stop checking.

If you want the annotation and the table to run as part of a build rather than a spreadsheet, describe the workflow and let a specialist set it up with you at Quanta. The calibration run itself is a script; the product is the reviewer experience around it.