Model Confidence Score vs. Your Own Classifier

It helps to read calibration the way the machine learning literature defines it. A model is confidence-calibrated if, for every confidence level c, it is correct c proportion of the time: among the samples where it says 0.7, roughly 70% should be right (Maja Pavlovic, arXiv). That is the formal test of honesty, and it is separate from ranking ability. A model can rank cases beautifully (high AUC) and still report numbers you cannot take at face value (Datanovia).

Self-reported confidence fails this test for a structural reason. Because it is generated text, it is anchored to surface fluency rather than correctness, so a model will say "95% confident" about a hallucinated citation in exactly the tone it uses for a verified fact (USQRD). Prompt wording moves it, and the high range (0.8 to 1.0) is usually compressed, which means the resolution you need for a gate is missing precisely where you need it.

Log probabilities fail it for a different reason. They are honest about the next token, but a fluency-boosting prompt or a retrieval-augmented setup sharpens the output distribution so the logprobs collapse toward confident-looking values regardless of whether the retrieved context supports the answer (USQRD). The model becomes fluent about the wrong thing, and the logprob stays high while the grounding rots underneath it.

A trained classifier measures the right target by construction. Its label is "the output was correct" or "the output was accepted," and its score is a fitted probability of that event. The catch is that it only knows what your labels taught it, so label noise and rare failure modes are its weak spots. But you can draw a reliability diagram on its held-out predictions from day one, which is more than you can say for a verbalized score.

Log probabilities arrive per token, so you have to reduce them to one number per output before any cutoff applies. The API returns the log probability of each output token, and summing those gives the joint log probability of the sequence, while averaging gives a per-token score that is easy to compare across outputs (OpenAI cookbook). Here is a minimal aggregation you can adapt:

import math

def seq_score(token_logprobs, mode="mean"):
    # token_logprobs: list of floats, each <= 0.0
    if mode == "mean":
        return math.exp(sum(token_logprobs) / len(token_logprobs))
    if mode == "joint":
        return math.exp(sum(token_logprobs))
    if mode == "min":  # worst-token: penalizes any low-confidence span
        return math.exp(min(token_logprobs))

The choice of mode matters more than people expect. Averaging is simple and robust but sensitive to outliers, while exponentiating first into linear probabilities gives a more intuitive scale at the cost of extra computation (Gautam Chutani). The minimum-token score is the strictest and often the best gate for extraction, because one low-confidence token in a critical field is exactly the failure you want to catch. Whichever you choose, fit and document it: the mapping from aggregation to calibrated probability changes with every prompt template and model version, so treat it as part of the model.

One practical guardrail: aggregate only the claim-bearing tokens, not boilerplate. A long polite preamble of highly predictable prose will pull the average toward confidence and mask a shaky answer.

Once you have a raw score, from any of the three sources, the job is to map it onto a probability that matches observed accuracy. Two methods cover most cases. Platt scaling fits a logistic curve to the score, which is a smooth correction that works well in small samples and cannot overfit badly. Isotonic regression fits a non-decreasing step function, which is more flexible but needs more data and can wobble when the score distribution is sparse (scikit-learn).

A tutorial on clinical prediction models makes the operational point clearly: you measure calibration on data the model never saw, using a reliability diagram and its companion summary metrics, then fit the correction on a separate held-out fold (PMC). Do not fit the correction on the same data you use to evaluate it, or you will report a curve that hugs the diagonal while production stays flat.

Two summary measures are worth reporting. Expected Calibration Error takes the weighted average gap between average accuracy and average confidence across bins, and Maximum Calibration Error takes the worst bin (Maja Pavlovic, arXiv). ECE has known drawbacks, including sensitivity to bin choice, so pair it with a visual reliability diagram: the diagram shows the shape of the miscalibration (over-confident or under-confident) while the number alone does not (MetricGate).

The Brier score is a useful third check because it decomposes into reliability, resolution and uncertainty, which means a lower Brier score can reflect better ranking rather than better calibration (scikit-learn). If your only goal is a gate, read the diagram and ECE in the region of the cutoff, not the whole curve.

from sklearn.calibration import CalibratedClassifierCV
from sklearn.isotonic import IsotonicRegression
from sklearn.linear_model import LogisticRegression
import numpy as np

# raw: held-out raw scores, y: correctness labels (0/1)
# Platt scaling: logistic curve on the raw score
platt = LogisticRegression().fit(raw.reshape(-1, 1), y)
cal_platt = platt.predict_proba(raw.reshape(-1, 1))[:, 1]

# Isotonic: non-decreasing step fit (more data needed, more flexible)
iso = IsotonicRegression(out_of_bounds="clip").fit(raw, y)
cal_iso = iso.predict(raw)

As the scikit-learn documentation puts it, calibration turns scores into probabilities you can interpret, and the library offers both estimators for exactly this purpose (scikit-learn). Pick Platt when your held-out set is small or the curve is roughly sigmoid, and isotonic when you have a few thousand labeled examples and the miscalibration is lumpy.

With a calibrated probability in hand, the cutoff is a business decision, not a default. The 0.5 threshold encodes the assumption that both kinds of error cost the same, which your problem almost certainly violates: a missed escalation costs more than an unnecessary review, or vice versa (Datanovia). The classification threshold maps a probability onto a binary action, and the right value depends on the relative cost of false positives and false negatives (Iguazio).

Here is a worked example that shows why calibration and threshold selection are separate knobs. Suppose 1,000 held-out outputs and your gate sends a case to human review when the calibrated probability of correctness drops below a cutoff. Careful: near the wrong tail, the numbers flip. Call the score your confidence and choose the cutoff for the action "auto-approve."

The model is the same in all three rows; only the cutoff moves. The $0.80 cutoff beats both neighbors once you price the two error types, and that crossover is the kind of number no default threshold will find for you. (The per-case and per-error figures are illustrative; substitute your own.) This is the cost-rule approach the calibration literature recommends, and the important discipline is to sweep the threshold on validation data and report final performance on a held-out test set (Datanovia).

Then audit it. Choose the threshold that maximizes Youden's J or minimizes expected cost only after you have seen the reliability diagram, because a single global threshold assumes the score is monotonically related to the true probability, and that assumption is worth checking rather than trusting (Valeriy Manokhin). Log the reported score, the calibrated score and the human verdict for every gated case. That log is what lets you re-fit the calibration and catch a cutoff that has quietly stopped matching the data.

Calibration is not a one-time fit. Prompt edits, retrieval changes and model updates all reshape the raw score distribution, so the mapping you fitted last quarter can quietly stop matching reality. The literature on LLM confidence is blunt that introspective signals are the least stable: external, verifiable signals such as agreement across several samples and entailment checks against retrieved spans beat self-reported confidence, and abstention thresholds should be refit empirically on your own reliability diagram rather than guessed (USQRD).

A practical monitoring loop has three checks. First, recompute ECE and the reliability diagram monthly on the gated cases whose verdicts you have, and alert when ECE climbs past a threshold you set from your baseline. Second, watch the approve rate and the review rate: a sudden jump in auto-approval usually means the raw score distribution shifted. Third, sample a small fraction of auto-approved cases for human audit, because the dangerous errors are precisely the ones the gate waved through.

If you run models from more than one provider, the plumbing gets annoying fast. Quanta Inference puts text, image, video, audio and embedding models behind one OpenAI-compatible key with prepaid US-dollar pricing, which keeps the logprob and scoring pipeline consistent across providers (https://quantaintelligence.ai/inference). For teams that need a second pair of eyes on the calibration and threshold design, a free 30-minute discovery call is available through Quanta Consulting (https://quantaintelligence.ai/calendar).

The rule of thumb that ties this together: prefer log probabilities or a trained classifier for any cutoff that gates an action, calibrate whichever you choose on held-out data, and treat a self-reported confidence score as a hint you can log but never as the number that decides.