Calibrating a Refusal Threshold from a Labeled Sample

You need rows where you know the correct value independently of the model: a human-verified gold set, an upstream system of record, or a manual audit. Two properties matter more than raw size.

First, the sample must contain a realistic mix of rows that need edits and rows that do not. If 98% of your sample needs no change, your threshold curve will be dominated by easy negatives, and the interesting region (rows near the boundary) will have too few examples. This is the same imbalance problem that motivates stratified sampling: select a representative sample from each stratum, such as document type, confidence band, or field type, rather than sampling uniformly the human review and calibration guide.

Second, the sample must be segmented. A single aggregate accuracy number can hide catastrophic failure on one segment: one documented case shows a 97% aggregate figure sitting on top of 60% accuracy for handwritten receipts and 45% for international formats, because the easy document type dominated volume the human review and calibration guide. If your edit system handles several row types, calibrate per type or accept that one threshold is a compromise.

Before you trust any score, check whether it is calibrated at all. A reported 0.90 might be correct 94% of the time on one field and 82% on another, and LLM self-reported confidence tends to be top-skewed, which creates a confidence floor you must account for when setting thresholds the LLM grader calibration paper. In one comparison across seven models and three datasets, self-reported confidence was the best-calibrated method, with average expected calibration error of 0.166 versus 0.229 for self-consistency voting, despite the latter costing 5x the inference the LLM grader calibration paper. The practical lesson: measure the reliability curve of your own scores on labeled rows before you assume a score of 0.8 means 80% correct.

If your pipeline extracts structured fields, make sure the score you calibrate is the one that gates the write, and treat the schema as fixed while you calibrate. Changing field names, meanings, or types mid-calibration invalidates the labels; schema evolution rules exist precisely because a renamed or re-typed field breaks consumers silently the schema evolution guide.

If you pick the threshold that minimizes cost on the same rows you used to draw the curve, you will overstate performance. The curve is noisy, and the minimum on a noisy curve tends to be a lucky dip. Split first:

Rules that keep the split honest:

• Assign rows randomly, but stratify by segment and by whether an edit was needed, so both sets resemble production.
• Keep rows from the same source document or the same account in one set only, otherwise near-duplicates leak information across the split.
• Never re-tune on the validation set after seeing its cost. If you do, it becomes a second calibration set and you need a third holdout.
• Freeze the sample and the labeling rules before you start. If a labeler decides a value was correct that the model had flagged as wrong, record that decision, do not quietly drop the row.

A practical guardrail: keep the validation set large enough that a cost difference between two candidate thresholds is visible. If your validation set has 200 rows and 10 of them need edits, the cost estimate swings wildly with one or two flips. Confidence intervals around accuracy estimates help communicate exactly this kind of uncertainty, and methods for constructing them for classifier performance are standard the confidence intervals guide.

Because you only get one honest look at the validation set, keep a record of the threshold and cost assumptions you tested. If you later add rows, append them to the validation set rather than reusing the calibration rows, and note that the estimate now covers a different set of rows.

For each candidate threshold t, walk the calibration rows and classify each one into one of four cells. Only two cells carry cost:

Then total cost = missed_edits * cost_miss + wrong_edits * cost_wrong. Sweep t over the observed score values (or a fixed grid of 0.01 steps), compute total cost at each, and pick the minimum. Note that a row with no edit needed but where the system refused is free, which is why a high refusal rate is not automatically bad: refusal is the safe default.

Here is the sweep in plain Python, kept short enough to read:

rows = [(score, needs_edit, applied_matches_gold) for score, needs_edit, applied_matches_gold in calibration]

def cost_at(t, cost_miss, cost_wrong):
    missed = wrong = 0
    for score, needs_edit, matches in rows:
        applied = score >= t
        if applied and not matches:
            wrong += 1
        elif not applied and needs_edit:
            missed += 1
    return missed * cost_miss + wrong * cost_wrong

grid = sorted({s for s, _, _ in rows})
best = min(grid, key=lambda t: cost_at(t, 1.0, 5.0))

Two implementation details that change results. First, ties: define whether a score exactly equal to t applies or refuses, and apply that rule in both calibration and production. Second, the "applied but no edit was needed" case: if the system applies an edit to a row that needed none, count it as a wrong edit, not a free action, because it changed correct data. When matching applied values to gold, decide how strict the comparison is (exact string, normalized whitespace, numeric tolerance) and fix that rule before the sweep.

The cost ratio is the lever. With cost_wrong at 5x cost_miss, the optimum moves toward refusing borderline rows. With cost_wrong equal to cost_miss, the sweep lands near the minimum-error point. This mirrors standard threshold analysis, where the operating point should reflect the cost of false positives versus false negatives rather than a blanket 0.5 the Evidently threshold guide. Precision-recall curves show the same trade-off across thresholds and help you see where a small change in cutoff sharply reduces precision for only a small gain in recall the scikit-learn precision-recall example. Precision is the share of applied edits that were correct, and recall is the share of needed edits that were applied, so plotting both against the cutoff makes the two error types visible as you sweep the scikit-learn precision-recall example.

One caution on curve-based shortcuts: picking the threshold that maximizes a summary statistic like Youden's J or minimizes error rate assumes equal costs and a single well-behaved boundary, and it can be suboptimal when the score is not calibrated or the decision needs more than one cut point the ROC threshold critique. Since you have labeled rows and stated costs, compute the cost directly.

Suppose the calibration set has 1,000 rows. 300 need an edit and 700 do not. You assign cost_miss = 1 and cost_wrong = 5, on the reasoning that a corrupted value costs about five times a stale one.

At three candidate thresholds, the confusion counts come out like this:

The naive instinct (catch every possible edit, threshold low) produces the worst cost because wrong edits are five times more expensive. The sweep picks 0.90 if these are the only candidates, but the true minimum may sit between 0.75 and 0.90, so run the grid rather than three hand-picked points. On the validation set, the chosen threshold costs, say, 220 against a calibration cost of 210. That gap is your estimate of optimism, and it is the number you report, not the calibration figure.

Now shift the ratio. If cost_wrong stays 1 and cost_miss rises to 4 (say, a missing edit triggers a compliance failure), recompute: 0.55 gives 160 + 90 = 250; 0.75 gives 340 + 30 = 370; 0.90 gives 680 + 8 = 688. The optimum flips to the low threshold. Nothing about the model changed. Only the cost statement did.

Sensitivity check worth running: recompute the optimum at cost ratios one step either side of your point estimate (for 5, try 3 and 8). If the chosen threshold does not move, your decision is robust. If it jumps by 0.15 or more, you need more labeled rows or a sharper cost estimate before shipping. The same discipline of measuring per segment applies when you deploy: track accuracy by segment and by confidence band, including high-confidence rows nobody reviews, because a systematic error on a high-confidence segment is invisible without sampling the human review and calibration guide.

When the model itself varies (multiple providers, model swaps, or a gateway across several APIs), keep the calibration pipeline model-agnostic so you can rerun it per model. An OpenAI-compatible layer over many model types lets you point the same sweep at each candidate and compare cost curves on identical rows; Quanta Inference is one such option at https://quantaintelligence.ai/inference.

The decision rule, in order:

1. State the cost ratio in writing, with a reason. "A wrong edit costs about 5x a missed edit because correcting bad data downstream takes a manual audit" is a decision others can challenge.
2. Sweep on the calibration set, record the full curve, not just the minimum.
3. Evaluate once on validation. If validation cost is much worse than calibration cost, your sample is too small or not representative; collect more rows.
4. Run the sensitivity check at adjacent cost ratios.
5. Ship the threshold with its cost assumptions, its validation cost, and the validation set size recorded alongside it.

Then monitor. Recalibrate when the mix of input rows shifts, when the underlying model changes, or when a new segment appears. Set a review cadence and a trigger, for example a monthly stratified sample of applied edits. If you route low-confidence rows to humans, order the queue by uncertainty, not arrival time, and reserve reviewer capacity for low-confidence and historically weak segments rather than spreading it evenly the human review and calibration guide.

A note on what not to do: do not choose the threshold by eyeballing the score histogram. Scores are frequently miscalibrated, and a visible cluster at 0.9 does not mean those rows are 90% correct the LLM grader calibration paper. The labeled sample is the only thing that ties a score to an error rate.

If the labels need to be refreshed but the model output is already flowing, keep the two apart during audit: sample production rows, have a person record the correct value, and add the pair to the next validation batch without retraining the model on those rows.

You do not need a huge sample, but you need enough of the rare, costly events. Two rules of thumb carry most of the weight:

• Count wrong edits, not total rows. If your validation set at the candidate threshold produces fewer than about 30 wrong edits, the cost estimate will be too noisy to distinguish thresholds. Size the sample so the expected wrong-edit count at your operating point clears that bar, then add rows.
• Split unequally when errors are rare. If only 5% of rows need edits, put 70 to 80% of rows in calibration and keep the validation set as large as you can afford, because recall on those rare rows is what moves cost.

Quick sizing table for a system where 10% of rows need edits and you expect roughly half of applied edits near the cutoff to be correct:

These are planning figures, not guarantees: your actual rates depend on the model and the data. The point is that the binding constraint is the count of errors, not the count of rows, and confidence intervals on the error rate make the uncertainty explicit when you present the result the confidence intervals guide.

Two more sizing traps. If a segment has fewer than a few hundred rows, the sweep for that segment will be too noisy to trust, so borrow a global threshold and monitor it rather than calibrating per segment. And if your labeled rows were collected from a preview of the first few hundred rows of a batch, they may not resemble later production traffic; resample after a few weeks and rerun the split.

Finally, decide in advance what you will do with rows in the ambiguous band just below the threshold. Options include queuing them for review, holding the old value and flagging the row, or attempting a cheaper second pass. This is the same selective-automation pattern used in grading and extraction systems: automate above a calibrated threshold, review below it, and sample the automated band to catch novel errors the human review and calibration guide.