Calibrating a Reason-Check Classifier Against Human Labels

Start by defining the decision precisely. A reason check usually answers a narrow question: does this draft contain an unsupported claim, a policy violation, a missing escalation disclaimer, a factual contradiction with the retrieved source? Write the definition as a rule a reviewer can apply consistently. Vague criteria produce noisy labels, and noisy labels make every metric downstream meaningless.

Then sample. Use a uniform random sample from the population the classifier will actually see, not a convenience sample of cases you already suspect are wrong. A stratified sample is fine, and often better: draw from each reason category and confidence band in proportion, then weight back to the real mix when you compute population estimates. Record the sampling frame and the weights so the numbers can be reproduced.

For size, you need enough positive cases to estimate a rate. If roughly one in ten cases is a true violation, 200 labeled cases gives you about 20 positives, and the confidence interval on a rate estimated from 20 events is wide. As a working rule, label until you have at least 30 to 50 cases in the smaller class, and more if the decision is high stakes. Labeling 300 to 500 cases with two reviewers is a day or two of work and buys you a defensible threshold.

Blind the review. Reviewers should not see the classifier's verdict, and where possible should not see each other's labels, so that agreement between them is a real check on the criteria rather than a social one. Have a third person adjudicate ties and record the adjudicated label as the reference.

The output of this step is a table with one row per case: case ID, sampled stratum, human label, classifier label, classifier confidence score, and reason category.

Raw agreement is the share of cases where the classifier and the human gave the same verdict. It is easy to explain and it is the number stakeholders ask for, but it is misleading in two ways.

First, it ignores chance. If 90 percent of cases are "no violation", a classifier that always says "no violation" scores 90 percent agreement while being useless. Cohen's kappa corrects for exactly this: it compares observed agreement to the agreement you would expect from the two raters' category frequencies alone (the kappa statistic explained). Kappa runs from about -1 to 1, with 0 meaning chance-level agreement, and it is the standard measure of interrater reliability for categorical labels (Cohen's Kappa tutorial).

Second, it hides direction. A classifier that says "violation" on everything and one that says "clean" on everything can have similar raw agreement while failing in opposite, differently expensive ways.

The formula, with p_observed as raw agreement and p_expected as chance agreement from the marginals:

kappa = (p_observed - p_expected) / (1 - p_expected)

A third metric belongs in every calibration report: recall on the positive class (the share of true violations the classifier catches) and precision (the share of its violation flags that are real). Kappa tells you whether the two raters behave like one another. Precision and recall tell you what the errors cost.

The interpretation of kappa is contested. Cohen's original bands treat 0.41 as moderate, which many clinical researchers regard as too lenient for consequential decisions (McHugh on kappa limitations). For a reason check that gates a customer-facing response, treat anything below roughly 0.6 as "not ready", 0.6 to 0.8 as "usable with human review on the low-confidence band", and above 0.8 as "trustworthy enough to act on, subject to drift checks". Those cutoffs are a decision rule, not a law, and you should write down the one you chose and why.

Here is a calibration run on 400 sampled cases, with the human label as reference. The classifier's confidence is a score from 0 to 1 for "violation"; threshold set at 0.5.

Raw agreement is (52 + 302) / 400 = 354 / 400 = 0.885, or 88.5 percent. That number will make almost anyone happy. Now correct for chance.

Expected agreement: the human flagged 80/400 = 0.20 as violations and 320/400 = 0.80 as clean. The classifier flagged 70/400 = 0.175 and 330/400 = 0.825. So p_expected = (0.20 × 0.175) + (0.80 × 0.825) = 0.035 + 0.66 = 0.695.

Kappa = (0.885 - 0.695) / (1 - 0.695) = 0.190 / 0.305 = 0.623.

The picture changes. Precision on the violation class is 52 / 70 = 74.3 percent. Recall is 52 / 80 = 65 percent. So one in four flags is a false alarm and one in three real violations slips through. Kappa of 0.62 is in the "usable with human review" band, not the "act on it" band.

Now ask where the errors live. Suppose the 46 disagreements break down as follows:

The aggregate score hides the real problem. The classifier is fine on tone and unsupported claims and poor on the escalation disclaimer, which is exactly the category where a regulatory requirement may attach, since consumer-facing chatbot rules increasingly require clear disclosure and a working path to a human (AB 1609 analysis). The fix is not a global threshold change. It is a prompt and rubric fix for that one category, which you then re-test on the same labeled set.

Category is one axis. Confidence is the other, and it is where the threshold decision lives. Bucket the labeled cases by the classifier's confidence score and compute precision and recall in each bucket.

Read this as a routing table, not a score. Above 0.9 the classifier is right 96 percent of the time and can auto-decide. Between 0.5 and 0.9 the error rate is too high for automation but the classifier's verdict is still useful as a prior for the human reviewer: it cuts review time even when it does not replace review. Below 0.5 you learn almost nothing.

This banded approach is the same logic that production support systems use, where intent-specific confidence thresholds and decay tracking decide whether to answer, clarify, or escalate, and where accuracy degrades without drift controls (chatbot best practices). The engineering difference is that you have measured the bands rather than guessed them.

With the band table in hand, pick the threshold by cost, not by round numbers. Assign a cost to a false negative (a violation the classifier missed) and a false positive (a clean case flagged and sent for review). Then, for each candidate threshold, compute expected cost across your labeled sample. The threshold with the lowest expected cost is your operating point.

A worked version: suppose a missed violation costs 10 units of risk and a false alarm costs 1 unit of reviewer time. At threshold 0.5, the 28 false negatives cost 280 and the 18 false positives cost 18, for 298. If you raise the threshold, recall rises but false alarms multiply. Plot the two cost curves against the threshold on the labeled set and read off the minimum. That is your starting threshold.

Then decide what happens in each band. A defensible design is three-way:

• High confidence: classifier verdict stands, with a 1 in 20 random audit to catch drift.
• Middle band: classifier verdict is shown to a human as a suggested answer, not a decision.
• Low band: classifier verdict suppressed, case goes to human review with no prior.

Write this policy down next to the kappa and band numbers it was derived from. When someone asks why the classifier was trusted, the answer is a table, not an opinion.

If your reason checks span several model families or you are comparing judge models, routing every call through one API keeps the calibration work the same across candidates. Quanta Inference exposes text and other model types behind one OpenAI-compatible key, which means changing the judge model is a config change rather than an integration project.

Two things will break a calibration that was fine when you ran it.

Imbalance is the first. Kappa is sensitive to skewed marginals, and severe imbalance can depress kappa even when the classifier is behaving reasonably. If the true violation rate is 2 percent rather than 20 percent, raw agreement will look excellent and kappa will bounce around. The remedy is to report precision and recall alongside kappa, to stratify the sample so the positive class is well represented, and to weight results back to the population mix before you quote them. Never quote a kappa computed on a rebalanced sample as if it came from the live population.

Drift is the second, and it is silent. The inputs change (new products, new policies, new phrasing from customers), the judge model changes, or the context the classifier reads changes. Any of these moves agreement without any alarm. The detection method is mechanical: keep the labeled set as a regression suite, re-run it on every prompt or model change, and run a small fresh-label audit on live traffic at a fixed cadence. Compare the fresh audit's kappa to the baseline and set a trigger, for example "investigate if kappa drops by more than 0.1 or precision on any category drops below 0.8".

A reviewer agreement check belongs in the loop too. If two humans agree with each other only 80 percent of the time, no classifier can exceed that ceiling in any meaningful sense, and you should fix the labeling rubric before tuning the model.

The same measurement discipline applies to agentic systems more broadly, where trajectory-level failures and tool errors escape output-only scoring and require repeated evaluation across the lifecycle (agent evaluation metrics; agent evaluation frameworks). A reason check is a smaller surface, but the rule holds: score against human labels, look at where the errors are, and re-score after every change.