Calibrating a Model Scorer Against a Human Judge

The sample is the experiment. Get it wrong and every number downstream is decoration.

Size. Two hundred to five hundred items is the working range. The lower bound gives you a usable point estimate but a wide interval; the upper bound tightens the interval enough to distinguish, say, kappa of 0.55 from kappa of 0.65. If you have three or more important strata (short items, long items, items with numbers, items in a second language), aim higher and sample within each stratum. Stratified sampling is how you avoid the classic trap of a sample that is 90 percent easy cases, which makes any scorer look competent.

Selection. Random, and documented. If you sample the most recent items only, you are testing a period, not the population. If you sample the items the model flagged, you have sampled on the outcome and cannot estimate agreement at all. Draw from the full population the scorer will see in production.

Blinding. The person labeling must not see the model's output, its confidence, or which items the model found hard. Show them the item, the rubric and nothing else. This is the single most common methodological error in scorer calibration, and it inflates agreement because a human who sees a suggested label anchors on it. In NLP annotation work, low inter-annotator agreement usually signals under-specified instructions or insufficient training rather than randomness, which is exactly why the human read has to be independent (inter-annotator agreement metrics).

Capture the human's certainty too. Ask the labeler for a confidence of high, medium or low on each item. When you later find the disagreements, you will want to know whether they cluster where the human hesitated.

Record every row in a table you can recompute from: item id, stratum, human label, model label, model confidence, human confidence. Store the rubric version alongside it. Rubrics drift, and a calibration result is only valid for the exact rubric and model configuration you tested.

The confusion table (also called the agreement matrix) is the whole analysis in miniature. Rows are the human's decisions, columns are the model's. Two labels give four cells, and each cell has a name that maps to a cost.

Take a worked example. You have a scorer for marketing email subject lines that decides "send" or "hold for rewrite," and you sampled 300 drafts. Here are the results:

Read the cells out loud before computing anything:

• 168 items both said send. This is the cell that makes the scorer useful.
• 24 items the human would send but the model held. These are false holds, and they cost reviewer time.
• 32 items the human would hold but the model sent. These are false sends, and they are the ones that reach the inbox.
• 76 items both said hold. The scorer is doing real filtering work here.

Already you know something a single accuracy figure hides: the scorer's errors are not symmetric. It holds good drafts more often than it sends bad ones, which may be fine or fatal depending on what a bad subject line costs you.

Raw agreement is the diagonal divided by the total. In the example, (168 + 76) / 300 = 0.813, or 81.3 percent. That number looks strong, and on its own it is misleading. Seventeen points of disagreement is 56 items out of 300, and the balance of those disagreements leans toward false sends.

The deeper problem with raw agreement is that it ignores the label mix. Suppose your corpus is 95 percent "send" because most drafts are fine. A scorer that always says "send" would score 95 percent agreement while doing nothing. Jacob Cohen introduced kappa in 1960 specifically because percent agreement cannot account for the agreement two raters would reach by guessing given their own label frequencies (McHugh on the kappa statistic). The formula is:

kappa = (Po - Pe) / (1 - Pe)

where Po is observed agreement and Pe is expected agreement by chance. Pe is computed from the margins: for each label, multiply the human's row total by the model's column total, sum those products, and divide by the square of the total count. This is the standard calculation, and a worked version of it on a 100-item table appears in Galileo's write-up of the metric.

Running the numbers on the subject line table:

• Po = (168 + 76) / 300 = 0.8133
• Pe = ((192 x 200) + (108 x 100)) / 300² = (38,400 + 10,800) / 90,000 = 0.5467
• kappa = (0.8133 - 0.5467) / (1 - 0.5467) = 0.2666 / 0.4533 = 0.588

That is a big drop, from 81 percent agreement to a chance-corrected 0.59. The scorer is clearly better than chance, but it is not yet in the range where most teams would let it replace a human read.

A code block you can run

import numpy as np
from sklearn.metrics import cohen_kappa_score

# rows: human label (1 = send, 0 = hold); cols: model label
human = np.array([1]*192 + [0]*108)
model = np.array([1]*168 + [0]*24 + [1]*32 + [0]*76)

po = (human == model).mean()
pe = sum((human == k).mean() * (model == k).mean() for k in (0, 1))
kappa = (po - pe) / (1 - pe)

print(f"raw agreement: {po:.4f}")
print(f"expected:      {pe:.4f}")
print(f"kappa:         {kappa:.4f}")
print(f"sklearn check: {cohen_kappa_score(human, model):.4f}")

The manual calculation and the library call should agree to four decimals. If they do not, you have a label encoding problem, not a scorer problem.

Reading the kappa value

Kappa runs from -1 to +1. Negative values mean the two raters disagree more than chance would predict, which usually means the rubric is inverted or misread. The conventional bands are 0.81 to 1.00 almost perfect, 0.61 to 0.80 substantial, 0.41 to 0.60 moderate, 0.21 to 0.40 fair and below 0.20 slight to poor (Galileo). McHugh argues Cohen's original bands are too lenient for consequential settings, where a value as low as 0.41 should not be treated as acceptable (McHugh). For a decision that changes what a customer sees, treat 0.6 as a floor and 0.7 as comfortable.

One caution before you fall in love with the number. Kappa is sensitive to the prevalence of each label. When one class dominates, kappa can look low even for a genuinely useful scorer, and the paradox has been documented well enough that some researchers argue for other coefficients altogether (the ResearchGate discussion of percent agreement versus kappa). Always report the confusion table next to the kappa. The table tells you what kind of error you have; the kappa tells you how much of the agreement is real.

A kappa of 0.588 from 300 items is an estimate, not a fact. Report an interval. The standard error of kappa can be computed directly; in scikit-learn and statsmodels you get it from the same call that produces the coefficient (Galileo's implementation notes). For the example above, the standard error lands around 0.05, so a 95 percent interval runs roughly 0.49 to 0.69. That interval straddles the 0.6 bar. The honest conclusion is "not yet demonstrated to clear the bar," not "passes."

Methodological guidance for agreement studies is consistent on this point: point estimates without uncertainty overstate precision, and reports should include confidence intervals, describe the rater design and account for label prevalence (inter-annotator agreement metrics). A cheap way to get an interval without a formula is the bootstrap: resample the 300 rows with replacement a thousand times, compute kappa each time, and take the 2.5th and 97.5th percentiles. It takes seconds and it handles the awkward cases where formulas assume more than your design provides.

Uncertainty also tells you how to spend your labeling budget. If the interval is too wide to make the call, more labels is the answer, and the width shrinks with the square root of the sample size. Doubling from 300 to 600 items tightens the interval by about 30 percent, which is often enough to separate a marginal scorer from a good one. Labeling is the expensive part of this work; sample size is where that expense buys certainty.

Agreement statistics do not make the decision. You do. The rule should be written down, in plain language, before you look at the production volume.

A rule that survives contact with reality has three zones. Take the model's confidence, calibrated on the same sample, and split on it:

Then add the hard overrides that ignore confidence entirely. If the model says "send" on an item that trips a blocklist, a length check or a legal term, a person looks regardless. Overrides are how you keep a confident but wrong scorer from doing damage. The structure is the same one email senders use when they set a hard ceiling on complaint rates rather than trusting an average: a bright line that applies no matter what the aggregate says (Yahoo sender best practices, Gmail's sender guidelines).

Two design notes. First, calibrate the confidence cut on data, not vibes: plot agreement by confidence decile, and set the high cut where agreement in that decile is at or above your bar. Second, keep the review zone wide at the start. A rule that sends 60 percent of items to review is still a 40 percent saving, and it generates the labeled data you need to widen the accept zone later.

The final step is the one teams skip. Your thresholds were fitted to the sample you looked at, so they will look better on that sample than they will in production. Draw a fresh sample, of the same size, from the same population, label it blind, and compute kappa again. If the fresh kappa holds within a few points, the rule is stable. If it drops ten points, you tuned to noise and need to widen the review band.

Then freeze the configuration: the rubric version, the model and prompt, the confidence cuts, the override list, the sample ids. Any change to the rubric or the scorer invalidates the calibration and requires a re-run. This is not bureaucracy. A rubric that was edited mid-study is no longer the rubric you measured, and a scorer whose prompt changed is a different scorer. Version everything in the same record.

Schedule the re-check on a cadence tied to change, not to the calendar alone. Re-run the calibration when the rubric changes, when the scorer changes, when the input distribution moves (a new product line, a new language, a new campaign style) and on a regular interval regardless. Keep the confusion table from each run so you can see drift rather than just detecting a sudden failure.

If you are running several scorers across different model providers, the plumbing gets tedious before the statistics do. Quanta Inference puts text, image, video, audio and embedding models behind one OpenAI-compatible API key with prepaid US dollar pricing, which means swapping scorers and re-running the same calibration harness does not require a new integration each time. That is a practical convenience, not a substitute for the work above: the sampling plan, the blind labels and the locked rule still decide whether you can trust the score.