When to Audit the Human Judge: Checking the Reference Standard

You cannot measure agreement on a label that two people would define differently. Before drawing any sample, write down three things in one short document:

1. The construct. The thing you actually care about, in one sentence, in plain language.
2. The decision rule. The observable test that turns an item into a label. "Does this reply answer the customer's question?" is a construct. "The reply contains a direct answer to the question asked, even if it also contains caveats" is a decision rule.
3. The edge cases. At least five worked examples from real items, with the label you expect and the reasoning.

This is the same discipline that good rubrics impose in education: clear criteria, described levels of performance, and examples so that two graders reach the same verdict (NC State rubric best practices). The education literature is useful here precisely because grading essays and grading model outputs have the same failure mode: a rubric that is clear in the author's head and ambiguous on the page.

The single most common cause of low agreement is not reviewer carelessness. It is an edge case the codebook never resolved, so each reviewer resolves it privately. Writing the decision rule first converts that hidden disagreement into something you can see and fix.

Sampling the first two hundred rows is the fastest way to learn nothing, because early rows rarely resemble the whole set. Sample at random within strata you already know differ.

Stratify on at least three axes where they exist:

The size question has no universal answer, but a useful default is enough items per stratum that a single disagreement does not swing the statistic by more than a few points. For a class with a 20 percent prevalence, 40 to 60 sampled items per stratum will usually separate "this label is solid" from "this label is a coin flip." If the minority stratum is tiny, oversample it deliberately and record that you did.

Two rules make the sample honest. First, the second reviewer must be blind to the first score and to any model output. Second, the sampler must not be the person who produced the labels, or you will get a sample of the items the first reviewer was confident about.

Two independent reviews of the same items produce a confusion matrix. Percent agreement is the first thing to look at and the first thing to distrust. If 90 percent of your items are labeled "fine," two reviewers who always say "fine" agree 90 percent of the time while telling you nothing.

Cohen's kappa corrects for that. It compares observed agreement against the agreement you would expect from the reviewers' own label frequencies, ranging from about -1 (active disagreement) through 0 (chance) to 1 (perfect) (McHugh, interrater reliability and the kappa statistic). The formula is (observed agreement minus expected agreement) divided by (1 minus expected agreement) (Galileo on Cohen's kappa for AI evaluation).

A worked example, using the standard 100-item two-rater layout from that same source. Suppose reviewer one labeled 50 items "pass" and reviewer two labeled 55. They agree on 40 passes and 35 fails.

                  Reviewer B: pass   Reviewer B: fail   Row total
Reviewer A: pass          40                10              50
Reviewer A: fail          15                35              50
Column total              55                45             100

Observed agreement Po = (40 + 35) / 100 = 0.75
Expected agreement Pe = (50*55 + 50*45) / 10000 = 0.50
Kappa = (0.75 - 0.50) / (1 - 0.50) = 0.50

A kappa of 0.50 is moderate. That is a warning, not a disaster: three quarters of labels agree, but the reference is not yet reliable enough to anchor a threshold on. Note the two off-diagonal cells. Fifteen items reviewer A passed were failed by B, and ten the other way. That asymmetry is where the diagnosis lives.

Computing it takes a few lines:

from sklearn.metrics import cohen_kappa_score, confusion_matrix

kappa = cohen_kappa_score(first_scores, second_scores)
print(f"kappa = {kappa:.3f}")
print(confusion_matrix(first_scores, second_scores, labels=["pass", "fail"]))

For more than two reviewers use Fleiss' kappa; for ordinal scales where a near miss is less bad than an opposite answer, use a weighted variant rather than the unweighted coefficient (numiqo on Cohen's kappa). And report a confidence interval, because a point estimate on 80 items hides how uncertain it is; the NLP annotation literature makes this a standard recommendation alongside a description of the rater design and the label distribution (James on selecting an inter-annotator agreement metric).

Kappa gives you a number. The confusion matrix gives you the reason. After computing kappa, walk the off-diagonal cells in order of size and read ten disagreements each. They almost always fall into four patterns.

The distinction matters because the remedy differs. Threshold and rater drift are fixed by retraining and recalibration, which is cheap. A definitional split means the codebook is wrong and some fraction of the labels must be redone, which is more expensive but still cheaper than discovering it after calibration.

There is a subtler point worth carrying into the design. Disagreement is not always noise to be stamped out. Some of it reflects legitimate ambiguity in the task, and collapsing it can throw away real information about where you are unsure (James on inter-annotator agreement). If two careful reviewers split evenly on a category, that category probably should be several categories or an explicit "unclear" bucket rather than a single forced choice.

Low agreement is information, not failure. The three responses, in order of preference:

Fix the codebook and re-audit. Rewrite the decision rule using the ten disagreements you just read, add them as worked examples, retrain the reviewer, and re-sample a fresh set of items. If kappa climbs, the label is savable. This is usually the right first move because the cost is a morning of writing.

Narrow the label. If the ambiguity is concentrated, split the class or add a third "unclear" option. A three-way label with clean agreement often beats a forced binary with muddy agreement, because you can then handle "unclear" explicitly downstream instead of laundering it into a false certainty.

Drop the label. If repeated audits fail to push agreement up, retire it. It cannot set a threshold or anchor a calibration curve. Use a proxy that does agree across reviewers, such as a downstream behavioral outcome, or accept a weaker claim about the system.

A practical decision rule: set your kappa floor before you look at the result, so you are not tempted to rationalize a number you already have. A common convention in applied AI evaluation is to treat anything below the "substantial" band as a signal to fix the codebook before trusting the labels, and even that convention is contested as too lenient for high-stakes contexts (McHugh, KNIME on kappa and common pitfalls). The point is not the magic number. It is that you agreed to it in advance.

Weight your efforts by class. A label with 95 percent prevalence can tolerate moderate kappa without shifting your threshold much, while the 5 percent minority class is where every threshold decision actually lives. Kappa's weakness is that it reports one number for a whole matrix, and it is easily dominated by the majority class, a caveat the KNIME guide spells out with a worked imbalanced example in which high accuracy masked poor minority-class performance (KNIME). Compute agreement per class and treat the minority class as the real result.

An audit is not a one-off. Labels drift as inputs change, as reviewers turn over and as the distribution of items shifts. The practical cadence:

Log the kappa and the confusion matrix alongside the model version so that a later drop in model performance can be traced to either the model or its reference. When it is the reference, that changes the fix entirely, and you will only know if you stored the audit.

When the audit is done and the reference is trusted, the next bottleneck is usually the surrounding work: running the same eval against several models, keeping results comparable and turning the output into something a business owner can act on. That is where a tool such as Quanta Inference, which puts text, image, video, audio and embedding models behind one OpenAI-compatible API key, removes some of the plumbing so the audit results can be checked across providers without rebuilding the harness each time.