Setting a Minimum Agreement Rate Before an LLM Judge Gates

Raw agreement is the fraction of cases where the judge and the human reference pick the same label. It is easy to compute, easy to explain to a release committee, and easy to game by a judge that always predicts the majority class. A comparison of percent agreement and kappa in LLM evaluation puts it directly: percent agreement counts matches but ignores the agreement that would happen by chance, which matters most when one label dominates the dataset.

Cohen's kappa corrects for that. It compares observed agreement with the agreement expected from each rater's label distribution, and the standard formula is kappa = (Po - Pe) / (1 - Pe), where Po is observed agreement and Pe is chance agreement. McHugh's overview of the kappa statistic explains why Cohen introduced it in 1960 as a critique of percent agreement, and notes the statistic ranges from -1 to +1. A walkthrough of kappa with a worked matrix gives the arithmetic: with 100 items, 40 agreements on "yes", 35 on "no", and 25 disagreements, observed agreement is 0.75, expected agreement is 0.50, and kappa is 0.50, which sits in the moderate band.

The worked example in the AWS grounded-evals guide shows why this matters for a gate: if 80 percent of responses are correct and the judge always says "correct," it scores 80 percent accuracy and a kappa of zero. It has learned nothing about the failures the gate exists to catch. Report both numbers, then gate on the stricter of the two. An LLM judge that correlates strongly with humans can still be systematically lenient, which is exactly the bias a correlation coefficient will not reveal; the Judge's Verdict study makes this argument and moves from correlation analysis to kappa precisely to catch systematic bias.

Two practical cautions. Kappa is affected by prevalence, so compute it per criterion rather than pooling criteria with very different base rates. And sample size matters: tightening a confidence interval around kappa gets expensive fast, and the calculation sizes on whichever bound lies nearer the expected value, so a target of 0.60 with a lower bound of 0.55 needs 1,625 cases regardless of how wide the upper bound is. Worked guidance from Bujang and Baharum on minimum sample sizes for kappa shows the same shape: when the two raters' marginal frequencies differ, the required sample is often more than double the balanced case. In practice, an evaluation set of a few hundred labeled cases per criterion is enough to detect a gross miss, and a deliberate doubling gives you margin for skewed label distributions.

Suppose a support-answer assistant is being gated on three criteria: groundedness (every policy claim is supported by the supplied policy text), policy compliance (the response does not promise an outcome the policy forbids), and actionability (the response names a next step available under the policy). Two human labelers annotate 300 held-out cases, disagreements are adjudicated to a reference label, and the judge scores the same 300 cases blind to the human labels. The table below is the kind of output that should sit in the release review.

Read the policy compliance row closely, because it is the one that matters. Raw agreement of 0.93 looks reassuring, but the base rate of compliant answers is high, so chance agreement is high too and kappa collapses to 0.48. The judge is passing almost everything, including the small number of forbidden promises that the criterion exists to catch. Kappa catches that; raw agreement does not.

Here is the same computation in a few lines you can drop into a test harness, using the reference labels and the judge's verdicts for one criterion.

from sklearn.metrics import cohen_kappa_score, confusion_matrix

# labels: 0 = fail, 1 = pass
ref = [1, 0, 1, 1, 1, 0, 1, 1, 1, 0]   # adjudicated human reference
judge = [1, 1, 1, 1, 1, 0, 1, 1, 1, 1] # judge verdicts on the same cases

raw = sum(a == b for a, b in zip(ref, judge)) / len(ref)
kappa = cohen_kappa_score(ref, judge)
print(f"raw={raw:.2f} kappa={kappa:.2f}")
print(confusion_matrix(ref, judge))  # rows = human, cols = judge

The confusion matrix is not decoration. If the judge misses a whole class, the matrix shows it as an empty column while raw agreement and kappa both look tolerable. The Arize alignment guide recommends reporting class counts and the confusion matrix alongside any agreement figure, and ensuring each important class has enough examples, not just the dataset overall.

There is no universal acceptable kappa. What matters is which error is worse. A false pass lets a bad answer through the gate and into production, which might mean a wrong refund promise or a fabricated deadline. A false block stops a release that would have been fine, which costs a day of engineering time and some patience. Those two costs are rarely equal, so the threshold on each criterion should not be equal either.

As a rough starting point, be explicit about which error the gate most needs to prevent, and set the bar so that the observed rate of that error cannot exceed what the organization tolerates.

The general convention for the higher-stakes criteria is substantial agreement. Interpretation bands place 0.61 to 0.80 as substantial and 0.81 to 1.00 as almost perfect, and the AWS guide maps those bands onto judge actions, treating 0.80 and above as the point where a judge can run autonomously in CI and 0.41 to 0.60 as usable only with human review on flagged cases. That mapping gives you a defensible default: set the bar at 0.70 or above for any criterion where a false pass is expensive, and accept lower kappa (with higher raw agreement) only where the cost of a false block dominates and a human sees the output anyway.

Also budget the sample. If a criterion's human-human kappa is only 0.66, the criterion itself is the weak link. The judge's 0.58 is measured against a reference that two humans only agree on 84 percent of the time, and confidence intervals at that label volume are wide. Tightening the bar without tightening the rubric first just produces a louder signal about the same ambiguity.

A miss is information. The worst reaction is to allowlist the failing cases and move on, because that converts a measurement gate into a negotiation. Rank the responses by how much of the remaining risk they remove.

1. Relabel. Check whether the human reference is solid before blaming the judge. Multiple human annotations serve three purposes: constructing a defensible reference, improving the criteria, and exposing ambiguity that a single labeler would have hidden. If two humans disagreed and the adjudication went the wrong way, the judge's kappa can move materially after relabeling. The one rule that matters: do not change past labels after seeing the judge's verdict, or the agreement number becomes a mirror. The reproducibility guidance applies the same principle to runs: fix the inputs and parameters so the measurement can be repeated.

2. Rewrite the rubric. Low human-human agreement is the signal that the criterion is underspecified. Words like "helpful" or "appropriate" need to become an observable question with anchored labels per score. The rubric design workflow sets out the five fields a strong criterion needs: name, question, scale, anchors, and a blocking rule that connects a score to the release decision. Oracle's prompt practices for agent systems make the related point for tool and agent descriptions: treat them like legal documents and define every term precisely, because vague descriptions flow straight into what the model reasons over.

3. Narrow the judge. If the rubric is solid and humans agree, but the judge still misses, cut scope. Use binary or a few named categories rather than a 1 to 10 scale, because judges are poorly calibrated on arbitrary numeric scales and a similar response can score 7 on one run and 9 on another. Narrowing also means one judge per criterion instead of one judge asked to weigh everything at once, and grounding checks that require the judge to quote the supporting policy text rather than assert that support exists. Arize's agent debugging walkthrough shows a grounding-check judge as a concrete pattern and reports a pass rate climbing from 40 to 90 percent as the harness is iterated, which is the kind of movement you should expect from rubric and prompt fixes rather than from threshold tweaks.

4. Block the release. If relabeling, rewriting, and narrowing still leave the criterion below its bar, the judge does not gate that criterion. Either the release waits, or the criterion ships with human review on every flagged case and no automated pass, or the product ships with that behavior removed. What you do not do is lower the bar to match the measurement.

The order matters because rewriting the rubric first and relabeling second wastes a cycle; you cannot tell whether a label is wrong until the rubric says what right means. Relabel against a fixed rubric, then rewrite if the criterion still wavers.

A judge is not a static artifact. Once it gates releases, the data it sees drifts, the rubric gets edited, and the model behind it changes underneath you. The Netflix LLM-judge lifecycle paper frames the judge as something that must be built, aligned, deployed and continuously monitored, with a human-in-the-loop alignment process that detects drift and triggers re-tuning behind a human review gate. The lesson for a smaller team is proportionate: re-run the agreement study on a fresh holdout every time the rubric, the judge prompt, or the underlying model changes, and treat a drop below the bar as a blocked gate, not a soft alert.

Monitoring also supplies the evidence for the next threshold decision. Log the judge verdict, the criterion, the confidence, and the human adjudication for every flagged case, so the next agreement run has a reference that was not built from the same samples used to set the bar. Persisting judge outputs and review traces is what makes agreement figures auditable and reproducible later, a point the NHI management group's guidance ties to audit log management controls.

If you are running several judges across several model providers, the plumbing starts to matter as much as the statistics: same prompt, same rubric, same logging, whatever model is behind the judge. That is a good place to consolidate model access behind one key rather than a client per provider, which is what Quanta Inference provides.