Two Judges and a Human: Three-Way Agreement as a Gate

Start from your dataset of cases, each with a binary label: pass/fail, correct/incorrect, grounded/ungrounded. The rule of thumb for the LLM judge setting is binary or a few clearly named classes, not a 1 to 10 scale, because models are not calibrated to produce consistent arbitrary scores, as Evidently's judge FAQ explains. A binary choice keeps the cube small and every cell interpretable.

For each judge J in {A, B}, build a 2x2 table against the human H:

Here a and d are the agreement cells; b and c are the disagreement cells. Cohen's kappa is:

κ = (Pₒ − Pₑ) / (1 − Pₑ)

where Pₒ = (a + d) / N, and Pₑ uses each rater's marginal distribution. A worked example on 100 items with 40/10/15/35 in the four cells gives Pₒ = 0.75, Pₑ = 0.50, and κ = 0.50, which the article calls moderate. Work each judge's table the same way and report the two kappas side by side with their confidence intervals, not as single numbers.

That last point is not pedantry. Sample size decides whether a kappa is even meaningful: a sample size calculator write-up shows that tightening the confidence interval around kappa is expensive, with distances of 0.20, 0.05 and 0.01 needing roughly 118, 1,625 and 38,203 subjects. A meta-analysis primer recommends aiming for a confidence interval narrow enough that you do not span one of Cohen's "levels of agreement." A practical kappa guide for LLM judges is blunter: below about 10 annotations the 95% interval is wider than ±0.30 and the number is statistically meaningless.

The two judges and the human give eight possible cells. Each case falls into exactly one. Count them:

Two aggregate numbers fall out immediately:

• All-three agreement share = (cell 1 + cell 2) / N.
• Judges-vs-human share = (cell 3 + cell 4) / N. This is the case the two judges agree with each other and disagree with the human.

The remaining four cells (5 through 8) are the "one judge tracks the human, the other misses" cases. They will not all be equal. If cell 5 dominates cell 8, judge A is closer to the human on pass cases and judge B is missing them; the reverse tells you to trust B.

A worked example makes the arithmetic concrete. Suppose N = 200 and the counts are:

All-three agreement is 130/200 = 0.65. The judges-together-against-human share is 35/200 = 0.175. That second number is the one to watch: nearly one case in five finds both judges lined up and the human outvoted. Even if each judge's two-way kappa against the human looks respectable (they can, because a large chunk of the data is trivial all-pass cases), the disagreement cell reveals a shared misjudgment that a judge-to-judge agreement score would have scored as success.

Do not average the eight cells. Each has a different operational meaning, and each points at a different fix.

All three agree (cells 1 and 2). This is your clean floor. When all three raters land the same way, the label is the most defensible. It is not proof of correctness (a human can be wrong too, and the kappa vs accuracy distinction applies to the panel like any rater), but it is the least contestable evidence you can produce. If you need a single gate number, this share is the one you should treat as your headline.

Both judges agree, human dissents (cells 3 and 4). This is the failure mode that two-way kappa cannot see. The judges are consistent with each other, so any judge-versus-judge metric rewards them. The human is the minority. Two questions decide how to act:

• Is the human plausibly wrong? Humans make mistakes too, which is exactly why multiple human annotations are worth collecting on a subset; a single human is a sample of one.
• Are the judges sharing a bias? The same Arize field guide notes that comparing human-human agreement to LLM-human agreement on the same examples is the right way to judge an LLM judge, and that human agreement is context, not a ceiling. If your two judges are the same model family, same prompt, or same rubric, they will share blind spots; the cell will not tell you the human is wrong, only that you have no independent second opinion.

The practical rule: a large judges-vs-human share is a prompt to inspect, not to override. Pull 20 examples from cells 3 and 4, read them, and decide whether the human or the judges are right. If the judges are right more often, the human label is the noisy piece, and you should reshape the reference (adjudicate, or collect a second human annotation). If the human is systematically right, your judges share a defect the two-way numbers hid.

One judge tracks the human, the other misses (cells 5 through 8). These separate judge quality. Compare cell 5 + cell 8 (A matches) against cell 6 + cell 7 (B matches). The judge with the larger "matches human" count is not necessarily better overall, but it is the one whose independent opinion carries more weight. Look at where the misses cluster: if judge B misses almost entirely on fail cases while A misses on pass cases, they have opposite biases and you may be able to use one to catch the other.

All three disagree is a data problem, not a model problem. A case where the two judges disagree and the human agrees with neither is a signal that the rubric or the case itself is ambiguous. The qaskills.sh rubric design guide frames this well: use disagreements to improve the criteria, and treat low agreement as a sign the rubric needs clarification, not that someone is wrong. Those cases belong in a held-out set and a written note on the decision boundary, not in the gate.

The table below compares what each number answers. None is a substitute for the others.

The last row deserves emphasis. Percent agreement can look excellent while meaning very little when one class dominates. An explainer on percent agreement vs kappa notes that a model that mostly predicts the majority class gets high percent agreement without real discriminative ability, and the GEDD kappa guide gives the stark version: a judge that always says "Correct" when 80% of the data is correct gets 80% accuracy and a kappa of 0.00. The three-way cube is essentially a way to decompose percent agreement so you can see which fish was caught in the dark.

For three or more raters on the same items, Fleiss' kappa or Krippendorff's alpha are the standard alternatives when every case has roughly equal coverage or some annotations are missing. An inter-rater reliability overview lays out the choice cleanly: Cohen's kappa for two raters, Fleiss' for three or more, Krippendorff's alpha when coverage is uneven. The reason to still compute Cohen's kappa per judge against the human is that you want to know which judge is out of line, not just the panel-average. The three-way cube gives you the diagnostic; a single multi-rater coefficient gives you the headline.

Here is a decision rule that turns the cube into a release gate. The numbers are illustrative starting points, not standard thresholds; you should calibrate the floors to the cost of the decision your evaluation supports.

1. Require a minimum N. If N is below roughly 10 per class, stop and collect more cases; the confidence interval on kappa makes the rest unreadable.
2. Compute each judge's kappa against the human and its confidence interval. If either judge's kappa is below your floor (for example, 0.60 for low-stakes automation, per the interpretation table in the LLM judge kappa guide), that judge is not fit for the gate.
3. Compute the all-three agreement share. This is your primary gate: the share of cases where the panel lands unanimously.
4. Compute the judges-vs-human share. If it exceeds your tolerance (say, more than 10%), the gate does not pass; route those cases to human review and log them, do not override the human.
5. Review a random sample of the human-matches-only cells (5 through 8) to check which judge is closer to the human and whether the misses are systematic.

Here is a compact script for the cube, assuming labels of 0 (fail) and 1 (pass):

import itertools

def three_way_counts(a, b, h):
    """a, b, h are equal-length lists of 0/1 labels."""
    cells = {f"{aa}{bb}{hh}": 0 for aa, bb, hh in itertools.product([0, 1], repeat=3)}
    for aa, bb, hh in zip(a, b, h):
        cells[f"{aa}{bb}{hh}"] += 1
    n = len(a)
    return cells, n

def shares(cells, n):
    all_agree = cells["111"] + cells["000"]
    judges_vs_human = cells["110"] + cells["001"]
    return {
        "all_three_share": all_agree / n,
        "judges_vs_human_share": judges_vs_human / n,
        "A_matches_B_misses": (cells["101"] + cells["010"]) / n,
        "B_matches_A_misses": (cells["100"] + cells["011"]) / n,
    }

Two details make the script trustworthy. First, pass the labels as plain integers; string cells are fine for reporting but easy to mis-key in a rush. Second, log the exact case IDs behind cells 3 and 4. The share is a statistic; the cases are the evidence a reviewer needs. A gate without the underlying cases to show a human is a gate you cannot defend.

If you are running judges through a model API to produce these labels, storing raw judge output and the case ID side by side is enough. For teams evaluating several model families side by side, one endpoint per class of model keeps the harness small; Quanta Inference exposes text, image, audio, video and embedding models through one OpenAI-compatible key, which reduces the plumbing spent on a judge harness and leaves room for the measurement work that actually changes a decision.

The most common mistake after extending to three raters is treating the human's label as final truth. The Arize field guide is explicit that human agreement should be used as context for the LLM comparison, not as a hard performance ceiling, and that the reference should be built by adjudication or a true majority vote while ties and low-consensus cases are preserved rather than painted over. A Netflix LLM-judge case study makes the same point operationally: a judge is a lifecycle, not a one-shot artifact, and monitoring with a continuous human-in-the-loop alignment process detects drift and triggers re-tuning behind a human review gate.

That has a direct consequence for the three-way matrix. Use cells 3 and 4 (judges against human) as a prompt to investigate, not an automatic verdict. Your human is a rater, not a ground truth oracle. If you have the budget, add a second human on a subset and measure human-human kappa first; that number tells you how much you can trust the human column at all. If human-human agreement is itself low, the judges are not the problem, the rubric is, and the three-way cube is telling you the task needs a sharper decision boundary before any automated gate is meaningful.

The panel logic also tells you when two judges are redundant. If both judges are the same model with the same prompt, cell 3 and cell 4 will capture their shared bias, not independent disagreement. A useful panel pairs two different judge families, or a deterministic rule and a model judge, so the "both judges agree, human dissents" cell is a genuine warning rather than two copies of the same opinion.