Inter-Rater Agreement Before You Compare Judge to Human

Forty items is a practical size for a pilot. It is small enough that two people can finish in an afternoon and large enough to show whether items cluster around a boundary. Example 1 in the PASS documentation works through a two-sided 95% interval with a target width of 0.1 and finds a sample of 574 needed for a precise estimate, which tells you a 40-item pilot is a smoke test, not a definitive reliability study Confidence Intervals for Kappa. You are looking for the obvious failure, not certifying the rubric.

Step 1: Write the rubric before recruiting anyone. Each level needs an observable anchor. "Excellent" and "good" force the rater to invent boundaries on every item, which is exactly where variance concentrates. A worked example from a support-answer rubric pairs each score with boundary evidence, such as "core diagnosis is sound, but an important limitation or step is missing" for a 3 and "correct and actionable with no material omission" for a 4 Testing LLM-Judge Rubric Score Consistency.

Step 2: Stratify the 40 items. Do not sample 40 good answers. Include clear failures, near-boundary cases and at least a handful of tricky items. Boundary cases are where disagreement lives, so a pilot weighted toward the middle tells you more per hour.

Step 3: Score independently and blind. The two raters do not discuss items and do not see each other's scores. Independence is a stated requirement for Cohen's kappa to mean anything Cohen's Kappa Calculator. If they talk, you are measuring consensus, not reliability.

Step 4: Record verdicts in a table you can copy into a script. A spreadsheet with one row per item and one column per rater is enough. Keep the original scores, including the ugly ones.

Step 5: Compute kappa on the raw scores. Use unweighted kappa for nominal labels (pass or fail, correct label or wrong label) and weighted kappa for ordinal 1-to-5 scales. scikit-learn exposes both through the weights parameter cohen_kappa_score.

from sklearn.metrics import cohen_kappa_score

# one score per item, same order, from rater A and rater B
rater_a = [4, 2, 5, 3, 3, 1, 4, 4, 2, 5]
rater_b = [4, 3, 5, 3, 2, 1, 4, 5, 2, 4]

raw = cohen_kappa_score(rater_a, rater_b)
ordinal = cohen_kappa_score(rater_a, rater_b, weights="quadratic")
print(raw, ordinal)

Step 6: Read the number against a stated scale, then decide. Landis and Koch is the usual reference: below 0.00 poor, 0.00 to 0.20 slight, 0.21 to 0.40 fair, 0.41 to 0.60 moderate, 0.61 to 0.80 substantial, 0.81 to 1.00 almost perfect Cohen's Kappa Calculator. Their original methodology paper is the source these bands are drawn from The Measurement of Observer Agreement for Categorical Data.

The number that matters is not the kappa alone; it is the kappa compared to the bar you set before running the pilot. Set the bar first, or you will move it once you see the result.

A workable rule for most content and agent-output work:

• Kappa below 0.40 (fair or worse): stop. The rubric is not ready. Rewrite anchors, not prompts.
• Kappa 0.40 to 0.60 (moderate): acceptable only for internal trend tracking. Do not gate releases on it.
• Kappa 0.61 to 0.80 (substantial): good enough to compare a judge to humans and to use for regression checks with spot audits.
• Kappa above 0.80: strong. A judge that hits this against humans is doing the job.

Percent agreement is the metric that will mislead you. If 85% of items are scored 4 and 5, two raters will agree a lot by accident, and percent agreement will look inflated. Kappa subtracts that chance agreement, which is why it is the preferred figure Wikipedia on Cohen's kappa. The formula is worth knowing:

kappa = (observed agreement minus expected agreement) / (1 minus expected agreement)

The Confidence Intervals for Kappa chapter shows the confusion matrix layout that makes the expected agreement easy to hand-check: row and column totals for the same categories, multiplied and divided by the square of the item count.

One caution. Kappa is sensitive to skewed prevalence, meaning when nearly every item gets the same score, and to bias between raters, meaning one rater is systematically harsher Cohen's Kappa Calculator. If your pilot shows near-zero kappa but high percent agreement, check the distribution before blaming the rubric.

Suppose 40 support answers, two reviewers, 1-to-5 scale on a "correctness" criterion. Reviewer A gives 22 items a 4, eight a 5, six a 3, three a 2 and one a 1. Reviewer B gives 18 items a 4, seven a 5, ten a 3, four a 2 and one a 1. They agree on 28 of 40 items. Percent agreement is 70%, which sounds respectable.

Expected agreement, computed from the marginals, is about 0.58 for this kind of distribution. Plug in: kappa equals (0.70 minus 0.58) divided by (1 minus 0.58), so 0.12 divided by 0.42, roughly 0.29. That is "fair" at best. The underlying problem is almost never the model; it is a pile of disagreements bunched at 3 versus 4.

Look at the disagreement table. If nine of the twelve disagreements sit between 3 and 4, the boundary between "usable with a gap" and "correct and actionable" is unclear. Rewrite those two anchors with concrete evidence requirements, rescore the same 40 items with the same two people, and recompute. A typical healthy outcome after one rewrite is agreement of 33 of 40, kappa around 0.62. That number is your ceiling: a judge cannot be expected to beat the humans it is calibrated against, and in practice strong judges land near human-to-human agreement levels How to Calibrate LLM-as-Judge.

The third row is the one teams want to see first and the one that matters least until row two is settled.

Using one "gold" human reviewer. LangChain's calibration guide is explicit that prompt iteration alone will not close the gap between a technically correct evaluator and a reliable one, and that the loop requires human corrections How to Calibrate LLM-as-Judge. One human is a single point of failure, not a gold standard.

Combining criteria into one score too early. A fluent but unsafe answer forces a single number to negotiate weights the rubric never stated. Score criteria independently first, then define the aggregation rule, which is how production rubric frameworks separate per-criterion evaluation from overall scoring Autorubric.

Letting the judge see the human scores. Few-shot calibration from human corrections is a legitimate later step, but the first comparison should be independent. A judge tuned on the same items it is scored on is not a fair test.

Forgetting repeatability. A judge can be reliable within itself and still wrong. Testing repeated-run variance on the same item is a distinct property from calibration Testing LLM-Judge Rubric Score Consistency. Measure both.

Skipping ordinal weighting. A 4 versus 5 disagreement is not the same as a 1 versus 5 disagreement. On ordinal scales, weighted kappa (quadratic or linear) reflects that ordinal structure cohen_kappa_score.

Once the humans clear 0.61, the judge becomes an extension of a working measurement, not a replacement for an unfinished one. The calibration loop is the same discipline the humans went through: collect corrections, build few-shot examples from the boundary cases, and track agreement over time How to Calibrate LLM-as-Judge. If the judge runs on a stack with any judge model behind a single interface, that is the natural place to plug it in: Quanta Inference gives you text and multimodal models through one API key, so swapping judge models to compare agreement does not mean rebuilding the pipeline (https://quantaintelligence.ai/inference).

Two extra checks belong in the same pilot, and they are cheap. First, preserve every repeated result so dispersion is visible instead of buried in an average. A judge returning 4, 4, 4, 4, 4 supports a release decision; one returning 1, 2, 3, 4, 5 does not, even though the means match Testing LLM-Judge Rubric Score Consistency. Second, rerun the frozen 40-item set after any prompt change to catch drift. The rubric itself becomes a regression asset: a product answer that scores a 4 today should score a 4 tomorrow unless something changed.

When the item set and rubric stabilize, the setup work moves from the pilot to the operating loop. Teams that want a working rubric, item set and scoring harness built out with them can start from a described spec and have a specialist pick it up (https://quantaintelligence.ai/). That is the natural second step once your humans and your judge are agreeing at a level you trust.

• Rubric written, with every score level tied to observable evidence, not adjectives.
• Forty items stratified with boundary cases included.
• Two independent raters, blind to each other, using the same written rubric.
• Cohen's kappa computed on raw scores; weighted kappa for ordinal scales.
• Decision rule recorded before the pilot, using the Landis and Koch bands.
• If kappa is below 0.61: rewrite anchors, rescore the same items, recompute.
• Judge prompt includes the revised anchors and the boundary examples the humans argued over.
• Judge scored on fresh items, not the pilot items, then checked for repeatability and drift.