Calibrating a Model Judge Against Human Labels

Before you touch the judge's prompt, write down what a human is judging. A rubric whose score levels are adjectives ("excellent", "adequate") forces both the human and the judge to invent boundaries on every call. Convert each level into an observable claim with boundary evidence: what would have to be present in the draft for it to earn a 4 rather than a 3 the rubric consistency guide.

For a draft-quality task, a workable four-level scale looks like this:

Two practical rules make the reference standard cleaner. First, do not mix correctness, style and safety into one holistic score; score criteria independently and decide separately how they combine. Second, include near-boundary examples in the rubric, not just a terrible 1 and a perfect 4, because that is where variance concentrates the rubric consistency guide.

Then collect the labels. Draw a random sample of drafts from the same distribution the judge will see in production. Have a human scorer apply the rubric without seeing the judge's output, so the two ratings are independent. The tooling matters less than the discipline: human judgment remains the gold standard for factuality, coherence, helpfulness and safety, and it is the only thing a judge should be measured against the Label Studio evaluation guide.

You also need to pick the decision the judge will feed. A dashboard trend, a hard release gate and a safety escalation tolerate very different levels of noise, so define the decision before you set an agreement target the rubric consistency guide.

The short answer is that 100 is a sensible starting floor, and the long answer depends on how close your observed kappa lands to the threshold you plan to enforce.

Kappa is an estimate, not a fact, and it has a confidence interval. A published worked example shows how widely the required sample moves with the data: estimating a kappa of 0.6 with a two-sided 95 percent confidence interval no wider than 0.1 needs 574 items when the proportion of raw agreement is 0.70, and 984 items when that proportion is 0.80 the NCSS kappa sample size documentation. The prevalence of your score categories has a large effect on the sample you need.

A simpler planning approach is to specify two kappas rather than a confidence-interval width: the value you would not accept (the null) and the value you expect. A textbook example tests a null of 0.3 against an alternative of 0.5 at 5 percent significance and 90 percent power, and arrives at a minimum of 173 subjects for two raters the Real Statistics kappa sample size page. If your rubric has many categories or your score distribution is skewed, expect the requirement to go up.

If you want the formal machinery, dedicated calculators implement the Donner and Eliasziw approach for testing and the Shoukri and colleagues approach for estimation, and add a dropout allowance on top the kappa sample size calculator. A tutorial on agreement studies sets out the same logic step by step the inter-rater sample size tutorial.

The decision rule that ties this together:

That last row is the one teams skip. If only 2 percent of drafts are unsafe, a 100-item sample contains roughly two unsafe drafts, which tells you almost nothing about whether the judge catches them.

Cohen's kappa measures agreement between two raters on the same items, with a correction for agreement that would happen by chance the Cohen's kappa calculator reference. The formula is:

kappa = (Po - Pe) / (1 - Pe)

Po = observed agreement (proportion of items where the two raters agree)
Pe = expected agreement by chance (from the raters' category marginals)

Worked example. Suppose you label 100 drafts by hand and the judge scores the same 100. Both agree on 78 items, so Po is 0.78. The human gave a "pass" to 60 drafts and the judge gave a "pass" to 55. The expected chance agreement is:

Pe = (0.60 * 0.55) + (0.40 * 0.45)
   = 0.33 + 0.18
   = 0.51

kappa = (0.78 - 0.51) / (1 - 0.51)
      = 0.27 / 0.49
      = 0.55

By the commonly used Landis and Koch interpretation scale, 0.55 sits in the moderate band, between 0.41 and 0.60, and well below the substantial band that starts at 0.61 the Cohen's kappa calculator reference. If your release gate required 0.6, this judge is not eligible, however good its outputs look in a chat window.

Three technical choices change the number you get:

• Use weighted kappa for ordered scores. If your rubric is 1 to 4, a judge that scores 1 where the human scored 4 is a worse disagreement than 3 against 4, and weighted kappa reflects that. The Autorubric framework includes Cohen's kappa, weighted kappa and correlation coefficients as reliability metrics for exactly this reason the Autorubric paper.
• Never report raw percent agreement alone. Two raters who both pass almost everything will agree often by chance; kappa removes that.
• State the confidence interval. Computing kappa without checking whether the scale is ordinal, or reporting correlation without assessing systematic bias, are the two inconsistencies the Autorubric authors call out in prior work the Autorubric paper.

A short script is enough to get started, and it forces you to keep the per-item data that section five needs:

from sklearn.metrics import cohen_kappa_score

# human and judge scores for the same 100 items, in the same order
kappa = cohen_kappa_score(human, judge, weights="quadratic")
disagreements = [(i, h, j) for i, (h, j) in enumerate(zip(human, judge)) if h != j]
print(f"kappa={kappa:.2f} disagreements={len(disagreements)}")

The kappa number tells you whether to worry. The disagreement list tells you what to fix, and it is where the real work happens. Split every disagreement into one of three causes before you change anything:

1. Judge error. The human label is right and the judge read the draft wrongly. This is a prompt problem: the criterion was under-specified, or the judge weighted length or position over substance.
2. Rubric defect. The human and the judge are both defensible under the written rubric. This is a rubric problem, and the fix is to sharpen the boundary with the disputed case added as an example.
3. Human error. The judge is right and the human slipped. This happens more often than teams expect on long sessions, and it is a good argument for a second scorer on the borderline band.

The published practice is to keep human corrections as assets. LangSmith's alignment loop collects human corrections, builds few-shot examples from them, and tracks agreement over time LangChain's LLM-as-a-judge guide. That turns each disagreement into a permanent piece of the judge's context rather than a one-off argument in a pull request.

Two review habits keep this honest. Review blind, so the reviewer sees the draft, the rubric and both labels without knowing which is which. And review a sample of the agreements too, not only the disagreements, because a judge that agrees with a careless human for the wrong reason will look fine on the disagreement list.

The consistency guide's warning applies directly here: if two senior reviewers cannot articulate whether a case is a 3 or a 4, the immediate task is adjudication and rubric repair, not prompt tuning the rubric consistency guide. Escalate those cases to a third reviewer and record the outcome as a rubric example.

Calibration tells you how closely the judge tracks your human standard on the sample you measured. It does not tell you the judge will track it on the next thousand drafts, and it says nothing about the cases your sample did not contain. Some decisions need a person regardless of the kappa.

Safety and policy vetoes. Full automation is not recommended for high-stakes applications, and a hybrid approach with human oversight is generally more reliable the Label Studio evaluation guide. If a failure could expose protected data or take an action the application was never authorized to take, express it as a veto that a person clears, not as a line in an average the rubric consistency guide.

Novel inputs outside the calibration distribution. Your human sample was drawn from drafts that resemble each other. A new product line, a new language or a new customer segment is a distribution shift, and the honest move is to re-label a fresh sample before the judge's scores are trusted again.

Regulated or contractual decisions. Where a rule or a contract requires a named human to make or sign off on the decision, the judge can rank and triage but not decide. This is a legal constraint, so confirm it with whoever owns compliance rather than inferring it from a benchmark.

Borderline bands. Even a well-calibrated judge is least reliable exactly where the decision is hardest. A router that sends the middle of the distribution to a person, and the clear cases to automation, keeps human effort where it changes the answer.

One more caution that matters for anyone who also uses first-impression research as a human reference: qualitative methods like five-second tests are designed to reveal immediate impressions, and a five-second test explicitly does not assess usability or detailed comprehension because participants do not interact with anything the CleverX five-second testing guide. If your human label comes from a method that measures a different thing than the judge is scoring, your kappa is comparing two tasks, not two raters.

Treat calibration as a pipeline you can rerun, not a one-time study.

1. Freeze the rubric. Version it, with anchors and near-boundary examples.
2. Draw a random sample. At least 100 items; more if a subgroup or a rare outcome matters.
3. Label blind. Human scorers do not see judge output. Use two scorers on the borderline band and adjudicate.
4. Score the same items with the judge. Store every per-item pair.
5. Compute kappa and its interval. Use weighted kappa for ordered scales.
6. Review disagreements. Classify as judge error, rubric defect or human error.
7. Repair, then re-measure. Add the disputed cases to the judge's few-shot examples and rerun on a held-out sample so you are not grading the judge on its own training examples.
8. Rerun on change. Any change to the model, the prompt or the rubric invalidates the previous kappa and requires a fresh check.

Step seven is where most teams overfit. If you add every disagreement to the judge's examples and then recompute kappa on the same items, the number will rise and mean less. LangChain's framing of the data flywheel, where production traces feed datasets, datasets power evaluations and evaluations drive improvements, assumes a held-out check at the end of the cycle LangChain's LLM-as-a-judge guide. Keep a locked holdout of labeled items you never tune against.

If you are comparing two prompts or two models as part of this loop, pairwise comparison, where the judge picks the better of two outputs against stated criteria, is a different job from the pointwise scoring described above LangChain's LLM-as-a-judge guide. Calibrate each judge type against its own human sample.

For teams running several models and want the judge calls and the scored outputs behind one API key, Quanta Inference routes text, image, video, audio and embedding models through a single OpenAI-compatible endpoint Quanta Inference, which keeps the calibration loop's per-item records consistent across judges. The estimates behind this article were checked against the sample-size calculators and worked examples cited above; if you are doing lots of affine arithmetic on lengthy real-world documents, a horizontal layer shift is a known lossy operation and raw scores can drift far from the numbers that existed under the old anchor the NCSS kappa sample size documentation.