How to Calibrate an LLM Judge Against Human Labels

Start from the population the judge will actually score. If the judge runs on production traces, pull from production traces; do not write tidy examples from imagination. Langfuse's guide to golden datasets treats this fixed set of test cases as the thing every meaningful change runs against, and it has to be versioned the same way.

Size and composition

A few hundred examples per judge task is a reasonable target. Raters with fewer than about 10 annotations per label cannot produce a kappa you should act on, and a confidence interval on kappa computed from a handful of rows is wider than any threshold you would set. For a three-label rubric, plan on at least 20 to 30 examples per label, then add volume where the business risk is highest.

Stratify on three axes:

• Rarity and risk. Over-sample the outputs that cause incidents (refunds, medical or legal claims, prompt-injection attempts) even if they are rare in traffic.
• Boundary cases. Include outputs that sit close to the rubric line, because this is where judge disagreement concentrates.
• Known edge cases. Every past bugfix should contribute a row, so the set grows as failures are found.

Google Cloud's framework for agent evaluation recommends starting with success criteria written as measurable statements, such as "grounded in known documents and free of claims outside them," so that the rubric you hand raters is the rubric the judge receives.

Keep prompt-tuning examples out

Never calibrate on the examples you used to write the judge prompt. OneUptime's walkthrough of kappa calibration is explicit about keeping the calibration set separate from the examples used to write or tune the judge prompt, and about freezing the judge itself: pin the model, the full prompt, the supplied references, the output schema and decoding settings.

Store each row with enough metadata to audit later. A minimal record looks like this:

{
  "case_id": "policy-018",
  "input": "...",
  "response": "...",
  "human_label": "fail",
  "human_reason": "Claims a refund is guaranteed after the policy deadline",
  "rubric_version": "policy-correctness/4"
}

Two raters per example is the floor. One rater cannot reveal ambiguity, and ambiguity in the human labels is exactly what an LLM judge will amplify. Innodata's overview of golden datasets frames these human-validated labels as ground truth against which model output is measured, and notes that quality of the label set caps the quality of everything downstream.

Give raters the same package: label definitions, authoritative references, boundary examples, and one decision per case. Ask them to label independently, without seeing each other's votes or the judge's output, and note a reason for every failing label. Do not change past labels once the judge's results are visible.

Then adjudicate. Where the two humans disagree, a third senior reviewer decides against the written requirement, and you keep all three votes in the record. Treat repeated disagreements as signal that the rubric is underspecified, not that one rater is careless. A judge cannot be calibrated to a rubric that two experts apply differently.

Worked example

Suppose you build a 300-case set for a support-response judge with three labels: pass, needs edit, fail.

Human-to-human percent agreement here is 277/300, or 92.3%. Exchangeable raters at that level are a workable calibration target for the judge.

Three numbers belong in every calibration report.

Percent agreement

The share of cases where the judge's label equals the human consensus label. It is intuitive and worth report, but it flatters judges whenever one label dominates. The AWS sample on Cohen's kappa for LLM judges gives the standard example: if 80% of responses are correct, a judge that always says "correct" scores 80% accuracy while contributing nothing.

Cohen's kappa

Kappa corrects observed agreement for the agreement expected from each rater's label frequencies. Formally, kappa = (Po - Pe) / (1 - Pe), where Po is observed agreement and Pe is the chance agreement implied by the marginals, as defined in the kappa guide. Use unweighted kappa for nominal labels, and weighted kappa when the labels are genuinely ordered.

from sklearn.metrics import cohen_kappa_score, confusion_matrix

ALLOWED = {"pass", "needs_edit", "fail"}
LABELS = ["pass", "needs_edit", "fail", "judge_error"]

def normalize(value: str) -> str:
    label = (value or "").strip().lower()
    return label if label in ALLOWED else "judge_error"

human = [normalize(x) for x in human_consensus]
judge = [normalize(x) for x in judge_outputs]

raw_agreement = sum(a == b for a, b in zip(human, judge)) / len(human)
kappa = cohen_kappa_score(human, judge, labels=LABELS)
print(raw_agreement, kappa)
print(confusion_matrix(human, judge, labels=LABELS))

Keep parse failures and refusals in the denominator as their own label. A judge that answers correctly on 90% of cases and errors on the rest has 90% conditional accuracy and 81% end-to-end agreement.

That arithmetic matters more than it looks. At 300 cases, a judge with 81% agreement has a standard error of about 2.3 percentage points, so its 95% interval runs roughly 76.5% to 85.5%. If your release threshold is 85%, the run cannot prove the judge clears it. Budget for 500 or more cases when the threshold sits close to your measured result.

Correlation

When the judge outputs a score rather than a label (1 to 5, or 0 to 100), report Pearson correlation if you care about the linear relationship and Spearman if you care that the rank order matches. Correlation catches a judge that ranks outputs sensibly while sitting systematically high or low, which kappa on a binarized score can hide. Report both the correlation and the mean offset.

Read the confusion matrix, not just the headline

For every calibration run, inspect four cells: human-pass and judge-fail, human-fail and judge-pass, and the two agreement cells. In practice, judge-pass on a human-fail case is the costly cell, because it lets defects through. If that cell is populated, tighten the failing definitions and add the cases to the few-shot examples before re-running. MLflow's LLM-as-a-judge documentation notes that its evaluation tooling supports aligning judges with human feedback so automated scores stay tied to the team's quality standards, which is the same inspect-and-repair loop.

A single kappa number is not a quality grade; it is an input to a decision about how much autonomy the judge gets. The widely used interpretation bands come from medical and social-science agreement research, and the common engineering reading is that 0.61 to 0.79 is substantial while 0.80 and above is near-perfect, per the kappa guide for LLM judges.

Pick your threshold before you see the first kappa, or you will negotiate with yourself. Then buy margin: if the lower bound of your confidence interval sits below the threshold, add labels rather than declaring victory. For experiments where you only need relative comparisons between two prompts, 0.6 is usually enough because systematic judge bias cancels; for decisions about refunds or safety, the judge needs the higher band.

The judge is a model in production and drifts for the same reasons your product does: prompts change, model versions get retired, traffic shifts, and the rubric's boundary cases get more common. LangChain's evaluation framework guide describes the production flywheel where real failures flow into annotation queues, get expert review, and return as regression tests, which is also how you keep a judge calibrated.

A practical monitoring loop:

1. Re-label on a schedule. Pull a fresh stratified sample of 50 to 100 production cases every month, or every two weeks while the product is changing fast, and have humans label them with the same rubric.
2. Compute kappa against consensus on each pull. Chart it on a control chart with the calibration baseline as the center line and limits at roughly plus or minus two standard errors.
3. Add alarms for the error cells. A rise in judge-pass on human-fail, or a sudden jump in judge errors and refusals, is an incident even if kappa looks stable.
4. Promote every confirmed miss to the permanent set. Regressions belong in the calibration set so they are re-tested on the next judge change.
5. Re-calibrate after any edit. Any change to the judge prompt, model or schema resets the clock and requires a fresh run on the untouched set.

Anthropic's guide to evals for agents makes the broader case: without evaluation infrastructure, teams catch issues only in production, where fixing one failure often creates another. A judge with a monitored kappa is what lets you stop firefighting.

If your judge scores a multi-step transcript rather than a single response, the same calibration mechanics apply with two changes. First, the human label must be attached to the same object the judge sees. Google's methodical approach separates agent success from process and trajectory, and points out that an agent can produce a correct output through an inefficient or wrong path, a silent failure. Second, sample traces, not final answers, so the judge and the humans are grading the same thing.

Because agents are non-deterministic, run the judge more than once per case and report both single-run agreement and label stability. Majority voting over three passes is a different judge configuration with its own kappa, a higher cost and often a different disagreement pattern, so calibrate the variant you intend to deploy rather than the one that flatters you. The Rise of Agent-as-a-Judge surveys this space and is clear on the boundary: agent-based judging complements human oversight, it does not replace it, and meta-evaluation of the judge is still an open problem.

The practical rule is the one that keeps teams out of trouble. Spend the annotation budget on the cases where a wrong judge verdict is expensive, and let the cheap deterministic checks (schema validity, length, required disclaimers) carry everything they can. Evidently's guide to LLM-as-a-judge is a useful reference for choosing between pairwise comparison, criteria scoring and reference-based scoring, because the calibration statistic has to match the judge's output type.

If you would rather run the judge on traces and scores that already sit in one place, Quanta Inference gives you one OpenAI-compatible endpoint for the judge model and any generation models you are comparing, so calibration runs and production scoring use the same path.