Sign in Meet your experts
AI agents

Reviewing a Judge Against a Reviewer: Disagreement and Its Cost

Antonio Evans9 min read

A split-screen illustration: on the left, a judge's gavel labeled 'LLM Judge' next to a green checkmark and a red X; on the right, a human reviewer's glasses labeled 'Human Reviewer' next to a matching checkmark and X.
Key points

Key points

  1. 1Record the judge's verdict and the human's verdict in the same row for every item, then count the rows where they differ.
  2. 2Classify each split by direction (judge too lenient or too strict) and by severity before deciding who is right.
  3. 3A single number like percent agreement or kappa describes the judge; a cost matrix tells you whether to let it gate a release.
  4. 4The judge wins overall when the total cost of its errors is lower than the reviewer's, measured on the same rubric and the same error weights.
Daily AI briefing

Daily AI briefing

Start by building a disagreement log, not a scoreboard

Most teams validate a judge by reporting how often it agrees with humans. That number is useful context and a poor decision rule. The first thing to build is a table with one row per item and both verdicts side by side, so that every disagreement is addressable.

A minimum schema that survives audit looks like this:

{
  "case_id": "refund-018",
  "stratum": "policy_boundary",
  "input": "...",
  "response": "...",
  "rubric_version": "policy-correctness/4",
  "judge_model": "pinned-model-id",
  "judge_verdict": "fail",
  "judge_reason": "Claims refund is guaranteed after the deadline",
  "human_verdict": "pass",
  "human_reason": "Deadline language is conditional, not a guarantee",
  "human_annotator": "rev-07",
  "cost_fp": 0,
  "cost_fn": 12
}

Two fields carry more weight than the rest. The rubric version pins what both raters were asked to decide, and the two cost columns encode what happens when each side is wrong. One calibration write-up recommends exactly this discipline: pin the judge model version, the full prompt, examples, output schema and decoding configuration, and keep judge errors and refusals in the denominator rather than excluding them, because a judge that is right on the cases where it produces a label but fails to produce one otherwise is less accurate end to end than it looks the OneUptime calibration guide.

Sample size comes before you read any split. A practical floor is 15 to 20 annotated items per error code, because with fewer than 10 annotations the confidence interval around kappa is wider than plus or minus 0.30 and the number is not actionable the AWS GEDD kappa guide. Stratify that sample instead of drawing it uniformly: include common traffic, rare high-risk cases, known failures and cases sitting on the rubric boundary the OneUptime guide. The general argument for stratification is that a uniform sample lets the dominant segment drown out small but decision-relevant ones, which inflates success metrics and hides regressions Statsig's stratified sampling guide.

One more rule before you look at the judge: have two humans label the same items independently. If the humans cannot apply the rubric consistently, no judge will fix the ambiguity, and you will spend weeks tuning a prompt against noise the Arize alignment field guide.

Three ways a split can go, and why direction matters

Every disagreement falls into one of a small number of shapes. Name them, because the fix differs for each.

Split typeJudgeHumanWhat it usually meansFirst fix to try
False positivefailpassJudge is over-strict or reads a rule too literallyAdd boundary examples and a note that conditional language is not a guarantee
False negativepassfailJudge is lenient or misses a policy breachAdd the missed failure to the rubric as an explicit case
Severity mismatchpartialfailBoth saw a problem, disagree on how badReplace the 3-point scale with binary plus an error code
Abstention or errorno verdictpass or failJudge refused, timed out, or returned unparseable outputTreat as its own outcome, never as agreement

Severity mismatch deserves special attention because it hides inside "both of us flagged it" and quietly wrecks any ranking the judge produces. A study of nine judge models against human annotations on TriviaQA found that even the two best-aligned judges, measured by Cohen's kappa, were outperformed at ranking exam-taker models by a trivial lexical matching method, which had up to 34 points lower human alignment the Judging the Judges paper. High agreement does not buy you correct ordering, and severity mismatch is the usual reason.

Direction also decides your remediation. A judge that is too lenient is dangerous in a CI gate because it passes bad outputs, while a judge that is too strict generates false alarms and trains the team to ignore it. Both can show the same percent agreement. That is why raw agreement is only descriptive and a chance-adjusted metric belongs next to it the NHI explainer on percent agreement versus kappa.

How to decide who wins a single disagreement

The short answer is that the verdict backed by the evidence and the rubric wins, and the losing side's error becomes a cost you record. Do this as a five-minute procedure per split, with a second person adjudicating anything that touches a high-cost error type.

  1. Read the item and the rubric version both raters saw. If the rubric does not decide the case, you have a rubric bug, not an error from either rater. Fix the rubric and re-label.
  2. Check the judge's stated reason against the response text. A reason that cites something absent from the response is a hallucinated justification and counts against the judge regardless of the verdict.
  3. Apply the cost columns. If the judge said pass and the human said fail, and a missed policy breach costs 12 dollars in refunds while a false alarm costs 12 minutes of reviewer time, the miss is the expensive error.
  4. Record the outcome as human-wrong, judge-wrong, rubric-ambiguous, or both-defensible. Both-defensible cases are the most valuable rows you own, because they are where you can tighten a boundary example.

Step 4 is where most programs fail. Adjudication should resolve the requirement, not conceal the disagreement, so keep the original reviewer votes even after you pick a reference label the OneUptime guide. Multiple human annotations exist to build a defensible reference and to improve the criteria, not to manufacture ground truth the Arize field guide.

Build the cost matrix before you debate the winner

A cost matrix turns a philosophical argument into arithmetic. List the error types your task can produce, assign each a cost in a unit you actually care about (dollars, support tickets, minutes of review, or a severity weight from 1 to 10), and keep the unit consistent across judge and reviewer.

Worked example. You have 400 calibration items, stratified as 300 routine and 100 boundary or high-risk. Human reviewers take 4 minutes per item at 45 dollars an hour, so a full human pass costs 400 times 3 dollars, or 1,200 dollars. The judge pass costs a few dollars of inference plus a one-off authoring effort. The disagreement log shows 48 splits:

Error typeCountWho erredUnit costTotal cost
Missed policy breach (false negative)9Judge40 dollars360 dollars
Spurious failure (false positive)22Judge4 dollars (reviewer minutes)88 dollars
Missed policy breach (false negative)5Reviewer40 dollars200 dollars
Spurious failure (false positive)12Reviewer4 dollars48 dollars

The judge's splits cost 448 dollars, the reviewer's cost 248 dollars on the same weights. The naive read is that the reviewer wins and the judge is not ready. The useful read is narrower: the judge is right about severity far more often than about existence, its expensive error is concentrated in one error code (missed breaches on refund policy), and that single code accounts for 360 of its 448 dollars. Fix that code with rubric examples, re-run the same 400 items, and the comparison is worth repeating. This is the decision rule: do not ask whether the judge wins overall. Ask which error code carries the cost, and whether a rubric change can move it.

Two cautions on the arithmetic. First, do not exclude judge errors or refusals from the denominators, or the costs shrink as reliability falls the OneUptime guide. Second, the same verdicts can support wildly different agreement numbers depending on the scale, retained cases and pooling choices, and one analysis found protocol choice alone moved reported accuracy from 0.551 to 0.899 without changing a single verdict the agreement metrics paper. Your cost matrix should therefore be published next to the raw counts, so a reader can recompute it.

Choosing the metric that sits beside the matrix

You need one agreement statistic to summarize the log and one classifier view to explain it. A working pairing:

NeedMetricWhy
Headline agreement for a stakeholderRaw percent agreementReadable, comparable to the human-human baseline
Agreement beyond chanceCohen's kappa (weighted if labels are ordered)Penalizes a judge that always picks the majority label
Where the errors areConfusion matrix plus per-class precision and recallShows direction and severity
Whether the rubric is even usableHuman-human kappa on the same itemsIf this is low, stop

The kappa logic matters because accuracy lies on skewed data. If 80 percent of your agent's responses are correct and the judge always says correct, it scores 80 percent accuracy and a kappa of zero, which is exactly the behavior that ships regressions through a green CI pipeline the AWS GEDD kappa guide. Report human-human agreement on the same items as context, not as a ceiling the judge must beat the Arize field guide. For multi-class or agent-trajectory judging, treat the judge as a classifier and report precision, recall, F1 and class counts alongside the matrix the Arize field guide.

One further wrinkle for binary verdicts: Pearson's r, Spearman's rho, Kendall's tau-b, the phi coefficient and the Matthews correlation coefficient are exactly the same statistic, so reporting several of them repeats one number under different names the agreement metrics paper. Pick one, state it, and spend the saved effort on the confusion matrix.

If you want to compare judges from different vendors or model families on the same log, the practical shortcut is one API for all of them. Quanta Inference (https://quantaintelligence.ai/inference) serves text, image, video, audio and embedding models through a single OpenAI-compatible key, which makes it cheap to re-run your frozen calibration set against a second judge whenever you revise the rubric.

Repair the rubric, then re-measure the same set

The output of a review cycle is not a verdict on the judge. It is a revised rubric, a set of few-shot boundary examples and a list of splits you expect to disappear.

Concrete repair moves, in the order that usually pays off:

  • Promote every both-defensible split into a labeled boundary example in the judge prompt. Few-shot prompting is one of the documented techniques for improving judge accuracy, alongside chain-of-thought reasoning and reference-based scoring the Confident AI guide.
  • Ask the judge for a short rationale before the verdict. Structured chain-of-thought in the evaluator prompt makes disagreements easier to diagnose because the judge states why it scored as it did LangChain's judge calibration guide.
  • Replace three-point severity scales with a binary verdict plus an error code where severity mismatch dominates. Nominal error codes are easier to agree on than an ordinal scale, and unweighted kappa is the correct statistic for nominal labels the OneUptime guide.
  • If the judge runs on sampled items rather than every trace, keep the same stratification so the pass and fail rates stay comparable Statsig.
  • Freeze the calibration set and never tune on it. Sample from the same workflows the judge will score, and keep that set separate from the examples used to write or tune the prompt the OneUptime guide.

Then re-run and re-cost. The signal to watch is not the headline agreement number, which can move for reasons unrelated to quality, but the cost total of the error code you targeted. If missed policy breaches drop from 9 to 3 and nothing else moves, you have earned the right to automate that judge on routine traffic with human review on the high-risk stratum. Teams that treat evaluation as continuous infrastructure rather than a one-time setup report materially better reliability than teams that abandon judges altogether Splunk's comparison of judge and human evaluation.

A note on where the disagreement log lives. If your judge evaluates agents rather than single responses, the unit of judgment matters: score the whole trajectory of tool calls and reasoning, not just the final message, because a correct answer can hide a broken process, and a judge calibrated on final outputs alone will agree with humans on the easy half and miss the rest LangChain's evaluation framework. Google's framing of the same problem names three layers worth testing separately: the final output and user experience, the trajectory and tool selection, and trust and safety behavior under adversarial input. Start with human evaluation on a debatable set, then align the judge to those labels before relying on it Google Cloud's methodical approach to agent evaluation.

FAQ

FAQ

How many labeled items do I need before I trust the comparison?

Aim for 15 to 20 annotated items per error code, and do not read a kappa value computed from fewer than 10, where the confidence interval is too wide to support a decision the AWS GEDD guide. Stratify the set so rare high-risk cases are represented, since a uniform sample will underrepresent exactly the cases your cost matrix cares about Statsig.

The judge and the human disagree. Who is right?

The verdict supported by the response text, the pinned rubric version and the evidence wins, and anything the rubric cannot decide is a rubric bug rather than a rater error. Record that case as ambiguous, keep both original verdicts, and promote the case into the judge prompt as a boundary example the OneUptime guide. Do not let an adjudicator overrule a case without writing down the requirement that decided it.

Why not just use percent agreement and skip the cost matrix?

Percent agreement tells you how often the two raters matched, not whether the match is meaningful, and it can look strong when one label dominates the data the NHI explainer. The cost matrix answers a different and more useful question: which side's mistakes are expensive, and which error code carries that expense. You need both, reported side by side.

Our judge scores well but ranks models badly. What is going on?

Ranking is more fragile than agreement. In one study, judges with strong human alignment by kappa were beaten at ranking exam-taker models by a simple lexical matching method with far lower human alignment the Judging the Judges paper. The usual cause is severity mismatch, where the judge and human agree that something is imperfect but disagree on how imperfect, so replace ordinal scales with binary verdicts plus error codes where that pattern appears.

Can I skip human review once the judge clears a threshold?

Only for the strata where the cost matrix says a judge error is cheap. Keep human review on the error code that dominates your cost total, and keep the calibration set frozen so you can re-measure after any prompt or model change the Arize field guide. If you would rather hand the operational side to someone who does this continuously, a consulting call through Quanta Experts (https://quantaintelligence.ai/agents) is a reasonable next step: each expert reads your context and returns finished deliverables, including a calibrated rubric and a scored disagreement log.

Sources

Sources

  1. How to Calibrate an LLM-as-a-Judge Against Human ...
  2. cohens-kappa-for-llm-judges.md - sample-GEDD
  3. Stratified Sampling for AI Evaluation: Improve Metrics ...
  4. How to Measure Human-LLM Judge Alignment
  5. Evaluating Alignment and Vulnerabilities in LLMs-as-Judges
  6. What is the difference between percent agreement and ...
  7. Agreement Metrics for LLM-as-Judge Evaluation
  8. LLM-as-a-Judge Simply Explained: The Complete Guide ... - Confident AI
  9. How to Calibrate LLM-as-Judge with Human Corrections
  10. LLM-as-Judge vs. Human Evaluation: When to Use Each ...
  11. LLM Evaluation Framework: Trajectories vs. Outputs - LangChain
  12. A methodical approach to agent evaluation | Google Cloud Blog
About the author

About the author

Antonio Evans
Antonio Evans

Antonio Evans is the founder of Quanta Intelligence and a fractional CTO who builds HIPAA-ready AI systems for health and life-science companies. Before that he was technical co-founder of FriendsEAT for twenty years, CEO of Socialdraft, and a data engineer at IBM. Antonio lives with his dog Santino in Lisbon, Portugal.

Related articles

All articles