When Two Reviewers Disagree: A Tie-Breaking Rule for Scored Pairs

Confidence is a statement about the reviewer's own certainty, not about the item's quality. Those are different variables and conflating them is the most common mistake in tie-breaking.

Research on AI conference reviews found that confidence and review text are highly consistent at the word, sentence and aspect level. It also found a negative correlation between reviewer confidence and paper outcomes, meaning higher confidence was associated with rejection in that dataset (Wu, Xi and Zhang on reviewer confidence consistency). Read that carefully: the finding is about the correlation between confidence and acceptance, not evidence that confident reviewers are wrong. It is a warning that confidence carries signal you did not necessarily intend.

For tie-breaking, you need three properties from the confidence field:

1. It was captured before discussion. If reviewers adjust confidence after seeing each other's scores, the field is contaminated. Capture it at submission.
2. The scale is defined. A three-point scale (low, medium, high) is enough. Five points invites false precision. What matters is that everyone means the same thing by "high."
3. It is independent of the score. A reviewer must be able to say "score 2, high confidence" and "score 4, low confidence" without feeling inconsistent.

The third point is where most rubrics fail. If your form implies that high scores go with high confidence, the field tells you nothing. Application rubric design guidance is blunt about this: anchor descriptions must describe what the evidence at a level looks like, and adjacent levels must stay distinct so reviewers are not inventing the difference between "good" and "very good" (Sopact on scoring rubric anatomy and anchors). The same discipline applies to the confidence anchors. "High" should mean "I can point to the specific evidence and I checked the whole item," not "I feel strongly."

A checklist-based review beats a holistic score on reliability. CheckEval, a checklist framework that decomposes evaluation criteria into binary questions, improved average agreement across evaluator models by 0.45 and reduced score variance, while remaining interpretable because each decision traces to a specific criterion (Lee et al., CheckEval). The lesson transfers directly to human reviewers: the note attached to a checklist item is evidence, and evidence is what a tie-breaker should weigh.

So the notes field is not a courtesy. It is the mechanism. A useful note names the checklist item and points at the passage that decided it. Something like:

checklist_item: 3.2 (cites a source for every factual claim)
decision: fail
evidence: paragraph 4 states a failure rate with no source
confidence: high

Compare that with a note that reads "felt thin overall." The first survives a tie-break. The second does not, and it should not, because it cannot be checked by a third party.

This is the point where reviewers often push back: "my judgment is holistic, I cannot itemize it." You can require itemization for disputed items only. Reviewers score freely, and when a disagreement appears, the resolver asks one question: which checklist item does your note cite? If the answer is none, that score loses the tie-break regardless of confidence. This keeps the routine case fast and puts the rigor exactly where the conflict is.

Here is the rule, stated so it can be pasted into a review guide.

Step 1: Compare confidence. The reviewer with the higher confidence level wins, and the higher-confidence score becomes the item score. If both reviewers chose the same confidence level, go to step 2. If the lower-confidence reviewer's note cites no checklist item, the item is closed here.

Step 2: Read the notes for a checklist citation. If the lower-confidence reviewer's note names a specific checklist item and points to the evidence that decided it, check whether the higher-confidence reviewer addressed that same item. Two outcomes:

• The other reviewer missed the item entirely, scored it differently, or did not mention it: the lower-confidence score wins. Confidence cannot compensate for an unchecked criterion.
• Both reviewers cite checklist items, with contradictory readings: go to step 3.

Step 3: Average and round down. When both notes cite checklist items and the readings genuinely conflict, take the mean of the two scores and round down. A 3 and a 4 become 3.5, which rounds to 3. A 2 and a 3 become 2.5, which rounds to 2. For binary pass/fail items, round down means fail.

The rounding direction is a deliberate policy, not a mathematical convention. An item two trained reviewers cannot agree on is not a clean pass. Rounding down encodes that, and it is consistent with treating unresolved questions as unresolved rather than converting uncertainty into approval (Rubrex on retaining an unresolved state).

The table below restates the decision in checklist form. Print it.

Two guardrails. First, never let a reviewer revise a note after seeing the other note without timestamping it; the whole rule depends on independence. Second, an escalated case (last row but one) is not a scoring problem. It is a signal that the rubric permits more than one reasonable reading, and the fix is a rubric change plus a decision on whether to rescore affected items. Mixing a label correction with a rubric change is how evaluation baselines quietly break (Rubrex on classifying the cause before choosing a resolution).

Two reviewers score a support-answer batch against a five-item checklist on a 1 to 5 scale. Four items disagree.

Item A. Reviewer 1 scores 4 with high confidence, note reads "solid answer, clear steps." Reviewer 2 scores 2 with low confidence, note reads "step 3 tells the user to delete cached credentials, which the policy forbids." Step 2 applies: Reviewer 2 cites checklist item 4.1 (policy compliance) and quotes the violating step. Reviewer 1's note addresses clarity only. Final score: 2. Roughly 40 seconds.

Item B. Reviewer 1 scores 3 with medium confidence, note cites item 2.3 (correct regional exception). Reviewer 2 scores 4 with medium confidence, note cites item 2.3 as satisfied by a different sentence. Both cite the same item with opposite readings. Step 3: mean is 3.5, rounds down to 3. Roughly 50 seconds.

Item C. Reviewer 1 scores 5 with high confidence, note reads "excellent." Reviewer 2 scores 3 with low confidence, note reads "seems incomplete." Step 1: Reviewer 1's higher confidence would win, but check Reviewer 2's note. It cites no checklist item. Reviewer 1's score stands at 5. This is the uncomfortable case: a vague low-confidence objection does not beat a confident high score, and the resolver should not rescue it. Instead, flag Reviewer 2 for calibration on what a usable note looks like. Roughly 20 seconds.

Item D. Reviewer 1 scores 2 with high confidence, no checklist citation. Reviewer 2 scores 4 with low confidence, note cites item 5.1 (states limitations) and quotes the passage. Step 2: lower confidence but a specific citation that Reviewer 1 never addressed. Final score: 4. Roughly 30 seconds.

Total: about two and a half minutes for four disputed items, no meeting. Note that the rule did not always favor the confident reviewer. It favored the reviewer who could be checked.

The rule resolves the item. The record should resolve the pattern. For every disputed item, log five fields: both original scores, both confidence levels, both notes verbatim, the step that resolved it, and the final score. That is the decision record, and its purpose is to answer a later question you cannot answer from the final score alone.

A useful test: could a third reviewer reproduce the resolution from the log without asking either original reviewer anything? If not, the note was not specific enough, and the next training cycle should say so.

Use the log for three things. First, calibration. Recount how often step 3 fires. A high rate of averaged disputes means the rubric has ambiguous anchors, not that reviewers are careless. Anchor sets are chosen precisely to make score points concrete where the rubric alone cannot, and calibration against shared examples is how readers get lined up (Angie Kratzer on anchor papers). Build a small set of disputed items that took step 3 and use them as anchor cases for the next round. Second, coverage. Recount how often step 1 closed a case because one note cited nothing. That is a note-quality metric, and it is more actionable than an agreement rate. Third, rubric repair. Every escalated case becomes a candidate rubric edit, and when you edit the rubric, version it, because a rubric change alters what counts as acceptable and may require rescoring (Rubrex on versioning rubric changes).

If your team is also evaluating model output at volume, the same log doubles as evidence of the human reference against which a model judge gets validated. The AI Engineer evals reference makes the point that a model judge should be checked against a defensible human procedure and that assessment coverage belongs beside performance numbers (AI Engineer's evals reference). Your disagreement log is that human procedure, written down.

Three cases fall outside it.

The item is a safety, policy or compliance gate. A confident reviewer cannot outvote a specific policy citation, so step 1 does not apply. A cited gate failure always wins, and a cited gate pass against a cited gate failure escalates rather than averaging. Averaging a compliance failure into a pass is exactly the wrong direction of error.

The disagreement is about the rubric, not the item. If the two notes cite the same checklist item with irreconcilable readings, averaging produces a number but not an answer. Treat it as a rubric defect and resolve the rule first.

The item carries outsized weight in a decision. When a single item can flip a selection outcome, proportionality argues for spending more time on it (NAIC's description of proportionality in AI evaluation). In that case, escalate to the decision owner once, and record the escalation.

Beyond those three, keep using the rule. Its value is that it is boringly predictable, and predictable beats persuasive when two honest reviewers disagree. If your team would rather hand the first pass to an expert that reads your context and returns finished work, Quanta Experts is built for that division of labor (https://quantaintelligence.ai/agents). Then your two reviewers only handle the cases the rule says need them.