The Arbitration Log: What to Record When Reviewers Clash

Build the log so that one row equals one resolved conflict. The primary fields are:

Two fields carry more weight than they look.

The reason field should be a closed list, not free text. Useful values include rubric_ambiguity, missing_evidence, factual_error, severity_difference and unresolved. The adjudication literature draws the same distinctions: a missed explicit rule is a label correction, a rule that permits two readings is a rubric change, and evidence that neither reviewer could establish stays unresolved (Rubrex on classifying causes). Encoding those as enum values lets you count them later. If rubric_ambiguity dominates, the problem is the rubric, not the reviewers.

The confidence field should never be discarded, even though it is imperfect. A study of review text found that reviewers sometimes give high confidence while expressing reservations, and low confidence alongside detailed, well-founded critique (Wu, Xi and Zhang on confidence consistency). Research also shows confidence can be estimated from review text itself, which suggests it is a real signal rather than a decorative checkbox (Bharti et al. on reviewer confidence). Logging it next to the score is what lets you ask, later, whether high-confidence outliers predict overturned decisions.

Keep the schema flat and append-only. Never mutate score_a or score_b after the fact. If a reviewer self-corrects, write a new row with resolves pointing at the original arbitration_id. Append-only tables are cheap to audit and impossible to quietly rewrite.

Anika scores a support-policy answer 2 (incorrect) because it states the general refund window but omits a regional exception. Marcus scores it 4 (correct) because the general policy is true and the question never named a region.

Read the confidences, not just the scores. Anika is confident her reading is right; Marcus is less sure his is sufficient. That pattern is itself evidence that the instruction is underspecified, not that one reviewer erred. The correct response is not to pick a winner and move on but to version the rubric and rescore affected cases, which is exactly the separation between label correction and rubric change that adjudication guidance recommends (Rubrex).

The decision rule here is concrete: if both reviewers have confidence at or above a set threshold and the reason is rubric_ambiguity, open a rubric change rather than a one-off correction. If one reviewer's confidence is low and the other's is high, treat the high-confidence review as the working interpretation until evidence says otherwise.

Marcus scores a methods summary 5 (strong pass). Priya, resolving, scores it 2 (reject) because the summary claims the experiment established causation, and the underlying study reports only association. This is not ambiguity; it is a checkable error.

Marcus's confidence of 5 is a red flag worth counting. A high-confidence reviewer who conflicts with a checkable fact is exactly the case a confidence-aware queue should rank first. This is also where a third state earns its keep: when neither reviewer can establish applicability, record unresolved rather than forcing a binary. Converting uncertainty into a passing answer because the release process expects one is a known failure mode (Rubrex on retaining an unresolved state). Use a separate column rather than hiding it in the note.

The log is only as good as the scale it records. Two design choices pay off.

First, prefer decomposed checks over holistic Likert ratings where you can. Evaluator models show low agreement and high variance on subjective Likert scoring, and a checklist of binary questions improved average agreement across evaluator models by 0.45 while reducing variance (CheckEval). Even if your reviewers are human, the lesson transfers: a score built from "did it cite a source, did it state scope, did it avoid an unsupported causal claim" is easier to arbitrate than a single "quality: 1 to 5."

Second, use anchored level descriptions rather than vague labels. "Poor, average, excellent" names levels without explaining how to assign them; an anchor describes what the evidence at a level looks like so a reviewer can point to a passage and say why it matches (Sopact on scoring rubrics). Anchor sets serve the same purpose in scored writing: they make a subjective process slightly more objective and are used to calibrate readers to a common standard (Kratzer on anchor papers). Store the rubric version in the log so a later reader knows which anchors applied.

There is a harder question hidden here: should the log measure reviewer conformity or decision quality? A position paper argues that AI evaluation should shift from solo performance against a human baseline to the performance of human-AI teams, since modern work is a joint cognitive system rather than a person versus a machine (AI Evaluation Should Work With Humans; McKenna on joint cognitive systems). Apply the same lens to reviewers. The metric is not "did the second reviewer match the first" but "did the two-plus-adjudicator system reach a defensible call." A log built to measure agreement alone will push reviewers toward consensus and hide real disagreement, which is the opposite of what you want.

You can also store machine judgments in the same table. If an LLM judge or a scripted check scored the item, log its output and confidence as just another reviewer row with a reviewer_type of model. Evaluation work increasingly combines code checks, LLM-as-a-judge, human review and production sampling, because no single method covers every failure mode (Arize on LLM evaluation). One table for all judges makes the arbitrations comparable.

# Same table for humans and model judges.
row = {
    "arbitration_id": "ARB-2050",
    "item_id": "DOC-91002",
    "reviewer_a_id": "rev_anika", "reviewer_type_a": "human",
    "score_a": 2, "confidence_a": 4,
    "reviewer_b_id": "judge_v3", "reviewer_type_b": "model",
    "score_b": 4, "confidence_b": 0.81,
    "tie_breaker_reason": "severity_difference",
    "resolving_reviewer_id": "rev_priya",
    "resolved_score": 3,
    "rubric_version": "rubric-2026-02",
    "decision_at_utc": "2026-03-04T16:40:00Z",
}

Normalize model confidence to a 0 to 1 float and human confidence to its native integer scale, and record the scale in the rubric metadata. Do not rescale in the log itself or you lose the raw value.

The tempting policy is to keep everything forever. The practical policy is tiered, and the decision rule is about replay.

Tier 1, full record, live window. Keep every field for a defined active window (30 to 90 days for a fast-moving queue, longer for regulated review). This is when you rerun, rescore and audit adjudications. Storage is trivial at this scale because arbitration rows are a small fraction of all review rows.

Tier 2, structured record, midlife. Drop verbose free text and evidence snapshots, keep every typed field including the tie-breaker enum and both confidence values. Rows stay queryable for trend analysis and drift detection.

Tier 3, summary only, long tail. Keep arbitration_id, item_id, tie_breaker_reason and decision_at_utc. This answers "how often did rubric ambiguity drive conflicts in that period" without keeping the blow-by-blow.

A simple decision table helps:

Two operational rules keep retention honest. First, version the rubric and store the version on every row, so a rescoring pass knows which cases fall under the old rule; a rubric change alters what counts as acceptable and can require rescoring baseline and candidate outputs (Rubrex). Second, treat the arbitration log as a derived table that can be rebuilt from immutable review submissions, so tiering down the log never destroys the source of truth.

If you are running these arbitrations at volume and need a second, consistent judge in the loop, Quanta Inference puts text, classification and embedding models behind one OpenAI-compatible endpoint, which makes it easy to add a model row to the same decision log (Quanta Inference).

Run these before the log becomes load-bearing:

• Replay test. Take ten past conflicts and rebuild the decision from the log alone. If any require asking a person what happened, a field is missing.
• Enum coverage. Measure the share of rows that land in an other reason. If it exceeds a threshold you set, the enum is too coarse; refine it rather than letting prose fill the gap.
• Scale check. Confirm the same score means the same thing across reviewers by comparing against anchor sets, and record the calibration date.
• Unresolved rate. Track how often conflicts end unresolved. A rising rate usually means the rubric or the evidence packet is failing, not the reviewers.
• Append-only enforcement. Verify at the database level that scored columns reject updates.

The goal is not a perfect record. It is a record another reviewer can read cold and reproduce the call, with the confidence signals and the reason preserved in machine-readable form.