Designing Reviewer Abstention: A Band Between Pass and Fail

Replace the single cutoff with a band. Pick a lower threshold L and an upper threshold H on the reviewer's score, where higher means more confident the item passes.

The band is the whole design. Its width is a product decision with two consequences: a wider band raises the abstention rate and costs you routing capacity, while a narrower band pushes more items into confident verdicts that are more likely to be wrong. Start wider than you think you need and tighten as you gather labeled outcomes. The human-in-the-loop design patterns guide makes the same recommendation for confidence thresholds generally: set them conservatively at first, err toward routing more cases to review, then tighten as you accumulate labeled corrections and can measure the actual error rate on borderline items.

A minimal implementation looks like this:

LOW, HIGH = 0.35, 0.65

def decide(score, item):
    if score < LOW:
        return "fail"
    if score > HIGH:
        return "pass"
    return "abstain"

That is not the interesting part. The interesting part is that L and H should not come from a symmetric guess. Derive them from consequences. If a false pass costs ten times more than a false fail, push H up until almost everything below it abstains or fails. Classifier calibration work explains why a single ROC-optimized operating point is the wrong tool here: it assumes one cutoff separates the classes optimally, and borderline cases are precisely the ones that assumption breaks on.

The score itself must be a probability you can trust, not a raw logit or an arbitrary rubric total. Scikit-learn's calibration guide describes reliability diagrams, which plot the fraction of positives against predicted probability per bin; if items scored at 0.6 actually pass 45% of the time, your band is misplaced. For LLM judges, calibration matters even more because scores are sensitive to prompt phrasing, label order, response length, and position, as the LLM-as-a-judge survey documents. An uncalibrated judge's 0.5 is a number, not a probability.

Abstention only helps if the score is honest about uncertainty. Three practical choices matter.

First, prefer direct scoring with explicit criteria over a single holistic number. Evidently's guide separates pairwise comparison from direct scoring by criteria, and notes that judges can be built for both offline and online evaluation. Direct scoring against a rubric gives you a number you can compare across items and tune; pairwise comparison gives you a ranking with no natural band.

Second, ask for the uncertainty explicitly. A judge prompt that returns only "pass" or "fail" cannot abstain coherently. Ask for a score and a one-line reason, then derive the band from the score. Arize's guide recommends starting from the failure you need to catch, using code for deterministic checks and an LLM judge only where interpreting meaning is required, and validating the judge against reviewed examples before relying on its scores. That validation step is where L and H come from.

Third, watch for the biases that flatten discrimination inside the band. Judges favor their own generations and drift with prompt casing, per Patronus's LLM-as-a-judge tutorial, which also lays out the components a judge needs: input contextualization, comparison to standards, multi-step reasoning, explanation generation, and score synthesis. A judge missing any of those produces scores that cluster mid-range, which widens your effective band without you choosing to.

A judge that cannot tell its own output from a correct answer is a special case of this. A repository-grounded judge trained to verify review comments reached 76.60% accuracy on a 359-instance test set, beating a strong general model by 6.13 points, according to CRJudgeBench. Even a purpose-built judge misjudges roughly one item in four. Design the band assuming that error rate, not zero.

An abstention is a data point, and if you do not record it you cannot tune the band, argue about it, or prove what happened. Log one row per decision, with abstentions flagged.

{
  "item_id": "pr-1842",
  "category": "auth-change",
  "score": 0.52,
  "band": [0.35, 0.65],
  "decision": "abstain",
  "reviewer_id": "reviewer-a",
  "model_id": "judge-model-v3",
  "prompt_hash": "9c1f...",
  "rubric_version": "auth-rubric-7",
  "reviewer_round": 1,
  "decided_at": "2026-05-14T09:12:44Z",
  "routed_to": "reviewer-b",
  "final_outcome": "pending"
}

The fields that earn their place:

Storing the band alongside the score, rather than only the thresholds, means historical rows stay interpretable after you retune. If you only store the score, a later reader cannot tell whether 0.5 was inside the band when the decision was made.

This is the same discipline the verification literature asks of AI reviewers generally. The verification guide frames review as a mechanical pipeline with the compiler, diff, and test suite acting as referee, and it treats auditability as part of the pipeline rather than an afterthought. An abstention you cannot reconstruct is not an audit trail.

The default route for an abstention should not be a human. Send it to a second reviewer first. The reason is the maker-checker principle: the reviewer that produced the original score shares its blind spots with any retry that uses the same model and same prompt. The adversarial code review guide documents the evidence that self-review fails, because LLMs measurably favor their own output, and it lays out a control model where a distinct checker with different context, instructions, and often a different model reviews the change.

So the second reviewer should differ on at least one axis:

• Different model. A different model family can repeat fewer of the first reviewer's errors. Use a gateway that reaches several model families through one key so the second reviewer is a config change rather than a new integration.
• Different prompt. A skeptical prompt that asks the reviewer to hunt for reasons the first verdict is wrong, rather than to re-score.
• Different context. A fresh context window without the first reviewer's reasoning, so it cannot anchor on it.

If the second reviewer returns a confident verdict (below L or above H for its own scale), accept it and record the round. If the second reviewer also abstains, or if the two reviewers reach opposite confident verdicts, the item goes to a human queue. The human-in-the-loop patterns guide distinguishes blocking from non-blocking review: a blocking pattern halts until a human responds and suits irreversible actions, while a non-blocking pattern queues low-confidence cases and continues processing. Route to the queue asynchronously unless the item is a privileged or irreversible change.

Human attention is not uniform either. Not all humans-in-the-loop are equal maps the human step on expertise and effort, from low-effort verification ("did it work?") to high-effort wisdom ("what should we do?"). Abstentions are usually verification work: a quick check against intent. Send them to someone who can confirm quickly, and reserve the expensive discernment work for disagreements between reviewers.

The rule you need is not "route abstentions to people." It is "route abstentions to people when a signal says the automated path is failing." Two signals do the work.

Per-category abstention rate. Track the share of items in each category that abstain, over a rolling window. Set a limit, say 20% for a category. When the rate crosses it, every new abstention in that category goes straight to the human queue and pages the owner. This is a circuit breaker against two failure modes: a model upgrade that shifted the score distribution, and a category the reviewer was never good at. Without it, a drifting reviewer quietly converts its whole workload into human work and nobody notices until the queue is days deep.

Reviewer disagreement. When two reviewers reach opposite confident verdicts, the automated path has failed on that item by definition. Send it to a human and tag it as a disagreement so you can count them separately from ordinary abstentions. A rising disagreement rate between two specific reviewer versions is the clearest early warning that one of them needs attention.

A worked example makes the arithmetic concrete. Suppose a reviewer handles 1,000 items a day across four categories, and a first pass lands 60% confident pass, 25% confident fail, and 15% abstain. That is 150 abstentions daily going to the second reviewer. If the second reviewer resolves 80% of them confidently, 120 items resolve automatically and 30 reach humans. Now suppose a model upgrade shifts the distribution so abstentions jump to 30% of a category. The per-category limit fires, that category's abstentions go straight to people, and the team learns about the shift on day one rather than after a week of queue growth. This kind of routing and logging is ordinary engineering, but it has to be built deliberately; the secure code review agent guide makes the broader point that an automated reviewer with access to source and CI is a privileged component and should be architected as one.

A band is not a setting you choose once. It is a parameter you estimate and then re-estimate as outcomes accumulate.

The process is simple. Take the abstentions your humans resolved and treat their verdicts as labels. Plot a reliability diagram: bin the items by the reviewer's score and compare the fraction of items the humans passed in each bin against the score. If items scored 0.4 pass 50% of the time, your lower threshold is too high. If items scored 0.7 pass 95% of the time, your upper threshold is too low. Move L and H to sit where the calibration curve crosses your acceptable error bars for each side.

Two rules keep this honest. First, never tune the band on the same items you used to set it, or you will overfit to a week of unusual traffic. Second, version the band the same way you version prompts, and log the version with each decision, so a score recorded under one band is not compared carelessly to one recorded under another. The calibration guide from scikit-learn is a good reference for the mechanics of reliability diagrams and for the calibration methods you can apply when a judge's scores are not probabilities at all.

There is a deeper design question underneath the tuning: what the band is for. The rubric research on LLM judges found that models generate interpretable, task-aware evaluation dimensions and apply them consistently within a model, but scoring reliability degrades in factual and knowledge-intensive settings, and it is fragmented across models. That suggests the band should be wider for knowledge-intensive categories, where a judge's score is less trustworthy, and narrower where the judge is well grounded by tests or compilers. A single global band across all categories ignores the evidence.

Finally, do not let the band absorb a broken reviewer. If abstentions rise because scores clustered, and the calibration curve shows the reviewer no longer discriminates, widening the band hides the problem. The band is a decision rule, not a repair.