How to Write a Reviewer Prompt That Abstains

The danger with cannot determine is that it becomes the easiest button to press. Judges drift toward it whenever a check requires effort, and you end up with a large pile of non-answers instead of a small, high-signal exception queue. Three prompt rules keep it honest.

Rule one: define the option by evidence, not by difficulty. The check should say that cannot determine is available only when the required evidence is absent from the material provided, and never when the evidence is present but ambiguous. Per Hebbia's description of its rubric design, each criterion should target a single, diagnosable failure mode. If a checker can point to the exact passage that would settle the question and that passage does not exist in the input, abstention is correct. If it can point to two conflicting passages, that is a fail, not an abstention, because the evidence exists and the work is wrong.

Rule two: require a missing-evidence field. The judge cannot submit cannot determine alone. It must complete a sentence describing what would have made the check decidable, such as "no changelog entry for any of the three versions in scope". This turns an abstention into a request that a human can act on.

Rule three: put the burden on the judge to prove absence. Ask it to list where it looked, for example "searched the reference section, appendix and inline citations; found none". A judge that must document its search cannot quietly skip the work.

One more constraint belongs in the prompt: cap abstentions per item. If more than about one in five checks cannot be determined, treat that item as out of scope for automated review rather than judging it partially. A partial review that passes the rest is worse than an honest escalation, because the numbers look better than the coverage.

A verdict you cannot count by criterion is a verdict you cannot learn from. The fix is mundane and takes ten minutes: name each check once, in a machine-readable form, and never reuse the name for a different meaning.

CHK-01 Summary states the document's main claim
CHK-02 Summary cites at least one primary source
CHK-03 Numbers in the summary match the source table
CHK-04 Tone follows the style guide's register rules
CHK-05 No factual claim appears without a reference

The identifier does three jobs. It lets you compute an abstention rate per check, which is the single most useful diagnostic you get from this setup. It keeps log records joinable across prompt versions, so when you rewrite CHK-02 you can compare its abstention rate before and after without guessing whether the criterion moved. And it makes routing specific: the reviewer sees CHK-02 and CHK-03 on an item, not "quality concerns".

This matches a wider lesson from rubric reliability work. When multiple raters apply a rubric and disagree, research on the statistical methods for rubric inter-rater reliability points out that the strength of measured reliability depends heavily on which statistic you choose and on how clearly the criteria are specified. A criterion that shifts meaning between versions destroys your ability to measure anything. Stable IDs are the cheap precondition for those measurements.

A useful side effect: per-check abstention rates tell you which criteria are badly written. A check with a 40 percent abstention rate is usually not a hard check, it is a vague one, and rewriting it to name the exact artifact it needs is the fix. A check with a near-zero rate across thousands of items may be too easy to be worth running.

The log is where most teams lose the benefit. A record that says "abstained" is useless. A record that says which criterion could not be evaluated, on which item, for want of which evidence, under which prompt version, is a work item.

CREATE TABLE review_abstentions (
  id            BIGINT PRIMARY KEY,
  item_id       TEXT      NOT NULL,
  check_id      TEXT      NOT NULL,   -- CHK-02
  missing       TEXT      NOT NULL,   -- the named absent evidence
  searched      TEXT      NOT NULL,   -- where the judge looked
  prompt_version TEXT     NOT NULL,
  created_at    TIMESTAMP NOT NULL,
  owner         TEXT,                 -- set by the router
  due_at        TIMESTAMP,            -- created_at + time box
  resolution    TEXT                  -- pass | fail | scope change
);

Five fields do the heavy lifting, and each earns its place. The check ID lets you aggregate by criterion. The missing evidence field is what the human reads first, so it should be one sentence in plain language, not a paragraph. The prompt version is the join key that lets you ask whether a rewrite reduced abstentions, and without it, a prompt change looks identical to a traffic change. The owner and due_at fields turn the log into an assignment rather than a report. The resolution field closes the loop, because an abstention where the human decided "pass, the evidence is in an appendix the judge was not given" is a different fix from one where the human decided "fail, the source is genuinely absent".

Two derived numbers should sit on a dashboard from day one. The first is abstentions per hundred items overall, which is your supervision load. The second is the share of resolved abstentions that ended in pass, which is your false-alarm rate. A high false-alarm rate on a single check means the check is asking for evidence you rarely supply, and either the input packaging or the criterion needs to change. This is the same discipline that evaluation guidance calls for when it warns that unambiguous criteria produce stable scores and ambiguous ones flip between runs, as Splunk's agent evaluation framework describes when it recommends validating rubric stability across independent runs before automating decisions.

The design goal is that an abstention never blocks the rest of the item. The other checks resolved, the item moved through those, and only the unresolved checks wait. That is the difference between a pre-triaged exception and a full re-review.

Route by check, not by item. If CHK-02 and CHK-03 are about sourcing and numerics, they go to whoever owns evidence standards for that content type. If CHK-04 is about tone, it goes to the editorial owner. This keeps each reviewer's context narrow, which matters because the supervisory load in review work is not the volume of items alone but the variety of unrelated judgments a person has to hold at once, as discussed in research on the supervisory paradox in AI delegation. Every extra category of question in one queue is a switch cost.

Give each route a time budget, and publish it. A workable default is one business day for content checks and four hours for anything touching a regulated claim. The time box is not a service-level target for its own sake; it caps how long an item sits in a partially resolved state, which is where review queues rot. If the box is missed, the item escalates to the route's backup owner rather than waiting.

Use a fixed review mode for abstentions. A reviewer resolving an abstention is not re-reading the whole item. They are answering one question: does the named evidence exist, and if so, where. That is a human-in-the-loop pattern in the narrow, per-item sense, where a person resolves a specific flagged decision before it is finalized, which the comparison of human-in-the-loop and human-on-the-loop describes as appropriate for high-stakes, lower-volume decisions where accuracy matters more than speed. Abstentions are exactly that slice.

Feed resolutions back into the input. The most common cause of repeated abstention is a missing document, not a bad judge. If CHK-03 keeps abstaining because the source table never reaches the reviewer prompt, the fix is to attach the table, and the abstention rate for that check drops without touching the model. Teams that track this find that a meaningful share of their exception volume disappears after two or three packaging fixes.

Keep a human calibration sample. Abstention logs tell you where the judge gave up, not where it was wrong. Set aside a small random sample of fully resolved items for human spot-checks, and compare. This is the same hybrid approach Hebbia describes, pairing deterministic property checks with rubric grading and human calibration so that rubric ambiguity surfaces as disagreement rather than as quiet error.

Suppose a pipeline reviews 10,000 documents a month against five checks, so 50,000 check verdicts. Before the change, 4 percent of items came back with a quality complaint, and the reviewers could not tell which checks caused them.

After adding cannot determine with a missing-evidence field, the first month shows 1,800 abstentions, or 3.6 percent of check verdicts. That looks like a large number until you break it down by check. Suppose CHK-03 accounts for 1,100 of them because the source table is attached only half the time. Fixing the attachment cuts those to roughly 200, and total abstentions fall to about 900, or 1.8 percent. Each of those needs one reviewer decision, which at three minutes per abstention is about 45 hours a month, spread across named owners with a published time box.

The insight is the gap between those two numbers. Without per-check logging, the team sees 1,800 exceptions and assumes the judge is weak. With it, they see one packaging bug wearing a judge's coat. The check ID, not the model upgrade, is what turned 1,800 problems into one.

Abstention is for missing evidence, not for hard calls. Three cases need a different design.

Subjective dimensions. If a check asks whether tone is "appropriate", no amount of missing-evidence discipline will make it decidable. Rewrite it into observable conditions, the way Hebbia's rubric principle converts a vague "good title" into a specific word-count and no-units rule. A decidable check rarely needs to abstain.

Contested ground truth. Where two experts would genuinely disagree about the right answer, the problem is the criterion, not the evidence. Send those to a calibration session and split the check until each part has one defensible answer. The human-in-the-loop scoring framework used in a national writing assessment makes this explicit: agreement between model and human varies by rubric dimension, and the workflow is designed to send to humans the cases where review adds the most value rather than all of them.

Very high abstention rates. If a check abstains on a third of items, stop judging it and audit the input instead. A criterion that is unanswerable a third of the time is usually measuring a document type you do not receive.

Once the prompt, the log and the routes are in place, the remaining work is operational: watch the per-check rates weekly, rewrite the worst criterion, and keep the human sample running. Teams that want one API key across the judge models and the downstream text, vision or embedding calls they need can point the pipeline at Quanta Inference and keep the abstraction stable while the judge model changes underneath.