Writing Checklist Lines a Model Can Score Automatically

Take any imperative or open-ended checklist item and ask what the reviewer would have to see to mark it wrong. If you cannot answer that in one sentence, the line is not ready for a judge.

Two rules make this work in practice. First, one question per line. The Turing Way list separates style, tests, security, and legal requirements into distinct sections, which is the right instinct: a reviewer who checks tests and security in one pass conflates them, and so does a judge (The Turing Way). Second, an explicit escape hatch. "Not applicable" is not a cop-out; it is the answer that flags scope. A migration file has no HTTP input validation, so a security line about input handling should return not applicable and never yes. Without that third option, judges tend to manufacture a yes to satisfy the prompt.

A checklist line is ready when a second person, given only the line and the diff, reaches the same answer independently. Test this on twenty past pull requests before you build anything.

Give the judge a fixed output shape. Free prose responses are readable and useless for counting. Ask for a short structured record per line, one record per checklist item:

{
  "line_id": "tests.assertions_deterministic",
  "answer": "yes | no | not_applicable",
  "evidence": "quoted text from the diff that the answer relies on",
  "confidence": 0.0
}

The evidence field is the load-bearing part. An answer without a quote is an assertion; an answer with a quote is a claim someone can check in seconds. The legal grading review reaches the same conclusion, noting that evidence should be "checked in code" rather than trusted from the judge's prose (SoK). You can enforce this mechanically: reject any record where the answer is "no" and the evidence field is empty or does not appear verbatim in the submitted diff. That single validation catches most fabricated findings.

Confidence needs a source too. If your provider returns token probabilities, use them. If it does not, ask the judge for a three-level certainty (high, medium, low) and calibrate the levels against your own labeled sample rather than trusting the number at face value. LLM-as-a-judge systems are documented to drift between runs and to favor long answers, so the confidence field is a routing signal, not a measurement (survey on LLM-as-a-judge).

A sample prompt skeleton, kept short on purpose:

You are scoring one review checklist line against a code diff.
Answer only yes, no, or not_applicable.
Answer not_applicable if the line's subject does not appear in the diff at all.
Quote the exact diff text your answer relies on. If you cannot quote it, answer not_applicable.
Do not infer intent. Score only what is visible in the diff.

The instruction to score only what is visible is the guard against a judge that reasons about the author's intentions. Judges are also susceptible to instructions embedded in the text they grade, so treat the diff as untrusted input: never let the diff dictate output format (SoK).

The most common cause of disagreement between a model and a human is a line that hides two questions. "Are tests present and meaningful?" fails a diff that adds a test asserting assertTrue(True). The judge answers "yes, tests present" and ignores meaningfulness. Split it:

• Does the diff add at least one test file in the project's test directory? (yes / no / not applicable)
• Does every added test assert on a value derived from the code under test rather than a constant? (yes / no / not applicable)

Now each answer has a distinct evidence trail, and a disagreement points at one specific criterion. This mirrors how the sample checklists in circulation are already structured: the Turing Way separates "do the tests conform to the standards" from "do the tests actually ensure the code is robust in its intended use" (The Turing Way). Keep that separation when you hand the list to a model.

Split until each line can be answered from a single quoted passage. If answering a line requires weighing three parts of the diff against each other, it is still two lines.

The economics of this pattern are the reason to attempt it. LLM judges agree with human reviewers at reported rates around 85 percent in published measurements, which is a useful pre-screen and a bad final decision (Confident AI on LLM-as-a-Judge). Design the pipeline around that gap.

Worked example. Suppose your team reviews 400 pull requests a week and each one runs 25 checklist lines. That is 10,000 judgments. A human reviewer spends an average of 20 seconds per judgment when the answer is obvious and several minutes when it is not. If the model settles 80 percent of lines without controversy and flags 20 percent, the human queue is 2,000 items, and the reviewer spends their time only where the answers were uncertain or contested. The number that matters is not model accuracy in the abstract, it is how many items reach a human and how many of those turn out to matter.

Three routing rules do most of the work:

1. Model says no or not applicable, human says yes: send to human, the model may have missed context outside the diff.
2. Model says yes, human review of a sample says no: this is a judge defect, not a code defect. Fix the line wording or the prompt before sending more to review.
3. Confidence is low on either side: route to human regardless of agreement.

Repeated runs on the same input also help. Run the judge twice and compare the two sets of answers; disagreements between two runs of the same model on the same diff are the clearest signal that a line is ambiguous and needs rewriting (survey on LLM-as-a-judge).

Track two numbers from the start: human overturn rate on flagged items, and escape rate, meaning defects a human later found in lines the model passed. If the escape rate stays near zero across a few hundred pull requests, you can widen autonomy. If not, the checklist is the problem, not the model.

When the reviewers cannot keep pace, running several model families behind a single API key removes the integration overhead of comparing judgments. Quanta Inference exposes text and embedding models through one OpenAI-compatible endpoint, which is the practical way to test two judges against each other before you pick one: https://quantaintelligence.ai/inference

Not everything converts. Some lines are judgment calls whose value lies in the discussion they provoke, and automating them produces confident noise. Sort your list into three buckets before writing any prompt.

The AI agent security literature offers a useful model for the middle bucket. A vendor evaluation checklist for agents asks buyers to demand a named person approves irreversible actions and to look for an action log an admin can open, which is the same evidence-first posture applied to a checklist (Hatchworks). An agent that changes production data needs a log a human can reconstruct; a checklist line that says "no" needs a quote a human can open. The pattern is identical: make every automated claim checkable by someone who was not there when it was made.

Leave the open bucket alone. Converting every line is a common failure mode, and it produces a checklist that scores everything and catches nothing.

A checklist that is never revised drifts away from the codebase. The judge makes this visible, because every line now has a history of answers, overturns, and disagreements. Review that history on a schedule.

Watch for three patterns. A line whose answer is "yes" on every pull request for a month is either always satisfied (retire it) or too easy to satisfy (sharpen it). A line with a high disagreement rate is ambiguous (rewrite it). A line whose evidence quotes are always outside the diff is being answered from context the reviewer did not provide (add the context or drop the line).

Assign ownership. A checklist without an owner decays into a list of things nobody checked. Quartely, or after any incident where a review missed something, run the failing change through the checklist as a judge would and see which line should have caught it. If none did, that is a new line. If one did and the model answered yes, the line needs a sharper test.