A Checklist a Reviewer Can Apply in Under Five Minutes

Every pair gets exactly seven fields, on one page, in this order. If a field cannot be filled in, the pair is not ready to score.

Two disciplines keep this from bloating. First, the prompt and response are pasted, never paraphrased. Second, the notes field is mandatory when the reviewer scores anything other than the highest level, which stops low scores from appearing with no explanation. Evaluation practice in general stresses that the record of what system was run, on what work and with what evidence must survive the review, not just the final number the AI Engineer evals guide.

Order is not cosmetic. A reviewer who reads the response first anchors on it and then reads the prompt to justify what they already believe. Presenting context first removes that bias and cuts reading time, because the reviewer knows what to look for.

Use this fixed sequence on the page:

1. Task ID and the one-line description.
2. The model and settings.
3. The prompt, in a bordered box, with the relevant constraints emphasized.
4. The response, in a separate box, unedited.
5. The rubric, one criterion per line, each with three anchor examples.
6. The confidence box.
7. The notes box.

Keep the whole thing to a single screen or a single printed page. If the prompt or response is long, show the first few hundred words in the box and put the full text behind a link. A reviewer who has to scroll between a prompt and a response loses the thread, and the five-minute budget goes with it.

One practical addition: put the response in a fixed-width block so line breaks and tables are visible. Formatting errors are a common failure mode in generated text, and a reviewer cannot see them in a reflowed paragraph.

TASK 14: Summarize a customer complaint thread for a support agent.
MODEL: production default, temperature 0.2

PROMPT
-------
Summarize the thread in under 120 words. List the customer's
requested resolution first. Do not invent order numbers.

RESPONSE
--------
Customer wants a replacement unit shipped this week. They report
the device failed on day 3. Order reference: unknown.
[...continues...]

RUBRIC (score each 1 to 3)
--------------------------
Facts accurate ......... anchors 1 / 2 / 3 below
Request captured first . anchors 1 / 2 / 3 below
Length within limit .... anchors 1 / 2 / 3 below
No invented details .... anchors 1 / 2 / 3 below
Usable as written ...... anchors 1 / 2 / 3 below

CONFIDENCE: high / medium / low    Reason (one word):
NOTES (required if any score is 1 or 2):

An anchor is a short description of what evidence at a given score level looks like. "Poor, average, excellent" is a label, not an anchor. The difference is that a reviewer can point to a specific part of the response and say why it matches the anchor Sopact's rubric anatomy guide.

Anchor sets work because they make a subjective process slightly more objective, and they are chosen by finding examples where experienced scorers agreed, then discarding the ones that caused argument Angie Kratzer on anchor papers. The same logic applies here, except your expert scores come from your own team rather than an exam board.

For each criterion, write three anchors on a 1 to 3 scale. Keep 2 as a genuine middle, not a polite version of 3.

Three rules make anchors usable. Use observable language, so the anchor names what is on the page rather than an impression. Keep adjacent levels clearly distinct, so nobody has to invent the difference between "good" and "very good". And make sure a blank or missing element is a defined state, not silently scored as a 2 Sopact.

If your team is weighing whether to score with a checklist of binary questions instead of levels, there is evidence for it. A checklist framework that decomposes criteria into yes or no questions improved agreement across evaluator models and reduced score variance, while staying correlated with human judgment CheckEval, EMNLP. For non-expert reviewers, binary items are easier to apply consistently than a five-point scale. A hybrid works well: binary checks for factual criteria, a 1 to 3 level for overall usability.

Every additional criterion costs reading time and adds a chance for the reviewer to drift. A practical ceiling of five criteria per pair keeps the judgment focused on the failure modes that actually change a decision. Guidance on LLM metrics makes the same point in the automated setting: pick a small set of metrics that match the task and its risk, because a single score hides important failure modes and too many scores hide the decision Confident AI's metrics guide.

Decide which five with a simple test. For each candidate criterion, ask what action the team would take if the score were low. If the answer is "nothing different", drop it. Typical keepers are factual accuracy, instruction following, completeness, format compliance and safety or tone. Typical droppers are style preferences and anything already checked automatically.

Two further rules:

• Separate eligibility from quality. If a response is missing a required element altogether, that is a gate, not a low score on an unrelated criterion. Mixing the two lets a well-written but ineligible response earn points Sopact.
• Do not double count. A response that is clear and well organized can collect points for clarity, professionalism and readability from one impression. Decide once whether presentation is part of the job.

Confidence is the field most teams skip and the one that saves the most time later. A reviewer who scores a response 2 with low confidence is telling you something different from a reviewer who scores 2 with high confidence. The first says the rubric is ambiguous or the case is hard. The second says the output is genuinely weak.

Use three levels and a one-word reason:

• High: the anchor clearly fits and the evidence is visible.
• Medium: the anchor fits but one detail needed interpretation.
• Low: the case sits between two anchors, or required information is missing.

When you aggregate, either drop low-confidence scores from the headline number or report them separately. Silently averaging them hides exactly the cases your experts should look at first. This mirrors the practice of evaluating deferral as a policy: a system that knows when to hand a case to a human is more useful than one that guesses the AI Engineer evals guide.

The notes field should be one or two sentences and should name the evidence, for example "invented an order number in sentence three". These notes are the raw material for prompt fixes and for improving the anchors.

A checklist is itself an instrument, and instruments need validation. The cheapest test is to take twenty pairs that already have expert scores, have your non-expert reviewers score them with the checklist, and compare.

Run it in three steps:

1. Select the twenty. Include easy cases, hard cases and at least three known failures. A set of easy examples tells you whether reviewers can follow instructions, not whether the checklist works SOLTECH's PoC guide.
2. Score blind. Reviewers must not see the expert scores.
3. Compare and diagnose. Look at where the two sets diverge, and ask whether the gap is a reviewer problem or an anchor problem.

Track two numbers: how often each reviewer matches the expert score, and how often reviewers agree with each other. Both matter. A stage-gate approach to pilots treats evaluation on a reference dataset as an exit criterion, not an afterthought, which is why the twenty scored pairs should exist before the review week, not after it Rubicon.

If your non-expert reviewers cannot reach a workable agreement even after two rounds of anchor revision, that is a finding, not a failure. It usually means the task needs specialist review, or that the criteria are too subjective to be scored reliably, and either answer is worth knowing before you scale the review queue.

Most failures are design errors, not reviewer errors.

The prompt is summarized rather than pasted. The reviewer then scores an interpretation. Paste it verbatim.

The response is shown with the model's own preamble. Extra commentary from the system can flatter a weak answer. Show the output as delivered.

The rubric has seven criteria. Reviewers skim the last two, and the scores become noise.

Anchors are adjectives. "Clear" and "complete" mean different things to different people. Replace each with a description of visible evidence.

Confidence is optional. It becomes blank, and the hard cases disappear into the average.

The checklist is never re-tested. Prompts and models change, and an anchor that was right for last quarter's output can be wrong now. Re-run the twenty-pair test whenever the task, rubric or model changes.

There is a broader reason to care about this step. AI pilots stall far more often for organizational reasons than technical ones, and a missing agreed definition of success is among the most common and most fixable. Projects with quantified success metrics defined upfront do much better than those without the Institute of AI PM's analysis. A reviewer checklist is the smallest possible version of that commitment: a written agreement, made before the results arrive, about what counts as a good output.

When you are ready to run the scoring at volume across several model families, one API key for text, image, audio and embedding models keeps the plumbing out of the review loop; Quanta Inference gives you that single key with OpenAI-compatible calls (https://quantaintelligence.ai/inference).