Antonio Evans
The most common failure in model reviewing is not a weak model. It is a loose rubric. If two junior reviewers on your team would grade the same output differently because the rubric is vague ("clear and professional"), a model reading the same rubric will be vague too, and you will never know whether a disagreement is the model's fault or the document's.
Start by collecting the rubric you actually use. Most review teams have one, even if it lives in a spreadsheet or a wiki. If your team is newer, a structured plan document is a good template to adapt: onboarding plans built on 30, 60 and 90 day checkpoints already force you to write responsibilities and measurements explicitly, which is exactly the shape a review rubric needs (AIHR's 30-60-90 template). The Management Center's goal-setting template is useful for the same reason: it pushes you toward measurable criteria rather than adjectives (goal-setting template).
Then convert each criterion into a decision the model must make. Three rules make this work:
- One criterion, one field. Do not ask for an overall "quality" score. Ask for a checklist of yes/no or 1 to 3 scores, each tied to a named criterion.
- Evidence required. Every verdict must quote the span of text that justifies it, or state "no evidence found."
- Fixed output format. Constrain the response to a schema so you can parse verdicts into a table instead of reading prose.
Structured output is a solved problem in most serving stacks: you can send a JSON schema and a response_format field, and the server constrains generation so the model cannot return anything that fails validation (Anyscale's structured output guide). That matters for a reviewer because malformed output is a silent source of missing data. Friendli's write-up on the same feature explains why: enforcing a schema with token filtering removes parse errors that would otherwise require brittle regex cleanup (Friendli on structured output). Ken Huang's chapter on schema-constrained generation covers the retry pattern in more depth, including feeding validation errors back to the model (schema-constrained generation).
A workable schema for a reviewer looks like this:
{
"type": "object",
"properties": {
"criterion": { "enum": ["factual_accuracy", "instruction_following", "tone", "completeness"] },
"verdict": { "enum": ["pass", "fail", "cannot_determine"] },
"evidence_span": { "type": "string" },
"confidence": { "type": "integer", "minimum": 1, "maximum": 3 }
},
"required": ["criterion", "verdict", "evidence_span", "confidence"]
}
Run the model once per criterion rather than once per output. It is more calls, but it keeps each prompt short and each verdict independent, and it makes the agreement math per category trivial later.
Two prompt-level choices matter more than model choice. First, include two or three worked examples per criterion, one clear pass and one clear fail, drawn from your own history. Second, forbid the model from inferring facts not present in the output or in the source material you attach. A reviewer that invents context will disagree with your juniors in a way that looks like judgment but is actually hallucination.
