Writing a Review Rubric an LLM Judge Can Apply

Take every adjective in your human rubric and convert it into a question that points to evidence in the item itself. "Accurate" becomes "does every factual claim match the supplied source text?" "Helpful" becomes "does the response state the next action available under the policy?" These questions overlap less and tell a rater what to inspect the QASkills rubric workflow.

A usable criterion has five fields. The first four are the same set practitioners recommend: a stable reporting name, an observable question, the allowed scale, and anchors for each label. The fifth is the one people skip: a blocking rule that connects the score to the decision contract.

Keep criteria orthogonal. If one fabricated date zeroes out groundedness, correctness, helpfulness, relevance, clarity and safety all at once, your dashboard reports six failures when there is one underlying defect the QASkills rubric workflow. Orthogonal criteria also make the agreement table readable, because divergence clusters around one question instead of spreading across six.

The task must be defined before you pick any metric. Write down the unit (a single model call, a multi-step trace, or a whole session), the label type (nominal categories versus ordered labels), the positive class for precision and recall, the reference policy that turns multiple human judgments into one label, and which error costs more, a false positive or a false negative Arize's guide to measuring human and judge alignment. That YAML-shaped checklist takes ten minutes and prevents months of scoring the wrong object.

Precision in wording is not decoration. Oracle's guidance on agent prompts calls vague definitions the top source of agent misbehavior and treats descriptions as legal documents: if a description says "retrieves worker information," the model does not know whether that means name, email or salary, so you specify exactly Oracle's prompt best practices. Your rubric is a prompt. Word it like a specification, not like a value statement.

An anchor is a one-line example that shows where the boundary sits. Without anchors, the label "partial" becomes a personal taste and the judge drifts.

The pattern below is a judge prompt you can run as is. It fixes the role, the checks, the anchors, the evidence requirement and the output shape. Note the last field: the judge must quote the exact span it relied on, or it returns evidence: null and the score is automatically fail.

You are a review judge for customer support replies.

Decision contract: decide whether this reply can be sent to a
customer without human review, given the supplied policy.

Criterion 1, groundedness
Question: are all policy claims supported by the supplied policy text?
pass: every claim traces to the policy.
  Example: "Standard returns close at 30 days."
partial: one minor claim is unstated but implied by the policy.
  Example: "You may want to contact support" with no policy basis.
fail: a material claim contradicts or invents policy.
  Example: "Refunds are accepted for 90 days."
Blocking: any fail blocks the reply.

Criterion 2, actionability
Question: does the reply name the next step the customer can take?
pass: one concrete action, with the condition that triggers it.
  Example: "If the item is defective, open a support ticket."
fail: no action, or several contradictory actions.

Rules:
- For each criterion return: label, evidence (exact quoted span
  from the reply), reason (one sentence).
- If you cannot quote evidence, return label "fail" and
  evidence null. Do not infer.
- Do not reward length. A longer reply is not a better reply.
- Return JSON only.

Output:
{"groundedness":{"label":"","evidence":"","reason":""},
 "actionability":{"label":"","evidence":"","reason":""}}

Two design choices inside that prompt are worth defending. First, binary or named categories instead of numbers: models are not calibrated to produce consistent scores on arbitrary scales, and named categories give more stable, interpretable evaluations the Evidently AI guide to LLM judges. Second, the evidence field. It converts the judge from a rater into a witness. When the judge quotes a fabricated refund deadline, you can check in two seconds whether the quote exists in the reply at all.

You also want reproducibility while you are debugging. Fixing the seed, setting temperature to zero and holding the model version constant removes the variation that makes prompt and rubric changes hard to isolate a practitioner guide to reproducible AI outputs. Use that setting for the calibration sample. Once the rubric is stable, measure how much the judge varies at its normal sampling setting, because that variance is part of what you are shipping.

Thirty to fifty items is enough to find a broken criterion. The McHugh paper on the kappa statistic treats under 30 comparisons as too few and observes that samples of 1,000 or more are likely to produce very small confidence intervals the kappa statistic paper. Thirty to fifty sits at the practical end: enough to see the pattern, small enough that two human reviewers will actually finish.

Assemble the case packet before the rubric goes out. Each item needs a stable ID, the input prompt, the supplied policy or source text, the response under review, and the metadata that changes correctness (locale, for example). Remove metadata that could bias a rater, such as which model produced the response the QASkills rubric workflow. Label the packet with neutral IDs, then score it three ways: human A, human B and the judge, all against the same rubric and the same packet.

Here is a compact, runnable way to compute per-criterion agreement. It reports raw percent agreement next to Cohen's kappa, because raw agreement alone can mislead: when 90 percent of items fall in one category, two raters making random choices at that base rate would still agree about 82 percent of the time a worked discussion of sample size for interrater agreement.

from sklearn.metrics import cohen_kappa_score

# one row per item, one column per criterion
humans = {
    "groundedness": [("pass","pass"), ("fail","fail"), ("pass","partial")],
    "actionability": [("pass","pass"), ("pass","pass"), ("pick","fail")],
}
judge = {"groundedness": ["pass", "fail", "pass"],
         "actionability": ["pass", "pass", "fail"]}

def agree(a, b):
    return sum(x == y for x, y in zip(a, b)) / len(a)

for name, pair in humans.items():
    a, b = zip(*pair)
    hh = agree(a, b)
    hj = agree(a, judge[name])
    k = cohen_kappa_score(a, judge[name])
    print(f"{name}: human-human {hh:.2f}  human-judge {hj:.2f}  kappa {k:.2f}")

Kappa corrects agreement for chance, ranges from roughly minus one to plus one, and is conventionally read in bands where 0.41 to 0.60 is moderate, 0.61 to 0.80 is substantial and 0.81 and above is almost perfect the kappa statistic paper. Sample-size guidance for agreement studies exists, and it gets expensive fast: shrinking the distance from your assumed kappa to the nearer confidence limit from 0.20 to 0.05 to 0.01 needs roughly 118, 1,625 and 38,203 subjects respectively the kappa sample-size walkthrough. That arithmetic tells you what the 30 to 50 item sample is for: finding which criterion is ambiguous, not proving a precise agreement value.

Keep the human-human number in view the whole time. It is the context for the comparison, not a hard ceiling for the judge, and no single number proves a judge trustworthy Arize's alignment guide. Reporting raw human-human agreement next to a chance-adjusted metric is the minimum a defensible calibration needs.

Lay the results out per criterion, not per item. The table below is illustrative, not measured, and it shows the shape of the readout you are aiming for.

The interpretation rule is the insight the first search result usually misses. A low overall score tells you almost nothing. Ambiguity shows up as disagreement on one specific criterion. Here, groundedness is healthy because both the humans and the judge agree, and the disagreements that remain are genuine edge cases worth keeping in the set. Actionability is broken. The humans agree with each other and disagree with the judge, which means the criterion means something different to them than to the model, and the anchor for "partial" is probably where the divergence lives. Tone is a different problem: the humans barely agree with each other, so the criterion is not yet measurable by anyone and may belong in a different decision contract entirely.

The same logic applies to why partial labels behave badly. On annotation setups with two annotators and categorical labels, Cohen's kappa is the standard starting point, and on ordered labels weighted Cohen's kappa applies Arize's alignment guide. Unweighted kappa treats a pass graded as partial exactly like a pass graded as fail, and that is usually wrong for a three-point scale: the near miss matters less than the reversal.

Reading the errors also matters. Treat the judge as a classifier against an adjudicated reference, and report precision, recall, F1, class counts and the confusion matrix, with enough examples in each important class rather than only overall Arize's alignment guide. If the positive class is "fail," a judge with high precision and low recall is conservative, which may be exactly right for a release gate and exactly wrong for a discovery sweep.

A worked example makes the cost concrete. Suppose 40 items, 10 of which truly fail on actionability. The judge flags 6 of the 10 and also flags 2 that pass. Precision is 6 of 8, or 75 percent. Recall is 6 of 10, or 60 percent. For a release gate where a missed fail reaches a customer, a false negative is worse than a false positive, so the 60 percent recall is the number to fix, and the fix is usually a tighter fail anchor, not a bigger model.

The rescue relationship is worth knowing too. Research on developer responses to agent-generated review comments found that the presence of an inline code suggestion is the strongest predictor of whether a comment gets resolved, while lengthy and complex comments are less likely to be acted upon the study of agent code review comments. The parallel for judging is direct: a judge output that carries the quoted evidence plus a one-line reason will be reviewed and acted on by your team, and a judge output that returns "score: 6, reason: quality could be better" will be ignored.

If your rubric scores agent behavior across a multi-step trace rather than a single reply, run the judge inside a traced pipeline. OpenTelemetry alone lacks the semantic conventions for LLM calls, tool calls and token counts, and the OpenInference extension supplies them so the judge's evidence can be attached to the exact span it scored the Arize AI debugging walkthrough. That same walkthrough shows an LLM-as-a-judge harness moving a pass rate from 40 percent to 90 percent across prompt iterations, which is the loop described next. The same trace-level thinking explains why long agent runs need subgoals and explicit replanning rather than one long stretch of execution, because error probability compounds as a task stretches and success rates fall off after about 35 minutes of human-equivalent work a survey of long-horizon agent planning. A judge scoring a long trace should score the active subgoal, not the whole run.

The revise and rescore loop is short. Take the broken criterion, rewrite the question and the anchors using the actual disagreement cases as examples, rescore the same 40 items with the judge only, and compare against the frozen human labels. Do not let the humans rescore, or you lose your reference.

Decide the stopping rule before you start, in numbers. For example: accept a criterion when human-judge kappa reaches 0.61, the floor of the substantial band the kappa statistic paper, and when the judge's recall on the blocking class reaches the level your decision contract requires. A release gate that misses a third of real failures is not ready regardless of its kappa. A regression monitor may accept lower absolute agreement if the labels are stable from run to run.

Then write down what the humans disagreed about. Disagreement is not noise to erase: it improves the criteria, and preserving ties and low-consensus cases rather than assigning an arbitrary label protects the reference from becoming fiction Arize's alignment guide. A pass graded as partial by one human and fail by another is a boundary you have not defined, and it will confuse the judge every time.

A last distinction to hold: the rubric and the ordering of work are separate decisions. In delivery planning, a horizontal slice of a story touches one layer and delivers no usable value on its own, while a vertical slice cuts through every layer and can be demonstrated Visual Paradigm on vertical and horizontal slices. Applied to eval design, a criterion that only a specialist can judge is a horizontal slice: it belongs in a specialist review, not in the automated gate. Small batches, sliced to be independently valuable, produce feedback you can act on a practical guide to story slicing. A rubric criterion should clear the same bar: small, self-contained, and useful on its own.

When the judge has to run across text, image and log evidence with one key and one bill, Quanta Inference gives you OpenAI-compatible access to every kind of model. When you would rather hand the whole calibration loop to someone who has run it, a consulting call through Quanta Intelligence covers the design and the readout.