How to Write a Second-Model Reviewer Prompt from the Same Rubric

Start from the rubric as written and split it until each line tests one thing. A line like "the summary is accurate and complete" tests two properties and will produce murky results. The rubric design guidance from teams running LLM graders converges on the same rule: one criterion, one failure mode, so that a failure points at a single diagnosable issue the Hebbia hybrid evaluation framework. The same source recommends pass/fail scoring over multi-level scales, adding more criteria rather than more score levels when you need granularity.

A second rule from the same body of practice is distinct coverage. If two checks tend to pass or fail together, they are measuring one thing, and keeping both distorts the aggregate and wastes evaluation work the Hebbia hybrid evaluation framework. Merge them or narrow one.

Hierarchical rubrics help here. A complex task can be scored by broad dimensions, then sub-dimensions, then fine-grained items that a reader can answer from evidence alone; a published benchmark rubric in this style uses seven primary dimensions, 25 sub-dimensions and 130 fine-grained items the Splunk agent evaluation framework. You do not need that depth, but the shape is right: the leaf items are what the model answers.

For a typical content or support-response rubric, the extraction pass looks like this:

Four lines became eight checks. That is normal, and it is the point: eight answerable questions beat four that require interpretation.

Convert every atomic check into a single question whose answer is yes or no, and require the model to copy the exact span it judged. The quote requirement does two jobs. It makes the verdict auditable, and it suppresses the model's tendency to answer from general impressions rather than the text in front of it. Rubric-based scoring guidance is explicit that giving a model a structured rubric, rather than asking it to just score the output, is what produces consistent and interpretable results the rubric scoring agent design write-up.

Specificity matters more than eloquence. "Is the title good" cannot be applied consistently. "Is the title descriptive, specific, and free of units or scale information" can the Hebbia hybrid evaluation framework. Where the rubric implies a fixed mapping, state the mapping: line chart for continuous trends, bar chart for discrete comparisons, stacked bar for part-to-whole across periods.

A prompt block for one check can be short enough to read at a glance:

CHECK 3 of 12: escalation_threshold
Question: Does the response escalate any refund above the
stated threshold to a human approver?
Answer: YES | NO | ABSTAIN
Quote: copy the exact sentence(s) that decide this. Max 40 words.
Reason: one sentence, max 25 words.

Keep the quote cap. It forces the model to select the decisive span rather than dumping context, and it keeps the reviewer's confirmation step fast.

Abstention is the part most prompt drafts skip, and it is where model-assisted review earns or loses trust. Write the abstain condition as part of the check, not as a general instruction at the top of the prompt. Good abstain conditions name the situation precisely:

• The decisive information is not present in the text being judged.
• The quote needed to answer would exceed the cap because the relevant content is scattered across more than one section.
• The check depends on a policy, threshold or source document not supplied in this prompt.
• Two passages support opposite answers and nothing in the rubric resolves which applies.
• The output is truncated or malformed in a way that prevents judgment.

Set aggregation so that abstains are visible and never silently converted. A check that abstains should count as unresolved, and any item with one or more unresolved checks should route to a full human review rather than a quick confirm. This mirrors the routing logic used in enterprise human-in-the-loop systems, where outputs above a risk threshold or below a confidence threshold go to a reviewer rather than proceeding the Elementum HITL comparison. In a large-scale writing assessment deployment, a human-in-the-loop decision flow was used specifically to identify which cases most needed expert review, which is the same principle applied to essay scoring the arXiv human-in-the-loop scoring framework.

Decide how check answers become a draft verdict, and write it into the prompt as a fixed rule. Three patterns cover most rubrics:

1. All-required: every required check must be YES. Any NO or ABSTAIN blocks a pass. Use for compliance, safety and policy checks.
2. Weighted score: required checks gate, additional checks add points. The Hebbia rubric design distinguishes required criteria, which represent the agent's service commitments, from additional criteria that mark advanced quality without causing failure the Hebbia hybrid evaluation framework.
3. Count threshold: a pass needs at least N of M checks. Simple, but it lets one serious failure hide behind several trivial passes, so pair it with a gating list.

Whatever you choose, have the model return a structured object rather than prose. A JSON shape keeps the output parseable and makes reviewer acceptance a one-click action:

{
  "item_id": "resp-1042",
  "checks": [
    {"id": "escalation_threshold", "answer": "YES",
     "quote": "Refunds above $250 require approver sign-off.",
     "confidence": "high"},
    {"id": "security_routing", "answer": "ABSTAIN",
     "quote": "", "confidence": "low",
     "abstain_reason": "No security issue appears in the response."}
  ],
  "draft_verdict": "REVIEW",
  "unresolved_checks": ["security_routing"]
}

Note the draft verdict is REVIEW, not PASS or FAIL. The aggregation rule sees an abstain and routes accordingly. Before you trust any of this, run the same sample several times and measure how often each check flips between runs; rubric stability across independent runs is the minimum bar before automating decisions, and a criterion that flips frequently needs rewriting rather than averaging the Splunk agent evaluation framework. Multiple grading passes per item, aggregated, reduce noise from grader non-determinism and expose ambiguous criteria the Hebbia hybrid evaluation framework.

Expect a calibration gap. Strong general-purpose judges reach agreement with human evaluators at roughly the rate humans agree with each other on general tasks, but off-the-shelf judges agree with domain experts only about two-thirds of the time on expert-knowledge tasks, because they lack the domain context the Splunk LLM-as-judge guide. The same guide reports that prompt wording alone moves accuracy widely on the same underlying model, which is why the check wording, not the model choice, is the main lever you control.

The prompt is only half the system. The reviewer needs to see, for each check, the question, the model's answer, the quoted span and the confidence flag, and must be able to accept, overturn or abstain with one action. Three design rules keep this from becoming a new bottleneck:

• Never show a single overall score first. Show the check table, so the reviewer forms a view from evidence rather than anchoring on the draft.
• Sort by confidence, ascending. Low-confidence checks go to the top. This is the same prioritization logic used in supervisory dashboards for AI voice agents, where alert prioritization by escalation potential reduces cognitive load and improves response times the emergingpub cognitive load study.
• Log every overturn. A check the reviewer overturns repeatedly is a badly written check. If graders frequently disagree on a criterion, that criterion needs refinement, not more grading passes the Hebbia hybrid evaluation framework.

The cognitive-load literature supports this restraint. Supervisors monitoring many AI interactions face information overload that leads to alert fatigue and missed escalations, and the proposed remedies are adaptive presentation and contextual alerting rather than more raw signal the emergingpub cognitive load study. Treat each check result as an alert with a priority, not as a data field.

Most weak second-model reviewers fail in one of five ways. The table below maps each failure to its fix and to the evidence you can inspect.

The bias problem deserves its own line. LLM judges are documented to favor irrelevant heuristics such as answer length and the order in which candidates are presented the arXiv dynamic rubric paper. In rubric-based evaluation at scale, teams are advised to actively mitigate verbosity, position bias and self-preference the Medium rubric-based evaluation overview. Practical countermeasures: cap the quote, randomize order when you compare two outputs, and keep the source document short enough that length cannot stand in for quality.

There is also a governance reason to keep humans authorizing. Analyst forecasts cited in evaluation guidance predict that a substantial share of enterprises will demote or decommission autonomous agents where governance gaps surface after production incidents the Splunk agent evaluation framework. A reviewer prompt that never lets the model make the final call sidesteps that risk entirely.

Some checks should stay human, and knowing which ones saves you from building the wrong thing. Skip the model, or route straight to expert review, when:

• The check requires knowledge outside the supplied text, such as the author's history or an unwritten team convention.
• The check is high-stakes and irreversible, and a false YES is expensive to reverse.
• The rubric line is itself disputed and has no agreed operational definition yet. Fix the rubric first.
• The item is so short that a reviewer reads it faster than the model output.

For the checks where the model does help, one API key covering many model families keeps the reviewer prompt portable: you can test the same prompt across several judges and compare flip rates without rewiring anything. Quanta Inference provides that single key with OpenAI-compatible access.

Finally, validate against humans, not against your own taste. Score a sample of items both ways, compute agreement per check rather than overall, and rewrite the checks with the lowest agreement. Rubric quality in multi-rater settings has been studied for decades, and the lesson transfers: validity and reliability are separate questions, and a rubric can be valid in principle while unreliable in application the FHSU inter-rater reliability paper. The same caution applies when the second rater is a model.