How to Write a Review Rubric That Makes Agent Supervision Faster

Most first-draft rubrics are written as qualities: accuracy, clarity, completeness. Those words feel right and review badly, because two reviewers can agree the output is clear while disagreeing about whether it passes. A rubric item should map to one diagnosable failure. Hebbia's evaluation team codifies this as one criterion, one failure mode, and argues atomic criteria make graders more consistent and debugging tractable Evaluating AI Agents: A Hybrid Deterministic and Rubric-Based Framework.

To build the list, work backward from incidents rather than forward from ideals:

1. Pull the last 20 to 30 failures that reached a user, a customer or an on-call engineer, including near misses caught by luck.
2. Write one sentence for each: what did the agent do, and what would have caught it?
3. Group the sentences. Most will collapse into a handful of patterns such as wrong scope, missing evidence, unsafe tool call, fabricated reference, or correct answer with an unnecessary retry loop.
4. Turn each group into a check with a pass definition you could verify from the transcript or artifact alone.

That fourth step is where rubrics usually fail. Good title is not checkable. Title is descriptive, specific, 5 to 10 words, without units or scale information is, and Hebbia uses exactly that contrast to illustrate grounded criteria the same Hebbia write-up. Rewrite every item until a new reviewer could apply it without asking you a question.

It also helps to separate outcome checks from trajectory checks. An agent can finish a task on an unsafe, expensive or unreliable path, and the gap stays invisible until customers report repeated tool calls or inconsistent answers. Four trajectory checks cover most of it: exact match of an approved tool sequence, in-order match of required actions, precision (penalizing unnecessary retries) and recall (catching omitted calls) How to Build an Agent Evaluation Framework for Production AI. For a review rubric, that means some items look at the artifact and some look at the run log.

The single largest speed decision in a review rubric is the scale. Weighted 1-to-5 scales feel more informative and review more slowly, because the reviewer has to hold a mental anchor for each level and argue with themselves about the difference between a 3 and a 4. Binary checks converge faster and are easier to validate against human judgment. Hebbia's stated rule is pass or fail only, and when more granularity is needed they add more criteria rather than more score levels Hebbia on rubric design.

The hybrid approach used in production evaluation systems pairs deterministic checks for verifiable properties (schema validity, required fields, data types, formatting) with rubric-based grading for properties that need judgment, such as coherence, completeness and domain appropriateness Hebbia's hybrid evaluation approach. Copy that split into human review. Anything a script can check should never cost a reviewer's attention. Reserve the human checklist for the items that genuinely require reading.

A useful compromise for severity: keep items binary, then assign each item a weight, and let the weighted total decide escalation. That gives you ranking without asking reviewers to rank.

Suppose you supervise an agent that drafts customer-facing responses to billing questions. Start with the failure modes you actually saw: wrong refund amount, promise of an action the system cannot take, missing policy citation, tone that reads as dismissive, and a response that is fine but took four tool calls and two minutes. Turn those into items and give each a pass definition.

RUBRIC: billing_reply v1        scale: pass / fail per item
Order is fixed. Stop at any HARD FAIL; record and escalate.

ID   W    Item                          Pass definition
H1   10   Amount correctness            Every figure matches the billing record; no rounding drift.
H2   10   Commitment safety             No promise of refunds, credits or timelines not issued by the system.
H3   8    Policy citation               Every policy claim names the clause; no invented clauses.
S1   4    Tone                          No blame framing; no sarcasm; acknowledges the customer's stated problem in the first sentence.
S2   3    Scope                         Answers the question asked; no unrequested upsell or account changes.
S3   2    Efficiency                    No repeated tool call with identical arguments; no more than 3 lookups.
S4   1    Format                        Under 180 words; no internal IDs or stack traces in the body.

Two details matter more than they look. First, the fixed order: hard fails first, so a reviewer who spots a wrong amount can stop in seconds instead of reading to the end. Second, the hard-fail flag: H1 and H2 escalate regardless of total score. Splunk's framework makes the same distinction between design, coordination and verification failures, each mapped to an observable check Splunk's failure taxonomy.

Then set the threshold. A reasonable starting rule for this example: escalate if any H item fails, or if the weighted total of the S items falls below 6 of 10. Calibrate the numbers against your own queue, not against this example.

A rubric without calibration produces confident, inconsistent decisions. The evidence from scored rubrics in education is directly transferable: inter-rater reliability can be achieved, but the measured strength varies substantially depending on the statistical method used, so teams must choose a measure deliberately rather than assume agreement Is it actually reliable?. A clinical documentation rubric study reached similar conclusions, treating inter-rater reliability as something to be measured and demonstrated rather than assumed Inter-rater Reliability of a Clinical Documentation Rubric.

Run calibration as a short, repeating ritual:

1. Build a calibration set of 10 to 15 agent runs, deliberately including two or three borderline cases and one clear fail.
2. Have each reviewer score the set independently, without discussion.
3. Compare item by item, not total by total. The total hides where the disagreement lives.
4. For every item where reviewers split, rewrite the pass definition or delete the item.
5. Re-score a fresh set of five. Repeat until splits are rare, then re-run the whole exercise whenever the agent changes materially.

The key diagnostic habit: disagreement is a rubric defect first and a reviewer defect second. If two experienced reviewers read the same output and reach different verdicts, the item is ambiguous, and no amount of retraining fixes an ambiguous item. Evaluation practitioners make the same point about automated graders: run the same input several times, and if grades frequently disagree on a criterion, that criterion needs refinement rather than more grading passes Hebbia on variance control.

For a lightweight reliability number, track simple percent agreement per item across reviewer pairs, plus a second measure for ordered totals (weighted kappa is the common choice). Report both, because the graduate-education study found the strength of reliability varies by method the FHSU reliability paper. If you would rather not compute statistics, a monthly sample where two reviewers score 20 runs and you count item-level splits gets you most of the value.

Weights are where a review rubric becomes an instrument instead of a form. The default mistake is to weight everything equally, which means an invented policy clause costs the same as a stray internal ID. Weight by the expected cost of the failure, approximated as frequency times impact.

A simple allocation method that survives contact with a real team:

• Hard fails (escalate, no score): failures that reach users, regulators or money.
• Weight 8 to 10: failures that damage trust but are recoverable, such as wrong tone in a sensitive case.
• Weight 3 to 5: quality issues a user would notice but tolerate, such as length or phrasing.
• Weight 1 to 2: hygiene issues you want to trend but never want to block on.

Then close the loop. Every production incident should become a permanent check. Splunk's framing is that a mature evaluation strategy converts each production incident into a regression test so the failure pattern cannot reach users again the Splunk evaluation guide. In human review terms, that means adding or reweighting an item within a day of an incident, and noting the change in the rubric's change log so you can tell whether review time moved because of the new item.

One caution on weight inflation. If every new incident bumps a weight, everything drifts toward 10 and the weighted total stops discriminating. Cap the number of hard fails at three or four, and when a new candidate hard fail appears, demote the least costly existing one to a weighted item.

You cannot claim a rubric makes supervision faster without two numbers: minutes per run and defects found after review. Instrument both before you change anything, or you will spend the next quarter arguing from memory.

Review time. Have reviewers log start and stop on a shared sheet for a week, or read timestamps from your review queue if it records them. Median time per run is the headline; the spread matters too, because a rubric mainly compresses the long tail of runs where reviewers had to reconstruct intent. Expect the first week to be slower, since reviewers are reading the rubric as well as the output.

Missed defects. This is the defect rate found downstream, after review passed an output, divided by runs reviewed. You need a source: user complaints, a second reviewer sampling a random 5 percent, or the failing checks in your monitoring. Pick one and keep it constant.

Agreement. Percent agreement per item across paired reviews, as described in the calibration section. If your team also compares against an automated judge, note that even low-temperature judging is not deterministic, so a single comparison run understates true reliability Hebbia on output variance.

The tradeoff between speed and thoroughness is real, not hypothetical. Hierarchical scoring exists because collapsing a complex trajectory into one pass or fail produces an unhelpful score, and the DEER benchmark framework decomposes evaluation into seven primary dimensions, 25 sub-dimensions and 130 fine-grained rubric items, with each branch ending in a question answerable from trace evidence Splunk on hierarchical rubrics. Your targeted review can stay far smaller, but the same logic applies: keep the rubric at the size where every item earns its minutes.

When a rubric gets genuinely right, most of the supervision collapses into pre-computed checks and reviewers only touch the exceptions. Composite evaluation designs that combine deterministic checks with structured scoring are already the norm for automated systems Hebbia's hybrid framework, and the same architecture works with people in the loop.

Three habits make a rubric survive beyond its author. First, publish a one-page battlefield guide: which failure modes are most common right now, what the current escalation threshold is, and the three most recent rubric changes with the reason for each. This is the equivalent of keeping a review board focused on the problems behind work items rather than the items themselves, the same principle behind project management meetings with clear, narrow purposes Slack on team meetings.

Second, keep the rubric in version control next to the agent's configuration. A rubric is a spec; if the agent prompt changes, its review spec should change with it, and the diff should be visible.

Third, run weekly 30-minute calibration on five fresh runs, the review equivalent of a short, recurring one-on-one focused on removing obstacles rather than status monday.com on one-on-ones. Frequency beats duration: short, frequent calibration catches drift before it becomes a disagreement about policy.

None of this reduces the fundamental load of supervision, which is worth being honest about. Every supervision queue with a rubric is still a queue of decisions, and sustained review-heavy work has well-documented costs: chronic fatigue, cynicism, and a reduced sense of efficacy that do not lift with a weekend Mental Health America, and three dimensions the World Health Organization classifies as an occupational phenomenon rather than a medical condition HelpGuide. A decade of practice also points to the value of naming the three patterns early and adjusting workload rather than waiting for performance to suffer a manager's early-warning field guide. A rubric that turns an hour of hesitation into ten minutes of scanning is, in the most literal sense, a preventive measure.

If your team is building this pattern across multiple agent products and wants a single API surface for the models behind them, Quanta Inference gives text, image, video, audio and embedding models through one OpenAI-compatible key with prepaid pricing in US dollars (https://quantaintelligence.ai/inference).