Grading Agent Output That Has No Assert: Rubrics and Judges

A rubric is a scoring guide that names the dimension, the scale and anchor descriptions that pin each level to something observable. Without anchors the judge interprets "1 to 5" however it likes on each call and the scores become noise. With anchors, the same rubric produces similar scores across runs, and agreement between judge and human rises from roughly 0.4 to about 0.78 Cohen's kappa as rubrics go from one line to anchored five-level scales, as reported in the Future AGI rubric glossary. Note the direction of that number: the model upgrade was not what moved agreement. The rubric was.

Practical construction order:

1. Write the criteria from failures, not from virtues. Pull twenty real agent outputs, including at least five you rejected. For each rejection, ask what specifically was wrong. That sentence becomes a criterion. Criteria derived from "what good looks like" tend to overlap; criteria derived from "what went wrong" tend to be independent.
2. Make each criterion atomic. One criterion should be scorable without deciding another. Bundling "accurate and well-toned" into one line forces the judge to average two independent judgments and hides which one failed.
3. Write three anchors per criterion, not five. Three levels (fails, partial, meets) are enough resolution for a decision and far cheaper to keep distinct. Five levels invite hair-splitting that human labelers cannot reproduce consistently either.
4. Anchor on observables. "Cites a source for each factual claim" beats "is well researched." "Names the files changed and why" beats "is clear."
5. Allow an explicit abstain. A CANNOT_ASSESS verdict is preferable to forcing a score when the output lacks the information the criterion needs; the Autorubric framework treats that uncertainty handling as a first-class feature in the arXiv paper on unified rubric evaluation.

A worked example for a coding agent's pull request, which you can adapt:

That table is essentially a graded version of the checks teams already run before merging: requirement fidelity, no weakening of quality gates, auth on protected paths and dependency provenance each appear as pass conditions in the Metacto pre-merge checklist in the Metacto standards guide. The rubric does not invent new standards; it makes the existing ones scorable at volume.

Weights matter more than people expect. Give a negative criterion (penalize hardcoded secrets, penalize test tampering) a weight large enough that it can fail the whole output on its own, or it will be averaged away by three passing cosmetic criteria. A simple rule that holds up: any criterion where a single failure is unacceptable becomes a gate, not a weighted score.

The judge is a model with a rubric in its prompt and a structured response format. Promptfoo's llm-rubric assertion is a concrete reference implementation: it sends the output and the criteria to a grading model and asks for JSON with a reason, a score between 0 and 1 and a pass boolean in the promptfoo documentation. Whether you use that tool or your own wrapper, three properties matter.

Structured output with a reason. Require the reason before the score in the output schema. A judge that commits to a score and then rationalizes it behaves differently from one that states evidence and then grades. Reasons also give you the audit trail a human reviewer needs when they disagree.

One call per criterion. Scoring all criteria in one pass encourages the judge to let a strong impression on criterion A bleed into criterion B. Per-criterion atomic evaluation exists specifically to prevent that conflation per the rubric framework paper.

A panel when the decision is expensive. Run three judges, ideally from different model families, and aggregate by majority or weighted vote. Ensembles with configurable aggregation (majority, weighted, unanimous, any-vote) are a documented mitigation, not an exotic technique in the same framework. The cost is three calls per output; the benefit is that judge-specific quirks (verbosity preference, self-preference for outputs from the same family) partially cancel.

Two biases to guard against explicitly. Verbosity bias means longer answers score higher for reasons unrelated to quality, so either penalize length in the rubric or tell the judge to ignore it. Position bias matters when you compare two outputs side by side; shuffling the order across runs is the standard mitigation per the same paper. A practical defense is boring and effective: include two or three outputs in your calibration set that are long, fluent and wrong, and confirm the judge scores them low.

If you are running multiple judge models, a single gateway helps keep credentials and prompt versions consistent; Quanta Inference exposes text, image and embedding models behind one OpenAI-compatible key, which is convenient when the judge and the system under test are different model families (https://quantaintelligence.ai/inference).

Calibration is the whole exercise. A judge that agrees with no human is a random number generator with good manners, and its scores will look stable while measuring nothing. The operational target is inter-rater reliability, not raw accuracy: Cohen's kappa corrects for agreement that happens by chance, which raw accuracy inflates on imbalanced labels per Galileo's calibration guide.

A calibration loop that fits in a week:

1. Label a stratified sample by hand. Take 50 to 100 outputs, deliberately over-sampling the rare and risky categories rather than sampling at random, because random sampling hides tail failures per Galileo. Have two people label independently so you can see how much humans agree with each other. That number is your ceiling; a judge cannot beat human-to-human agreement by much.
2. Score the same sample with the judge. Compare per criterion, not just overall. One criterion with weak agreement is a rubric problem, not a judge problem, nine times out of ten.
3. Rewrite the anchors where they disagree. If two humans disagreed, the anchor is ambiguous. If humans agreed and the judge did not, the anchor is probably fine and the judge prompt needs the disambiguation, or the judge is too weak for the task.
4. Record kappa per criterion and keep the trend. A judge can agree closely with reviewers at first and drift silently after a provider-side model update; one older model version update caused a 9.6% accuracy drop for one prompt and a 5.1% increase for another on the same task per Galileo citing the arXiv study. Dashboards stay green while the measurement quietly stops meaning anything.

Rough thresholds from practice: a criterion at or above 0.6 kappa is usable for triage; between 0.4 and 0.6, use it only to route outputs to a human, never to auto-pass; below 0.4, do not ship the criterion, rewrite it. And remember that a clean single run proves nothing: one team reported a hallucination fix as verified at zero of five failures, then re-ran under harder conditions and got 66.7% in the freeCodeCamp account of spec-verify. Judge calibration has the same property. One comparison against one golden set is a snapshot, not a guarantee.

The judge gives you scale. The human sample tells you whether the scale is real. Automating past that point is how teams ship a green dashboard over a real regression. Three conditions should force a human review regardless of judge score:

High cost of a wrong pass. Money movement, auth, PII, infrastructure and public APIs belong to an accountable human, even when the judge approves. The consistent position across the security literature is that review cannot rest solely on the automated layer per CSET, and practitioners recommend feeding corrections back into the review process rather than treating any single check as final per the Cloud Security Alliance writeup. A useful framing is that AI output can be treated as pre-reviewed only when your system can prove it, and the merge decision stays with a human on trust-boundary changes per Metacto.

Rubric still being written. While a criterion's anchors are under revision, its judge scores carry no weight. During that period, the human labels are the ground truth and the judge is being tested, not trusted.

Judge and human disagree. Log every disagreement with the output, the criterion, the judge's reason and the human's reason. Route it to the rubric owner weekly. A recurring disagreement pattern is the fastest rubric-improvement signal you will get.

For code specifically, a graded rubric does not replace reading the diff. The human checks that require context the judge does not have are the ones review guides emphasize: does the change do what the task asked, is there a named owner who can explain each file, is the design appropriate to this codebase rather than merely idiomatic per Google's what-to-look-for guide and the reviewer guide's emphasis on design and functionality covered in the BSSw summary of Google's guidance. Depth also degrades with volume: a long diff gets less scrutiny per line, which is why splitting agent-authored changes before grading them is worth the friction.

A practical sampling rule that balances cost and risk: auto-pass outputs that clear every gate; send outputs that fail any gate to a human; and send a random 5% of clean passes to human review anyway, because that sample is how you catch a judge that has drifted. When that 5% starts disagreeing, widen the sample until it agrees again.

Here is the sequence with numbers, for a team of four engineers and a support bot that writes replies and a coding agent that opens pull requests.

• Week one. Collect 60 past outputs, 20 of them known-bad. Write four criteria for the support bot and four for the coding agent, each with three anchors. This is one engineer-day per rubric and the highest-leverage day in the project.
• Week two. Two engineers independently label all 60. Their own agreement lands around 0.7 to 0.8 kappa; anything below 0.6 means the anchors are still ambiguous and you rewrite before touching a judge.
• Week three. Score the same 60 with a single judge, per criterion. Expect one or two criteria in the 0.4 to 0.6 band. Rewrite those anchors, rescore. Add a second judge only for criteria that stay below 0.6 or that gate on risk.
• Week four. Turn on the pipeline: gate criteria fail the output automatically; scored criteria route to a human queue when below threshold; 5% of clean passes sampled for review. Cost per output is one or three judge calls, which is trivial compared with the human time you were spending on every change.
• Ongoing. Weekly disagreement review, monthly recalibration on a fresh stratified sample, and a kappa trend chart per criterion. When the trend bends, recalibrate before you widen automation.

The insight that most teams miss is where the effort belongs. Recalibration looks like a pipeline review and front-loading the rubric looks like a delay, but the rubric and the anchors are the measurement instrument, and every hour spent on them buys more agreement than any judge swap. When the judge and the code under test come from the same model family, a panel of judges from a different family reduces self-preference; Quanta Inference is one way to reach several model families through a single prepaid, dollar-priced key (https://quantaintelligence.ai/inference).

Where the rubric work touches domain judgment outside your team's expertise, for example health, legal or financial review criteria, the criterion set is often better drafted with someone who already does that review. Quanta Experts read your context and hand back finished deliverables, which is a reasonable fit for a first-draft rubric plus anchor set on a domain your engineers do not own (https://quantaintelligence.ai/agents).