Scoring Agent Output Against a Human Baseline: A Rubric

Start by listing what actually matters for the task, then split the list along one line: things that are true or false, and things that are better or worse. Mixing them into a single score is the most common rubric defect, because a fluent, confident, wrong answer can outscore a blunt, correct one.

The Label Studio guide to scaling evaluation recommends a short set of clear dimensions applied consistently rather than a long checklist that leaves room for interpretation. Four to six criteria is usually right. Useful splits for agent work:

Two rules make this table useful. First, one dimension per rubric: faithfulness and tone are separate rubrics, not one combined score, because a reviewer asked to trade an ungrounded but pleasant answer against an accurate but blunt one will resolve that tradeoff differently every time Arize. Second, write each level as an observable condition rather than an adjective. "Every factual claim appears in the retrieved context" is checkable. "Mostly accurate" is not.

Where a check is deterministic, do not use a human or a model at all. Google's framework separates code-based evaluation, which is the cheapest and most deterministic, from human evaluation, which establishes ground truth for subjective quality Google Cloud. If the output must be valid JSON, or a number must fall in a range, that is a function call, not a rubric line.

A definition drifts in interpretation; an example anchors it. Every level of every graded criterion needs one worked example taken from real outputs, not invented ones Arize.

The education literature has done this for decades and the mechanics transfer directly. Texas's appraiser guidance recommends that teachers and appraisers collaboratively build "look fors" for each rubric descriptor: concrete lists of what the teacher and students are doing when a given level is present TEA inter-rater reliability FAQ. The same exercise for an agent rubric means writing, under each level, what the output contains or omits.

A worked example for a single criterion, groundedness in a support agent that answers return-policy questions:

• Level 0: the answer states a return window that does not appear in any retrieved policy document.
• Level 1: the answer is broadly consistent with the policy but omits the exception for opened items, which the user needed.
• Level 2: the answer states the correct window and the opened-item exception, both traceable to a retrieved document.
• Level 3: as level 2, and it also flags that the customer's order falls inside the window, which the user did not ask about but needed.

Note what level 3 does: it earns its place by surfacing a constraint rather than by being more polite. That is a design choice you should make explicitly, because otherwise raters will quietly reward length and polish.

Keep the anchor set small and versioned. The ground truth set that anchors a rubric is deliberately smaller than the full benchmark, because its job is to be a reusable reference point as reviewers change Label Studio. Ten to twenty anchored examples per criterion is enough to expose ambiguity; a hundred is a maintenance burden nobody keeps current.

This is the step teams skip, and it is the one that makes the score defensible. Decide in advance what number counts as a pass, write it down with a date, and share it with whoever will challenge the result later.

Two shapes of pass bar work:

• Absolute threshold per criterion. Correctness must reach a stated rate; safety must be 100 percent on the sample. Anything below fails regardless of other scores.
• Non-inferiority against the human baseline. The agent may be worse than the human on a criterion by no more than a stated margin. This is the right shape when the question is "can we remove the human from the loop" rather than "is the agent good."

Then define the gate as a combination. For example: safety at 100 percent, correctness at or above the human baseline minus a fixed margin, and no single criterion below its floor. Google frames this as a quality gate that a system must clear before promotion, with success criteria specific enough to lead directly to measurable metrics Google Cloud. A vague goal like "the agent should be helpful" cannot gate anything; a statement like "factually correct, concise, and fully grounded in known documents" can.

Pre-registering the bar also protects you from a specific bias. If you set the threshold after seeing scores, you will unconsciously set it just below whatever the agent achieved. The number stops being a standard and becomes a description.

Score a sample, not the whole log. Start with thirty to fifty output pairs drawn to cover routine cases, hard cases, and edge cases, since a golden set should include real-world inputs across complexity levels DeepEval's dataset documentation.

Give both raters the same package: the rubric, the anchors, the evidence rules (what they may look at, for example whether they see the agent's tool trace or only the final text), and the scoring sheet. Then keep them independent. No discussion during scoring, no shared spreadsheet where one rater can see the other's marks.

A minimal scoring sheet keeps the comparison honest:

case_id, system (agent|human), criterion, level (0-3), evidence_quote, rater_id, notes

Two practical details raise agreement before you have even measured it. First, tell raters to score from the trace, not from memory, because taking notes during observation reduces cognitive load and gives a reference point for justifying a score later TEA. Second, blind the system label. If a rater knows which output came from the agent, expectations leak into the score, and the comparison stops measuring quality and starts measuring prior belief.

If your volume makes two human raters per case expensive, the standard compromise is a calibrated model judge used at scale, with humans on a stratified sample. LLM judges apply the same criteria to every output and can reach high agreement with human evaluators, but they must first be aligned to human judgments on a labeled set MLflow's LLM-as-a-judge guide. That alignment step is the same calibration exercise described below, with the model in the rater chair.

Now you can compute something. Score agreement per criterion rather than overall, because a single blended figure hides the one dimension that is actually ambiguous.

For graded criteria, report the share of cases where the two raters assigned the same level, and also the share within one level. For binary criteria, report the simple agreement rate and, because chance agreement inflates that number, a chance-corrected statistic such as Cohen's kappa. A rough reading guide that holds up across domains:

Treat every disagreement as a bug report about the rubric. The Label Studio calibration workflow says exactly this: disagreements usually trace to rubric language that allows multiple interpretations, tasks without enough context, or judge instructions that weight the wrong signal, and the fix is to tighten wording or add the contested case Label Studio. Research on examination scoring found inter-rater reliability is negatively impacted when no formal or consistent rater training and rubric calibration is performed, which is the same conclusion reached by a completely different field Saenz, ERIC.

One warning before you celebrate a high number. Raters can agree with each other while both drifting away from the rubric. Texas's guidance distinguishes calibration with each other from calibration to the rubric itself, and notes that consistent inflation means scores are reliable but not valid TEA. Guard against this by keeping a small set of expert-scored anchor cases in every batch; if the raters drift from the anchors, agreement is meaningless.

With a frozen rubric and converged raters, the comparison itself is trivial arithmetic. The interpretation is where the thinking goes.

Report per criterion, not just in aggregate. A typical result looks like this: the agent matches the human baseline on correctness, trails by a visible margin on concision because it restates the question before answering, and leads on surfacing unrequested constraints. That profile tells you what to fix. A single blended score tells you only whether to argue.

Run a significance check before you call a difference real. Agent outputs are stochastic, and quality comparisons on subjective dimensions need far larger samples than a button-color test to separate a real lift from run-to-run noise; some practitioner guidance puts the requirement in the thousands of trajectories per arm for subjective metrics Maxim's A/B testing guide. On a thirty-case calibration sample, treat differences as directional, not proven. Use the calibration sample to validate the rubric, then score a larger sample to make the claim.

Also decide what "human baseline" means in your scoring. The cleanest version is the historical output the human actually produced on the same input. That is a real comparison against the incumbent, not against an idealized human. If you score a hand-polished gold answer instead, you are measuring the agent against a standard no human in production meets.

Finally, check whether the human baseline itself is stable. If two humans scoring the same historical outputs disagree as much as the agent and human do, you have not found an agent quality problem; you have found an undefined standard. Fix that first, because no agent can clear a bar that moves.

When the work moves from scoring a sample to running the comparison at production volume across several models, a single API for every model type removes a whole layer of plumbing, and Quanta Inference (https://quantaintelligence.ai/inference) exposes text, image, video, audio and embedding models through one OpenAI-compatible key, which is useful once your judge and your candidate agents are different vendors.