How Many Outputs to Label: Sizing a Human Review Sample

The margin of error is the most expensive dial you control, because the required sample scales with its square. Halving the margin of error from 5 points to 2.5 points quadruples the sample.

The decision rule: match the margin to the size of the effect you need to detect. If you are watching for a regression from a 10 percent error rate to a 20 percent error rate, a 5-point margin is adequate and a 2.5-point margin is wasted money. If you are trying to prove that a high-risk workflow sits below a 2 percent failure rate, a 5-point margin cannot answer the question at all, because the interval will straddle zero and the target. In that case you need either a much larger sample or a different design, such as stratified sampling that overweights the high-risk stratum.

There is a second trap. A sample sized for a proportion gives you an estimate of the overall error rate, not a reliable read on rare failure modes. If one failure type occurs in 1 in 500 outputs, a sample of 139 will almost never contain it. That is what stratification is for.

Stratified sampling splits the population into groups that behave differently, then samples within each group. In AI evaluation this matters because error rates are not uniform: a policy question in a regulated workflow and a tone-of-voice rewrite for a marketing page will not fail at the same rate, and the failures carry different costs. Stratification ensures every subgroup, including small ones, gets enough representation to be measured rather than drowned out by high-volume traffic (Statsig on stratified sampling for AI evaluation).

A workable design uses two dimensions. First, risk tier: high (financial, legal, medical, security), medium (customer-facing support and account changes), low (internal drafts and formatting). Second, output type: generated text, structured extraction, tool calls or actions, and retrieved-answer summaries. You do not need every cell; pick the cells that carry risk.

Then allocate. Proportional allocation mirrors traffic, which is right for an overall error estimate but under-samples rare risk tiers. Disproportionate allocation deliberately oversamples high-risk cells and then reweights when you report the overall rate. A simple rule that works in practice: give high-risk cells the full base sample for that cell, medium-risk cells half, and low-risk cells a floor of 30.

The practical multiplier is straightforward: if your base sample is 139 and you split into three risk tiers with separate estimates, budget roughly 3 times 139, about 417, minus overlaps where a cell already exceeds it. This is the single biggest driver of cost, so keep the number of strata small. Three risk tiers plus one split for your highest-risk output type is usually enough.

Stratification also tells you how to build your evaluation set, which is a different artifact from the review sample. Evaluation datasets should cover diverse inputs, varying complexity and edge cases, with clear scope and objectives (DeepEval dataset guidance). A golden dataset with human-validated reference answers is the ground truth against which your review findings get scored, and domain-specific sets are more useful than broad ones (Innodata on golden datasets). Surface reviewed failures back into that set so the same bug cannot return unnoticed.

The review sample has two jobs: estimate the error rate, and anchor your automated judge. Most teams now run an LLM judge continuously because human review of every output does not scale, and strong judges reach agreement with human evaluators roughly comparable to the agreement between two humans (LangChain on LLM-as-a-judge calibration). That does not make a judge trustworthy on your task out of the box.

Measure agreement with a chance-corrected statistic, not raw accuracy. If 80 percent of outputs are correct and your judge always says "correct," it scores 80 percent accuracy while learning nothing about the failures you care about. Cohen's kappa corrects for agreement expected by chance:

kappa = (Po - Pe) / (1 - Pe)

Po = observed agreement between judge and human
Pe = agreement expected from their label frequencies

The interpretation bands from medical and social science research, mapped to judge actions:

These bands and the 0.80 threshold come from the Landis and Koch convention adopted in NLP evaluation (AWS sample repo on Cohen's kappa for judges). Size the annotation set for the estimate you need: fewer than about 10 cases per error code gives a confidence interval wider than plus or minus 0.30 on kappa, while 20 to 30 per code is enough to make deployment decisions.

Two details make kappa honest. Keep parse failures and refusals in the denominator as an explicit error label, because a judge that is right on the cases where it answers is not right end to end (OneUptime on calibrating a judge with kappa). And if your judge is stochastic, repeat it and report label stability alongside single-run agreement, since majority voting is a different configuration with a different cost.

Expect the first kappa to disappoint. Research comparing nine judge models against human annotations found most judges poorly aligned, with strong models reaching kappa in the high 0.70s to mid 0.80s while human-to-human agreement was higher, and also showed that percent agreement can mislead: judges with high agreement still assigned very different scores (the Judging the Judges paper, Arize summary of the same paper). Improve iteratively by feeding human corrections back as few-shot examples and tracking agreement over time. Popular techniques include chain-of-thought prompting, reference-based scoring, confining outputs to a fixed label set and swapping candidate positions in pairwise comparisons (Confident AI guide to LLM-as-a-judge).

Judge design is not a solved problem, which is why calibration stays in the loop. An agent-based judge can inspect intermediate steps that a single-output judge misses, but that review literature is explicit that agentic judging complements rather than replaces human oversight (When AIs Judge AIs). Plan your sample accordingly.

A support assistant handles 40,000 outputs a month. It drafts replies and occasionally changes account attributes. Your prior error rate from last quarter's spot checks is 8 percent. You want 95 percent confidence and a plus or minus 5 point margin.

Base count: n = 3.8416 * 0.08 * 0.92 / 0.0025 = 452.8, so 453. That assumes a 0.5 prior; with p = 0.08 the formula yields 113, because the variance term p(1-p) shrinks sharply. Use 113 as the base.

Stratification: three risk tiers, each needing its own estimate. High-risk (account changes, billing) gets the full 113; medium gets half, 57; low gets a floor of 30. Total 200.

Cadence: re-review weekly while the prompt or model changes and monthly once stable. Two runs a month at 200 samples is 400 reviews.

Time: 5 minutes per sample is typical for a single rubric check on a short reply. 400 * 5 = 2,000 minutes, about 33 hours per month. Add one hour of adjudication for every 20 disputable cases, plus 4 hours a month maintaining the judge and rubric.

Multiply hours by your loaded reviewer rate to get cost. At an internal rate of 60 US dollars an hour, 37 hours is about 2,220 dollars a month. Now the trade-off: if you cut the margin of error to 2.5 points, every cell quadruples, and the bill roughly quadruples to the high four figures with no change in the schedule. If you halve the cadence from twice monthly to monthly, you save half the review hours but you detect a regression one cycle later, which is the real cost of a cheaper plan.

The cost trade-off you want to show your stakeholders is not sample size versus quality; it is detection latency versus spend. A small sample every week beats a large monthly sample when prompt changes ship weekly, because the monthly sample may arrive after the bad version has already been retired. Price the two options and let the decision owner pick the latency they can live with.

Keep the reviewed outputs. Each cycle produces a labeled set; failures flow into your regression tests, and the review effort compounds instead of evaporating (LangChain on trajectory evaluation and the production-to-dataset loop). The same reserved set lets you score any judge against human labels before trusting it in CI.

Cadence follows change, not the calendar. Four triggers should force a review cycle regardless of schedule: a model swap or provider change, a prompt or system message edit, a new tool or retrieval source, and a spike in user complaints or escalations. Routine cadence then covers drift between those events.

A simple rule set that survives contact with reality:

• Weekly while anything ships weekly, using the full stratified sample.
• Monthly once the system is stable, using the same strata but the smaller margin budget.
• Immediately, on a targeted slice, after any change to prompts or models, sampling only the affected output type.
• Quarterly, on a fixed stratified sample, to re-establish the baseline and refit the judge.

If your delivery process runs agents that make tool calls and multi-step decisions, review the trajectory as well as the final answer. Correct final answers can hide broken reasoning, and a sequence of decisions offers more places to fail than a single response (Google Cloud on methodical agent evaluation). Human evaluation establishes ground truth and identifies failure modes; LLM judging and code-based checks scale them, in that order.

Finally, make the cadence cheap to sustain. If review requires a reviewer to export data, read it in a spreadsheet and hand-tally, you will cut corners by month two. Route sampled outputs to a queue with the rubric visible and one-click labels, and keep a running kappa so agreement is visible without extra work. The repeated labor is the cost you are optimizing, not the statistics.

When the model mix itself is the moving part, one API key across vendors makes the comparison sample easier to draw, because you swap the model and rerun the same evaluation set instead of rewiring providers; Quanta Inference is built for that pattern. For teams that want the review and rubric work owned end to end, a Quanta expert can run the calibration cycle and hand back the labeled set.