Sizing a Shadow Run from Rubric Variance

You cannot size a run from a rubric you have not tested. The cheapest test is to have two or more people independently score the same 30 to 50 examples and compare, treating disagreement as evidence the rubric is ambiguous rather than the reviewers according to Arize's rubric testing guidance. Label Studio's evaluation guide makes the same recommendation in more detail: build a small ground truth set of expert-scored tasks that exercises every rubric dimension, keep it as a reusable anchor, and reserve a subset for calibration in the Label Studio scaling guide.

From that pilot you extract two numbers.

1. The standard deviation of scores for a single case, which captures both true quality differences and rater disagreement combined.
2. The disagreement between raters on identical cases, which is the part you can shrink by adding raters.

If you have the pilot scores, the second number is easy to estimate as the average absolute difference between raters on the same case, or by fitting a simple variance components model. The important discipline is to write both numbers down before the real run starts. Sizing from memory or intuition is how teams end up with a run that cannot answer the question it was built for.

The Texas Education Agency's inter-rater reliability guidance describes the same practice in a different domain: calibrate appraisers against the rubric itself, not just against each other, because a group can be internally consistent and still be consistently wrong per the TIA inter-rater reliability FAQ. Consensus is necessary but not sufficient.

Here is the arithmetic, using round numbers.

Suppose three raters score a pilot set of 30 cases on a 5-point rubric. The standard deviation of the scores is 0.8. That 0.8 is the spread you see for a single case scored by a single rater.

Averaging raters is a variance-reduction move. The standard error of the average rating for one case is the single-rater spread divided by the square root of the number of raters:

SE_case = 0.8 / sqrt(3) = 0.462

That is the noise you carry for any individual case after averaging three raters. It is still large relative to a 5-point scale.

The number you usually care about is the mean score across all cases, because that is what a promotion decision rests on. The standard error of that mean drops again with the number of cases:

SE_mean = SE_case / sqrt(n) = 0.462 / sqrt(n)

A 95 percent confidence interval is roughly 1.96 standard errors on each side, which for practical sizing you round to 2. So to get a margin of error of plus or minus 0.2 on the mean score, you need a standard error of the mean of 0.1:

0.1 = 0.462 / sqrt(n)
sqrt(n) = 4.62
n = 21.3, so 22 cases

The finite population correction barely matters here unless your pool is tiny, but it is worth knowing the calculator exists and adjusts for both finite population and clustering as the Statulator single-proportion calculator documents.

Two things about this arithmetic surprise people.

First, the case count is small relative to the effort of labeling. Twenty-two cases scored by three raters is 66 grading decisions, which is a day of work for a small team. If your decision needs a tighter margin, the cost rises fast because n grows with the square of the ratio.

Second, adding raters has diminishing returns once raters are already averaging. Going from one rater to three cuts the per-case standard error from 0.8 to 0.462, a 42 percent reduction. Going from three to six only takes it to 0.327, a further 29 percent, and costs twice the grading time. The table below makes the trade-off visible at a fixed target: a 95 percent margin of 0.2 on the mean, with the same 0.8 single-rater spread.

The total work is nearly flat across the row. What changes is the shape of the risk. Fewer cases with more raters protects you against bad luck in case selection less well, because a smaller case set may miss whole categories of failure. More cases with fewer raters gives better coverage but leaves each individual verdict noisier. The decision rule I would use: pick three raters as a default, because it gives you a disagreement signal on every case and keeps the case count large enough to cover your failure taxonomy. Drop to two only when grading time is truly scarce, and go above four only when you need per-case verdicts at high confidence, not just a portfolio mean.

The worked example sized for a margin. The more useful question is whether your affordable run can detect the difference that would change your decision.

Suppose the incumbent process scores 3.6 on the rubric and you will promote the candidate only if it scores above 3.8. The gap is 0.2 points. Your run can distinguish two means reliably when the confidence interval around their difference excludes zero, which for two independent groups of equal size and a per-case standard error of 0.462 needs roughly:

n per group = 2 * (2 * 0.462 / 0.2)^2 = 2 * 21.3 = 43 cases per group

That is 86 cases total, more than double the 22 you needed for a single estimate. This is the most common sizing failure in shadow runs: teams size for a description of the candidate and then try to use it for a comparison. A comparison of two arms costs roughly four times the cases of a single-arm estimate at the same margin, because the variance of a difference is the sum of two variances.

For agent and model comparisons, the numbers get larger still. Maxim AI's testing guide notes that agent A/B tests on quality metrics typically need 10,000 or more trajectories per arm to separate a real quality lift from model stochasticity, especially when the metric is subjective per Maxim AI's A/B testing article. That figure assumes small effect sizes and high per-sample variance. If your rubric has a single-rater spread of 0.8 on a 5-point scale and you only need to detect a 0.5 point difference, the case count falls dramatically, which is an argument for defining a decisive effect size before you start.

There is also a raters-side inversion. If you cannot afford more than 15 cases, you can ask what rater count makes 15 cases meet the margin:

SE_mean = 0.462_r / sqrt(15) where 0.462_r = 0.8 / sqrt(r)

Setting the result to 0.1 gives 0.8 / sqrt(r * 15) = 0.1, so r * 15 = 64, or about 4.3 raters. Round up to five. Pair the same case set with five raters, and total grading decisions are 75, in the same neighborhood as the 66 above. The volume and rater dials trade off against each other along a curve of roughly constant total work, so the real design question is coverage versus per-case precision.

The cheapest way to shrink a shadow run is to shrink the spread before you size anything. Rubric design does that directly.

Microsoft's rubric reference for Copilot Studio uses a standard 5-point scale from "needs improvement" to "exemplary" with observable definitions at each level, and it defines alignment between an automated grade and a human grade with an explicit formula rather than a verbal judgment in the Microsoft Copilot Studio rubrics guide. That kind of definition is what keeps raters from collapsing toward the middle of the scale.

Hebbia's evaluation writeup argues for going further: pass/fail criteria rather than graded levels, one criterion per failure mode, and criteria grounded in specifics like exact chart-to-data mappings, because binary decisions are faster for graders to converge on and easier to validate against human judgment per Hebbia's hybrid evaluation framework. Their reasoning is directly relevant to sizing. On a binary rubric, the variance you measure is variance in a proportion, and the sizes follow the standard proportion formula, which is easy to publish in advance and compare across runs. On a 5-point scale, the spread depends on how raters use the middle, which is exactly where they disagree most.

The practical decision rule: if the shadow run is deciding pass or fail, score pass or fail. A binary rubric with ten independent criteria gives you ten proportions with clear confidence intervals and no averaging ambiguity. Reserve graded scales for cases where the size of the miss matters, and even then, keep the scale short. Ambiguous criteria produce different scores even with the same judge, prompt, and temperature, and the right response is to revise the rubric rather than average the noise away as Splunk's evaluation framework guide notes on rubric validation.

Rubric calibration is not a one-time event. Saenz's study of examination scoring across five years found that inter-rater reliability falls when no formal or consistent rater training and calibration is done, and that neither faculty status nor length of employment made a significant difference to mean scores per the ERIC record of the study. The lesson for a shadow run is that every rater should be calibrated on the rubric before they touch a scored case, and re-calibrated when the rubric changes.

Sometimes the question is not "what is the mean score" but "do raters agree well enough to trust the rubric." That is a different calculation with its own sizing rules.

If you use Cohen's kappa as the agreement statistic, size on the distance from your expected kappa to the nearer confidence limit, not on the full interval width. Worked numbers show the cost: with an expected kappa of 0.60, requiring the interval to reach 0.55 needs about 1,625 subjects, while reaching 0.40 needs about 118 per the kappasizeci documentation. That is an 11-fold difference between two requirements that both look modest on paper. Decide the kappa you need to demonstrate before you commit to a rater count.

A tutorial on sample size calculation for inter-rater agreement studies makes the same point in a formal setting, stressing that the design has to specify precision in advance and that both the number of raters and the number of subjects enter the calculation in the Indian Journal of Psychological Medicine tutorial. The tutorial's framing matches the shadow run problem exactly: you are estimating a reliability property, you want a standard error, and both raters and subjects buy it.

For a practical sanity check, McHugh's heuristic of at least 30 comparisons, with 1,000 or more giving very small confidence intervals, is often cited as a floor as summarized in Lukas Wallrich's writeup on interrater agreement sample sizes. Treat 30 as a minimum for any agreement claim at all, and expect to pay the kappa-precision price above that floor if the claim is going to gate a release.

Work in this order, and write the numbers down before you run.

1. Write the decision. Name the threshold, the incumbent score, and the gap that would change your mind.
2. Pilot the rubric. Score 30 to 50 cases with two or more raters, and record both the single-case standard deviation and the rater disagreement per Arize's rubric testing guidance.
3. Fix ambiguous levels. Rewrite any level where the pilot showed systematic disagreement, and add the disputed case as a worked example per the Label Studio guide.
4. Compute the per-case standard error as the pilot spread divided by the square root of the rater count.
5. Invert for cases, using a margin that matches your decision, and remember that a two-arm comparison needs about four times the single-arm case count.
6. Check the budget. If total grading decisions exceed what you can afford, trade cases for raters along the constant-work curve, or loosen the decision to a single-arm description instead of a comparison.
7. Calibrate every rater on the final rubric before the scored run, and re-calibrate if the rubric changes per the ERIC study of examination scoring.

If the run itself is expensive because it depends on model outputs, one practical way to keep grading cost down is to generate candidate responses through a single endpoint rather than juggling per-provider keys; Quanta Inference puts text, image, video, audio and embedding models behind one OpenAI-compatible API with prepaid dollar pricing, which makes the case-generation step a line item you can estimate before you size the run. That matters here because the case count you calculate should be a number you can actually pay for.

Where the run needs a decision written up for stakeholders, from the sizing memo through to the final promotion recommendation, an AI expert on the relevant line of business can take the design and hand back a finished deliverable rather than a draft. Describe what you need at Quanta and a specialist will pick it up with you.