Sizing a Review Sample: Confidence, Error Rate and Volume

The formula is easy. Choosing p, the confidence level and the margin is where teams argue, so here is a decision rule for each.

Pick p from the consequence, not the average. If a wrong answer in a customer email is embarrassing but correctable, a 5 percent detectable rate is reasonable. If a wrong answer touches payments, access provisioning or anything irreversible, the rate you care about may be 1 percent or lower. A 1 percent detectable rate at 95 percent confidence and a 1 point margin needs around 38,000 samples if you review at random, which tells you immediately that random sampling cannot be your only control for that tier. That is a finding, not a failure: it tells you to gate the action instead of reviewing after the fact.

Use 95 percent unless someone is paying for 99. The jump from 95 to 99 percent confidence increases the required sample by about 73 percent (1.96 squared versus 2.58 squared). Reserve the higher level for the tier where a miss becomes a regulatory or safety event, and be explicit that this is why the number is larger.

Set the margin by the decision you need to make. If your remediation trigger is "the true rate is above 5 percent," a 3 point margin lets you see a sample result of 7 percent and know the true rate is plausibly 4 to 10 percent, which is enough to act. A 1 point margin is what you need to argue that quality improved from 5 to 4 percent.

One practical caveat: these formulas assume the sample is random and that outputs are independent. Agents running the same workflow in a loop produce correlated outputs, so the effective sample is smaller than the count suggests. This is a strong argument for spreading the sample across sessions, tenants and task types rather than taking 203 consecutive runs.

Say a support triage agent produces 2,000 outputs a week across three categories. The team currently reviews 10 percent, which is 200 outputs, split evenly by whoever is free. That looks roughly right by luck, but the split is wrong.

Start with the target: catch a 5 percent failure rate at 95 percent confidence with a 3 point margin. That requires 203 outputs. Because 2,000 is far larger than 203, you can treat the weekly volume as effectively unlimited and skip the finite population correction; the calculators note that the adjustment barely changes results for populations above roughly 100,000, and here the ratio of sample to volume is about one in ten, so the correction would shave only a few outputs. Round to 210 for scheduling convenience.

Now allocate across tiers. Use risk and volume together, not volume alone:

That is 210 reviews and it changes what the team learns. Under the old even split, Tier 1 got about 67 reviews and was undersampled relative to its risk, while Tier 3 was over-sampled for a tier that already has a human gate. The tier allocation is a judgment call, and you should write down the reasoning because auditors and incident reviews will ask why each number is what it is. Human-in-the-loop design guidance makes the same point about approval routing: match oversight to blast radius and reversibility, not to a uniform percentage.

If the week's Tier 1 reviews find 5 failures out of 100, your observed rate is 5 percent with a 95 percent interval of roughly 1.6 to 11.3 percent. You cannot conclude the process is fine, and you should not conclude it is broken either; you should increase the Tier 1 sample next week and, in parallel, add a pre-send check on that path. That is the value of doing the arithmetic: it stops a single bad week from triggering either panic or complacency.

At very high volume, even a correctly sized random sample becomes expensive, and some tiers produce more output than any team can review. Three tactics keep the control honest.

Triggered review plus a random audit. Send every output that trips a runtime flag (guardrail block, low confidence, unusual tool call, policy near-miss) to a human, and add a random sample of the rest. The guardrails literature describes runtime checks that block or escalate when an agent drifts, and operational guidance stresses that access and action decisions should be logged as they happen. Triggered review is fast and targeted; the random audit is what tells you whether the triggers are missing failures. Without the random leg, you only ever see what the flags already caught.

Gate the action instead of reviewing it. If an agent can do something irreversible, review belongs before execution, not after. Approval gates on irreversible or regulated actions are the recommended pattern in Dataiku's design patterns for governed agents, and the matching approval record (who approved, when, against which request) belongs in the audit trail, as Permit's audit trail guidance sets out. For that tier, the sample size is 100 percent of the action, by design.

Use double sampling to save reviewer time. When a first small sample is inconclusive, take a second rather than starting large. The NIST handbook's treatment of double sampling shows how a first sample with accept and reject thresholds avoids a second sample in most cases, and how the average number inspected drops as a result. Applied here: review 120 Tier 1 outputs first; if failures are at or below 3, stop; if at or above 8, stop and escalate; if in between, review another 120. In a stable week you inspect less than the full 210 and learn the same thing.

A small script makes the weekly number reproducible, which matters when someone asks why the sample changed:

import math

def weekly_sample(p, z, margin):
    n = (z**2 * p * (1 - p)) / (margin**2)
    return math.ceil(n)

print(weekly_sample(p=0.05, z=1.96, margin=0.03))  # 203
print(weekly_sample(p=0.05, z=2.58, margin=0.03))  # 351

Run it whenever the failure rate you care about, the confidence level or the margin changes, and record the output in the review policy so the number is traceable.

A sample size is only half the control. The other half is interpreting what the sample found without overreacting or underreacting. Two rules cover most weeks.

Report the interval, not the percentage. A week with 5 failures in 100 looks alarming next to a target of 2 percent, but the interval is wide. A week with 18 failures in 210 is a different situation: the point estimate is 8.6 percent and even the optimistic end of the interval sits above your 5 percent threshold. Reviewers should write down "failures, sample, interval" every week, not just a pass or fail, because human evaluation guidance is clear that no single signal, whether an automated eval or a small human sample, is sufficient on its own; you combine human sampling with traces and production signals.

Re-size when the process changes, not on a calendar. A prompt change, a new model behind the agent, a new tool permission or a new customer segment all invalidate your estimate of p. Re-run the formula with the new assumption and adjust the sample for a few weeks after the change. This is the same logic agent evaluation frameworks apply to autonomy levels: the oversight needed depends on the agent's behaviour and risk, not a fixed schedule.

If the team finds itself doing this sizing by hand for several agents and several tiers each week, and pulling traces from more than one provider to do it, the arithmetic is the easy part and the plumbing is the hard part. Quanta Inference puts text, image, video, audio and embedding models behind one OpenAI-compatible API key, which at least keeps the trace source consistent while you sample.