When the Review Sample Is Too Small to Trust

Run this list against any review report, internal or external, before you act on its conclusion. Each row is a check you can complete in a few minutes.

Two rows deserve emphasis. First, percentages without denominators are the most common way small samples get dressed up as big findings: "60% of reviewers preferred the new interface" sounds firm until you learn it was 3 of 5. Second, a missing stopping rule turns a small sample into an unreliable one, because optional stopping lets a team keep adding cases or re-cutting subgroups until something crosses the threshold. The fix is preregistration, specifying the research plan in advance and submitting it to a registry so planned and unplanned analyses can be told apart the Center for Open Science. Its guidance notes that the same data cannot be used both to generate and to test a hypothesis without reducing the credibility of the result the same page.

There are two separate mechanisms, and conflating them causes most confusion.

The first is arithmetic. The standard error of a mean is the sample standard deviation divided by the square root of n, so uncertainty shrinks with the square root of sample size. Quadruple the sample and you roughly halve the width of the interval. This is why halving a survey's margin of error approximately quadruples the required sample under the usual proportion formula a sample-size planning guide with the formula and worked values. That page gives a concrete planning number: at 95% confidence with an anticipated proportion of 50% and a margin of plus or minus 5 percentage points, the large-population calculation is 384.16, rounded up to 385 completed responses same source. The same page warns that the margin covers sampling uncertainty under the stated design and does not include leading questions, missing coverage or systematic nonresponse, so a precise estimate of a biased group can still mislead same source.

The second is distributional. With small samples and unknown population standard deviation, substituting the sample standard deviation into a normal-based formula is no longer valid, and the standard fix is Student's t distribution with n minus 1 degrees of freedom, which has heavier tails a worked treatment of small-sample confidence intervals. That switch matters in practice: it makes the interval wider, which is exactly the correction a small review needs. The t interval assumes the underlying values are roughly normal, so with very small samples also check for extreme outliers that would break that assumption same source.

One terminology caution. A pilot or feasibility study is not designed to estimate an effect precisely; it exists to test whether the design, measures, recruitment and procedures will work at scale, and its sample size should be justified by that goal rather than by a formal power calculation Kunselman's overview of pilot studies. Reviewers frequently cite a pilot's apparent effect as if it were a confirmatory result. It is not, and a common warning is that pilot effect estimates are unreliable inputs for powering the main study same source. If a review is explicitly a pilot, judge it on feasibility questions, not on the size of the gap it reports.

Suppose an internal review compares two onboarding flows. Group A (current) has 8 reviewers; Group B (redesign) has 8. Each reviewer completes a timed task, and the outcome is seconds to completion. The summary: Group A mean 100 seconds, standard deviation 20 seconds; Group B mean 88 seconds, standard deviation 22 seconds. The reported "12% faster" looks like a win.

Compute the 95% interval for the difference in means. With small samples and unknown population standard deviations, the standard error of the difference is the square root of (s1 squared over n1 plus s2 squared over n2):

s1 = 20, n1 = 8   ->  s1^2 / n1 = 400 / 8  = 50.0
s2 = 22, n2 = 8   ->  s2^2 / n2 = 484 / 8  = 60.5
sum = 110.5  ->  standard error = sqrt(110.5) = 10.51 seconds

difference = 100 - 88 = 12 seconds

For a small-sample interval, use the t critical value. A conservative choice is degrees of freedom equal to the smaller group size minus one, which is 7; the 97.5th percentile of t with 7 degrees of freedom is about 2.365. A more precise approach uses the Welch-Satterthwaite approximation to the degrees of freedom, which lands near 14 and gives a critical value near 2.16. Take the conservative one:

margin = 2.365 * 10.51 = 24.9 seconds
interval = 12 +/- 24.9 = (-12.9, 36.9) seconds

The interval runs from about 13 seconds in favor of the redesign to about 37 seconds in favor of the current flow. It contains zero. Under the usual reading, this review has not shown a difference: the data are consistent with the redesign being meaningfully faster, slightly slower, or identical. Reporting "12% faster" as a conclusion is exactly the error the checklist is designed to catch.

Now the decision rule. Suppose the team agreed in advance that the redesign would ship only if the lower bound of the interval showed at least a 5 second improvement. The observed lower bound is negative, so the rule says do not ship, and the honest follow-up is a larger sample, not a reinterpretation. To halve the margin from roughly 25 seconds to roughly 12.5, you need about four times the observations per group under the square-root relationship, so plan for around 32 per arm. That number is a planning approximation from the formula above, not a guarantee. The right way to set it is a power calculation that states the effect you care about detecting, the alpha level and the target power. Power is the probability of detecting a difference that genuinely exists; a study with too few subjects is underpowered and may miss a real effect a primer on power analysis and its role in replicability. Sample size planning tools make the arithmetic routine by letting you set anticipated means, variance, alpha and power ClinCalc's sample size calculator. The drivers are the same everywhere: rarer outcomes, higher variance and smaller effects all demand more subjects the same calculator's explanation.

One more subtlety worth knowing before you argue about a result. Choosing the sample size after seeing the data is not a fix; it converts a planned test into an unplanned one. If you must collect more, treat it as a new study with its own preregistered plan, or model it explicitly as a sequential design, which preregistration guidance treats as a distinct case from optional stopping the Center for Open Science.

A short routine keeps the diagnosis consistent across reviewers and vendors.

1. Find the denominators. Write down n per group and the number analyzed after exclusions. If either is missing, stop and ask.
2. Find the spread. Locate the standard deviation, standard error or interval. If only means or percentages appear, request the raw data or the variance.
3. Compute the interval for the difference yourself. Use the conservative t value if group sizes are small; the t table entry for the relevant degrees of freedom is the number you need the small-sample interval walkthrough.
4. Check whether the interval clears your pre-agreed margin, not just zero. A result that excludes zero but sits far below your practical threshold is detectable and operationally irrelevant.
5. Check the analysis history. Was the endpoint fixed before collection? How many subgroups were examined? Every extra comparison raises the chance that one looks impressive by luck.
6. Record the verdict as one of three: usable, underpowered but informative for planning, or uninterpretable.

That third category is not a failure of the review. It is a finding about the review's design, and it should route to the same kind of structured learning that incident reviews use: a blameless postmortem produces a written account of what happened and a list of steps to prevent recurrence, and its value comes from institutionalizing continuous improvement rather than assigning fault PagerDuty's guide to postmortems. The same framing applies to AI-assisted reviews, where a structured postmortem with a standard format, tracked action items and dissemination beyond the immediate team is what turns an incident into organizational learning, with a note that timing matters because too soon and emotions interfere while too late and memory fades Pertama Partners on AI incident postmortems. An underpowered review that is documented as underpowered, with the specific numbers and the sample size needed next time, saves the next team from repeating it.

Reviews of model-assisted workflows add two wrinkles that inflate the risk of a too-small sample.

The first is variance. Model outputs vary run to run and prompt to prompt, so the spread within a group can be much larger than reviewers expect from human-only work. Higher variance widens intervals and demands more observations for the same precision ClinCalc's explanation of the drivers of sample size. A review that ran each prompt once per reviewer is measuring the reviewers, not the system's variability.

The second is iteration. Agentic workflows produce intermediate steps, tool calls and retries, and each of those can be a place where an evaluation quietly becomes exploratory. Teams building agents already have to engineer for cost and observability, tracing every reasoning step and tool call across an execution cycle, precisely because the loop's behavior is not visible from the final output Oracle's description of the agent loop. That tracing is also what makes a review auditable: if you cannot see how many steps ran and which tools fired, you cannot tell whether the sample was comparable across arms.

A third wrinkle is scope. Autonomy is a spectrum, and more autonomy is not automatically better; useful autonomy is bounded by permissions, budgets, approvals and explicit stopping rules a guide to autonomous agents. A review that compares a bounded configuration against an unbounded one is comparing two different systems, and the difference in outcomes may reflect the guardrails rather than the model. Fix the configuration and the stopping condition before you sample, or the interval you compute will be measuring the wrong thing.

When a team needs to run these reviews across text, image and audio models without rebuilding the evaluation harness for each provider, one option is a single OpenAI-compatible endpoint with prepaid, dollar-denominated billing, which keeps the sampling and logging code identical across model types: Quanta Inference.