How Many Evals Are Enough? A Sample Size Rule for Model Testing

Suppose your product requirement is simple. A model version passes if its success rate on the core task is at least 90 percent, and you want to be confident that a version which truly sits at 90 percent does not slip below the bar by accident.

You decide you can tolerate a half width of 5 percentage points. Run the shortcut: n equals 1 divided by (0.05 squared), which is 400. The square root of 400 is 20, and 0.5 divided by 20 is 0.025, so the 95 percent half width is 1.96 times 0.025, or roughly 0.049. That is 4.9 percentage points, just inside your tolerance (Statology's formula confirms the arithmetic, and San Jose State gives the n equals 1 over d squared shortcut explicitly).

Now run it with real numbers. You evaluate 400 examples and 356 pass. The observed pass rate is 356 divided by 400, or 0.89. The standard error is the square root of 0.89 times 0.11 divided by 400, which comes to about 0.0156. The 95 percent interval is 0.89 plus or minus 1.96 times 0.0156, so 0.859 to 0.921. The interval straddles 0.90. You cannot yet say whether the version clears the bar.

Here is the interesting part. If the same observed rate had come from 600 examples instead, the standard error would fall to about 0.0128 and the interval would be 0.865 to 0.915. Still straddling. The point is that the observed rate matters as much as the sample size, and a result sitting right on the threshold is expensive to resolve.

A clearer example. Suppose version B passes 380 of 400 examples, a rate of 0.95. The interval is 0.929 to 0.971, entirely above 0.90. That is a clean pass. Now suppose version B had produced that same 0.95 from only 100 examples. The standard error would be about 0.0218, and the interval would run 0.907 to 0.993. Technically above the bar, but barely, and a 2 point miss in the next sample would flip the decision. That is the case where more examples genuinely change the call.

The pattern in the middle column is the lesson. Each halving of the margin quadruples the work. Budget accordingly, and spend the money where the decision actually lives.

A pass rate on its own tells you almost nothing. The number that matters is the distance between your observed rate and the threshold that changes what you ship. Statisticians call this the gap between your estimate and the null value. Practitioners should call it slack.

There is a useful way to think about it. If your threshold is 0.90 and your observed rate is 0.96, you have 6 points of slack. If your interval half width is under 3 points, the entire interval sits above the threshold and the decision is stable. If the half width is 5 points, the interval dips to 0.91 and you are one unlucky batch away from a flip. So the rule is not "collect 400 examples." The rule is "collect enough that your half width is smaller than your slack."

This is exactly the distinction that matters in agentic systems, where a headline accuracy number can hide failures that matter at volume. A 1 percent failure rate on 10,000 daily tool calls is 100 wrong actions a day, which is why raw accuracy misleads on imbalanced or high stakes workloads (Galileo's accuracy metrics guide makes this argument and covers precision, recall and F1 alongside it). Your threshold should be set on the failure rate you can actually absorb, not on a round number you copied from a slide.

The same discipline shows up in customer research. Practitioners are told to focus on what people actually did rather than what they say they would do, because hypothetical answers are cheap and noisy (The Mom Test notes summarize Rob Fitzpatrick's distinction between compliments, fluff and concrete past behavior). An eval case where you asked the model a leading question and accepted a plausible answer is the model equivalent of a compliment. It passes, it feels good, and it tells you nothing. Real cases, drawn from actual traffic, are the equivalent of "tell me about the last time you did this" (the desert and Pacific I-Corps guide makes the same point about past behavior predicting action far better than hypotheticals).

So the decision boundary has two components: a numeric threshold, and a definition of what counts as a valid case. Get the second one wrong and no sample size saves you. A neat test for this: would a skeptical colleague agree your 400 cases are representative of production traffic? If the answer is "well, we wrote them ourselves," you have measured your imagination, not your model.

Every number above assumes your examples are independent draws from the population you care about. Violate that and the arithmetic flatters you. Three failure modes are common.

The first is clustering. If you generate 400 cases but 300 of them are paraphrases of ten underlying scenarios, your effective sample size is closer to ten than 400. The interval you compute will be far too narrow. The fix is to count distinct scenarios, not rows, when you do the sizing.

The second is contamination. If your eval cases leaked into training data, or if a case appears in both the tuning set and the final eval set, the pass rate is inflated and the variance assumption collapses. Keep a held out slice you never look at until the final gate.

The third is threshold shopping. If you add examples, check the result, add more, check again, and stop when the number looks good, you have quietly inflated your false positive rate. Decide the sample size before you run, in the same way a clinical trial pre-registers its analysis plan. The qualitative literature is blunt on the adjacent problem: saturation is a property of the data, not something you can know in advance, so you plan a target and code as you go rather than declaring victory at a convenient point (Koji's sample size guide covers the data versus theoretical saturation distinction and why the two milestones differ).

A related trap is over-ranking. If you compare five model versions on the same 100 examples, the chance that one of them looks best by luck is substantial. With a half width of about 10 points at n equals 100, differences of a few points between neighboring versions are noise. Either shrink the number of comparisons or grow the eval.

For teams running several model families through the same harness, the plumbing should not compete with the statistics. Wiring text, image, audio and embedding endpoints behind one OpenAI compatible key removes a class of integration bugs that quietly change what you are measuring, which is the point of Quanta Inference when the eval set spans modalities.

Here is the rule I would hand to a team on day one. Write it on the wall.

1. Name the decision and the threshold. "Ship if the core task pass rate is at least 0.90" is a decision. "Check quality" is not.
2. Pick the smallest margin that would change your mind. If 3 points of difference would not alter what you ship, do not pay to measure to 1 point.
3. Compute n as 1 divided by the square of the half width, using the worst case pass rate of 0.5. Round up.
4. Take a pilot of 50 to 100 cases and estimate the observed rate. Recompute the interval using the real p, not the worst case. If p is far from 0.5, the required n drops.
5. Compare your slack to your half width. If the half width exceeds the slack, either collect more or lower the threshold ambiguity.
6. Stop when the interval no longer straddles the threshold, or when the cost of more cases exceeds the cost of a wrong call. Both are legitimate stopping reasons.

The pilot step is the one people skip and the one that saves the most money. A preliminary run is also how qualitative teams avoid wasted interviews: the suggestion is to practice and pilot before a full round, and to review each conversation before scaling up (Commoncog's field report on JTBD interviews describes limiting early interviews to recent buyers precisely because that is where the signal is easiest to read).

If you want a quick sanity check on a specific rate you already have, the standard error calculator will preview the interval for you (Pearson's standard error calculator shows the formula and the p hat plus minus z times SE preview), and Statistics How To's overview is a reasonable refresher on why sample size narrows the distribution. None of these replace a decision rule. They just make the arithmetic trustworthy.

The question in the title has a precise answer, and it is not "when you have 400." It is: more cases stop changing the call when the confidence interval is entirely on one side of the threshold.

Put differently, you are done when the lower bound clears the threshold with room to spare, or when the upper bound falls short of it. Anything else is a coin flip wearing a percentage.

There are three signals to watch.

• The interval clears the threshold. Stop. More cases will narrow the interval further but cannot change the verdict.
• The interval straddles the threshold and is not narrowing fast enough to matter. This happens when the observed rate sits almost exactly on the threshold. More cases will not save you; sharpening the case definition or splitting the metric into submetrics will. A single pass rate that averages two very different failure modes is a classic case for splitting.
• The interval straddles but the decision has a deadline. Ship with a documented interval and a monitoring plan, and say plainly what the evidence supports.

The second signal is where the framing about outcomes helps. Jobs to be Done research argues that the decision to switch is triggered by a measurable gap between the current state and the desired one, and that understanding the trigger is more useful than understanding the product (Strategyn's framework overview sets out that logic, and dscout's interviewing primer covers the trigger, first hire and switch questions). An eval threshold is the same object: not a number on a slide, but the point at which your organization would actually change behavior. If nobody would act differently at 0.89 versus 0.91, your threshold is wrong, and no sample size fixes a threshold nobody obeys.

That is the one step further. Do not ask how many examples you need. Ask what decision you are trying to make, what margin would change it, and how much uncertainty you can carry into that decision. Then the sample size falls out of the arithmetic, and the interval tells you when to stop.

For teams that want to spend their time on the decision rather than on the harness, an AI specialist can help set thresholds and evaluation design for a specific line of business, and a free discovery call is the fastest way to scope it: Quanta Consulting.