When to Stop a Smoke Test Early: A Sequential Stopping Rule

The rule has exactly three pre-registered inputs. Write them into the test document before the first visitor arrives, and do not edit them afterwards.

1. A maximum sample size. This is the number of visitors you would collect if you never stopped early. It comes from a standard power calculation: baseline conversion rate, the minimum detectable effect you care about, your chosen false positive rate and your desired power. The inputs and their trade-offs are the same ones behind any sample size estimate described in this guide to A/B sample size calculation. Cro Metrics recommends planning around a full business cycle and notes that a minimum of 100 conversions is a common floor, though the number should follow from the calculation rather than a rule of thumb in Cro Metrics' sample size guide.

2. A short, fixed checkpoint schedule. Keep it small. Three looks at 25, 50 and 75 percent of the maximum, plus the final look at 100 percent, is plenty for a smoke test. More checkpoints mean more error budget spent and a stricter boundary everywhere. The guidance on group sequential designs is to fix both the number and timing of looks in advance in this comparison of stopping rule families.

3. A stopping boundary computed for that schedule. The boundary answers one question at each checkpoint: how far from the target rate must the observed rate be before we accept that the gap is real? It is computed once, from the maximum sample size, the checkpoint fractions and the target error rate, and it stays fixed even if traffic arrives slower than planned.

A fourth, non-negotiable rule: if a checkpoint is missed because traffic dried up, you do not get to test at whatever N you happen to reach. That is peeking with extra steps.

Smoke tests produce binary outcomes: a visitor clicks or does not. The beta-binomial model is the natural fit because it treats the click count as binomial given an unknown click rate, with a beta prior on that rate. The practical version for a stopping rule is short.

Let p be the true click rate. Your decision is between two hypotheses. H0 says p is at or below the threshold you would not act on, for example 2 percent. H1 says p is at or above the target that justifies building, for example 3 percent, a 50 percent relative lift. The gap between them is your minimum detectable effect. Converting that effect into a sample size via the usual power inputs gives the maximum N using the same inputs as a standard sample size calculation.

At each checkpoint, compute the posterior probability of H1 given the clicks and visitors seen so far, starting from a weak prior. Stop for a win when that probability crosses an upper boundary; stop for a kill when the posterior probability of H0 crosses the matching lower boundary. The boundaries are set so that, under H0, the chance of ever crossing the upper boundary at any checkpoint is at most your chosen alpha, and symmetrically for the kill side. This is the same error-budget logic as alpha spending: you allocate the total false positive risk across the planned looks rather than spending it all at the end as explained in the stopping rule comparison.

Here is the decision procedure in plain steps, using a maximum of 20,000 visitors, checkpoints at 5,000, 10,000 and 15,000, and a final look at 20,000:

1. Before launch, record the maximum N, the checkpoint fractions, the win boundary and the kill boundary.
2. At each checkpoint, compute the cumulative visitors and cumulative unique clickers.
3. Compute the posterior probability of each hypothesis under a weak prior.
4. If the win posterior crosses the win boundary, stop and record a win. If the kill posterior crosses the kill boundary, stop and record a kill. Otherwise continue.
5. At the final checkpoint, apply the same boundaries. There is no additional test; the schedule already spent its error budget.

The single most useful insight is that the boundary at the 25 percent checkpoint is severe. You need a click rate far above the target to stop that early, because a small sample carries little information. That severity is the price of the option to stop, and it is what keeps the overall error rate honest.

Early stopping is an option, and like any option it has a price. The table below summarizes the trade-offs.

The honest expectation from the research is that early stopping helps most when the effect is large, and barely at all when the truth sits near the boundary. Group sequential designs preserve high power at the fixed maximum sample size when you can estimate that maximum in advance according to Spotify's comparison. Sequential methods generally trade a little power at the end for the ability to stop early, and the sources are explicit that this trade-off cannot be avoided when a strict cap on false positives is enforced per Statsig's evaluation. Always-valid methods, which allow truly continuous monitoring, typically give up slightly more power at a fixed sample size as the family comparison notes.

A worked example makes the savings concrete. Suppose your maximum is 20,000 visitors and the gap you care about is 2 percent to 3 percent. If the true rate is 4 percent, the posterior crosses the win boundary comfortably at the 25 percent checkpoint, so you stop after roughly 5,000 visitors and save 75 percent of the planned traffic. If the true rate is a marginal 2.3 percent, it will almost certainly not cross until the final look, and your expected saving is close to zero. The rule is asymmetric by design: it pays out on obvious answers and charges full price on ambiguous ones.

A pre-registered rule is only as good as its audit trail. If the analysis lives in someone's dashboard filter, the next person to look will not be able to reconstruct why the test stopped.

Log the following at every checkpoint, and keep the log with the test document:

• The checkpoint fraction and the planned sample size for that checkpoint.
• The cumulative visitors and cumulative unique clickers.
• The observed click rate and the posterior probability of each hypothesis.
• The boundary values in force at that checkpoint, and whether the observed statistics crossed them.
• The decision taken, and by whom, with a timestamp.

A structured log entry is enough to make the audit trivial:

{
  "test_id": "pricing-page-smoke-01",
  "checkpoint_fraction": 0.5,
  "planned_n": 10000,
  "observed_visitors": 9940,
  "observed_clicks": 198,
  "observed_rate": 0.01992,
  "posterior_win": 0.08,
  "posterior_kill": 0.71,
  "win_boundary": 0.95,
  "kill_boundary": 0.90,
  "decision": "continue",
  "logged_at": "2026-01-14T10:00:00Z"
}

The log serves a second purpose beyond audit. When the rule stops a test early and the answer turns out to be wrong later, the log tells you whether the boundary was violated or whether you simply had a bad draw inside the agreed error budget. Those are different failures with different fixes.

One related risk deserves a line in the log too: repeated exposure to the same audience. If a segment has already seen a fake door, a second one erodes trust, and product teams are advised to space such tests at least six months apart per user segment according to Userpilot's guidance on when not to run fake door tests. A stopping rule cannot protect you from that; only the test registry can.

The rule is not universal. Four situations argue for running to the fixed maximum with a single look.

When the idea is complex or multi-step. A single button cannot represent a process that takes many steps, so click data will not reflect real interest, and prototypes or usability testing are better tools as Userpilot argues. In those cases the entire smoke test design is wrong, not just the stopping rule.

When a regression is the thing you fear. If the offer sits on a live funnel and a low conversion rate costs money every hour, early stopping for a kill is valuable. That is the case sequential methods were built for: detect harm early rather than waiting for the full sample as the sequential testing overview describes. If you only care about confirming a win and there is no cost to continued exposure, the added complexity buys little.

When your expected effect is small relative to noise. If you are trying to detect a 10 percent relative lift on a 2 percent baseline, the sample size becomes large, and the boundary at early checkpoints becomes so strict that you will almost never stop early anyway. The traffic figures in this guide show the scale of the problem: a 2 percent page aiming to detect a 10 percent lift needs tens of thousands of visitors per variation per Mida's traffic guide. You pay the complexity without the option.

When the test is also a message test. A smoke test often tests whether people understand the promise at all. If you are checking whether visitors understand the problem and want the solution, as KickoffLabs frames the validation question, you want enough data to segment the audience and see which sources convert, not the earliest possible answer in KickoffLabs' landing page validation guide.

One honest caveat on the fake-door variant: click-through rate measures curiosity, not commitment, and users who click a button have not proved they would pay or change their workflow as Userpilot notes. A tight stopping rule makes the measurement sharper; it does not make the metric mean more than it does. Pair the number with follow-up conversations with the people who clicked.

For teams building the model-driven parts of a validation site, such as generated copy variants or automated scoring of signups, one API key for text, image and embedding models keeps the experiment plumbing separate from the statistics. That is what Quanta Inference is for: https://quantaintelligence.ai/inference. If you would rather have the whole validation site, the waitlist and the reporting built with you, a specialist can pick it up through https://quantaintelligence.ai/.