How to Design a Smoke Test That Produces a Decision

Both are smoke tests in the broad sense, because both expose an offer before the product exists. They answer different questions and have different failure modes.

The landing page test asks whether strangers will respond to a promise at scale. Todd Jackson's interviews with founders at companies including Flexport and Vanta show that the strongest early validation came from a specific, differentiated promise rather than a broad category claim in Lenny's Newsletter. KickoffLabs puts the operational version plainly: start with one specific promise, design the page around the signup, and be honest about what exists today in its landing page chapter.

The concierge test asks whether people will accept and pay for the outcome when you deliver it manually. Harvard Innovation Labs describes it as a manual process where you deliver the service personally, then automate the most time-intensive parts only after the problem and solution are confirmed in its demand validation guide. For a service business, a manual delivery is also a genuine smoke test: nobody has to pretend the product exists.

The decision rule for choosing between them is straightforward. If your offer can be described in one sentence and delivered by software later, use a landing page, because you need volume to get a statistically usable signal. If the offer involves judgment, bespoke work, or a relationship, use a concierge test, because a click means almost nothing when the buyer expects a human. If you have no audience and no ad budget, concierge wins by default: ten real conversations beat a page with forty visitors.

Fake door tests sit inside the landing page family. ConfigCat defines the method as presenting a feature as if it exists and tracking how many users attempt to engage with it, with the warning that you must be transparent when someone clicks in its fake door explainer. The Koji guide adds the useful refinement that a click rate means little without a benchmark or control decided before launch in its fake door guide.

The most common design error is choosing an action that costs the visitor nothing. Email signups with no qualification are nearly free to give, so they measure curiosity. A booked calendar slot, a deposit, or an application form with three screening questions measures intent, because it costs time or money.

Rank candidate actions by cost to the participant:

1. Page view or scroll depth. Cost: nothing. Signal: almost none.
2. Email address only. Cost: a few seconds. Signal: weak.
3. Email plus two qualifying questions (role, team size, current tool). Cost: a minute. Signal: moderate.
4. Booked 20 minute call, no payment. Cost: a calendar slot and a small social commitment. Signal: strong for B2B.
5. Refundable deposit or prepayment for early access. Cost: real money, small amount. Signal: strongest available before you build.

Match the action to the size of the decision you are making. If you plan to spend three months building, a booked call is a reasonable bar. If you plan to raise money on the strength of the idea, a deposit is more honest.

The Good's smoke testing guide gives a concrete consumer example: an online media site that wanted to know whether a "snooze my subscription" feature would reduce cancellations added the button and measured clicks, sending clickers to a page saying the feature was coming soon in its five step guide. The action was a click, appropriate because the cost of building was small and the metric was directly tied to the behavior of interest.

For a B2B test, the equivalent is a call booking. For a marketplace, it is a listing or a demand-side post. For a paid tool, a card-authorized pilot signup is often the only number a skeptical colleague will accept.

Two diligence points make the action trustworthy. First, deduplicate: count one conversion per person or per company, not per session. Second, instrument the source: record which channel produced each conversion, because a blended rate hides that all your qualified signups came from one newsletter mention KickoffLabs recommends comparing traffic sources and conversion rates.

Where the build itself is the bottleneck, a short internal prototype costs less than a badly run test. Quanta can take a described page, form, or small app and build it with you, which is often faster than wiring analytics and a qualification flow by hand: https://quantaintelligence.ai/.

A threshold with no baseline is a guess. A baseline with no threshold is a report. You need both, written down before the first visitor arrives.

The baseline is the rate you would expect if the idea were mediocre but not dead. There are three honest ways to get it:

• Your own prior pages. If your last three offers converted 1.8, 2.2 and 2.6 percent of qualified visitors to a signup, start there.
• A comparable public page. A competitor's pricing page traffic is not public, but the shape of your channel is: if the keyword volume is 900 searches a month and you expect a 20 percent click-through on an ad, you know your ceiling.
• A control offer. Run two versions of the same page, one for your new idea and one for something you already sell. The difference is your signal, and it controls for channel quality.

The threshold is the minimum improvement that would justify building. This is a business decision, not a statistical one: the minimum detectable effect should reflect the smallest change that justifies the cost of building and the opportunity cost of the test as explained in this sample size walkthrough. If building costs three engineer-months and a 0.2 percentage point difference would not change your plan, then the test needs to detect more than 0.2 points or it is not worth running.

The arithmetic gets uncomfortable fast. At a 2 percent baseline, detecting a 10 percent relative lift, meaning 2.0 to 2.2 percent, at 95 percent confidence and 80 percent power needs roughly 80,700 visitors per variation worked through in Mida's traffic guide. Aim for a 20 percent relative lift instead and the same page needs about 21,000 per variation. That is the trade you are making when you set the threshold: a bigger required lift needs far less traffic and the same relationship is spelled out in Cro Metrics' sample size guide.

A worked example makes it concrete. Suppose you sell a bookkeeping service to design studios. You expect a mediocre offer to convert 1.5 percent of qualified visitors into a booked call. You decide that anything below 3 percent means the offer is not strong enough to build a product around, and anything at or above 3 percent means you will run a paid pilot. That is a doubling, a 100 percent relative lift. Doubling the required lift cuts the sample dramatically compared with the 10 percent case above, which is why early smoke tests should chase large effects. If the required effect is large and the traffic is small, you do not have a test. You have a reason to run a concierge test instead.

Set two secondary rules at the same time. A stopping rule, for example "we stop at 600 qualified visitors or 14 days, whichever comes first," prevents the peeking problem, where you keep checking results and stop when they look good which inflates false positives, as described here. And a segment rule: decide whether you will report the blended rate or the rate for the narrow target segment, and do not switch after seeing the numbers.

If your traffic cannot reach the threshold in a reasonable window, say so in the written plan. Then either accept a qualitative read from a concierge test, or accept that your test is a directional signal and label it that way in the memo.

Vanity signals are the ones that rise whenever anyone does anything. A thousand page views is a distribution result, not a demand result. A 40 percent email capture rate from a LinkedIn post is a measure of your post, not the market.

Others worth flagging:

• Raw waitlist size. A waitlist without qualification tells you how well the page traveled, nothing about who joined.
• Apparent click-through on a fake door with no control. A click rate is only meaningful against a benchmark you set in advance as the Koji guide notes.
• Friends and network asks. KickoffLabs separates network validation from landing page validation in its guide structure, and the reason is that your network converts for you, not for the offer see the chapter list.
• Repeated exposure. Userpilot warns that a segment that has already seen a fake door will grow skeptical if you show another, and recommends spacing tests at least six months apart per segment in its fake door analysis.
• Survey enthusiasm. "Would you use this?" answers are cheap. The Mom Test principle is to ask about past behavior and current workarounds, not future intentions as summarized by Harvard Innovation Labs.

There is also a cost to running the test badly. Fake door testing carries a real reputational risk if users feel tricked, which is why the transparency step is not optional as Chameleon's guide emphasizes. Show a clear "in development" message, offer a genuine way to be notified, and never let a clicker land on a dead end.

One structural check: if the action you chose could be produced by bots or by an unqualified audience, assume it was. A signup form that accepts any email domain, or an ad campaign targeting broad interest keywords rather than job titles, will inflate your numerator. Restrict the audience, verify the email domain, and ask one qualifying question.

Execution is short if the design is done. A minimum viable smoke test run looks like this:

1. Publish a single page with one promise, one action, and a visible note about what exists.
2. Set up one traffic source with a fixed budget and a fixed keyword or audience definition, so you can compute your denominator.
3. Track visitors, qualified visitors, conversions, and source in one table from day one.
4. Hold the test for the pre-agreed duration or visitor count. Do not extend it because the line is close.
5. Write the memo the day you close the window.

The memo is where most teams lose the decision. It should be one page and contain the hypothesis as written before launch, the actual counts, the conversion rate with its confidence interval, the threshold, and the verdict. Adding a confidence interval matters: at 120 conversions, an observed 3.1 percent rate has a range wide enough that a 2.6 percent result is entirely possible. If your threshold sits inside that range, the honest verdict is "inconclusive, and here is what we would need to resolve it."

Teams that measure carefully but decide by feel often discover afterward that the result would not have changed anything. Writing the kill condition in advance is the only defense. It also gives you a secondary benefit: a documented negative result is reusable. When someone proposes the same idea in eight months, you have a file.

Once the smoke test passes, the next real decision is what to build and how fast. This is where a decision needs a delivery path, and the sensible move after a pass is to scope the smallest version and get it into users' hands. A short discovery call is the low-risk way to pressure test your interpretation of the result before you commit build time: https://quantaintelligence.ai/calendar.