A Quickstart That Stays Working After Every Deploy

A quickstart breaks because of changes that are individually reasonable. A dependency gains a required environment variable. A command's default output path changes. A service defaults to a different port. A setup step that was implicit becomes explicit. None of these break the unit tests, because the unit tests do not start from a clean machine and walk the documented path.

This is the gap smoke testing exists to fill. Smoke tests check whether a build is stable enough to test further, running a small number of critical checks after each build rather than exhaustive scenarios Smoke Signals: A Comprehensive Guide to Smoke Testing in Software Development. A well-scoped suite is deliberately small: one practitioner's example of an e-commerce smoke suite lists eight checks completing in roughly 23 seconds and explicitly excludes features the core purchase flow does not need Smoke Testing in CI/CD Pipelines: What Actually Matters. That is the right size for your quickstart test too: one path, a handful of assertions, seconds rather than minutes.

There is a second reason these bugs slip through. Authors test from their own machine, where the quickstart has already been completed once. State left behind by a previous run hides every missing step. A clean container deletes that advantage, which is why it is not optional.

CircleCI's writeup on smoke tests makes the general point well: all the tests can pass and the deployed application still not function as expected, because unit and integration tests do not exercise the deploy environment Smoke testing in CI/CD pipelines. Your quickstart test is the same idea one layer up the funnel. It tests the environment a stranger is in, not the environment your tests construct.

The failure mode to avoid is a hand-copied script. It passes, the README gets edited, the script does not, and now you have two documents that disagree and one test that lies. Generate the script from the README instead.

Give the quickstart a recognizable marker, then add a small extraction step in CI so the script is always a faithful copy:

```

git clone https://github.com/example/project.git
cd project
./scripts/install.sh
./scripts/start.sh --port 8080

```

The fenced block is the source of truth. A short extractor pulls every block tagged `quickstart` into an order-preserving script. Keep the extractor dumb: read lines, strip the fence markers, concatenate. Commentary lives outside the tagged block so it never leaks into the script.

Two rules keep extraction honest. First, if the quickstart needs a placeholder value such as a token or a project name, write it in the README as an environment variable reference rather than a literal, and let CI supply a test value. Second, if a step cannot run unattended (an interactive prompt, a browser-based OAuth consent), that is a finding rather than an exception: either the step gets a non-interactive path or the quickstart is wrong for the audience it claims to serve.

Keep the extracted file out of the repository, or commit it with a generated header plus a check that it matches. A committed copy invites someone to fix the script instead of the README, which puts you back to two documents.

The container choice matters more than most people expect. You want the base image a user would plausibly start from, not a heavyweight CI image that already has your toolchain, your language runtime and a warm package cache. If the image preinstalls the thing your quickstart installs, the test proves nothing.

Clone the repository fresh inside the container and run the extracted script there. Cloning inside the container, rather than mounting the workspace, also catches a subtle class of bug: files referenced only because they exist uncommitted on someone's machine.

The clean-start requirement has a useful side effect. It makes the test sensitive to geography. A new contributor in another region hits a cold cache, a different default locale and possibly different network conditions. Your container will not replicate all of that, but starting from nothing catches most of it.

Practical details that save time later:

A related project makes a point worth stealing for structure: a quickstart smoke test that runs without real provider credentials by standing up a local mock server, starting the application against it, and checking the main endpoints plus at least one error path the Free-Way issue. The pattern generalizes. Any third-party service in your quickstart gets a local stub, so the test runs on a fork with no secrets and no spend.

This is where most quickstart tests go soft. Running the commands and checking that the shell did not error proves only that nothing crashed. A user needs more.

Four assertions carry most of the value:

1. Exit code. The script exits 0. Obvious, and it still catches a surprising number of failures, because a failure in the middle of a multi-command block is often silent.
2. Listening port. The expected port is open and accepting connections within a deadline. Poll rather than sleeping a fixed amount of time; a fixed sleep is either flaky or slow and often both.
3. Named output file. The file the README promises (a build artifact, a generated config, a local database) exists and is not empty. This catches the case where a tool "succeeds" and writes nothing.
4. Expected output line. A specific string the README shows appears in the logs or in a health response. Choose a stable phrase, not a timestamp or an identifier.

Two more assertions are worth adding when they are cheap. First, an idempotency check: run the start step twice and confirm the second run either succeeds cleanly or fails with the documented message. Second, a negative path: confirm that one documented error case produces the documented error. The Free-Way test treats a malformed request as part of the smoke scope, which is a good default because error handling is where new users land first the Free-Way issue.

Write the assertions against the README's own promises, not against whatever the code does today. If the README says the server listens on 8080, the test asserts 8080. When the test fails, you then decide which one is wrong, and that decision is a product decision rather than a test-maintenance chore.

Worked example: a variable, a read-only home, and a red check

Consider a project whose quickstart is four commands: install dependencies, generate a config, start the server, open a page. A team adds a variable for a cache directory, defaulting to a path derived from the user's home directory. The unit tests set it explicitly, so they pass. The README is unchanged.

Three days later, a newcomer runs the quickstart in a container where the home directory is read-only. The generate step fails with a permission error that has nothing to do with the project's purpose.

Without a quickstart test, the first report arrives as a support message, and someone reproduces it by hand, possibly not. With the test, the failure arrives as a red check on the pull request that introduced the variable, with a log pointing at the generate step. The fix is a default that does not assume a writable home directory.

The instructive part is the size of the change. The test is not a browser test of the whole product and not a coverage exercise. It is one extraction step, one container run and four assertions, and it caught a bug that every existing test was structurally blind to.

A quickstart test that drifts from the quickstart is worse than none, because it manufactures false confidence. The extraction approach from step 1 is the main defense. Two smaller habits finish the job.

Make the README part of the review surface. A pull request that changes a command in the quickstart should change the extracted script as a mechanical consequence, and reviewers should see both. If a quickstart change cannot pass the smoke test, that is a merge blocker, not a note for later.

Second, keep the prose honest about what the test asserts. If the README shows an expected output, the test should check for it. If the test cannot check for it because the output is inherently variable, say so in the README rather than showing a fake exact result. The Berkeley library guide's README checklist is a useful sanity list for the surrounding content: a brief description, installation instructions and a short example or tutorial How to Write Good Documentation. Your smoke test covers the installation and tutorial parts. Everything else on that list is prose, and prose is where the boundaries in Diátaxis matter, since a tutorial overloaded with reference detail distracts the learner Start here - Diátaxis in five minutes.

One more check worth having: make sure the test runs from the commit a reader would get. A quickstart can be technically correct and still fail because it points at a branch, a package version or a URL that only resolves on the maintainer's setup.

A quickstart smoke test proves the documented path works from a clean start. It does not prove the product is correct, it does not cover every configuration, and it will not catch a confusing sentence. Keep it narrow on purpose. The smoke-testing literature is consistent that the value comes from running a handful of critical checks quickly on every build, not from broadening scope until it duplicates the regression suite Smoke testing in CI/CD pipelines. A quickstart test that takes twenty minutes gets skipped, and a skipped test protects nobody.

Where the quickstart touches an API, the documentation ecosystem has a related habit worth borrowing: treat the specification as the single source of truth and generate from it so examples cannot drift from reality API documentation best practices guide. Your extracted script is the small local version of that principle.

If the quickstart calls a hosted model alongside your own service, the clean-container run is also the cheapest way to notice that a credential or endpoint assumption has changed. Quanta Inference puts text, image, video, audio and embedding models behind one OpenAI-compatible API key, which simplifies the stub-versus-real decision: stub the local service, and keep any paid endpoint call out of the per-commit path.

Here is the checklist to ship this week:

A quickstart that survives every deploy is not a documentation achievement. It is a test suite achievement that happens to live in a Markdown file.