Writing a Smoke Test Suite the Pipeline Runs on Every Deploy

The short answer from the intro holds up: health, auth, one read from each critical service, one reversible write. Here is how each category earns its slot.

Health and readiness. The main URL returns a success status, the health or readiness endpoint returns success, and the app can reach its database. QA Wolf's smoke checklist covers exactly these, plus dependent services such as auth, payments, queues and third-party APIs.

Authentication. One real login through the same path users take, including token issuance. Broken sign-in is the classic example of a failure that makes deeper testing pointless, and it is cheap to assert.

One read per critical service. One warm request per service you depend on, checking status and a couple of fields in the body. Gift the failure a schema check: status 200 with an error payload is still broken.

One write that rolls back. Create a record through the real API, assert it exists, delete it, assert it is gone. This catches wiring and permission failures that read-only checks miss, and it leaves no residue in production data.

A sensible inventory looks like this:

That lands comfortably inside the 10 to 20 check range. TestMu AI describes a typical smoke suite as 20 to 50 cases finishing in 5 to 15 minutes (TestMu AI's smoke testing guide). That is fine as an upper bound, but a pipeline gate you want people to leave enabled should aim much lower, which the next section covers.

Every minute you shave off a build is a minute saved for each developer on every commit, and GitLab cites Martin Fowler's ten-minute build guideline for continuous integration (GitLab's CI best practices). Even ten minutes is too slow for a gate that blocks every deploy. The target for post-deploy smoke is one to two minutes end to end, which is what keeps it from being the step people disable.

Get there by spending the budget deliberately:

• Run checks in parallel. Independent HTTP checks finish in the time of the slowest one, not the sum. Start all checks at once, then fail fast on the first hard failure.
• Force one serialized chain. Auth depends on login, and the write depends on auth, so keep that chain linear and make everything else concurrent.
• Cap every request. A timeout of 10 seconds per check with two retries turns a 30 second hang into a 30 second worst case, and slow is a failure too.
• Skip the browser unless the release touched the front end. HTTP-level checks cover most deploys in seconds; save a single headless page load for releases that change rendering.
• Warm the target first. The first request after a cold start can take several seconds; fire one throwaway request to /health before the timer starts.

A worked example: login at 1.5 seconds, auth-only /me at 0.4, four service reads in parallel at 0.9 each, one write at 1.2 with delete at 0.3, and static assets in parallel at 1.1. Serial time inside the dependent chain is about 3.4 seconds plus the parallel block, so wall time is roughly 4 seconds plus runner setup. Even with a cold start and retries you stay well under a minute. If your suite runs past 120 seconds, something in it is a regression test.

Treat flakiness as a bug, not noise. Teams that ignore maintenance end up with unreliable smoke tests that fail for the wrong reasons (QA Wolf). When a check flakes twice in a week, either fix its setup or delete it; a gate that cries wolf gets bypassed.

Smoke testing exists to check that a build is worth testing further. It does not validate that every feature works. Duplicating pre-merge coverage post-deploy buys nothing and costs the trust you need.

That comparison follows the standard distinction between the three: smoke is broad and shallow on a new or unstable build, sanity is narrow and deep on a specific change, and regression is full coverage before release (TestMu AI). Anything already covered in the merge pipeline does not need to rerun against production with production data.

Three categories in particular deserve to stay out:

• Load and performance testing. It needs a separate environment and a sustained load plan; a smoke run that doubles as a load test will be flaky by construction and slow by definition.
• Edge cases. Invalid inputs and validation logic are precisely the cases TestMu AI says a smoke suite deliberately skips. If an input is invalid, a passing smoke test would never have caught it anyway.
• Anything requiring manual steps or seeded business data. If a check needs a human to log in and click through a flow, it belongs in exploratory or acceptance testing.

The dividing line is simple: if the failure would not justify a revert at 2 a.m., leave it out.

A smoke failure should read like an incident report, not a stack trace. Harness's smoke testing guidance stresses that strong smoke suites stay small and dependable, with clear assertions and rich failure artifacts, and that they verify the deployment using real health signals where possible (Harness on smoke testing in CI/CD). The cheapest way to get there is to make every check log one line on failure, in a fixed shape.

FAIL check=login_read GET https://api.example.com/me
  status=401 expected=200
  body={"error":"token expired"}
  target=smoke-7f3a deployed=e3b0c442
FAIL check=write_create POST https://api.example.com/items
  status=500 expected=201
  body={"error":"database is read-only"}
  target=smoke-7f3a deployed=e3b0c442

Four details do most of the work:

• The request line. Method, URL and the check name, so a reader can reproduce it in one command.
• The observed against the expected status. The pair is the assertion, and the expected value should be printed literally rather than implied by the check's name.
• The response body, truncated. Twenty lines is enough to see "read-only database" or "expired token."
• The deployed identifier. The commit SHA or version tag ties the failure to an artifact and a diff.

Print that identifier into the run itself, not just the log. In GitHub Actions, the run-name key lets the workflow title include expression values from the repository context (GitHub's workflow syntax reference), which means the deploy SHA can appear in the list of runs without anyone opening a log. Pair it with a required check so a failing smoke job blocks promotion, which is what harness calls turning smoke tests from slow manual steps into deployment gates (Harness).

One more rule: fail once, fail loud, and do not bury it. A smoke failure that needs a second human interpretation is a rollback you will debate for twenty minutes.

The mechanics are boring, which is the point. Workflow files live in .github/workflows and run when an event such as a push or a deployment fires (GitHub Actions quickstart). Place the smoke job after the deploy step in the same workflow, and scope it to deployed environments.

name: deploy-and-smoke
on:
  push:
    branches: [main]
jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v6
      - run: ./scripts/deploy.sh
      - run: ./scripts/smoke.sh
        env:
          SMOKE_BASE_URL: ${{ vars.SMOKE_BASE_URL }}
          SMOKE_TOKEN: ${{ secrets.SMOKE_TOKEN }}
          DEPLOYED_SHA: ${{ github.sha }}

Two operational habits matter more than the YAML:

1. Gate on the smoke job. If the step fails, the pipeline fails and the release does not proceed. That is the whole value of a gate; a check that only warns is documentation, not control.
2. Never point smoke at production data you cannot afford to touch. The reversible write must use a dedicated smoke account and delete what it creates, and the check should be idempotent so a retry does not double-create.

Keep credentials in secrets, not in the script, and keep the suite's own runtime predictable: a smoke suite with wild timing is unreliable and slow. If the post-deploy check has become a job you outsource, a QA automation or release-engineering consultant can rewrite it in a day or two. Quanta Intelligence can put an AI expert for a line of business onto that kind of scoped work, or hand the smoke-suite refactor to a specialist through Quanta.