Splitting a Smoke Suite by Blast Radius: Read-Only, Writes, Money

The tiers are defined by what a check touches, not by how important the feature feels. A login test is read-only if it verifies the login page renders and the session cookie is issued against a dedicated account. It becomes a write the moment it stores a new user or mutates a profile record.

The read-only boundary is stricter than most teams assume. A production smoke suite must never create permanent data, modify user accounts, trigger billing events or send real notifications, and every check should be read-only or use dedicated smoke accounts with sandboxed permissions the QASkills production smoke suite skill. A test that accidentally charges a customer is worse than having no smoke tests at all. Write the rule into the tier definition and enforce it in review, because the temptation to "just add a quick order test" never goes away.

The money tier deserves its own boundary too. It is not "anything important." It is specifically anything that moves money, changes a quota that gates usage, or emits an invoice or receipt. A quota check belongs here even if no card is charged, because an off-by-one on a quota limit either denies paying customers or gives away product, and both have a dollar figure attached.

The core ordering rule is simple: run read-only first, write second, money last, and stop at the first failing tier. But stopping at the first failure is exactly how a read-only failure masks a money failure, so you need two refinements.

First, fail fast on read-only but do not fail silently on the tiers behind it. When the read-only tier fails, do not run write and money checks against the same broken build, because their results will be noise. Instead, mark the skipped tiers as "blocked by tier 1" in the report rather than "passed." A skipped test that reports as green is how a broken billing probe reaches production with a clean dashboard.

Second, make the tier order a pipeline dependency, not a convention. In GitHub Actions, jobs run in parallel unless one job declares that it needs another, in which case it waits for the dependency to finish before it starts the freeCodeCamp GitHub Actions guide. Use needs so the write tier cannot start until read-only is green, and the money tier cannot start until write is green the GitHub Actions workflow syntax reference. That single declaration turns ordering from a habit into an enforced rule.

The third rule: if the read-only tier fails, still run a tiny, separate money probe on a schedule. A read-only outage suggests the app is down, but it does not prove billing is fine, and the money surface has its own dependencies (the payment provider, webhook receivers, ledger jobs) that can break while the web tier is healthy. A scheduled money probe catches that divergence without ever running money checks against a broken build on the commit path.

The fourth rule covers retries. Retry read-only failures automatically, because transient latency and CDN caching are normal, and the smoke suite should tolerate minor variation with generous timeouts and assertions on ranges rather than exact strings the QASkills production smoke suite skill. Never retry a money check automatically without idempotency keys. A retried charge is a second charge unless the payment API deduplicates on a key you supply, and that is the single most expensive mistake a smoke suite can make.

jobs:
  read-only:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Health, reachability, render checks
        run: npm run smoke:readonly
  writes:
    needs: [read-only]
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Create and clean up sandbox records
        run: npm run smoke:writes
  money:
    needs: [writes]
    if: github.ref == 'refs/heads/main'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Test-mode checkout, quota move, invoice lines
        run: npm run smoke:money
        env:
          IDEMPOTENCY_KEY: ${{ github.sha }}-smoke

Notice the if on the money job. Running money checks on every pull request invites rate limits, test-data exhaustion and flaky failures that train the team to ignore pages. Run them on the release candidate and on a schedule.

A deploy gate and a page are different controls, and conflating them is where most of the pain lives. A blocked deploy is a state decision: the artifact does not move forward. A page is a human decision: someone must look now.

Quality gates in CI are typically enforced by exit codes. A test command returns zero when everything passes and a non-zero status when anything fails, and the pipeline turns that status into a required check that must pass before a merge or promotion the Momentic CI/CD pipeline testing guide. That mechanism can block a deploy, but it cannot make a human wake up.

Two hidden rules make this work in practice. The first is that a money failure freezes the surface even if the failing change is unrelated. If the billing smoke is red, nobody ships anything that touches pricing, plans or quotas until it is green, because the failure might be caused by an earlier unflagged change and a second change removes your ability to bisect. The same logic drives rollback decisions: you cannot roll back one change in isolation, all changes roll back or none do the Octopus rollback strategies guide. A freeze protects that option.

The second is that a rollback path should trigger on pipeline failure, not on someone noticing. Pipeline tooling supports failure strategies that roll back the whole pipeline when a stage or step fails, configured by failure type the Harness pipeline rollback documentation. Wiring your money tier into a rollback strategy is the difference between a gate and a report.

There is a subtlety about what a smoke pass means. A passing smoke run does not mean the build is correct; it means the build is testable the Minitap automated smoke testing guide. Keep the tier definitions honest: a read-only pass says nothing about whether a write works, and a write pass says nothing about whether a charge settles. Each tier's green light carries a narrow claim.

Take a checkout service with a 40-check smoke suite. The team currently runs all 40 on every commit, the suite takes 14 minutes wall clock, and the on-call rota ignores smoke alerts because roughly a third of them are transient failures on the payment provider sandbox.

Tag the checks by tier. Suppose 22 are read-only, 14 are writes, and 4 are money. Move the money checks off the commit path and onto the release candidate and a 15-minute schedule. The commit-path suite drops to 36 checks, and because the read-only tier runs first and fails fast, the median commit spends about 40 seconds in smoke on a healthy build and stops there when the app is down.

Now attach costs. Say a transient money-check failure wakes an engineer for a mean of 20 minutes at an on-call rate that works out to about 60 dollars for the interruption once you include context switching. If the current setup produces about 30 false money alerts a month, that is roughly 600 dollars a month in pure noise, plus the harder-to-count cost of the team learning to ignore the alert. Moving money checks to the release candidate does not just save the money; it restores the signal, because a money alert now arrives rarely enough to be believed.

The blocking side is asymmetric on purpose. Read-only failures cost a rerun; write failures cost a blocked merge; money failures can cost a customer refund and a finance reconciliation. Tiering by blast radius means the routine cost stays routine and the expensive failure keeps its alarm.

If the team wants one place to route model-backed probes, for example an LLM-judged check that a generated invoice description is sane before it reaches a customer, a single OpenAI-compatible endpoint for text models removes the need to wire several SDKs into CI (Quanta Inference, https://quantaintelligence.ai/inference). Keep that kind of probe in the money tier, since it inspects billing output.

The suite stays honest only if tags are enforced mechanically. Pick one convention and check it in CI:

1. Put the tier in the test name or a metadata annotation, for example smoke:money:invoice-lines-generate.
2. Add a lint step that fails the build if any smoke test lacks a tier tag, so new tests cannot land untiered.
3. Add a second lint step that fails if a read-only check issues a non-GET request to the target service (mock servers excepted).
4. Require every write check to register its cleanup, and fail the run if cleanup did not complete.

That last rule matters more than it sounds. Write checks that leave records behind slowly poison the sandbox until a "passing" test fails on a duplicate constraint, and the team blames the code. Teach maintenance as part of development: when a feature changes, update its smoke test rather than deleting it, because neglected suites produce flaky tests that fail for the wrong reasons the QA Wolf smoke testing guide.

Keep the suite small. A typical smoke suite holds 20 to 50 test cases and finishes in 5 to 15 minutes, and a failure rejects the build before regression begins the TestMu AI smoke testing guide. Tiering helps keep it there: when a check takes too long, first ask whether it belongs in smoke at all, then ask which tier it really is. Many "money" checks are actually read-only assertions about pricing pages that never needed a charge.

If a coding agent is generating or updating these checks, a ready-made skill can install the production smoke pattern directly into the agent rather than re-prompting it every session (Quanta Skills, https://quantaintelligence.ai/skills). Give the agent the tier rules as context, and it will tag and order new checks consistently.

The money tier is the one that crosses an organizational boundary, and it fails politically more often than technically. Before you gate deploys on a billing check, agree four things with the people who own the money surface.

First, which environment. Money checks should run against the payment provider's test mode or a sandboxed ledger, never against live cards, and never against live quota counters that affect customer accounts. Second, the idempotency story. Every money mutation needs a key derived from something stable, usually the commit or the run ID, so a retry cannot double-charge. Third, who gets paged. Put the billing owner on the escalation path, because the on-call engineer may not know whether a failed invoice line is a code bug, a provider outage or a config change.

Fourth, what the failure actually means to finance. A money smoke failure does not necessarily mean customers were charged incorrectly, but it does mean you cannot prove they were not. Say that plainly in the runbook, and include the exact query or dashboard that shows today's charge volume, so the first responder can answer "is money moving incorrectly right now" within minutes.

Do not roll back a database to fix a money failure. Database rollback scripts and restore-from-backup both risk data loss, and the guidance is blunt: do not roll back the database unless a catastrophic event justifies it the Octopus rollback strategies guide. Forward-fix the money surface, and use compensating writes for records that were created incorrectly.

Finally, revisit the tier assignments on a schedule. A check promoted to money because it once charged a real card should stay there. A check demoted to read-only after the query was made safe can move forward in the order. The point of the tiers is that the response is agreed before the failure, and that agreement only holds if it is written down and reviewed.