How to Route Smoke Test Results Between a Model and a Human

There is direct evidence that language models summarize what a program looks like it should do rather than what it actually did. Researchers at Waterloo and Carnegie Mellon tested summaries by mutating programs and checking whether the summary changed to match the new behavior. Accuracy fell from 76.5 percent for single functions to 17.3 percent for multi-threaded systems, and on human-written programs the rate was 49.3 percent, as reported in the mutation-analysis study. The authors describe the pattern plainly: models often describe algorithmic intent while missing the logic change that defines actual behavior.

Smoke test reporting is exactly this shape. The checkpoints have familiar names (health endpoint, login, core read, safe write), and a model that has read a thousand CI logs will happily write "all core paths healthy" because that is the intent of the suite. The actual state may be that the write path returned a 500 and the runner skipped the assertion.

Summaries also invent detail when the source is thin. Reporting on summarization research notes that a substantial share of summaries contain fabricated facts and that most named entities in them can fall outside the source document's scope, a pattern described in this analysis of why LLM summaries fail. A missing checkpoint is a thin source. The model fills the gap.

The lesson for smoke tests is not "do not use a model." It is "never let a model be the only reader of a failing gate." The mutation study's own framing supports this: an inaccurate or incomplete summary can lead developers to draw the wrong conclusions about how code works, which is exactly what a plausible smoke verdict does to a release decision.

Route by consequence of being wrong, not by how tidy the summary looks. Concretely, three conditions must all hold before a model summary may stand as the record of a smoke run:

1. The raw counts are attached to the summary, per checkpoint, with pass, fail and skip numbers, plus the checkpoint list from the run log.
2. The totals reconcile: the sum of per-checkpoint results equals the total the test runner reported. A mismatch of even one test blocks the summary.
3. Every checkpoint in the plan reported a status. A checkpoint absent from the output is a missing status, not a pass.

If any condition fails, or if any checkpoint failed, a person reads the raw lines before anyone acts. The model may still draft the summary; it just becomes a draft that a human signs, not the artifact the team acts on.

Two rows deserve comment. "All pass, totals do not reconcile" is the sneaky case: a runner that reports 18 of 20 tests as passed and silently drops two looks identical to a clean run in prose. "Any skip" matters because a skipped write-path check is not a pass, and treating it as one is how a broken build reaches staging.

If you want the rule to be machine-checkable, a short pre-check is enough to gate the model call:

def may_summarize(run):
    if run.failed > 0 or run.skipped > 0:
        return False, "fail or skip present"
    if run.reported_total != run.expected_total:
        return False, "totals do not reconcile"
    if set(run.results) != set(run.plan):
        return False, "checkpoint missing a status"
    return True, "summary may stand"

That function is the whole rule. Everything else is formatting.

Teams that route by impression tend to use prose quality as a proxy for correctness. That proxy fails for the same reason code summaries fail: fluency and accuracy are separate properties, and the mutation study found the gap grows with system complexity, from 76.5 percent accuracy on single functions down to 17.3 percent on multi-threaded systems (mutation-analysis study). Your smoke suite is a multi-component system by definition, because it spans auth, data, network and flags. It sits at the hard end of that curve, not the easy end.

Smoke testing guides are consistent that the gate is binary and that failing means reject the build and fix it first. The Momentic smoke testing guide lists the usual critical paths (launch, login, navigation, checkout, APIs) and states that if a smoke test fails, the team rejects the build. The Ranorex guide makes the same point about smoke tests acting as a quality gate after deployment to test and staging environments, and it frames the smoke suite as superficial in depth but broad in scope, which is why losing a line from the report costs you a whole area rather than a single case.

There is a useful analogy in product validation. A smoke test in the startup sense, as described in this overview of smoke, concierge and MVP strategies, measures whether people click a pre-order button. The click rate is meaningful only against the traffic that saw the page. A summary that says "strong interest" without the denominator is a story, not a result. Test reporting has the same denominator problem: "all passed" without the checkpoint count is a story.

None of the above argues against using a model. It argues for scoping the task. Three jobs are safe and useful:

• Normalizing the log. Turning three runner formats into one table of checkpoint, status, duration and error class. This is transcription, and transcription is checkable against the source.
• Drafting the human-readable paragraph that sits under the table for release notes or a ticket comment, with the counts embedded so a reader can audit it.
• Clustering failures across runs. Grouping the last twenty failed smoke runs by error signature so an engineer can spot that fourteen of them are the same expired token. This is a search and grouping task where wrong guesses are cheap because a person is already in the loop on the fail path.

The unsafe job is the one people reach for first: generating the verdict. A verdict is a decision, and the rule should own decisions.

This is the same routing habit that applies to model selection elsewhere in production work, where you pick a model by the cost of its mistakes rather than by the polish of its output. When a team needs one endpoint for several model types behind a single key, a gateway such as Quanta Inference keeps the transcription and clustering calls in the same billing and auth path as everything else, which matters when the logging pipeline is what feeds the routing rule.

A routing rule written after the run is a rationalization. Write it into the smoke test plan, next to the checkpoint list, and make it part of the pull request that adds a checkpoint.

The practice is the same one that keeps experiments honest. Sequential testing methods exist precisely because looking at data repeatedly and stopping when it looks good inflates false positives; Spotify's engineering write-up on choosing a sequential testing framework gives the canonical example of a 5 percent false positive rate climbing toward 10 percent after just two looks. Fixing the decision rule before collecting data is what makes the result trustworthy. Here the "data" is the smoke output and the "decision" is whether a model summary may stand, but the discipline is identical.

A stopping rule is an error budget, as this comparison of stopping rules frames it: you start with a total tolerance and decide how to spend it. Your smoke reporting rule is an error budget too. You are deciding, in advance, how much unreviewed machine summary you will tolerate before a person must look. For a binary gate on the critical path, that budget should be zero for any fail, skip or mismatch.

A sample-size mindset helps calibrate the other direction. When a test is underpowered, the honest answer is "collect more data," not "the trend looks good," a point made repeatedly in A/B sample size guidance and in this breakdown of how traffic determines detectable effects. The smoke equivalent is: when a checkpoint is missing, the honest answer is "rerun or read the raw log," not "the rest passed."

Suppose the smoke plan has 20 checkpoints. The runner reports 18 passed, 1 failed, 1 skipped, and its own total line says 19 executed. The model writes: "Smoke run broadly healthy, 18 of 19 checks passed, one checkout assertion to review."

Apply the rule:

• Failed greater than zero: yes, so the fail path triggers. A person reads the raw lines.
• Skipped greater than zero: yes, second trigger.
• Reported total 19 against expected 20: mismatch, third trigger.

The summary's phrasing did real damage. "Broadly healthy" is a character judgment, and "18 of 19" hides the 20th checkpoint entirely. Under the rule, this summary is a draft for a human, and the human's first job is to find out what the twentieth checkpoint was and why the runner did not execute it. Then to read the failing checkout assertion, because a failure on the write path is exactly the case where nobody acts on a prose verdict.

Now change one variable. Same 20 checkpoints, 20 passed, runner total says 20, all plan entries present. The model summary may stand, and the counts sit in the table beneath it. That is the only case where the first reader can be a person who reads only the summary, and even then the counts are one glance away.

The rule is boring on purpose. Boring rules get followed at 3 a.m. before a release.

A short checklist to paste into the test plan:

• Plan lists every checkpoint, and the count is stated in the plan.
• Runner writes a total that can be compared to the plan count.
• The report includes per-checkpoint pass, fail and skip counts.
• The route function runs before any model call, not after.
• Any fail, skip, mismatch or missing status goes to a named person, not to a queue.
• The model's draft is labeled as a draft whenever the route function returned false.

Teams that would rather not maintain the plumbing around this themselves can describe the reporting job and have a specialist build it: Quanta takes a description of the site, app, document or plan you need and pairs you with someone who builds it with you.