How to Spot a Test That Passes for the Wrong Reason

Before reaching for tooling, learn to spot the shapes by eye. Almost every false pass falls into one of four buckets.

Tests that assert nothing. The test calls the function and stops. It may print the result, snapshot it loosely, or end with expect(true).toBe(true). If the function returned null instead of an object, the test would still pass.

Assertions that only check for no exception. The test is wrapped in expect(() => fn()).not.toThrow(). That confirms the code runs, not that it computes the right answer. It cannot distinguish a correct total from a zero.

Mocks that replace the logic under test. When the thing being verified is the calculation itself, mocking the calculation tests the mock. The mock assertion study published at FSE 2025 examines 4,652 test cases from 11 Java projects and finds mock assertions useful for side effects and control flow, but also documents "aggressive mocking," where excessive verification of interactions overfits the implementation. The same study cites earlier work reporting that 31% of tests generated by one tool raised false alarms because of aggressive mocking.

Fixtures that encode the bug. The factory or fixture data was built to match the wrong behavior. A date fixture in the wrong timezone, an ID that was supposed to be a UUID, a discount rate copied from the buggy constant. The test and the code agree with each other and both are wrong.

A quick triage table helps a reviewer decide what to ask for:

The QA guide on deleted tests and weakened assertions makes the case that a green remaining suite cannot prove that removed proof was redundant. It recommends treating test deletions and loosened matchers as a high-priority finding until new tests prove the same flow at the same revision.

Mutation testing runs in six steps: confirm a green baseline, generate mutants by applying operators across the source, run the suite against each mutant, classify each as killed or survived, compute the mutation score, and strengthen the tests that let mutants survive. A killed mutant caused a test failure, so the suite caught the break. A surviving mutant changed observable behavior and no assertion noticed.

The states you will see in reports matter more than the headline score. Augment Code's guide to mutation testing for AI-generated code lists the five states used by Stryker, including killed, survived, timed out, no coverage, and error. It also reports that Meta deployed LLM-based mutation testing across its apps from October to December 2024, and that privacy engineers accepted 73% of the generated tests. That number is worth sitting with: even machine-generated tests aimed at killing mutants were rejected about one time in four.

The score is not the goal. Mutation testing "does not test the software directly, but rather the tests," as researchers quoted in the same guide put it. A score is evidence about the specific mutations the tool applied, under the specific tests it ran. It cannot prove that a missing feature works, because there is no production code to mutate.

The practical problem with mutation testing is cost. Running a full suite once per mutant scales badly, which is why a long-running Stryker feature request asked for mutations limited to changed lines. The issue thread shows a workaround that mutates only files changed against the merge base:

npx stryker run --mutate $(git diff --name-only HEAD \
  $(git merge-base HEAD origin/main) -- "*.ts" ":(exclude)*.spec.ts" \
  | tr '\n' ',' | sed 's/,$//')

Two cautions from that thread are worth repeating. First, the file list duplicates the globs already in your Stryker config, so keep them in one place or they will drift. Second, the workaround still mutates whole files, not only changed lines, so it is a filter, not a scalpel. For a focused review, prefer instead to check out the diff, identify the functions it touched, and mutate only those source files with your tool's --mutate or equivalent setting.

A review workflow that fits agent-generated pull requests:

1. Establish the baseline. Run the full suite on the branch and confirm it is green. A failing baseline turns mutant results into noise, as the CircleCI walkthrough warns.
2. Scope the run. Mutate only source files the diff changed, excluding test files. For a medium-sized repo this usually keeps a run to single-digit minutes rather than hours.
3. Run and collect survivors. Export the JSON or HTML report. The mutant-survival-triage skill documents how to normalize survivor records across StrykerJS, PIT, mutmut, and Mull into one shape with file, line, mutator, original text, mutated text, and the tests that covered the line without killing it.
4. Classify each survivor. It is either a missing case, a weak assertion, an equivalent mutant (behavior unchanged), unreachable code, or a flaky run. The same skill treats equivalence as a judgment call, because deciding whether a mutant is equivalent is undecidable in general, so a small residual survivor rate is expected rather than a defect.
5. Write the test that kills it, not the test that raises the score. For each genuine survivor, name the input that separates the mutant from the original and the assertion that rejects the mutant.

That last step is where reviewers add the most value, and where a decision rule from the PracHub mutation testing guide helps. For each survivor, ask four questions in order: did a test reach the modified line, did the input make the original and mutant behave differently, did that difference reach something observable, and did an assertion actually reject the changed result? The answer tells you whether to add an input, reduce a mock, assert on persisted state, or tighten the assertion.

Take a shipping calculator from the same PracHub exercise. The contract: for nonnegative order totals in cents, shipping costs 500 cents below 5,000 cents and zero at or above 5,000.

def shipping_cents(total_cents):
    return 500 if total_cents < 5000 else 0

Now suppose the tests, maybe written by an agent after the implementation, assert only that shipping is nonnegative for totals of 1,000 and 6,000. Both branches execute. A coverage tool reports full branch coverage on the function. Yet changing the fee from 500 to 1 still passes, so a constant-mutation of 500 survives.

Change < to <= and both sample totals still produce the expected answer. Even exact assertions at 1,000 and 6,000 would miss that boundary mutant, because the only input that distinguishes the two programs is exactly 5,000. The test that kills both mutants states the contract at the transition:

def test_shipping_contract():
    assert shipping_cents(4999) == 500
    assert shipping_cents(5000) == 0
    assert shipping_cents(5001) == 0

Count the mutations concretely. Suppose the tool generates 10 mutants for this tiny function. The two nonnegative tests kill the ones that make the result negative, perhaps 2 of 10, and leave 8 alive. The contract test kills the boundary mutant and the fee-constant mutant, and any mutant that changes the zero, bringing the kill count up sharply. The score moves from roughly 20% to near 100% on this function, but the number is a byproduct. The real change is that the test now encodes the business rule, and a refactor to a different correct implementation still passes because the test never asserts that an if statement exists.

The same chain applies to an assertion weak enough to accept both answers. Asking whether shipping_cents(5000) >= 0 would kill the boundary mutant answers no: the input reaches the distinguishing case, but the assertion accepts both zero and 500. Better data and stronger assertions solve different problems, and you usually need both.

A mutation score is a ratio of mutants killed to mutants run, and it is easy to game. Do not accept "mutation score went up" as evidence without reading the survivors. The useful output is the list, not the percentage.

The scoping point bites. When you mutate only changed files, a score computed over a different file set is not comparable to last week's score for the whole repository. Keep the comparison fixed to the same scope or you will chase noise. The Stryker diff-runner discussion shows exactly this tension between running the full suite daily and mutating only changed files per pull request; teams that adopt both need to label each result by its scope.

One more reviewer habit, borrowed from the separation argument in Yegor Bugayenko's post on code and tests in different pull requests: when the tests and the code arrive in the same diff, it is hard to tell whether the author adjusted the requirement to fit the implementation. Reviewing tests first, or at least diffing the test files separately, makes weakness visible before you read the fix. The GitHub pull request review documentation gives you the mechanical tools (file filters, splitting the diff into viewed and unviewed, comments on a line range), but the decision to read tests before code is yours. If you want a skill that triages survivors for you and drafts the killing test, an agent with Quanta Skills installed can carry that loop while you review.

Mutation runs are a spot check. The cheaper fix is to make the agent produce tests that would survive a mutation run in the first place.

Get the order right. Martin Fowler's description of TDD keeps the red step as the core: write a test for the next bit of functionality, write code until the test passes, refactor. He also notes that sequencing tests is a skill, and that neglecting refactoring is the most common way teams wreck TDD. For agent work, a practitioner course page on TDD with Claude Code recommends one extra instruction: after writing the regression test, run it and show the failing assertion before changing production code. That single sentence prevents a lot of fake coverage, because the agent cannot quietly produce a test that was born passing.

The TDD guide for Claude Code suggests encoding the rules in project configuration, including assert-one-behavior tests named for the expected outcome, the Arrange-Act-Assert pattern, and a rule that the agent writes the failing test first. Put the same expectations into review: every test in the diff should name the behavior, contain an assertion that would fail on an obvious mutation of the changed line, and avoid mocking the module the change is in. The QA risk guide adds a fail-closed default: a diff that removes the only runnable proof for an important flow stays open until new proof exists at the same revision.

And keep the green suite in perspective. Every statement in this article rests on one fact from the CircleCI walkthrough: mutations that survive name the code location that no assertion noticed, and that is where the missing test lives. Run the campaign narrowly, read the survivors, write the test that kills each genuine one, and accept that a small residue will remain. That is a stronger promise than any green dot on a dashboard.