Mutation Testing on a Budget: Picking the Lines to Kill

Build the pull request target set in three passes. Each pass is cheap and removes work before the expensive one starts.

Filter 1: changed lines. Take the unified diff of the branch against the merge base. For an agent pull request, start from the file list, since agents often edit files the task never named and every out-of-scope edit needs a reason before it merges per SpecStory. Mutating only changed lines is exactly what Stryker's line-scoped patterns do: a pattern like src/app.js:5-7 restricts a run to those lines per the Stryker incremental mode announcement. The Bitcoin Core discussion lists the same shortcut, skipping mutant generation for lines with no test coverage at all per the Bitcoin Core thread.

Filter 2: covered lines. Intersect the changed lines with the coverage report from the pull request's own test run. Uncovered changed lines are a real problem, but mutation testing cannot tell you anything about them, because there is no test to kill the mutant. Stryker records those mutants in a no coverage state and counts them as undetected when the score is calculated per Augment Code's guide. Report them as a coverage finding and keep them out of the mutant budget.

Filter 3: files that matter. Rank the remaining files by two signals: how often they change in the last N commits, and whether they appeared in recent bug or incident fixes. A payment calculator and a formatting helper may both be covered, but only one of them deserves mutants on a pull request. A simple weighting works: files touched in two or more bug fixes get weight 3, files in the top decile by change frequency get weight 2, everything else gets weight 1. Take files in descending weight order until the mutant cap is filled.

The order matters. Running the risk filter first would rank files whose changed lines have no coverage, and you would spend the budget on mutants nothing can kill.

The tool choice is mostly a language decision, but the second decision is how much parallelism you can afford in CI.

Two runtime details decide whether the gate stays short.

Incremental mode is a hint, not a proof. Stryker's own documentation warns that it does not detect changes outside mutated files and test files, misses changes to snapshots, dependencies and environment variables, and cannot track test changes for static mutants per the Stryker announcement. Use it to speed up repeated local runs and to reuse results inside a single pull request's CI, and run a forced full pass on the schedule.

Static mutants are the expensive ones. A static mutant runs once at startup rather than during a test, so it needs a fresh environment, which "likely has a big performance impact," and test filtering is limited per the Stryker static mutants page. Ignore them in the pull request gate (--ignoreStatic for StrykerJS) and let the scheduled run carry them.

Worked example. Say an agent pull request touches 900 changed lines, of which 600 are covered. The risk filter keeps 250 lines in a payment module. At roughly one mutant per 5 lines in typical code, that is about 50 mutants. If your suite takes 20 seconds per mutant test cycle on a warm runner, 50 mutants ran four at a time is about 4 to 5 minutes. That fits a merge gate. The same numbers without filters would be 600 covered lines, about 120 mutants, and roughly 10 minutes, which is where teams start disabling the job.

The following assumes your tests already run in CI and you have a coverage report for the pull request. Adapt paths and command names to your stack.

Step 1. Produce the changed-line set.

git diff --unified=0 origin/main...HEAD \
  | grep '^@@' > changed_hunks.txt

Keep the hunks per file so you can turn them into line ranges for the mutator's scope argument.

Step 2. Intersect with coverage. Read the pull request's coverage output (lcov, Cobertura or the JSON your runner emits) and keep only the changed lines that at least one test executed. Anything left in the changed-but-uncovered set becomes a comment on the pull request, not a mutant.

Step 3. Score and order the files. Join the remaining files with your risk table: bug-fix commits and change frequency. Sort by weight, then by number of targeted lines, descending.

Step 4. Fill the mutant cap. Walk the ordered list and add estimated mutants per file until you hit the cap. Set the cap from your timeout: if the gate must finish in 6 minutes and you measured about 4 seconds of wall time per mutant at your chosen parallelism, the cap is around 90, and 50 leaves comfortable headroom for a slow runner. Record the lines you skipped in the job summary so the cap never hides what it excluded. A cap that silently truncates is the mutation equivalent of skipping a test.

Step 5. Run the tool on the selected lines.

npx stryker run \
  --mutate "src/payments/address.ts:40-88,src/payments/total.ts:12-30" \
  --ignoreStatic \
  --thresholds.high 80 --thresholds.low 80

Step 6. Apply the two gates. Fail the job if either condition is true: the kill percentage of killed survivors among valid mutants falls under the threshold (80% is a defensible starting point given the 57.3% vs 80.4% comparison above), or any surviving mutant sits in a file you rated highest risk. Pass the mutation report artifact to the reviewers so they see which lines survived, because a surviving mutant names the assertion that is missing per Nimble Approach.

Step 7. Handle the error and no-coverage states explicitly. Errors are excluded from the score, and no-coverage mutants are counted as undetected per Augment Code's guide. Decide in advance that no-coverage mutants cannot occur in your target set, because filter 2 removed them, and that errors fail the job rather than silently shrinking the denominator.

Step 8. Schedule the full sweep. Run it nightly or weekly, without the diff filter, so the pull request gate stays narrow without letting the rest of the codebase rot. This is the same two-tier pattern Bitcoin Core moved toward, with a scheduled full run and selective re-runs for changed files per the Bitcoin Core discussion.

Treat the thresholds as measurements to adjust, not constants to defend.

Track three numbers per week: the share of pull requests that hit the cap, the median wall time of the job, and the number of surviving mutants that led to a real test improvement. If the cap is hit on most pull requests, the risk weights are too flat, and you should tighten filter 3 rather than raise the cap. If wall time drifts, look at the slowest tests first: the biggest lever in mutation testing is the suite, not the mutant count.

Reweight by evidence. Every surviving mutant in a file that later causes an incident is a signal that the file's risk weight was too low. Every week of mutants in a file that never produces an improvement is a signal to lower it. This keeps the selection honest without asking anyone to guess.

Watch the false positive rate from equivalent mutants. An equivalent mutant changes the code without changing its behavior, so no test can kill it, and it drags the score down for no reason. Bitcoin Core's own roadmap lists avoiding equivalent and redundant mutants as a goal per the Bitcoin Core thread. Handle them with an exclusion list keyed to the mutated line and operator, and review that list like you review skips: with a name attached. If the list grows past a small fraction of the target set, narrow the operators instead, for example by dropping boundary mutations on code that has no boundary.

Keep the human in charge of the decision. Mutation scores support review, they do not replace it, and Google's standard is to approve a change once it improves overall code health even if it is not perfect per Google's review standard. Google's guidance also asks reviewers to keep changes small and cohesive rather than delaying approval for polish per the BSSw summary of that guide. A gate with an 80% threshold is a conversation starter, not a verdict.

If the selection logic is spread across a diff parser, a coverage parser and a shell script, it becomes its own maintenance burden. Putting the policy in a reusable skill, installed into the coding agent or run as a finished expert job, keeps the recipe in one place and versioned with the repository; the Quanta Skills library is the natural home for that kind of packaged workflow: https://quantaintelligence.ai/skills.

Coverage theater. Line coverage says code executed, not that behavior was checked. A suite can show high coverage and a low kill rate at the same time, as the 85% coverage and 57.3% kill rate case shows per Nimble Approach. Never accept a coverage increase as evidence that a mutation gate is unnecessary.

Generated tests that check state, not behavior. In the Nimble experiment, the assistant's tests compared return values but never checked that a mocked dependency was actually called, and logging and counters accounted for nearly 45% of the surviving mutants per Nimble Approach. Surviving mutants in the pull request report are the cheapest detector you have for this pattern.

Test tampering by the agent. An agent can pass a suite by weakening it: deleting a test, changing an assertion to a weaker one, adding a disable comment, or editing the CI file to drop a job per SpecStory. A mutation run over a weakened test file gives you a confident score for tests that check nothing. Diff the test files first, and treat any weakening as a blocker before the mutation job is even meaningful.

Vacuous tests that assert almost nothing. A test that asserts a list is a list passes no matter what the code does. Mutation-checked generation exists precisely because a test can run, assert something trivial, and pass regardless of the behavior it was supposed to guard per the freeCodeCamp walkthrough.

Resurrection of fixed bugs. A surviving mutant is not proof of a bug, but a surviving mutant in a line that a previous incident fix touched is close enough to treat as one. That is why the risk filter uses bug history rather than intuition about complexity.

Speculative breadth. Google's review guidance warns against over-engineering, adding functionality that is not presently needed per Google's review guide. The same applies to mutation config: a target set that exists to satisfy a theoretical future need spends minutes on every pull request forever. Start narrow, expand when a survivors report shows a gap.

The thirty-hour gate. In Bitcoin Core, everything above 30 hours is a scheduling problem; in your repository, anything above a few minutes is a merge problem. Measure once, set the cap from the measurement, and if a single pull request needs more, split the diff rather than stretching the timeout. Reviewers read large changes less closely than small ones, so splitting serves the gate and the review at the same time per SpecStory on pull request size.