Scoping the Gate to Changed Lines: Coverage and Mutation

Everything downstream depends on this step, and it is the step teams most often get subtly wrong.

Compute the diff against the merge base with the target branch, not the branch tip of a parent commit. On a long-lived branch where the base has moved, git diff main returns everything that landed on main since the fork, expressed as reversals. For gate purposes you want the lines the change introduced relative to where it forked:

BASE=$(git merge-base origin/main HEAD)
git diff --unified=0 "$BASE"...HEAD -- 'src/**' > /tmp/change.diff
git diff --name-status --find-renames "$BASE"...HEAD > /tmp/changed-files.txt

Three filters belong in that command. Restrict to source paths, because a lockfile or a binary asset in the diff produces nonsense targets. Exclude vendored and generated directories. And record deletions separately, because they carry no lines to cover but they do change behavior.

From the diff you want a map keyed by file, with the added or modified line ranges in the new file:

The last row is the useful one for mutation. Line-level targeting exists, and StrykerJS accepts a mutator pattern with an explicit line range, for example --mutate src/app.js:5-7 (the incremental mode announcement). But a mutant on a line you never touched tests assertions you did not write, so the practical target is the enclosing function of each touched hunk. Widening to functions costs little and removes a whole class of false negatives where a mutant on line 119 survives only because line 120 changed.

The coverage half is the cheap one, and diff-cover is the reference implementation. It compares a coverage report in Cobertura, Clover, JaCoCo or LCOV XML with git diff output and reports the percentage of new or modified lines that are covered (diff-cover on GitHub). It has been used exactly this way on a large Python codebase: run pytest with a coverage XML report, then pipe it through diff-cover against the compare branch (a walkthrough of that pipeline).

The step everyone skips is narrowing the test selection as well as the reporting. Measuring changed-line coverage from a full test run still pays the full runtime. Two reductions keep the cost flat:

# 1. Derive touched source files -> test files via import graph or naming convention
# 2. Run only those tests, then measure
pytest $(cat /tmp/selected_tests.txt) --cov=src --cov-report=xml
diff-cover coverage.xml --compare-branch="origin/main...HEAD" --fail-under=90

The --fail-under value is the gate. Ninety percent on changed lines is a common floor because a small diff with one uncovered branch still passes at 95, while a diff that is entirely untested fails immediately.

Useful calibration numbers, from the edX team: converting from a target list to diff coverage moved project-wide coverage from under 50 percent to 87 percent over ten months, without ever gating on the global number (the Open edX writeup). The mechanism is simple. If touching a line obliges you to cover it, project coverage rises as a side effect.

A caveat worth internalizing: diff coverage is a line-executed metric and says nothing about whether the assertions are any good. The same writeup says this plainly, noting that in a tightly coupled system a change to one component can have far-reaching effects even when the changed code is fully covered. That is the job of the second half.

Mutation testing answers the question coverage cannot: would the tests fail if the logic broke? The mechanism is to inject small faults, run the tests against each mutated version, and classify the result. A mutant is killed when a test fails and survives when all tests pass; some tools count timeouts as killed and treat mutants with no coverage as undetected (Augment Code's guide, which cites the upstream tool documentation).

Why this matters disproportionately for agent-written code: when the same model writes the implementation and the test, the test tends to encode the implementation's assumptions rather than an independent specification. One team's experiment on a Python module is instructive. Twelve generated tests passed with roughly 85 percent coverage, and mutation testing found a kill rate of 57.3 percent, meaning nearly 43 percent of single-fault bugs went undetected; after human refinement the suite reached 100 percent coverage and an 80.4 percent kill rate with 33 tests (the Nimble Approach experiment). The largest category of survivors was state assertions without behavior assertions: a test that asserts a returned dictionary passes even if the function never called the collaborator it was supposed to call.

Full-repository mutation runs are brutal. Reporters on the Stryker issue tracker described runs of roughly 3,500 mutants taking about five hours on one configuration (the incremental analysis issue). Bitcoin Core's mutation pipeline takes over 30 hours on its current setup, and its maintainers' first optimization target was skipping files that had not changed, along with skipping mutants on lines with no test coverage (incremental mutation testing in Bitcoin Core). That is the same conclusion from a very different codebase: scope to the change.

The gate you want:

The 70 to 80 percent band is deliberately not 100. Aiming for 100 on every hunk invites equivalent mutants, where a change is semantically identical to the original and no test can kill it, and that becomes a permanently red gate people learn to ignore.

Here is the argument in numbers. Suppose the full coverage suite takes 30 seconds per source module and full mutation takes four minutes per module. A codebase of 100 modules costs 50 CPU-minutes per pull request on coverage alone. At 900 modules it costs 450. A target-list policy multiplies those numbers by the fixed size of the list; a diff-scoped gate multiplies by the number of modules the pull request touched, which for agent-sized changes is one to three.

Working through an example diff of 180 added lines across four files:

• Diff coverage runs selected tests for four files, roughly 120 seconds of wall clock.
• Diff mutation targets the enclosing functions of 14 hunks, which generate perhaps 140 mutants. At the 5-hours-for-3,500-mutants rate from the Stryker thread, that is about 12 minutes, so shard across four runners and it lands near three minutes (the incremental analysis issue).
• Total gate time: about five minutes, whether the repository holds 100 modules or 9,000.

That last line is the whole point. Runtime becomes a function of change size, and change size is bounded by reviewability. A gate slower than the review it backs is a gate people route around.

Each of these has bitten a real team, and none of them shows up as a red build.

Deleted code. A deleted line has nothing to cover, so diff coverage is 100 percent for a change that only removes code. If the agent deleted the only call site of a validation function and the tests for that function sat in the same diff, the tests go too, and the mutation score for the remaining hunks is perfect. The bug is an absence, and diff-scoped metrics cannot see absences. Mitigation: fail the gate when a diff lowers the total count of test cases or assertions by more than a small margin, or require an explicit label on diffs that delete test code.

Moved lines. A function moved between files, or a block reordered inside a module, appears in the diff as a deletion plus an addition. The addition has no coverage until selected tests run, and the selection heuristic can miss the move entirely if the test file imports the old path. Mitigation: enable rename detection, and treat any file with more than roughly 60 percent of lines added as a whole-file target rather than a range target. Whole-file mutation scope is acceptable precisely because moves are rare.

Generated files. Protobuf output, ORM migrations, bundled clients and vendored dependencies are all changed lines no human wrote and no test will kill mutants in. Left in scope, they produce a permanent mutation failure that someone eventually suppresses by lowering the threshold. Mitigation: an explicit exclusion glob checked into the repository next to the gate config, so it is reviewable.

Changed dependencies and environment. StrykerJS documents that incremental mode will not detect changes in files other than the mutated files and test files, and will not detect dependency updates, environment variable changes, snapshot changes, or the like (StrykerJS incremental documentation). The same blind spot applies to any diff-scoped gate. A dependency bump that changes a function's behavior touches one line in a lockfile, so the gate scopes to nothing and passes. Mitigation: route lockfile and build-config changes to the full-run path.

Untouched callers. This is the big one. A changed function's own lines can be fully covered and fully mutation-killed while a caller in another module passes arguments the old contract allowed and the new one does not. Static mutants make it harder: a mutant executed once at startup rather than during test execution has no per-test coverage, so test filtering cannot be applied to it and tools handle it differently, sometimes ignoring it and sometimes giving it a fresh environment at high cost (static mutants). Mitigation: compute the reverse dependency set for each changed function and run the covering tests of its direct callers, a far smaller expansion than the target list.

A diff-scoped gate needs a companion, not a replacement, for the full run. The pattern that works is a weekly or nightly full mutation run over a rotating slice, tracked as a trend rather than enforced as a hard threshold. Bitcoin Core's weekly run over a selected set of important files follows exactly this shape and is used to find test improvements rather than to block merges (incremental mutation testing in Bitcoin Core).

Three rules keep the diff gate from becoming noise:

1. Baseline the surviving mutants. A mutant that survived before the change and still survives after it is not the diff's fault. Compare the diff report against the baseline for the same functions and fail only on newly surviving mutants. This mirrors the reuse logic incremental tools use, where a previous result is kept when the killing test still exists and has not changed (StrykerJS incremental documentation).
2. Let full-run failures create follow-up work instead of blocked merges. A nightly run that files a ticket per newly surviving mutant gives you interaction coverage without putting a forty-minute job in front of every pull request.
3. Keep the exclusion list and the thresholds in the repository under review, and require a second reviewer on changes to that file. A gate configuration an agent can also edit is a gate agents will eventually relax.

One more point on adversarial agents. If the agent can edit the test suite, it can make the gate pass by writing tests that assert nothing, a variant of the vacuous-test problem: a test that runs, asserts something trivially true, and passes regardless of the code (freeCodeCamp on vacuous tests). Mutation score on the changed hunks is the specific defense, because a vacuous test kills no mutants. That is why the mutation half earns its runtime cost even when coverage is already green.

The published figures explain why the pressure to ship agent code needs a gate that scales with it rather than against it: research cited by Endor Labs puts the share of AI-generated code solutions containing security flaws above 40 percent (Endor Labs). A gate that makes every pull request forty minutes long will be bypassed. A gate that takes five minutes and looks only at what changed is the version people keep.

For teams already leaning on agents to write both code and tests, the natural next step is putting the agent on a harness where verification is part of the job. An expert that reads the change, runs the diff-scoped checks and hands back a finished deliverable, rather than a diff for someone else to validate, is what Quanta Experts are built to do.