Writing Test Suites an AI Agent Cannot Fake

Return values are the easy half. The half agents skip is the set of observed effects: what was written, what was called, what was logged, what was emitted. An experiment with generated test suites reported by Nimble Approach found that this gap was the single largest source of missed faults: their generated tests checked returned state but ignored interactions, and logging and counter checks accounted for nearly 45 percent of the mutants that survived.

Three rules close most of it.

Assert interactions that matter, once. If the function is supposed to write to an audit table, assert one row with the fields you care about. Do not assert call counts on every collaborator, since that turns tests brittle against refactoring.

Assert side effects through a real boundary when the boundary is cheap. A temporary file, an in-memory HTTP client, or a test database asserting a row is more convincing than a mock asserting a call.

Before writing negative cases, read the authorization path. A Cloud Security Alliance article on AI code security gives a concrete example of the failure mode: a generated check written as user.role == "admin" instead of "admin" in user.roles works for single-role users and silently over-permits for multi-role users. A test with a two-role user, asserting both the allowed and the denied path, catches it. Without that user, the code passes review and passes its own tests.

For anything touching trust boundaries, confirmation emails, or configuration changes, the Metacto pre-merge checklist treats failure-path tests as a required pass condition, owned by both author and reviewer. Adopt that as a hard rule: no merge without at least one test that exercises malformed input, an empty state, or a permission failure.

A fixture's job is to make the test's world small and known. If a test can only pass on a developer laptop, it does not prove anything about the code, only about the laptop. Agents amplify this because they tend to build objects from scratch with every attribute specified instead of using existing factories, which is a pattern the Thoughtbot reviewers flag as a signal that the underlying implementation wants a refactor.

A working fixture policy has four parts:

1. Freeze time. Inject a clock or freeze it in the fixture, and assert on timestamps you control.
2. Isolate external services. Point HTTP clients at a local stub, and assert the request the code made. Never let the suite depend on a live third-party API.
3. Build data through one factory. When a field is added to the model, one file changes, not fifty tests.
4. Tear down state. Each test starts from a database, temp directory, or message queue that is empty or reset.

The review signal is the inverse of what reviewers usually watch for. Huge setup blocks are not a sign of a thorough test. They are a sign that the unit under test needs too much of the world. When a test needs a long list of mocks to run, the Google review guidance would call that complexity in the test, and complexity there is a maintenance cost with no product value.

There is a second hidden dependency that has nothing to do with the code. Generated imports can name packages that do not exist, and an attacker can publish a package under such a name. The GitHub guidance on reviewing AI-generated code names both hallucinated packages and slopsquatting, and asks reviewers to verify every suggested dependency yourself. A test-only dependency runs with the same privileges as production code, so audit those in the same pass as the fixtures.

Coverage answers one question: which lines did the suite execute. It does not answer the question you actually care about: would the suite notice if the code were wrong. Mutation testing answers the second question by inserting small faults into the source and rerunning the suite for each one. The Augment Code guide to mutation testing for AI-generated code describes the loop and the result states the tooling reports: a mutant is killed when at least one test fails against it, survives when every test still passes, and a mutant that causes an infinite loop is counted as killed. Mutants that throw an error are excluded from the score, and a mutant that no test reaches is counted as undetected.

The numbers from a published experiment are worth carrying into a review conversation. Nimble Approach had a model write unit tests for a Python module. Twelve tests passed, with roughly 85 percent coverage. The mutation kill rate was 57.3 percent, meaning nearly 43 percent of single-fault bugs went undetected. After humans refined the suite, coverage reached 100 percent and the kill rate reached 80.4 percent, with 33 tests. The jump from 12 to 33 tests is the interesting part: the suite was small because it was shallow, not because the code was simple.

A mutation run also has an operational cost, so scope it. Run it on changed files only, on a schedule, or on the modules that handle money, permissions, and data writes. A usable loop looks like this:

# Python, using mutmut on one module
mutmut run --paths-to-mutate src/pricing.py
mutmut results          # lists survived mutants by line
mutmut show 14          # shows the diff of mutant 14

Read the survivors, not the score. Each survivor is either a missing assertion, dead code, or an equivalent mutant that cannot be killed, and only the first category needs a new test. A team that sets a kill-rate target without reading the survivor list will chase equivalent mutants and stall.

The decision rule for a threshold: pick the current measured kill rate on a stable module, require new or changed code not to lower it, and treat any survivor on a trust boundary as a blocker. That keeps the bar objective without demanding a rewrite of the whole suite.

A skipped test is a promise nobody is checking. The three common forms all look the same in a diff: @pytest.mark.skip, it.skip, and an expected-failure marker such as xfail. Some are legitimate, such as a known upstream bug or a test for a platform you cannot run in CI. Most are a test an agent could not make pass and quietly disabled, and that outcome leaves the behavior unproven.

Make the rule mechanical. Every skip or expected-failure marker carries four pieces of information:

• A reason with a ticket reference, so the next reader can find the issue.
• A named owner, a person, not a team inbox.
• A condition that ends the skip, such as a version of the dependency or a date to re-check.
• Confirmation that the behavior is not tested anywhere else.

@pytest.mark.skip(
    reason="flaky: upstream queue drops acks, see OPS-412",
    # owner: dana
)
def test_retry_after_ack_loss():
    ...

Both the SpecStory checklist and the Metacto standards treat any skipped, deleted, or weakened test as a merge blocker until a person signs off, and it belongs near the top of the review, because reading the test diff first tells you whether the rest is worth reading. A weakened assertion is the same problem wearing a different coat: replacing a length check with a defined-ness check makes a broken result pass.

The freeCodeCamp walkthrough of mutation-checked test generation describes the failure mode exactly. A test that asserts a list is a list passes no matter what the underlying bug does, so it provides no coverage of the behavior despite looking like every other green test. A skip is the same object with the assertion removed.

Two more review habits finish this off. First, confirm a human owner can explain each changed file; SpecStory cites the OWASP secure coding with AI cheat sheet as requiring that AI-generated code have a human owner. Second, list the touched files rather than trusting the pull request summary, which the agent wrote: git diff --name-only origin/main...HEAD exposes the files the task never named, and each of those needs a stated reason. The Google standard of code review sets the tie-breaker for those disputes: technical facts and data overrule opinions and personal preferences, so a survivor list or a failing mutation run settles an argument about whether a test is strong enough.

The full gate is short enough to put in a pull request template, which turns the rules into something a reviewer checks rather than remembers. The SpecStory guide notes that GitHub fills the pull request body from a pull_request_template.md on the default branch, so the checklist shows up automatically.

Tests
- [ ] No test was deleted, skipped, or weakened; every skip body has a reason, owner, and expiry.
- [ ] Each added test names an input and an expected output, and would fail if the named behavior broke.
- [ ] Side effects are asserted: writes, calls that matter, logs with values.
- [ ] Fixtures freeze time, isolate services, build data through a factory, and reset state.
- [ ] Mutation run on changed modules: no new survivors on trust-boundary code.

Then run the checks in the order the GitHub review tutorial recommends: automated tests and static analysis first, then intent, then quality. That order is not arbitrary. If a test is weakened, passing it means nothing, so the test diff checks come before the code reading.

There is a wider point worth reading into reviews of generated code. The Georgetown CSET report on cybersecurity risks of AI-generated code evaluated snippets from five language models on the same prompts and found that almost half contained bugs that were often impactful and could lead to malicious exploitation. That report also notes that evaluation benchmarks for code models often focus on functional correctness without assessing secure code, which may incentivize deprioritizing security over functionality. The report's own caveat is that these results are limited to a narrow evaluation, and that is exactly why the gate above needs to be mechanical rather than a matter of vigilance.

When your team runs the same suite across many modules and needs a second pair of eyes on the spec and the exemptions, a consulting call is a reasonable next step. If you would rather have an expert review the suite and the mutation configuration directly, the Quanta experts can pick up the review as a finished deliverable.