When an Agent Writes the Mutant: Prompt and Sandbox

Constraint design is easier if you write down what is fixed and what is free before you write the prompt. The table below is the contract.

The timeout deserves a concrete rule rather than a vibe. A mutation tool that runs mutants in parallel has to account for contention: run the baseline suite once alone, then run it N times concurrently and take the slowest result as the contended baseline, then set the mutant timeout at roughly 1.5 times that baseline with a floor of a few seconds the mutator configuration guide. You can pass that computed number into the prompt and the agent never has to guess.

On operators, three families earn their keep as a starting set:

• Arithmetic replacement: swap + with -, * with /, or drop one operand of a sum. Catches tests that never pin a computed value worked arithmetic examples.
• Conditional boundary: move < to <= and > to >=. Catches missing boundary assertions PIT conditional boundary mutator.
• Return value: return null, false, zero or an empty collection where the original returned something. Catches callers whose tests only check that no exception was thrown.

Leave literal-replacement operators out of the first run. They generate a lot of equivalents (a constant that only feeds a log line cannot be killed), and the equivalence problem is undecidable in general, so you will spend reviewer time on mutants no test could ever kill Marsit et al. on estimating equivalent mutants.

The template below is written for a general-purpose coding agent that has shell access to a checkout. It assumes a tool or script named mutate that accepts a file, an operator and a timeout. Swap in your own runner. The important part is that every constraint arrives as a value the agent must echo back, and that the test directory is declared read-only.

ROLE
You are running mutation testing. You do not write, edit, delete or
reformat any file under tests/ or any fixture file. That directory is
read-only for this task.

FIXED PARAMETERS (do not change, do not substitute)
operators: arithmetic_replacement, conditional_boundary, return_value
files (may mutate only these):
  src/billing/invoice.py
  src/billing/proration.py
timeout_per_mutant_seconds: 45
baseline_command: python -m pytest tests/ -q

PROCEDURE
1. Run baseline_command once. If it is not fully green, stop and report
   BASELINE_RED. Do not proceed.
2. For each file and each operator, choose mutation sites yourself.
   Prefer branches that guard money, dates, quantities and permissions.
   Skip sites whose mutated value cannot reach an output.
3. For each mutant: apply exactly one operator at exactly one site, run
   baseline_command with the timeout, record the outcome, revert.
4. Classify each mutant as killed, survived, timeout, compile_error or
   runtime_error. Do not merge categories.
5. Rank survivors by blast radius: how many callers reach the line, and
   whether the line touches money or state.

OUTPUT
Two artifacts:
  (a) a JSON array with one object per mutant: file, line, operator,
      original, mutated, outcome, first_failing_test or null
  (b) a ranked list of survivors, each with a one-sentence explanation
      of what assertion would kill it.
Do not emit a mutation score. The pipeline computes it.

Three details in that template matter more than they look.

First, "do not emit a mutation score" removes the reward the agent would otherwise optimize. If it reports the score, it can also quietly decide which mutants count. Let the pipeline compute killed / (killed + survived). Note that a timeout is counted as detected by at least one established tool, on the reasoning that an infinite loop would hang CI too Stryker mutant states, but you should decide that convention once and keep it.

Second, "skip sites whose mutated value cannot reach an output" is where the agent earns its keep. A static analysis pass cannot tell that a particular return value is only consumed by a formatting helper, but a reader of the code can. This one instruction is the difference between a run full of unkillable equivalents and a run with a short, actionable survivor list.

Third, the baseline gate comes first. Mutation results on a red suite are noise: a mutant is only "killed" if a test fails because of the mutation, not because of an unrelated pre-existing failure CircleCI's step-by-step description. Failing fast on the baseline costs one test run and saves a whole report.

Hashing is the enforcement half of the contract. The prompt says the tests are read-only; the hash says whether that was true.

# before the agent runs
find tests -type f \( -name '*.py' -o -name '*.json' -o -name '*.snap' \) \
  -print0 | sort -z | xargs -0 sha256sum > /tmp/tests.before

# ... agent runs, produces mutants.json ...

# after the agent runs
find tests -type f \( -name '*.py' -o -name '*.json' -o -name '*.snap' \) \
  -print0 | sort -z | xargs -0 sha256sum > /tmp/tests.after

if ! diff -q /tmp/tests.before /tmp/tests.after > /dev/null; then
  echo "TAMPERED: test files changed, discarding run" >&2
  exit 2
fi

Four practical points on the script.

The sort -z matters. Without a stable order, find can emit paths differently between runs on some filesystems and the diff fails for no reason. The inclusion of *.snap and fixture JSON matters because snapshot and fixture files are tests in everything but name. Include anything the test runner reads.

Also compare the working tree against the commit you started from, not just the test directory. The agent's only legitimate product is a JSON report; a git status --porcelain that shows a modified source file after the final revert means the revert logic is broken and the next mutant ran against dirty state.

Finally, the check must run outside the agent's control. If the agent's own shell script computes the hashes, a sufficiently confused or adversarial agent can recompute them after editing. Run the before-hash on the host, hand the agent a checkout, and run the after-hash on the host.

The decision rule that follows is blunt and should be: any hash mismatch discards the entire run, not just the affected mutants. You cannot know which survivors were evaluated against a weakened suite, so a partial result is worse than no result.

Consider a proration.py module with a 60-line billing function and a suite that takes 40 seconds to run. An operator pass targeting conditional boundaries and arithmetic in that one file finds 30 eligible sites. Assume the baseline is green, the contended baseline is 70 seconds, and the timeout is set to 105 seconds.

Now suppose a typical mix:

The score over detectable mutants is 18 + 2 divided by 28, which is about 71 percent. Read it as "the suite catches roughly seven of every ten plausible defects in this function," not as a grade. The 8 survivors are the deliverable, and the ranking step should put the two money-rounding survivors first.

Cost sanity check: 30 mutants at roughly 70 seconds each is about 35 minutes serially, and roughly 9 minutes across four workers. That is why per-file scoping and incremental runs exist. Google's production approach mutates only changed code during code review, filters out mutants unlikely to matter and caps mutants per line, which turns an intractable full-repo run into something that fits a review cycle Google, Practical Mutation Testing at Scale. Stryker's incremental mode does the same trick from the other end, reusing prior results when neither the mutated file nor the covering tests changed Stryker incremental mode.

If your team already gates review on diff coverage, the two metrics fit together well. Diff coverage answers "did the new lines run and get covered," computed from a coverage XML report against git diff diff-cover. Mutation results answer "would the new lines' tests notice a defect," computed against the same diff. One team reported lifting line coverage from under 50 percent to 87 percent over ten months by making the per-change number, not the global number, the thing reviewers looked at the edX write-up on diff coverage. Mutation testing scoped to the same diff is the natural next increment.

Two parts of the pipeline are pure judgment, and both are reasonable uses of a model.

The first is ordering survivors so the highest-value gap floats to the top. Random selection among survivors wastes reviewer attention. A learned prioritization model trained on mutant subsumption graphs outperformed random mutant selection in an LLM-driven test generation study, letting teams spend limited effort on the survivors that imply the most other mutants PRIMG on arXiv. You do not need the subsumption graph to get most of the benefit: "how many callers reach this line" and "does this line touch money" are cheap heuristics an agent can apply while reading the code.

The second is explaining what kind of test would kill a survivor. That is a translation task: from a code location and a transformation, produce a sentence like "assert that prorating a half-day invoice does not round up twice." Those sentences go back to the human who owns the test file, and that human writes the test. The loop closes without the agent ever touching the suite.

The research on agent-generated tests is a caution rather than a green light here. Work combining LLMs with mutant prioritization found that LLM-authored test cases needed iterative refinement to reach syntactic and behavioral correctness the PRIMG abstract in the ACM record. Refinement is a human-in-the-loop activity in most teams, and it belongs after the mutation run, not inside it.

Start with one file and one operator family, not with the repository. A single file keeps the run under half an hour and lets you calibrate the timeout against real contention before scaling.

Keep the scope tied to the change set once you have the loop working. The technique is expensive by construction: every mutant costs a suite run, and a full-suite pass over many mutants is measured in hours for a large codebase, which is why selective mutation and test prioritization were studied as cost reducers FaMT on test prioritization and reduction. Mutating only what the pull request touched, with a per-mutant timeout, is what keeps it inside a review window.

If your team wants this running across several service repositories without rebuilding the harness each time, an agent that ships with the mutation skill installed and reads your repository context is the shortest path: Quanta Skills lets you install the skill into your own coding agent or hand the job to a hosted expert. Either way, keep the hash check and the fixed parameters in your pipeline, not in the agent's hands.