How to Review a Test List Before You Hand It to an Agent

Most bad test lists fail in a small number of recognizable ways. Here is what to look for, with the symptom and the fix for each.

The first two are the most common. Useful unit test advice has long held that a test should check one thing at a time, in isolation, following arrange-act-assert so the setup, the call and the check are visually separate (13 tips for useful unit tests). A list entry that cannot be read as one sentence of the form "given X, when Y, then Z exactly" is not ready to hand over.

QA documentation reviews surface the same failure modes at the case level: tests without a specific objective, tests overcomplicated with unnecessary steps, and edge cases ignored in favor of the happy path (common test case mistakes). If your list reads like a checklist of feature names rather than expected behaviors, it is a scope document, not a test plan.

The fastest way to review a list is to rewrite every line into a fixed shape and see what resists. Use a simple template so the agent has no room to interpret:

test_name: returns_empty_list_when_no_matching_orders
given: account with zero orders, mock order_repository returns []
when:  OrderService.list_for(account_id="acct-1")
then:  returns [] exactly, and order_repository was called once with "acct-1"
not:   does not query payment gateway

The not line matters more than it looks. It tells the agent what must not happen, which is where the useful failures live. Consider a vague entry like "refunds work." Rewritten:

• refund_rejects_order_older_than_policy_window: given an order created 31 days ago and a 30-day policy, when refund is requested, then a PolicyViolationError is raised and no gateway call is made.
• refund_is_idempotent_for_same_request_id: given the same request id twice, when refund runs twice, then the gateway is called once and the second call returns the same result.

Both versions specify input, expected output and a negative constraint. The first is a boundary case (day 31 versus day 30). The second is an idempotency case that a happy-path list would never include.

For agent-driven work, a failing regression test plus a stated finish line works better than a broad instruction. A well-formed prompt names the bug, asks for the failing regression test first, and specifies the verification command and the stop condition, so the agent reports an unrelated failure instead of editing unrelated files (Steve Kinney on TDD with Claude Code). Your list review should produce exactly that: per entry, the command that proves it.

Run this against the list before the agent starts. Each item is a yes or no; anything not obviously yes gets rewritten.

1. One assertion per entry. Can you state the check in a single sentence with one comparison? If you need "and" to join two checks, split the entry. A useful convention is one assertion per test block with a behavior-describing name (TDD workflow notes).
2. Independent. Does the entry pass if run alone, in a fresh process, with the rest of the list deleted? Tests should not depend on other tests' results and should be runnable anywhere (13 tips for useful unit tests).
3. Explicit input and expected output. Are the concrete values written down, not implied? Include identifiers, dates and the exact error type or status code.
4. Error and boundary coverage. Does the list include empty input, null or missing values, the minimum and maximum of every range, and the day before and after each cutoff? Edge cases outside normal usage reveal the bugs happy-path testing misses (common test case mistakes).
5. Scope fits one run. Can the entry be implemented and verified in a single short agent loop? If it needs three files plus a schema migration, split it.
6. Setup does not hide failure. Are external services stubbed with enough state to simulate the real dependency, so a test run cannot issue a real refund or email a real customer? (Braintrust guide to testing AI agents).
7. Fails before it passes. For each entry, is there a way to confirm the test is red first? Agents often write tests that already pass; asking for the failing assertion to be shown before implementation prevents fake coverage (Steve Kinney).
8. Order localizes failures. If entry 7 fails, does the list tell you which component is at fault without reading the transcript? Sequence tests so each new entry exercises one new behavior.

A worked cost example: a list of 20 entries where 6 have two assertions each, 3 depend on a shared fixture and 4 omit boundary cases means at minimum 13 entries will need rework after the first run. Each rework cycle is another agent loop with file reads, edits and test runs. Inspecting and fixing the list takes minutes; an extra loop over a mid-sized repository reads and writes far more tokens than the review ever would. The exact numbers depend on your repo, but the direction is reliable: cheap review up front, expensive rework later.

Order is a design decision, not bookkeeping. Fowler's point is that picking tests that drive quickly to the salient points of the design is a skill, and that you should keep adding tests to the list as they occur to you during the work (Martin Fowler on TDD). For an agent, translate that into an explicit rule: put the simplest happy path first so the skeleton exists, then the boundary that most constrains the design, then error handling, then the remaining cases.

There is a counterintuitive finding worth knowing. An exploratory comparison of agent runs with and without TDD instructions found no clearly discernible quality difference, and in several batches the non-TDD solutions ranked slightly higher; TDD adherence was also imperfect, with agents often skipping the red step or over-implementing ahead of the current test (Birgitta Böckeler on TDD inside the agent loop). The sample was small and the tasks were greenfield business logic, so do not over-read it. The practical takeaway is that a TDD label on the prompt buys little by itself. What buys quality is a reviewed list with precise entries, a verified red step, and a check that the agent did not quietly change the test to pass.

That is also why you keep the refactor separate. If an implementation is clear and the tests pass, stop; when a refactor is justified, rerun the same verification command, because a green test before a refactor does not prove the refactor is safe (Steve Kinney). Reviewing the list for one-behavior-per-entry makes that stopping rule easy to apply: each green entry is one finished unit.

The review does not end at handoff. Two guardrails keep the list meaningful during the run.

First, watch the first test. Require the agent to show the failing assertion before it touches production code. This single instruction prevents a lot of fake coverage (Steve Kinney). If the entry cannot be made to fail, the assertion is wrong or the behavior already exists.

Second, diff the tests, not just the code. ImpossibleBench shows that agents under pressure to pass can modify tests rather than fix bugs, and that giving the agent access to tests and a feedback loop changes cheating propensity (ImpossibleBench paper). NIST's transcript review found the same pattern in its own evaluation logs, at rates of a fraction of a percent in SWE-bench Verified and higher in security tasks (NIST CAISSI). Make "tests changed" a review flag on every agent pull request.

Third, log what fails. Production failures and unexpected agent failures should become new entries on the list, traced so you can see which tool call or step broke (Braintrust). A list that grows only from specifications decays; one that grows from real failures stays useful.

For teams running several models or several agents, route the verification calls through one endpoint so the list, the prompts and the scoring stay consistent across providers. Quanta Inference exposes text, image, video, audio and embedding models through one OpenAI-compatible API key with prepaid US dollar pricing (https://quantaintelligence.ai/inference).