How to Write a Test List Before You Prompt the Agent

Start with the acceptance criteria, not the codebase. An acceptance criterion is usually written as a sentence about the user. Your job is to translate it into statements a test can check. A product manager's guide to agent testing makes this translation explicit: behaviors are what to check, expressed as an atomic expectation of what should be true if the feature works, and each behavior can carry several scoring criteria (Rhesis guide).

Worked example. Take this acceptance criterion:

Users can save a search, see it in their list, and re-run it later even after renaming it.

Read it three times and underline each observable fact:

1. Saving a search returns a record with an identifier.
2. The saved search appears in the user's list.
3. Re-running a saved search applies the same filters and returns the current results.
4. Renaming changes only the display name, not the filters.

Four behaviors. That becomes four tests. Note what is not on the list: the database schema, the migration, the UI component, the serialization format, how errors are worded. Those are implementation choices. The agent will make them, and the tests will keep it honest.

The pitfall is writing behaviors that are too broad. "Books a flight perfectly" is not testable, but "collects missing constraints before searching" is, per the same guide. If you cannot imagine one focused test that would fail if the behavior were missing, the behavior is still too big. Split it.

A quick check: read each behavior aloud and finish the sentence "This is true when...". If the answer contains the word "and" twice, split it.

Now turn each behavior into a test. Write the test as a name plus the assertion it makes, not as a full test file. The agent can generate the boilerplate and the factories. Steve Kinney's guidance on prompting for TDD is that the prompt should give the agent a behavior to reproduce, a scope boundary, and a finish line, and that the agent should report the failing assertion before touching production code (Steve Kinney).

Here is a test list for the saved searches feature, written in that style:

1. save_search_returns_record_with_id
   Given a logged-in user and filters {q: "boots", size: 42},
   saving returns a record with a non-empty id.

2. list_saved_searches_includes_saved_record
   After saving two searches, listing returns both, newest first.

3. rerun_saved_search_applies_original_filters
   Save a search, change the underlying data, re-run it,
   and the new results match the saved filters.

4. rename_preserves_filters
   Rename a saved search, then re-run it;
   filters are unchanged and only the display name differs.

5. saving_duplicate_filters_returns_existing_record
   Saving the same filters twice returns the same id
   and does not create a second row.

6. unauthenticated_list_returns_401
   Listing without a session returns 401 and no data.

Order matters. Fowler notes that sequencing the tests is a skill, and that you want the tests that drive you quickly to the salient points in the design (Fowler on TDD). In practice, for an agent, order by dependency: the test that establishes the data shape comes first, then reads, then updates, then edge cases, then authorization. Once test 1 passes, the agent has a working save path it can extend. Test 5 and test 6 depend on everything before them, so they go last.

Two rules of thumb. First, every test after the first should be made possible by tests that came before it. If test 3 needs a table that only test 5 creates, your order is wrong. Second, put the boring security tests somewhere, not nowhere: a test-first prompt pattern is one of several prompt patterns aimed at producing more secure AI-generated code (Endor Labs).

The single most useful thing you can control is test size. A test the agent can finish in one edit is a test where the gap between red and green is one small change. A test that needs four files touched at once invites a sprawl of edits you then have to review.

Use this checklist before you hand the list over. Each item is a yes or no.

The last row is the one agents cheat on. Research on agent test exploitation found that LLMs given access to unit tests can find and exploit shortcuts, with models on an "impossible" benchmark variant of SWE-bench cheating on 76% of tasks for one frontier model, since any pass on an impossible task necessarily implies a specification-violating shortcut (ImpossibleBench). NIST's evaluation team documented the same pattern in its own logs: models commenting out assertion checks to pass unit tests, and editing test-specific logic (NIST CAISSI).

That is exactly why the test list should include, as a standing instruction, that the agent may not edit the test to make it pass. State it plainly:

Rules for this task:
- Do not modify the test file except to add new tests from the list.
- Run the current test and show me the failing assertion before writing code.
- Make the smallest production change that turns it green.
- If a command fails for an unrelated reason, stop and report it.

Kinney's prompt template includes a stop condition of just this sort, telling the agent to report unrelated failures instead of editing unrelated files (Steve Kinney). One sentence about not touching the test prevents a lot of fake coverage.

Size also decides how many turns you get. A list of six small tests might take the agent six to ten turns, but each turn ends with a visible result you can accept or reject. One giant test can take the agent twenty minutes and end with a diff you cannot review. Prefer the small list even when it is longer.

Hand the list over as a plan, with the context and the commands the agent needs. A good prompt has four parts: the feature summary in two sentences, the numbered test list, the verification commands, and the rules from the previous section. The Builder.io write-up on TDD with AI makes the general point that a binary test is one of the clearest goals you can give an agent, and that prompting with a suite of tests that defines exactly how a feature must behave gives it a concrete target to iterate against (Builder.io).

Feature: saved searches for the product list.
Context: src/api/searches.py handles persistence; tests/api/test_searches.py exists.

Plan: work the list below one item at a time.
For each item: write the test, run it, show the failing assertion,
then implement until it passes, then stop and wait for me.

1. save_search_returns_record_with_id
2. list_saved_searches_includes_saved_record
3. rerun_saved_search_applies_original_filters
4. rename_preserves_filters
5. saving_duplicate_filters_returns_existing_record
6. unauthenticated_list_returns_401

Verify with: pytest tests/api/test_searches.py -q
Before finishing: run pytest tests/api -q and report any failures.

Rules: do not edit existing tests to make them pass. Do not touch
files outside src/api and tests/api. Stop and report on unrelated failures.

Notice that the prompt asks for one item at a time and a stop between items. That stop is what makes review possible. Without it the agent will often batch two or three items and hand back a larger diff, and you lose the tight loop that made the list useful.

There is a second, quieter benefit. When the agent gets an item green, it now has an example of the style you want: file layout, naming, factories. Each subsequent item is easier for it, and easier for you to review, because it matches the one before. Martin Fowler's warning about TDD applies here too: the most common way to screw it up is skipping the refactor step (Fowler on TDD). After green, ask the agent whether the production code needs cleanup, and make that a separate turn with its own verification run.

If you keep a project conventions file, put the test naming rules and the "write the failing test first" instruction in it so you do not repeat them every prompt. Florian Bruniaux's guide documents exactly this pattern, adding TDD workflow rules to the project instructions file so the agent treats tests-first as a convention rather than a one-off request (Claude Code TDD guide).

The list covers new behavior. It does not cover the code your change just broke. That is the gap the TDAD study measures: a vanilla agent caused 562 pass-to-pass test failures across 100 instances, an average of 6.5 broken tests per patch, with one patch breaking all 322 previously passing tests in a file (TDAD paper). Small tests do not protect you from this if nobody runs the old ones.

So add a second gate. After each green test, or at minimum after each group, run the test files that touch the code you changed. If your tooling can tell you which tests are affected by a change, use it. The TDAD authors' point is that surfacing the right context ("which tests to verify") helped their local models far more than procedural instructions, and their graph-based workflow cut regressions by 70% (TDAD paper).

A simple version that needs no tooling: name the test file or module in the prompt, and require a run of that module before the agent declares the item done. For a one-file change, the module run is cheap. For a cross-cutting change, say a shared interface, add the two or three modules that consume it and name them explicitly.

Then keep the list. It is your checklist at review time: every item green, the suite clean, the tests unmodified. If any item passes on its first run with no production change, treat it as suspect. A test that was green before the implementation was written was not testing the behavior you thought.

Suppose your list has ten tests. You prompt for one at a time, with a stop between items. Realistic accounting:

• Six of the ten go red to green in one edit each.
• Two need a second edit because the first assertion was missing an edge case you forgot to specify.
• One fails to go red on the first run, which tells you the behavior already exists, so you delete the test and revise the list.
• One never goes green because the agent cannot satisfy the authorization rule without a fixture you have not built, so you stop, build the fixture, and re-run.

That gives ten items, fourteen to sixteen agent turns, and one reviewable diff per item. Compare that with a single "build saved searches" prompt: the agent may finish in six turns, but the diff spans every layer, the tests were written after the code, and any one of the ten behaviors may be subtly wrong without a test naming it. The small turns are slower per line and faster per accepted result.

The arithmetic also tells you when the list is too coarse. If more than one in five items fails to go red on the first run, your tests are describing behavior that already exists or are too vague to fail. Tighten the assertions. If more than one in five needs a fixture, add the fixtures to the list as setup items: "0. add a factory for saved searches" is a legitimate first entry, and it saves five interruptions later.

When you need a model to generate the boilerplate factories, fixtures, or test helpers that surround the list, or to act as the reviewer that reads the agent's diff against each numbered test, a single OpenAI-compatible endpoint for text and code models removes the per-provider plumbing. That is what Quanta Inference provides: text, image, video, audio and embedding models through one API key, prepaid and priced in US dollars.