How to Write Acceptance Checks an Agent Can Run Itself

The translation is mostly a matter of naming an observable. Below are five typical acceptance criteria from tickets, next to the check that replaces each one.

Two rules separate the left and right columns. First, the observable must be something a program can read: an exit code, a file, a stdout byte, a JSON field. Second, the pass condition must be stated as a comparison, not an adjective. That second rule is what the Gherkin-style Given/When/Then format enforces, which is why agents handle it well: the requirement is split into a precondition, an action and an expected outcome (the maturity matrix guide).

A useful smell test comes from Anthropic's "new employee" framing: show the criterion to a colleague with no context on the task and ask them to follow it. If they would be confused about what to run, the agent will be too (Anthropic prompting best practices).

Every check gets the same five fields. Write them in a Markdown block, one block per check, in a file the agent can read (ACCEPTANCE.md works well). The template is deliberately boring so the agent never has to interpret it.

1. Precondition. What must be true before the action. A clean database, a built artifact, a running server, a fixture in place. The precondition is itself runnable, usually an idempotent setup command.
2. Action. One command line, exactly as the agent will run it. Not a description of a command.
3. Observable. What the action produces that will be inspected, and where it lands: stdout, a file path, a served response.
4. Pass condition. The comparison that yields exit code 0. Prefer the comparison to be executed by the command itself (grep -q, diff, pytest, a jq -e assertion) so the agent does not have to parse free text.
5. Timeout. A wall-clock limit in seconds. On expiry the check fails with a distinct exit code so the agent can tell "wrong" apart from "slow."

A filled-in block for a real task, adding response-time logging to an API route:

### AC-03: /api/orders returns request_id header

- Precondition: `docker compose up -d api && ./scripts/wait-healthy.sh api`
- Action: `curl -sS -D headers.txt -o body.json -w '%{time_total}' localhost:8080/api/orders`
- Observable: `headers.txt`, `body.json`, and the printed total time
- Pass condition: `grep -qi '^x-request-id:' headers.txt` exits 0, AND `jq -e 'has("items")' body.json` exits 0
- Timeout: 30 seconds

Notice that the pass condition is itself two commands whose exit codes are ANDed. The agent runs a one-liner that combines them, or the runner does. Either way, there is no judgment call.

The runner is the piece most teams skip, and it is what makes the checks autonomous. Its job is to execute each check, capture stdout and stderr, enforce the timeout, and emit a machine-readable summary. Here is a minimal version in bash that reads a manifest and prints one JSON line per check.

#!/usr/bin/env bash
# run-checks.sh: reads checks.tsv (id<TAB>command<TAB>timeout)
set -uo pipefail
while IFS=$'\t' read -r id cmd timeout; do
  [[ -z "$id" || "$id" == \#* ]] && continue
  out=$(timeout "$timeout" bash -c "$cmd" 2>&1)
  code=$?
  jq -cn --arg id "$id" --argjson code "$code" --arg out "$out" \
    '{id:$id, exit:$code, status:(if $code==0 then "pass" else "fail" end), output:$out}'
done < checks.tsv

Two details carry most of the value. First, timeout from coreutils returns 124 when it kills a command, so exit 124 becomes your "slow" signal, distinct from exit 1. Second, dumping the captured output into the JSON means the agent sees the failing assertion text, not just a number, and can usually fix the problem in one iteration.

For anything beyond a handful of checks, put the manifest in your test framework instead. pytest exits non-zero on failure and prints a readable report; Playwright and Cypress do the same for browser checks. The template does not care which runner executes it, only that the exit code is honest.

The order of operations matters more than the wording. A study of LLM-generated acceptance tests in an industrial setting used a two-step pipeline: user stories became Gherkin scenarios, and scenarios became executable Cypress scripts against the page HTML. Practitioners rated the generated scenarios helpful in 95% of cases, and of the generated test cases, 60% were usable as produced, 8% needed minor fixes, 24% needed regeneration with more input, and 8% were discarded (the industrial case study). The interesting number is the 32% that needed another pass. That is the human checkpoint: generated checks are a draft you review before the agent implements, not after.

The practical sequence:

1. Read the ticket and write the five-part blocks first.
2. Run every check against the current code. All behavioral checks should fail; any that pass were already true or are wrongly scoped.
3. Hand the agent the blocks and the instruction "make the failing checks pass without editing the checks."
4. Review the diff against the blocks, not against the prose ticket.

Step 2 is the one people skip, and it is the one that prevents the circularity problem. A check that passes before any work is done tells you either the feature already exists or the check is not testing what you think.

There is a real limit here. Not every ticket converts. Refactoring, infrastructure and performance work do not map cleanly to behavioral assertions, and the guidance is to reserve acceptance-check generation for feature and bug tickets and keep the mature cases manual (the maturity matrix guide). For performance work, substitute a numeric threshold check (time_total under a stated bound) rather than a behavioral one.

Three failure modes show up again and again, and each has a cheap fix.

Too many checks. Every check is a chance for the agent to thrash. Five to eight per task is usually enough; if you have twenty, the task is really two tasks. This mirrors the general advice to break larger features into small, independent prompts and to scope each one to a single component or file (the Agentic Coding Handbook).

Checks that depend on order. If check 4 only passes because check 3 left a file behind, the suite is not reusable and the agent's reruns will confuse it. Each check should carry its own precondition, even if that means duplicating a setup command. Make the setup idempotent.

No overall budget. Per-check timeouts do not bound the total run. Set an outer limit and an iteration count: for example, stop the agent after twelve reruns or twenty minutes, whichever comes first. When the budget runs out, the agent should report which checks still fail and the captured output, not keep going. Anthropic's guidance on long-running agents addresses the same failure: agents over-claim progress on long tasks, so an external, non-negotiable stopping rule is safer than trusting the agent's own sense of completion (Anthropic prompting best practices).

A quick self-review checklist before you hand checks to an agent:

The same mechanism scales past the local loop. Once checks are files with stable IDs, they can run in CI, gate a merge, or serve as the reproduction for a bug. Berkeley's position paper on software autonomy draws the line usefully: pipeline autonomy, where humans state a demand and evaluate resulting behavior rather than reviewing code, depends on automated validation being trustworthy enough to replace inspection (Berkeley RDI). Acceptance checks an agent can run are the first piece of that trust. They are also the reason a spec-first workflow works: a spec is only as executable as its success criteria, and the criteria are exactly the checks (Addy Osmani on specs).

If you are running this with a coding agent inside an IDE or terminal, the checks file belongs wherever that tool reads persistent context, so it survives between sessions. Teams that run several agents across different model providers often want the same acceptance file executed against each provider's output; a single OpenAI-compatible endpoint such as Quanta Inference removes the per-provider plumbing so the checks and the runner stay identical.