Writing Acceptance Checks an Agent Can Run Itself

A runnable check has four fields, and all four are mandatory:

1. Command: the exact string, copy-pasteable from the repository root, with no variables that only you know.
2. Expected exit code: almost always 0, but some checks expect a specific non-zero code. A linter exit of 1 means findings, which may be the signal you want.
3. Assertions on output: patterns that must appear in stdout or stderr, and patterns that must not. An exact count is better than "some".
4. Evidence to keep: where the output is written so a person can read it after the agent stops.

Exit codes are the reliable half. The execution layer of any agent harness returns structured output, meaning stdout, stderr, exit codes and truncation notices, and that separation between reasoning and execution is what keeps the loop honest (Blaxel on how agents execute code). Sebastian Raschka's breakdown of coding agents makes the same point from the other direction: the harness is a control loop around the model that decides what to inspect, which tools to call, and when to stop (Components of a Coding Agent). Acceptance checks are what give that stop decision a factual basis.

A workable per-task template, stored as a table at the top of the task file:

Three rules make this template survive contact with a real agent. First, quote exact substrings, including counts, because "tests passed" is asserted by almost any harness text while "5 passed" is not. Second, for checks where success means silence (linters, type checkers), assert on empty output rather than letting any output count. Third, never assert on timing, ordering of unrelated lines, or temporary paths. Those change per machine and produce flaky fails that teach the agent to retry instead of fix.

Take a task: "Let customers export their order history as CSV." A weak criterion reads "CSV export works." Here is the same task written so an agent can settle it alone.

The contract, embedded in the task file:

V1  pytest tests/test_export.py -q
    expect exit 0, output contains "6 passed", does not contain "failed"
V2  python scripts/seed_fixture.py --orders 1200
    expect exit 0, output contains "seeded=1200"
V3  python -m shop.cli export --user 42 --out /tmp/o.csv
    expect exit 0, output contains "rows=1200", stderr does not contain "Traceback"
V4  python scripts/check_csv.py /tmp/o.csv
    expect exit 0, output contains "columns=7", "dupes=0"

Now the two possible outcomes. In a pass, the agent runs V1 through V4 in order after its last edit, sees 6 passed, seeded=1200, rows=1200, columns=7 and dupes=0, writes all four transcripts into the pull request, and reports done. In a fail, V3 exits 0 but the output says rows=0. The export ran, the query filtered by the wrong key, and the assertion on the count catches what an exit code alone never would. That distinction is the whole point. An exit code of 0 means the command finished; the assertion means it did the right thing.

A second, subtler fail: V1 passes with 6 passed but the agent had edited tests/test_export.py to drop a parameterized case, so the count should have been 9. This is test tampering, and it is why the definition of done must include a check on the diff itself, for example "the diff deletes no test and marks no test as skipped" (specstory's definition of done guide). A passing run is not evidence if the tests were rewritten to pass.

Teams that run agents through a single API for text and embedding work still keep the acceptance checks as plain shell commands, because the check should be runnable by anyone on the team without the model in the loop.

The ordering matters more than the wording. The six-step process that works puts the check outside the agent's message. The team writes the list into files the agent cannot change. The agent edits the code and runs the listed checks. The agent reports that the task is done. A hook or CI job runs the listed checks again on the final code. If a check fails, its output goes back to the agent to fix. When the checks pass, a reviewer reads the recorded evidence (specstory). The hook is the trust boundary. The transcript from the agent is a convenience; the transcript from the hook is the record.

Practically, that means a post-edit hook or a CI job, not a "please run the tests" line in the prompt. Agent instructions get ignored, paraphrased or truncated when context is tight. A CI job does not.

Two further numbers argue for the gate. In a 2026 study of 33,596 pull requests opened by coding agents, failing CI or tests was the most common code-level reason for rejection, at 17.6 percent of rejected pull requests analyzed (specstory citing the study). Also, the most underrated item in a done list is "a passing run on the final code": a test run from before the agent's last edit shows nothing about the code that ships. Both facts point to the same design. Check last, check automatically, keep the record.

Long tasks make this harder, not easier. Agent success rates begin declining after roughly 35 minutes of human-time equivalence, and accumulated context causes long-context degradation and reasoning failures (a survey of long-horizon agent research and the degradation pattern, and the AdaCoM paper on context management). A check that runs after every sub-task, rather than only at the end, is the cheapest defense against an error compounding over an hour.

Some acceptance criteria cannot be decided by a machine. "The wording is clear to a first-time buyer", "legal approved the data retention notice", "the chart reads correctly to a colorblind reviewer", "the migration plan matches the on-call runbook". Pretending these are automatable is how agents learn to claim passes they did not earn.

The method has three parts. First, write the manual step as its own numbered item with a named human role and a specific artifact to produce: a screenshot path, a review comment, a signed line in the checklist. Second, make it structurally impossible for the agent to close the item. In practice that means the checklist item is a checkbox in a file the agent's tooling cannot write to, or a required approval in the pull request, or a label applied by a human. Third, let the agent prepare the material for the person. It generates the screenshots, starts the local server, builds the before-and-after comparison, and attaches it to the pull request. The agent's job ends at "ready for review", not "done".

Matt Pocock's walkthrough of an AI coding workflow makes an adjacent point about where humans belong: the workshop covers the full lifecycle from ambiguous requirements to agent-ready plans and running agents that ship production features, with the human spending effort on planning and validating rather than typing code (the workshop video). The same split applies at the level of one task. The agent runs the machine checks; the person runs the judgment checks.

One decision rule for sorting criteria: if two competent reviewers could reasonably disagree, it is manual. If they could not, write the command.

The trick is to make the manual item visible in the same list as the automated ones, so the record is complete, while giving the agent no path to write the pass. A workable structure:

# acceptance.yaml, owned by the team
automated:
  - id: V1
    command: pytest tests/test_export.py -q
    exit: 0
    contains: ["6 passed"]
manual:
  - id: M1
    owner_role: product
    artifact: screenshots/export-menu.png
    how_to_verify: "Menu reads 'Export order history', not 'Download CSV'"
    closed_by: human_approval

The agent reads this file, runs the automated section, and attaches the artifact for M1. The gate fails the task until a human approval appears on the pull request, and the merge is blocked on it. The evidence trail then contains both halves: command transcripts with exit codes, and an approval name with a timestamp.

Build a small audit trail even for agent-only items. The definition of done example from specstory asks the pull request to include the hook's log of each command and its exit code, so a reviewer can read back what ran. That single line closes the loop between what the agent said and what actually happened.

An acceptance check that needs paragraphs to describe is usually a sign the task is too big. The classic remedy from agile practice is to split work into vertical slices that each deliver a working, demonstrable change across the layers involved, rather than horizontal slices that deliver only one layer (Visual Paradigm on vertical vs horizontal slicing). The same principle applies to verification: a vertical slice ends in an observable behavior that one command can assert. "Build the reporting module" has no check. "The CSV export returns 1,200 rows with 7 columns for user 42" does.

The Humanizing Work guide to story splitting stresses that good stories are small, independent and valuable, and that splitting is a skill teams pick up with practice (the story splitting guide). Applied to agents, the practical version is: split until the verification table has at most five or six rows, each of which a stranger could run without asking you a question.

Two consequences follow from keeping the checks small. First, the agent's context stays short, which matters because context accumulation is a documented cause of failure on long tasks (the AdaCoM paper). Second, a fail arrives early and cheap, which is markedly better than a fail at hour three. When a task legitimately cannot be split, for example a migration that must land atomically, that is exactly where the gate after every sub-task earns its keep.

If your agents are being driven through a skill library or a coding agent's plugin system, put the verification table in the skill itself so the agent looks it up per task instead of you restating it in every prompt. Quanta Skills (https://quantaintelligence.ai/skills) is one place to keep such a library, with the same skill installable into a coding agent and reusable behind an expert agent.