Turning a Manual QA Checklist into Agent-Runnable Checks

Every runnable check has the same three parts, and any step you cannot express in all three parts is not ready to automate. The command is what executes: an HTTP request, a browser action, a shell command, a database query. The assertion is the expected outcome, written so a machine can compare actual against expected. The exit code is the signal that travels back to the agent, and the convention that matters is the simplest one in computing: zero means pass, anything else means fail. A standardized execution interface returns structured output of exactly this kind, stdout, stderr, and exit codes, which is why a plain script is a valid agent tool 13.

Here is a check written as a small shell script, which is the lowest common denominator any agent can run:

#!/usr/bin/env bash
set -euo pipefail

BASE="${BASE_URL:-http://localhost:3000}"
EMAIL="qa+${RANDOM}@example.com"

# Command: create an account through the public API.
response=$(curl -sS -w '\n%{http_code}' -X POST "$BASE/api/accounts" \
  -H 'Content-Type: application/json' \
  -d "{\"email\":\"$EMAIL\",\"password\":\"correct-horse-battery\"}")

body=$(echo "$response" | head -n -1)
status=$(echo "$response" | tail -n 1)

# Assertion 1: the API accepted the request.
[ "$status" = "201" ] || { echo "FAIL: expected 201, got $status"; exit 1; }

# Assertion 2: the response carries an account id.
echo "$body" | grep -q '"id"' || { echo "FAIL: no id in response"; exit 2; }

# Assertion 3: the new account can log in.
login_status=$(curl -sS -o /dev/null -w '%{http_code}' -X POST "$BASE/api/sessions" \
  -H 'Content-Type: application/json' \
  -d "{\"email\":\"$EMAIL\",\"password\":\"correct-horse-battery\"}")

[ "$login_status" = "200" ] || { echo "FAIL: login returned $login_status"; exit 3; }

echo "PASS: signup and login"
exit 0

Three things make this agent-friendly. It is deterministic in its verdict: the same code and the same environment produce the same exit code. It isolates failures by using different exit codes per assertion, so the agent's retry loop gets a specific signal instead of a generic error. And it prints a human-readable reason on failure, which the agent can feed back into its next edit, the same loop used in a definition-of-done workflow where a failing check's output returns to the agent and a passing run reaches a reviewer 12.

The same structure holds for UI checks. Where a browser-driving tool is available to the agent, the assertion becomes a DOM query or a visible-text check rather than a curl status. What does not change is the contract: the check returns zero or not-zero, and it says why. One useful pattern from agentic QA tooling is to design acceptance criteria and scenarios before writing the runnable test, so the check traces back to a named criterion rather than to an expectation invented at coding time 19.

The table is itself a deliverable. Once a team fills it in, the "automate now" column is usually obvious, and the argument moves from philosophy to scheduling.

Not all steps deserve the same treatment. Three properties decide the order, and they are multiplicative rather than additive, so a low score on any one property pushes a step down the list.

Frequency is how often the step runs: every commit, every release, or once a quarter. Stability is how often the expected outcome changes for reasons unrelated to bugs, which is a proxy for how much maintenance the check will cost. Risk is the damage if the step silently breaks in production. A signup flow fails all three tests well: it runs constantly, its expected behavior rarely changes, and a break costs revenue. A marketing page's typography fails stability and frequency both, so it stays manual.

A simple scoring rule works: give each step 1 to 3 on each property, multiply the three, and automate in descending order. A signup check might score 3 × 3 × 3 = 27. A visual regression on a page that is being redesigned every sprint might score 1 × 1 × 2 = 2. Between those, the checkout end-to-end test scores high enough to justify the harness it needs. This is the same logic as splitting stories late and only when you are about to work on them: splitting has overhead, so you spend it where it pays 9.

Two practical caveats. First, the first automation of any flow costs more than the estimate, because you have to build fixtures, test accounts, and a clean starting state. Budget for it and do not let a single expensive check stall the rest. Second, do not automate a flaky step at the top of your list. A check that fails at random teaches the agent to retry endlessly and teaches humans to ignore red builds. Fix the flakiness or mark the step manual until you can. Mobile test tooling makes the same point in the other direction: agents should be given a stated starting state before they author a test, because ambiguous setup is where repeat runs diverge 19.

Some steps genuinely need a person: judging whether copy reads naturally, whether a visual hierarchy feels right, whether a legal disclosure is adequate. The mistake is leaving those steps as prose the agent might try to satisfy anyway. The better move is an explicit gate: a stopping point where the agent records what it did, states what it cannot verify, and waits for a named person to sign off. Acceptance criteria are the contract between the person and the agent, and a vague criterion will be interpreted however the agent chooses 11.

A manual gate has four fields, and writing them down is the whole job: the trigger (which check or stage the gate follows), the question the human must answer, the evidence they need in front of them, and the criteria for pass and fail. Here is an example as a small structured file the agent can read and write:

gate: copy-review
after: checkout-redesign-page-loads
question: "Does the new refund policy copy state the 14-day window and the restocking fee?"
evidence:
  - url: /policies/refunds
  - screenshot: evidence/refunds-page.png
  - diff: evidence/refunds-copy.diff
pass: "Both facts present, no contradictory statement elsewhere on the page"
fail: "Either fact missing or contradicted; file a ticket against the copy source"
owner: content-lead
blocks_merge: true

This structure does two things an ordinary checklist does not. It gives the agent a machine-readable pause point, so the run halts cleanly instead of guessing. And it gives the reviewer a checklist that is already specific, which raises the chance that two reviewers reach the same verdict. The reference workflow for agent quality keeps a single human checkpoint open on purpose, not because you intervene on every run, but to guarantee that you can 20.

An agent-run check is only worth something if the result is recorded and re-readable after the agent stops. That means capturing the command, its exit code, and a timestamp for every check in every run, and attaching that log to the change under review. This is the core of a definition of done for agent work: evidence that someone can read back, not a sentence the agent wrote 12.

Two failure modes deserve specific defenses. The first is test tampering: an agent asked to make a check pass may edit the check instead of the code. The defense is to make the check files read-only to the agent during a run and to add a rule that the diff must not delete, skip, or weaken an existing check, which is an explicit item in the Acme Co. example definition of done 12. The second is the stale run: a check that passed before the agent's last edit says nothing about the code that ships, so the suite must run again on the final state before the verdict counts 12.

There is also a context problem at the level of the whole run. As an agent works through many checks and retries, its working context fills with output noise, and long-horizon agents begin to lose track of what has already been done as their active context grows, which is the constraint Context Folding is built to address by folding completed subtasks into short summaries 2. Research on adaptive context management finds that pruning stale content while preserving task constraints and progress improves performance, and that the right amount of preservation depends on how strong the underlying agent is 3. The practical version for a QA suite: keep each check's output short and structured (exit code plus one line of reason, with full logs on disk), and have the runner write a fixed-format status file after each stage so the agent reads a compact summary instead of replaying every command.

Take a realistic manual item: "As a returning customer, I can update my shipping address during checkout, and my cart contents are preserved." The conversion has five passes.

First, split the sentence into separate behaviors. The address update and the cart preservation are two assertions, and the second one is the one humans miss. This is the same vertical slicing discipline that keeps stories small enough to finish and verify 9, applied to checks: one behavior per check wherever the behaviors can fail independently. Teams often ask whether it is acceptable to break a vertical story into horizontal tasks, and the working answer is that tasks are for whoever does the work, while the thing you accept is judged on the vertical behavior 10. The same holds for checks: however many helper commands a check needs, it should be accepted or rejected on one behavior.

Second, define the starting state. The check needs a known customer, a known cart, and a known address. Write the fixture as data, and write a reset step that returns the environment to that state. Checks that share mutable state with other checks are the main source of false failures.

Third, write the command and assertion. Drive the update through the interface the agent can reach, then assert both that the address changed and that the cart line items are identical to the expected list. The second assertion is the one that would have caught the failure in the Acme Co. example, where the address saved but the cart emptied, and the failure only surfaced when an end-to-end check ran after the agent reported done 12.

Fourth, pick exit codes and a timeout. Codes 1 and 2 above are the two assertions; the timeout handles the case where the environment hangs. A check with no timeout is a check that can stall a whole run.

Fifth, write the gate. If the flow includes any judgment item, such as whether the confirmation screen's copy is accurate, add a manual gate after the automated checks pass. The agent's deliverable for the whole item is then: the check script, the fixture, the reset step, the recorded run log, and either a pass verdict or a filed gate request. That is a finished unit of work an agent can complete and a reviewer can accept without re-running anything by hand.

Running this conversion across a typical checklist is often a good first task for a coding agent, because it is mechanical and easy to check. If you want to hand the conversion itself to an agent with a defined working style, install a skill into your coding agent or put an expert on the job at Quanta Skills. The output should still be reviewed by a person the first time, since the value of the check depends on the assertions being right.

Do not switch the checklist off on day one. Run the automated checks alongside the manual process for a few releases and compare verdicts. Disagreements are the most valuable data you will get: either the check is wrong, the human is, or the step was ambiguous all along. Fix the ambiguity first, because it will otherwise migrate into every future check you write.

Sequence the rollout in this order: read-only checks (no side effects), then checks that write to a test environment, then end-to-end checks in a fresh environment, then schedule them on every change. The order matters because it limits blast radius. An agent that can run unrestricted commands against shared infrastructure is a liability; one running inside an isolated environment with scoped permissions and audit logging is a tool that automates meaningful work 13. The same separation applies at the design level: keep generation and verification in different hands, so the thing that writes the code is not also the thing that certifies it passed 20.

Finally, keep a written index of what is automated, what is gated, and what is still manual, with an owner and a last-reviewed date for each row. The index is the difference between a test suite that improves and one that decays into a pile of skipped checks that nobody trusts. When a check fails twice in a row for reasons unrelated to the product, the rule should be to fix it or demote it to manual within the same sprint, never to leave it red.