Turning a Written Instruction into a Runnable Checklist

The core unit is a check with three parts: a criterion (what is being verified), an expected result, and an evidence field (how the executor shows what happened). Then every check needs two cases, one that must pass and one that must fail. The fail case is not optional: a check nobody has ever seen fail is a check whose failure mode is unknown.

The pattern below is the smallest useful shape. It works whether the executor is a person, a script or a model.

- id: C-03
  criterion: "Output file exists at the path named in step 4"
  evidence: "Listing of the output directory, pasted into the report"
  pass_case:
    given: "Step 4 run to completion"
    expected: "output/report.csv present, non-zero bytes"
  fail_case:
    given: "Step 4 run with the source path misspelled"
    expected: "No output/report.csv; executor reports the missing input instead of creating an empty file"
  source: "instruction.md#step-4"

Two design choices in that block matter more than they look.

First, evidence is a first-class field. Test execution practice is consistent on this point: results should be recorded clearly enough to distinguish pass, fail, blocked and rerun, and a high pass rate alone proves little without coverage and known risks (TestRail). Asking the executor for a pasted listing, a diff, a screenshot or a log excerpt does two things at once: it gives the reviewer something to inspect and it discourages the executor from reporting success it did not observe.

Second, the fail case describes the wrong result you expect to see, not just "it fails". "Missing file plus a plain-language error naming the input" is a testable expectation. "Fails gracefully" is not.

For tasks where the instruction includes examples, derive checks from the examples first. This is the same logic behind few-shot prompting: showing the model one or two example input-output pairs anchors tone, format and level of detail far better than describing them (Mirascope). When you convert an example into a check, you get a regression test for free. If a later run no longer matches the example, either the instruction changed meaning or the executor drifted, and both are worth knowing.

A quick completeness pass on any candidate check:

The last row is the decision rule. If a second person cannot reach the same verdict from the same evidence, the check is still a note, not a check.

Not every expectation reduces to present or absent. Format constraints, tone, length and factual accuracy are all real requirements, and pretending otherwise produces a checklist of trivial items while the hard requirements go unchecked. Three patterns cover most of it.

Thresholds beat adjectives. "A short description" becomes "3 to 5 sentences", which is checkable by counting. Prompt guidance is blunt about this: replace fluffy, imprecise descriptions with measurable ones, because vague instructions push the executor to fill the gaps with its own assumptions (OpenAI). The same article makes the second point that matters here: instead of only saying what not to do, say what to do instead, because a check phrased as a prohibition gives the reviewer no positive standard to apply.

Rubrics need anchors. For judgment calls, write three anchors (unacceptable, acceptable, strong) with a one-line description of each, and require the executor to cite the anchor. A rubric with anchors behaves like a five-point scale with the ends defined; a rubric without them invites scoring drift between runs.

Separation beats cleverness. Split a compound expectation into several simple checks rather than writing one elaborate check. This mirrors the long-standing advice for instruction writing: break complex tasks into simpler parts so each part can be confirmed on its own (Excelsior OWL). It also makes failures diagnosable, because a single failing check points to a single requirement.

On phrasing, keep checks in plain language and in the active voice, with the executor addressed directly. That is ordinary plain-language guidance, and it reduces the chance that a check is misread by the person or model running it (OPM). Short paragraphs and bulleted lists are recommended for the same reason (University of Michigan).

The order in which the checklist is used changes how it should be written. Most stuck checklists are stuck because the author never chose between the two modes.

This read-do versus do-confirm distinction is a standard way of describing checklist types, and the reason it matters is practical: a read-do checklist pushes execution in front of verification, while a do-confirm checklist pushes verification to the end (Varsity Tutors). If your instruction covers a task the executor performs weekly, put the checks at the end as a confirmation gate. If it covers something rare and high-stakes, make each check mandatory before the next step begins.

A related habit worth adopting: prefer checklists written so each check forces a look at the artifact, not a recollection of it. Photo-proof steps exist in operational tooling precisely because a signature can be produced from memory, while a timestamped photo cannot (Whale). The software equivalent is a pasted command output or a diff.

The short answer is a structured file, in the same repository as the instruction, versioned with it. That is a stronger recommendation than it may sound, and there is a well-documented reason.

When a legacy C++ codebase lost the three engineers who knew the build sequence, one team recovered by writing the sequence down as a checklist and checking it into source control alongside the code that the checklist operated on. Every phase of discovery was recorded in files, experiments were checked in next to the checklist that produced them, and both process and code changed in the same commit. The result was a graded path from manual checklist to partially automated checklist to full automation (Deep Roots).

That structure gives you free properties worth naming explicitly:

• Diffs are reviewable. A change to the instruction and a change to its checks appear in one pull request, so the reviewer sees both together.
• Rollback is trivial. If a new check turns out to be wrong, revert it without losing the instruction.
• Discovery is navigable. Branches let you explore a process change without contaminating the main checklist, then merge what worked.
• Automation is incremental. A check that you later automate is a small change to an existing step, not a rewrite.

A structured file (YAML

or JSON) is worth the small formatting cost over free text because the fields (id, criterion, evidence, pass and fail cases, source reference) become queryable. You can then compute useful numbers: how many steps have no check, how many checks have no fail case, how many checks have never once failed in the last fifty runs. A Markdown table of checks also works, and is easier to read in review, but it resists automated checks on itself.

There is one more storage decision: whether the executor sees the checklist up front or only uses it at the end. Hiding the checks reduces the risk that the executor works backward from the test instead of forward from the instruction, and keeps the instruction the primary specification. Showing the checks up front makes the requirements explicit and reduces ambiguity. Choose deliberately, and state the choice in the instruction's introduction, which is exactly what an introduction is for: end result, how to use the instructions, prerequisites and hazards (Proofed).

If the work is done by a model rather than a person, keeping the checks in a structured file matters more, not less. Treat the prompt itself as a versioned artifact with tests that run known inputs and validate outputs, rather than as a string hardcoded into a deploy (LaunchDarkly). One API that reaches every model family, priced in dollars and compatible with OpenAI clients, makes it practical to run the same checklist against several models and compare failures rather than betting on one (Quanta Inference).

A failure report is evidence, and it comes in two flavors. The executor failed the check, or the check failed the executor. Your job in review is to decide which, using the recorded evidence rather than the tone of the report.

Run the review in this order:

1. Re-read the check and its evidence together. If the evidence does not actually demonstrate what the criterion asks for, the failure is in the check. Add the missing evidence requirement.
2. Re-run the pass case. If the pass case now fails, something upstream changed and the check is stale, not the executor.
3. Re-run the fail case. This is the one teams skip. If the fail case no longer produces the wrong result you expect, either the executor improved or the check stopped discriminating. Both are useful facts, and you cannot obtain either without running it.
4. Ask whether the instruction is ambiguous. If two reasonable executors would read the step differently, fix the instruction and tighten the check to match. This is the most common fix and the one that compounds in value.
5. Only then classify the executor. If the check and the instruction survive all four questions, you have a genuine execution failure.

Two rules keep this process from being gamed. First, never delete a check because it keeps failing; downgrade it to a known-defect check with an owner and a note, or retire it explicitly with a reason in the commit message. Second, count failures per executor and per check rather than globally. A check that fails only for new executors is an onboarding gap; a check that fails randomly is probably under-specified. Exactly what test execution practice warns about with pass rates applies here: the number tells you little without coverage and known risks alongside it (TestRail).

One more habit, borrowed from delegation practice: set a review cadence, then step back. A short series of scheduled reviews early on, followed by a lighter rhythm once the checklist is stable, prevents the reviewer from hovering over every step forever, which teaches the executor to wait for rescue rather than fix the obvious problems (Sarah M. Hoban). A quarterly pass through the "never failed" list is usually enough to retire checks that have stopped carrying information.

If several people run the same checklist and you want a shared library of checks with one credit pool and per-person keys, that is what a team plan is for (Quanta for Teams).

Take an instruction with this step and this example. Step 3 reads: "Export the summary as a CSV using the same column order as the sample below." The sample shows five columns in a fixed order. Step 4 reads: "Check the row count is between 100 and 500 and note it in the report."

Converting it takes about ten minutes:

• C-03a: Header row equals the five sample columns, in order. Pass case: sample file used as input, header matches exactly. Fail case: run against a file with the columns in a different order, expect a named mismatch error listing both orders.
• C-03b: File has no blank lines and no duplicate header rows. Pass case: clean export. Fail case: input containing a trailing blank row, expect the executor to reject or strip it and report which.
• C-04a: Reported row count is a number, and it falls in the stated range. Pass case: an input of 250 rows produces "250". Fail case: an input of 40 rows produces an explicit out-of-range note, not silence.

Six checks from two steps, half of them fail cases. The review that follows is now a matter of comparing four expected values against four reported values, which is a five-minute job rather than a debate. Note that C-04a also fixes a defect the instruction had: it never said what to do when the count is out of range, which is precisely the gap that a stop-and-escalate rule is supposed to close (Operations1).