When to Let an Agent Mark Its Own Checklist

A check belongs here when the failure is mechanical, the signal is unambiguous, and deciding pass or fail needs no interpretation. The canonical examples:

• Schema validity. The output parses against the schema you defined. It parses or it does not.
• Exit codes. A command returned 0. A build succeeded.
• Unit tests. A named suite ran and every test passed.
• Hash match. The artifact content matches the expected digest.
• File existence and size. The output file is present and non-empty.
• Deterministic lint. A linter reports zero violations against a fixed rule set.

None of these require judgment about quality. They require running a tool and reading a boolean. That is the whole test: if a five-line script returns an identical verdict, the check is tier one.

Format instructions make tier one possible. Delimiters, numbered steps, and an explicitly specified output format are the basics that make responses consistent enough to parse (LaunchDarkly). If you cannot parse the output, you cannot check it mechanically, and the check slides up to a higher tier.

An agent asked to produce a configuration file runs checks like this:

import json, hashlib, subprocess

raw = open("config.json").read()
cfg = json.loads(raw)                                 # invalid JSON raises
assert set(cfg) == {"region", "retries", "timeout"}    # exact keys
assert cfg["retries"] in range(0, 10)                  # bounded value
digest = hashlib.sha256(raw.encode()).hexdigest()
assert digest == open("expected.sha256").read().strip()
assert subprocess.run(["config-lint", "config.json"]).returncode == 0

Every assertion is binary. The interpreter either raises or it does not, so there is nothing for the agent to be generous about.

The failure mode that promotes a tier one check

Tier one breaks in one predictable way: the check is mechanical but the signal is not. An agent that writes its own test can write a test that trivially passes. An agent that picks which files to hash can pick the wrong ones.

The rule: a tier one check is only tier one if the agent cannot influence both sides of the comparison. The schema comes from outside the agent. The test suite is pre-existing or reviewed by someone else. The expected hash is computed independently. When the agent authors the check and the work, the check has moved to tier three even though it looks mechanical, because verifying the check needs a person.

Test practitioners have long known that a high pass rate proves little on its own: you also need coverage, remaining risks, and the quality of the tests themselves (TestRail). An agent is exactly the kind of test author that inflates pass rates with weak tests.

Tier two is where most real quality lives, and where teams make the expensive mistake of self-grading. The failure needs judgment (is this grounded, does it match the rubric, is the tone right?) but a second model, given the input, the output, and the source material, can catch it.

The rule: route the check to a different model than the one that produced the work, and give the judge a written rubric plus any reference material it needs. An LLM judge returns labels, scores, or comparisons for criteria such as correctness, relevance, and groundedness, using the task input and reference material (Arize).

Why the judge must be a different model

A model reviewing its own draft tends to agree with it. It wrote the text under a set of internal assumptions, and when it re-reads the text it brings them back. A second model does not share them. A well-prompted judge can approximate human evaluation closely enough to be useful at scale, and multi-agent frameworks reduce the bias any single judge carries (the agent-as-a-judge review).

A worked example

A retrieval-augmented support agent answers refund questions. A different model judges it.

This mirrors a documented judge pattern (Arize). The point is not that the judge is clever. The point is that it is external: it reads the evidence and compares, which the producing model does not reliably do against its own output.

Start from the failure you need to catch. Use code for deterministic checks and an LLM judge only where evaluation requires interpreting meaning (Arize). That sentence is the boundary between tier one and tier two.

A tier two check costs a second inference call per item. That is the honest trade: tokens spent to avoid a human review, spent only where interpretation is needed. Quanta Inference (https://quantaintelligence.ai/inference) puts text, image, video, audio and embedding models behind one OpenAI-compatible key, which is one way to point the judge at a different model than the producer without running two integrations.

The failure mode that promotes a tier two check

1. Vague rubric. "Is this good?" produces noise. The judge needs explicit pass/fail criteria and reference material, or it invents standards.
2. Position bias. Judges favor answers by position in the prompt, and this bias is not random: it varies across judges and tasks and is strongly affected by the quality gap between candidates (the position bias study). Swap the order of compared outputs. If verdicts flip, the check is unreliable.
3. A sympathetic judge. A judge from the same model family, given no reference material, rewards plausible-sounding text. Give it the source and require it to quote it.

If a tier two check keeps producing verdicts a person would overturn, it is a tier three check wearing a costume, and the costume costs you a false sense of coverage.

Validating the judge before you trust it

Validate the judge before relying on its scores: compare it with reviewed examples, inspect missed failures, and keep the evaluator version and evaluation data in view when comparing results (Arize).

False passes are the dangerous error. A judge that misses bad work is worse than no judge, because it moves work into a pipeline without review. Track false passes as a first-class number.

Judges handle pairwise comparisons and direct scoring, offline and online, and their success depends on the prompt, the model, and the complexity of the task (Evidently AI). That dependence is not a caveat; it is a design instruction. Complex tasks need decomposed rubrics, not one holistic score.

A check belongs with a person when the failure is about consequence, intent, or an irreversible side effect.

• Consequence. The action moves money, grants access, deletes data, publishes to the public, or commits the organization.
• Intent. The question is not "did it follow instructions" but "did someone mean this." Agents cannot read intent behind an ambiguous request, and a person has to confirm what was meant.
• Irreversibility. There is no undo. The check cannot be "mark done"; it has to be "someone who can live with the outcome says yes."

A judge can evaluate text; it cannot accept the consequence of acting on it. The agent-as-judge literature says as much: agent-based judging can complement but not replace human oversight (the agent-as-a-judge review).

Plain-language discipline transfers neatly here. A checklist item should answer the reader's question, use "must" for requirements, and avoid ambiguity like "shall" (U.S. Department of Labor). Write tier three items as yes/no decisions with a named approver. "May this refund of the stated amount go to account Y? Approver: finance owner" is an item. "Check the refund looks reasonable" is not.

Designing a gate people actually use

1. Show evidence and consequence together. The approver needs the diff, the amount, the destination, and the trigger on one screen.
2. State the safe default. The item says what happens if nobody approves, and the answer must be: nothing irreversible happens.
3. Log who approved and when. An approval without a record cannot be audited. Operational checklists that record who ran each step turn incident reviews into a document instead of a scramble (Whale).

Shop-floor work instructions make the same point: a precise description of the normal sequence with no response plan is the common failure. People then know the target process but not what to do when something is missing or out of tolerance (Operations1). Give every tier three item a stop-and-escalate rule: stop, mark, quarantine, escalate to a named person.

The most common misclassification

The most common error is putting a reversible-looking action in tier one because the agent can do it and any single instance is small. A thousand small refunds is not small. Aggregate consequence is a consequence. Any check whose cost of being wrong compounds across runs belongs with a person.

Ask what happens if the check passes wrongly on every run for a week. If the answer is "we notice and fix a few files," tier one is fine. If the answer is "we have a compliance problem," it was never tier one.

Run the questions in order.

Pass 1, the mechanical test. Can a five-line script produce an identical verdict with no interpretation? If yes, and the agent cannot author both the check and the work, the item is tier one. If the agent authors the check, send it to a person for review first.

Pass 2, the judgment test. Does the failure need interpretation but have a written rubric and reference material a second model can apply? If yes, the item is tier two. Name the judge model, the rubric, the reference material, and the validation set in the item itself.

Pass 3, everything else. Consequence, intent, irreversibility. Name the approver and the safe default.

Then set a cadence. Sample tier one items to confirm the checks still test what you think. Sample tier two verdicts against human labels on a regular interval. Review tier three approvals for patterns, such as one approver rubber-stamping everything, which usually means the gate fires too often or the evidence is hard to read.

The arc from manual work to a partially automated checklist to a fully automated one is well known in operations, and it holds for agents: automate what you are confident about, keep people on the rest, and move items up a tier only when the evidence says you can (Deep Roots).

If you are designing these gates across a whole line of business rather than one agent, Quanta Experts (https://quantaintelligence.ai/agents) hands the work back as finished deliverables, a natural place to put a tier two rubric and tier three approval pattern in front of a practitioner.

A checklist item is a contract between the agent and whoever marks it. Weak items invite self-grading.

Prompt structure matters as much as checklist structure. Putting the task first, separating instructions from context with delimiters, and being precise about the output format make responses consistent enough to check (Accelirate). An item that cannot be checked against a structured response will be graded by vibes.