How to Set a Stop Rule for an AI Pilot Before It Starts

You can monitor twenty things. Three will end almost every failing pilot on their own, because they map to the three ways an AI workflow damages a business: it is wrong, it is slow, or it is expensive.

Error rate. The share of outputs that fail your quality bar, not the share the model scores badly on a benchmark. Define the bar in business terms: an invoice coded to the wrong cost center, a support reply that promises something the policy does not allow, a document summary that omits a material clause. Wavect's scorecard treats straight-through completion rate (successful actions with no human correction or escalation, divided by eligible attempts) as a core pilot KPI, with a suggested green band at 80% or above and red below 60% (Wavect).

Review time. The minutes a human spends checking and correcting each output. This is the metric teams most often forget to baseline, and it is the one that quietly eats the business case. If the AI saves four minutes of drafting and adds five minutes of review, the pilot is negative value no matter how good the output looks. Wavect's suggested bands put human correction time at 20% of baseline task time or less for green, and above 50% for red (Wavect). Set your own numbers against your own baseline, but measure review time from the first day, at the individual task level, not as an aggregate impression.

Cost per task. Not cost per token, and not cost per call. Taskade's guide lays out the formula that matters: price per token times tokens per attempt, plus tool fees, times attempts per accepted result, plus one-time costs such as human review (Taskade). DoiT makes the same point with the retry math: expected cost per solved task equals cost per attempt divided by success probability, and a model that is 10% more expensive per attempt but succeeds far more consistently can be much cheaper per correct task once retries and cleanup are counted (DoiT).

That last point is the decision rule most teams miss. When you set a cost-per-task threshold, set it on the accepted-result basis, not the attempt basis. Otherwise a model that fails cheaply looks better than a model that succeeds reliably.

There is no single correct format. The format you choose should match how much uncertainty the pilot carries and how granular your measurement can be. Here is how the three common formats differ in practice.

The composite format carries a specific trap that Wavect names directly: never let a high average hide an unacceptable safety, false-positive or auditability failure (Wavect). If you use a composite, attach hard floors to any metric where a single breach is disqualifying, and state those floors as vetoes rather than as low weights.

The staged gate format has the strongest structural support in the literature. Rubicon's stage-gate model defines each gate as a decision meeting with the business owner, IT, security and the delivery team, where the options are go, change or stop; and it states plainly that stopping at a gate means the process worked (Rubicon). That framing is worth borrowing even if you choose a different format. A stop that follows the written rule should be recorded as a success of governance, not a failure of the team.

The two-page pilot charter approach makes the same demand in document form. A charter should carry logistics, purpose and scope, quantified objectives with baselines and targets, and explicit success and exit criteria, all fixed before launch (ROSE). The stop rule lives in the exit criteria section, and it is two sentences long at most.

An accounts payable team is piloting an AI document review workflow on supplier invoices. Here is how the stop rule gets built, with the arithmetic shown.

Baseline first. The team samples 100 invoices processed manually. The loaded cost of the AP clerk is 45 dollars per hour, and average handling time is 12 minutes, so labor is 9.00 dollars per invoice. Rework and error loss (mis-coded invoices found and corrected downstream) adds 1.20 dollars per invoice on average. Baseline task cost is 10.20 dollars per invoice.

Set the thresholds. The team picks a composite score with veto floors:

• Error rate: target at or below 3% of invoices with a material coding or vendor error. Stop floor: above 5% across a rolling 100-invoice window.
• Review time: target at or below 3 minutes per invoice (25% of baseline handling time). Stop floor: above 6 minutes per invoice.
• Cost per accepted result: target at or below 7.00 dollars per invoice (roughly 70% of baseline). Stop floor: above 10.20 dollars per invoice, meaning no saving at all.

Run the arithmetic on a bad week. The model costs 0.18 dollars per attempt on average, attempts 1.6 times per accepted invoice, and review runs at 8 minutes, well past the floor. Cost per attempt is 0.18 times 1.6, which is 0.288 dollars, so 0.29 dollars of model spend per accepted result. Review cost is 8 minutes at 45 dollars per hour, which is 6.00 dollars. Total cost per accepted result is 6.29 dollars. That looks like a saving against 10.20 dollars, and a team watching only cost would keep going.

Now add the corrections. Suppose 9% of invoices carry a material error, and each one costs 14 dollars to find and reverse downstream. Error burden is 0.09 times 14, which is 1.26 dollars per invoice. Total cost is 7.55 dollars, review time is 8 minutes against a 6 minute floor, and both the review floor and the error floor are breached. The composite score might still look green because cost improved. The veto floors are what end the pilot. That is the entire reason to write them as vetoes.

Set the sample size before you start. The short answer to "error rate above 5%" needs a denominator. Say so in the rule: 100 tasks for a first signal, and a rolling 100 for ongoing checks. A window of 20 tasks cannot distinguish 3% from 9% with any confidence, and arguing about that after a bad week is exactly the conversation the stop rule exists to prevent.

Where the numbers come from. Every threshold above is derived, not borrowed. The 70% cost target follows Wavect's suggested green band of cost per successful action at 70% of baseline or less (Wavect). The 25% review target follows the same source's 20% guidance adjusted upward for a document-heavy workflow. Do not copy the numbers. Copy the derivation method: measure the baseline, pick a target that makes the investment worthwhile, and set the stop floor at the point where the pilot is no longer worth running.

A worked cost check you can reuse. If your vendor quotes a token price and you want the accepted-result figure, the shape of the calculation is this:

cost per accepted result
  = (price per token x tokens per attempt + tool fees per attempt)
    x attempts per accepted result
    + human review minutes x loaded rate / 60
    + error rate x average cost to correct one error

Taskade's guide gives the same structure line by line, including a 20-task test to measure your own numbers rather than forecast them after the pilot starts (Taskade). If you want one API key across text, image and embedding models so the token side of that formula is a single invoice rather than four, Quanta Inference exposes text, image, video, audio and embedding models through one OpenAI-compatible endpoint: Quanta Inference.

Name one person. Not a committee, not a steering group, not "the sponsor." A named individual, with a named backup, written into the charter.

The reason is mechanical rather than political. A committee can only stop a pilot at a meeting, and meetings can be deferred. A single owner can stop it on the day the threshold is breached. Alex Lieberman's readiness checklist separates two things that teams routinely merge: the business owner who owns the result, and the people who know who approves changes, resolves issues, and stops the test (Alex Lieberman).

A workable split of authority looks like this:

The pilot lead having immediate stop authority is the part most organizations get wrong. If the only route to stopping is an escalation, the pilot keeps running while the escalation is scheduled. Make the lead's authority explicit and time-boxed to a working day, with the business owner confirming the formal decision afterward.

Two governance safeguards are worth writing in. First, the sponsor should not be the sole decision maker on their own pilot. The Spark Academy framework names letting the sponsor decide on their own pilot as one of the standard mistakes, and its rule is blunt: three weak answers out of four means stop (Spark Academy). Second, stopping should be a recorded outcome, not an embarrassment. Rubicon frames the gate meeting as go, change or stop, and states that stopping at a gate means the process worked (Rubicon). Put that sentence in the charter.

The decision record is what converts an argument into a procedure. It should be one page, written at the time, and filed with the pilot charter.

Include the rule as written before launch, the measurement window and sample size, the observed numbers for each metric, which veto floor or threshold was breached, who decided, when, and what happens next (stop, change and re-run, or continue to the next gate). Add one line on what was learned and where the artifacts go, because a stopped pilot that leaves behind a clean dataset, a scoring rubric and a measured baseline has still paid for itself.

Fix the review cadence in the charter too. A weekly check of the three metrics against the rule, and a formal gate review on a fixed date, is enough for most pilots. Wavect's scorecard recommends accumulating 30 days of representative shadow or live usage before the formal kill-or-scale readout, and warns that a month spent waiting for permissions or test data is evidence about data readiness, not a valid sample of task performance (Wavect). Record blocked days separately. Do not let them count toward the measurement window.

Finally, tie the stop rule to the production question. The Institute of AI PM reports that 61% of pilots were approved with no plan for what happens if they succeed, and recommends defining the production path before starting (Institute of AI PM). Write the "if it passes" branch at the same time as the stop branch. A rule that only describes failure gives the team no reason to aim for the other outcome.

If you would rather work through these decisions with someone who has set them up before, a free 30-minute discovery call covers your goals and a tailored recommendation: Quanta Consulting.