Turning a Review Sample Into a Stop or Go Signal

Three inputs drive everything, and each has a business meaning.

The threshold (p0). This is the highest true failure rate you would accept and still ship. Derive it from consequences, not from taste. If a wrong refund is reversed in minutes, the threshold can be loose. If an agent emails a customer a false statement, the error is permanent once sent, which is why external email sits in the supervised bucket in these approval patterns.

The confidence level. The probability that the procedure produces an interval containing the true rate. Ninety-five percent is common; ninety-nine is for high-stakes releases. Minitab's treatment of confidence and reliability makes the trade explicit: a 95% confidence level means a 5% risk of incorrectly concluding you met your reliability goal based on the units in your sample in this explanation of 95/95 demonstrations.

The consequence of being wrong. Two errors are possible. Shipping a bad release (the bound was too optimistic) or stopping a good one (the bound was too pessimistic). Confidence level is how you weight them. If a false go costs ten times a false stop, use a one-sided bound at a high confidence, not a two-sided interval. Common levels and their z-scores are tabulated alongside the formulas on this sample size reference, so the choice is a lookup rather than a debate.

A decision table helps teams agree before sampling:

You have n reviewed items and x failures. The observed rate is x/n. That number is not your answer; it is the center of a range. The upper end of the range is what you compare to the threshold.

For larger samples the normal approximation is workable. The standard error of the proportion is the square root of p(1-p)/n, and the interval is the observed rate plus or minus a z-score times that error, with z of 1.96 at 95% as set out in this LibreTexts treatment of proportion intervals. The normality check there is that both the success and failure counts are at least 5.

For small samples or rates near zero, the normal approximation breaks down: it can produce intervals that overshoot past 0 or 1, or intervals of zero width when there are no failures as documented for the Wald interval. Use an exact method or the Wilson interval instead; a calculator that reports both the binomial exact and the normal approximation side by side is a quick way to see the gap on this UCSF-hosted proportion calculator. For the zero-failure case there is a clean closed form: the upper bound is 1 minus the confidence level raised to the power 1/n.

Here is a short script that computes a one-sided upper bound using the exact binomial method, which works at small n and at zero failures:

from scipy.stats import beta

def upper_bound(failures, n, confidence=0.95):
    # one-sided exact (Clopper-Pearson) upper bound
    return beta.ppf(confidence, failures + 1, n - failures)

for x, n in [(0, 20), (1, 20), (3, 40), (1, 80), (6, 200)]:
    print(x, n, round(x/n, 4), round(upper_bound(x, n), 4))

The output is the decision input. Read the last column, not the third.

(Values computed from the formula above; recompute for your own confidence level.)

The insight the first search result misses: the observed rate being below the threshold tells you almost nothing. In the 1-in-80 row the observed rate is 1.25%, comfortably under 2%, and the honest conclusion is still "cannot decide." Reviewers who stop at the third column will ship that release and be wrong at least some of the time by construction. The bound is what protects them.

Compare the upper bound to the threshold. There are exactly three outcomes.

Bound at or below threshold: go. The sample is consistent with a true rate under your limit at your chosen confidence. Record the sample size, failure count, confidence level and bound in the release notes, so the claim is reproducible later. Reproducibility is the same property audit trails need: an auditor sampling a sensitive action should be able to reconstruct the decision from immutable, correlated records rather than rerunning the model per this audit trail guide.

Bound above threshold: stop. The sample does not support shipping. This is not a judgment about the agent's quality in general; it is a statement that you cannot demonstrate the rate is under your limit with this evidence. Fix the top failure mode and resample.

Bound straddles the threshold: the sample is too thin. The 1-in-80 and 0-in-20 rows above are this case for a 2% threshold. Two honest responses exist. Collect more samples until the bound resolves, or accept the risk explicitly: ship behind a human approval gate, narrow the scope to a low-risk user segment, or log the decision with the bound attached and a named owner. What is not acceptable is quietly treating the observed rate as the true rate. Writing the acceptance down turns an unstated risk into a managed one, which is the same logic behind runtime approval gates for irreversible or high blast radius actions in these design patterns.

A note on what the confidence level means. A 95% interval does not mean there is a 95% chance the true rate is inside this particular interval; it means the procedure covers the truth in 95% of repeated samples, with the true value treated as a fixed unknown as defined for confidence intervals generally. Practically: your decision rule is right 95% of the time it is applied, not 95% right on any single release.

You can flip the question around: how many reviews do I need before the bound can possibly clear the threshold? The standard formula for a proportion, solved for n, is z squared times p(1-p) divided by the margin of error squared, rounded up with a worked voter example here. When you have no prior estimate, use p of 0.5, which maximizes the product and gives the largest, safest n as the sample size calculator notes. A population-proportion calculator that returns a recommendation and then shows how the number moves with margin of error and confidence is a fast way to sanity-check your plan from this Select Statistics tool.

For threshold decisions the cleaner target is the zero-failure plan. To claim a true failure rate at or below p0 at confidence c with zero observed failures, you need n of at least ln(1-c) divided by ln(1-p0). For p0 of 2% and 95% confidence that is about 149 clean samples; for 99% confidence, about 228. This is the same C=0 planning logic used in reliability demonstration, where the sample must be large enough that zero failures still supports the claim discussed in this reliability guide. Sandia's report on sample sizes for confidence limits works the same trade for zero-failure acceptance in this DOE report.

Two practical corrections. First, if your observed failure rate is meaningfully above zero, a zero-failure plan is the wrong tool; you need the general bound from the previous section. Second, if one review pass is expensive, a two-stage plan can save effort: take a first sample, accept if failures are at or below a first cutoff, reject if at or above a second, and sample again only in the ambiguous middle. That structure is standard double sampling as described by NIST, and it maps neatly onto review batches where most releases are obviously fine or obviously broken.

Every number above rests on what you call a failure. Two traps.

Ambiguous failures inflate the rate. If reviewers disagree on whether a draft "contains an error," your x is noisy. Define failure as a checkable property: refund amount differs from the order record, customer name differs from the CRM, a quoted policy does not exist in the knowledge base. Agent evaluation guidance makes the same point: pick metrics that match the product, and note that tool-level correctness and task completion are separate signals that can disagree in this evaluation overview. Choose the one that matches your threshold's business meaning.

A transcript that says "done" is not a success. Evaluation guides call this false task completion: the agent reports completion while nothing changed as covered in this agent evaluation guide. If your reviewer only reads the transcript, your failure rate measures the transcript, not the outcome. Where possible, verify the outcome (the record was written, the email left the outbox) and count a missing outcome as a failure.

Human review needs calibration too. A sample reviewed by one person is a sample of that person's judgments. Spot-check a subset with a second reviewer and record disagreements; if the disagreement rate is high, your effective sample is smaller than n. The Dataiku survey cited in their governance piece found only 5% of data leaders say AI output is traceable 100% of the time per that report, which is a good argument for capturing the evidence as you review rather than reconstructing it later.

Keep the record short enough that people actually fill it in. One row per release:

• Threshold and confidence, stated before sampling.
• Definition of failure, one sentence.
• Sample size, failure count, observed rate.
• Upper bound and the method used (exact, Wilson, normal).
• Verdict: go, stop, or not enough evidence.
• If shipped on a straddling bound: named risk owner and the mitigation in place.

That record is also the raw material for the next release. Pool the failure counts from several reviews of the same agent, recompute the bound on the pooled sample, and the bound tightens with every review you were doing anyway. Teams that run this loop on agents handling consequential actions are effectively building the evidence layer that auditors now expect for agentic systems, where the question is not just what the model said but whether the action was permitted and reviewed as argued for authorization decision records.

If you are reviewing several agents across several models, the review pass is only half the work; the other half is running the same agent against varied inputs in the first place. Quanta Inference puts text, image, video, audio and embedding models behind one OpenAI-compatible key, so swapping the model under review is a config change rather than an integration project: https://quantaintelligence.ai/inference. The statistics do not change; only the number of releases you can afford to sample does.