How to Set a Disagreement Budget for Human Reviewers

The core decision rule is a ratio. Take the expected cost of one wrong automated decision, divide by the fully loaded cost of one manual review, and you have a number that says how many reviews one correct automated decision is worth.

The budget sits at the inverse of that ratio, scaled to one hundred: budget = 100 / ratio. Worked example: if a wrong automated decision costs 20 times a manual review, then 100 divided by 20 gives a budget of 5 overrides per hundred cases. The reasoning is symmetry. You would pay for about five manual reviews to spare yourself the expected cost of one wrong automated decision, so if the replay shows more than five in a hundred cases going wrong, the automation is not yet buying you more than manual review costs.

Two cautions on the inputs. First, "cost of a wrong decision" should be expected cost, not worst-case cost: probability of harm multiplied by the size of harm, including downstream cleanup, customer remediation, and staff time spent reversing it. Worst-case numbers produce budgets so tight that no replay ever passes, which is its own failure mode. Second, the manual review cost must be fully loaded: reviewer time, queue overhead, tooling, and the delay the review adds to the process. Understating it makes automation look worse than it is.

If the same wrong decision can land in several categories (a missed refund, a broken customer relationship, a compliance finding), sum the expected costs across categories rather than picking the scariest one. Write the arithmetic down and keep it next to the budget. Teams that skip the write-up end up renegotiating the number every time a result is inconvenient.

A budget is only useful if you can tell whether the observed rate is above or below it. That requires a sample large enough that the uncertainty around your estimate is smaller than the distance from your budget. The working rule from the short answer: about 400 randomly selected cases puts the 95 percent confidence interval at roughly plus or minus 2 percentage points around the observed rate.

Why 2 points? A confidence interval for a proportion is widest around 50 percent, and at that point a sample of roughly 400 gives a half-width close to 2 percentage points. If your budget is 5 overrides per hundred and your sample of 400 shows 4 overrides per hundred (16 overrides in the sample), the interval runs roughly from 2 to 6 per hundred. That interval straddles the budget, and the correct verdict is inconclusive. If the sample shows 6 per hundred with the same interval (4 to 8), the whole interval sits above the budget, and the evidence supports rejecting the candidate.

Sample size thinking is a familiar discipline in other settings. The inter-rater agreement literature sizes studies on the nearer limit of a target interval, not on total width, and tightening the nearer limit gets expensive fast, as shown in this sample size guide for observer agreement studies. A similar logic applies here: pick the precision you need relative to your budget, then stop. Rounding rules of thumb like "review 20 percent of cases" are not a substitute for a precision target, as this analysis of interrater agreement sample sizes points out.

A short sketch of the arithmetic you can run in a notebook:

import math

def interval(k, n, z=1.96):
    p = k / n
    se = math.sqrt(p * (1 - p) / n)
    return p * 100, max(0, (p - z*se) * 100), (p + z*se) * 100

# 16 overrides in 400 cases
print(interval(16, 400))
# rough output: (4.0, 2.03, 5.97) in percent

Randomness is not optional. Pull the sample from all replayed cases, not from the cases reviewers already found interesting. Reviewers naturally gravitate to the cases the model handled oddly, so a hand-picked sample overstates the override rate and makes good candidates look bad. Stratify if the case mix varies (for example, by risk tier), and report the budget check per stratum, because a 5 percent budget overall can hide a 20 percent rate inside a high-stakes tier.

"Replay evidence is inconclusive" deserves a precise definition, otherwise it becomes a polite way of saying "we argue about it." A workable rule uses the confidence interval against the budget:

• Conclusive against: the lower bound of the interval is above the budget. Reject or redesign.
• Conclusive for: the upper bound of the interval is below the budget. The replay supports moving forward.
• Inconclusive: the interval contains the budget. Neither verdict is justified.

Under this rule, an inconclusive result is not a failure. It is a classification with its own next actions, and this is exactly the logic that decision-support research recommends for policy launches from logged data. That replay to launch readiness study shows that replay-only and guardrail-only pipelines all pick the same candidate policy but would wrongly recommend direct deployment, while the full decision-support framework picks the same policy and changes the action to online validation because propensities, bidder response, and interference remain unresolved. A disagreement budget does the same job at the reviewer level: it keeps a promising result from being overclaimed.

When the interval straddles the budget, you have three honest responses. Widen the review. Collect more cases from the same distribution; if the true rate sits clearly on one side of the budget, an interval of plus or minus 1 point (roughly 1,600 cases at a rate near 50 percent, tighter at low rates) will settle it. Narrow the decision. Restrict the automated policy to the strata where the interval clears the budget, and route the rest to humans. Or hold and investigate. Break overrides down by reason code to see whether one systematic failure drives the rate.

A budget with no consequence attached is decoration. Decide in advance what each verdict triggers, and who has authority to overrule the budget. Overrides of the budget should themselves be logged, with a reason.

Reviewer overrides are only usable if each one is recorded as a structured artifact. A trace shows tokens and latency, and an application log shows an HTTP 200, but neither explains why a decision was allowed or what changed; the structured record that does is what this guide to AI agent decision logs calls a decision log, linking the task, bounded evidence, policy version, chosen action, and independently verified outcome. For budget purposes, the override record needs the case identifier, the system's proposed action, the reviewer's action, a reason code from a fixed list, and the reviewer identity.

Two-phase recording improves the quality of the comparison. The decision log guide recommends writing the planned decision before the side effect and appending the verified outcome after it, so that attempted, verified, and unknown outcomes stay distinct. In a replay this means the reviewer is shown the planned action and the evidence links, not a tidy summary written after the fact. Reviewers shown post-hoc narratives tend to agree with them.

Version the evidence context, too. A decision can be acceptable under one prompt release and blocked under the next, and without identifiers you cannot explain a historical override honestly; this is the case made for prompt versioning in this best practices guide. Every override in the sample should cite the release, model, and policy version that produced the decision. Otherwise a spike in overrides after a prompt change gets misread as model drift.

The governance layer has the same shape as general decision tracking. A decision log captures the decision, owner, date, alternatives, and what would reopen it, and it stays useful because it records the destination rather than the conversation, as this project decision log template puts it. The budget decision itself belongs in that log: the ratio used, the resulting budget, the sample design, and the verdict, so a later team can see why the number was chosen. Architecture decision records offer a compact format for exactly this, with status, context, decision, and consequences, per this engineering playbook on decision records, and this Zitadel decision log shows how much useful context a few structured fields carry.

The override rate is only as trustworthy as the reviewers producing it. If two reviewers would flip the same case in opposite directions, the count is measuring reviewer variance, not model error. Set a calibration threshold alongside the budget: before the main sample, have reviewers independently score a shared set of cases and compare. If their own disagreement on that set is in the same range as the budget, the budget cannot discriminate and needs to come down or the review process needs to tighten.

Watch for drift in the other direction as well. Reviewers who see many cases in a row become faster and more lenient; a budget measured in week one and week six may not be measuring the same process. Rotate the sample and re-run calibration periodically.

The human-oversight model matters here. In a human-in-the-loop setup a person authorizes each action before it executes, while human-on-the-loop keeps a human monitoring and correcting outcomes, and the choice maps to workflow risk, as this comparison of the two models explains. A disagreement budget fits human-on-the-loop well: the system acts, reviewers sample and correct, and the budget governs whether that posture is acceptable. If the budget is breached repeatedly, the workflow probably needs to tighten toward mandatory pre-execution review.

A support triage system proposes routing tickets to refund, billing, or escalation. A wrong route costs, on average, 20 times one manual review when you add up the remediation call, the customer credit, and staff time. Budget: 100 / 20 = 5 overrides per hundred cases.

Draw 400 random cases from the replay. Reviewers override 13. The observed rate is 3.25 per hundred, and the interval runs roughly 1.5 to 5.0. The upper bound touches the budget, so the verdict is inconclusive, not "pass." Break it down: nine of the thirteen overrides carry the same reason code, "refund eligibility misread." The fix is a narrower policy on refund routing; route those to humans and re-run the replay on the remaining 320 cases worth of strata. The new sample shows 6 overrides in 320, a rate near 1.9 per hundred with an interval well under 5, and the verdict becomes conclusive for the narrowed scope.

Running these replays and the second opinion pass across several models at once is where a single API for many model types saves setup time, and the inference platform exposes text, image, video, audio, and embedding models through one OpenAI-compatible key priced in US dollars, which makes budget comparisons across candidate models a matter of switching a model name in the replay script.

The budget machinery only works if the replay can run at all. Some teams still run manual replay because their workflows span several systems; that is a setup problem worth fixing early, and a short conversation about where the review layer should live can save months of recalibration later.