What a False Pass Costs After It Ships

The case for ranges is not statistical purity. It is that a guessed point estimate hides its uncertainty, and a hidden uncertainty gets discovered by the person most motivated to doubt you. Three values force you to write down what would have to be true for each.

The same logic appears in evaluation work. Reporting a single accuracy figure conceals how much the answer depends on the measurement protocol: on one benchmark with per-criterion human labels, changing only the protocol moved reported accuracy from 0.551 to 0.899 without altering a single verdict Rao and Callison-Burch on agreement metrics. If a headline number can swing that far from protocol choices alone, a bare point estimate for a dollar figure deserves the same suspicion.

A practical rule: write the assumption in the same cell as the number. "High: 1 in 8 affected customers churns, because the customer who was overpaid still had a wrong promise corrected publicly" is defensible. "$2,400 churn" alone is not.

Someone has to undo the ticket. The error was approved, so the correction takes more steps than the original answer: find the ticket, re-read the policy, contact the customer, walk back the promise, issue a corrected resolution, file the note.

Build the line from hours:

• Low: 0.5 agent hours plus 0.25 supervisor hours.
• Best guess: 1.5 agent hours plus 0.5 supervisor hours.
• High: 3 agent hours plus 1 supervisor hour, if the customer pushes back or a second approval gets tangled in.

At a loaded agent cost of $35 per hour and a supervisor at $65, the best guess is $52.50 plus $32.50, or $85. Low comes to $33.75 and high to $170. Service desk benchmarks treat personnel as the dominant share of cost per contact, which is why loaded hourly rates are the right input rather than salary alone HDI on cost per ticket.

Rework looks trivially small next to the refund line. Keep it anyway. It is the one line you can instrument within a week, and it anchors the whole estimate in something real.

This is the line that scares people, and it needs the most discipline.

The overpayment is $4,200 minus the $350 cap, or $3,850. The question is how much of that you get back. Recovery depends on whether the money has cleared, whether the customer cooperates, and whether the payment method allows a reversal. Refund and policy abuse is common enough that merchants report it as a growing problem: 57% of surveyed merchants said refund and policy abuse increased over the past year the Merchant Risk Council's payments and fraud report.

For the estimate, split the overpayment by recovery probability:

The refund line ranges from $850 to $3,850, best guess $2,350. Note that the low case is not zero. Treating it as zero is the single most common way these estimates get inflated into uselessness and then ignored.

Churn is the line people inflate most, because it is the hardest to check. Keep it modest and tie it to the affected population.

The affected population is one customer in this incident, but the decision rule should scale. Use a churn probability applied to the annual value of the customer, and state the probability:

• Low: 0% churn. The corrected answer and a small goodwill credit keep the relationship.
• Best guess: 25% churn. A quarter of customers who get a promise walked back leave.
• High: 50% churn, with a referral effect counted as one additional lost customer at the same value.

If the customer's annual value is $9,600, best guess churn is $2,400. High, with the referral, is $9,600. Low is $0.

The zero in the low column is deliberate. It gives the reviewer a floor to argue from, and it keeps the range honest about the fact that most corrected errors do not end relationships. Retention work around returns and refunds is exactly where goodwill decisions pay off, which is why the low case is plausible rather than wishful Loop's retention benchmarks report.

A wrong answer generates contacts. The customer calls to confirm, a colleague asks about the policy, a second agent re-checks the account. Repeat contacts are the hidden multiplier in support economics: a 2.3 contact-per-issue average means the real cost per issue is 2.3 times the cost per contact Lorikeet's cost-per-ticket benchmarks.

For this incident, the extra contacts are on top of the rework hours already counted:

• Low: 1 extra contact at $6.
• Best guess: 3 extra contacts at $18.
• High: 6 extra contacts across two channels at $36.

The per-contact figure should come from your own cost per ticket, computed as total operating expense divided by resolved tickets GHDSI on evaluating cost per ticket. A SaaS desk often lands in a much higher band than a retail one, so do not borrow someone else's number without checking the category.

Here is every line in one place, with the arithmetic visible:

The spread is wide, roughly 15 times from low to high, and that is the point. If you report only the best guess, you will be accused of cherry-picking. If you report the range and the assumptions, the conversation moves to which assumption to test first.

Notice which line dominates each end. At the low end, the refund is 96% of the total. At the high end, churn is 70%. That tells you where to spend measurement effort: recovery rates first, churn probability second. Rework and support time are almost noise.

Now scale it. If the control that let this answer through misses one incident in twenty and the volume is 400 such answers a month, the best-guess exposure is 20 incidents times $4,853, or about $97,000 a month. That number is what justifies a review step, and it is the number a false-pass budget should be built around.

The method is short enough to put in a template. Work in this order.

1. Define the unit and the failure. One sentence: what escaped, and what did it authorize.
2. Write the four lines. Skip any line that is genuinely zero, but say why.
3. For each line, write the low, best guess and high, each with its assumption in the same row.
4. Total the columns, not just the best guesses. Keep the three totals.
5. Identify the dominant line at each end and name the measurement that would tighten it.
6. Convert to a monthly figure using an escape rate and a volume, and label both as estimates.

The last step is where most teams stop, and where the estimate earns its keep. An escape rate is itself a guess, and the same three-point treatment applies. Low 1 in 100, best guess 1 in 20, high 1 in 5.

One caution on the evidence you gather. If you plan to check your numbers with an automated reviewer, validate that reviewer first. Layered judges compound error: an evaluator with a 5% error rate checked by another with a 5% error rate yields a combined accuracy of about 0.9025, not 0.95, so error rises from 5% to nearly 9.75% GovTech's write-up on validating annotation agreement. Report human-to-human agreement alongside any automated agreement figure, and use a chance-adjusted metric when the decision depends on it Arize on measuring human-LLM judge alignment. Percent agreement alone overstates quality when one label dominates NHI Mgmt Group's comparison of agreement metrics.

If you want to run the review itself through one interface rather than stitching vendors together, Quanta Inference puts text, image and embedding models behind a single OpenAI-compatible key.