Measuring the Cost of a Missing Fact: Recovery, Refusal, Re-Ask

Model the recovery path as a sequence of stages with a probability attached to each. A user who hits a miss either re-asks, escalates, or leaves (sometimes in that order, sometimes jumping straight to leaving).

The right way to combine them is expected value:

E[cost per miss] = p_reask * C_reask
                 + p_escalation * C_escalation
                 + p_churn * C_churn

The probabilities do not have to sum to one in your tracker if you allow a user to re-ask and then escalate in the same episode. If you want a single number per episode, either count the episode once at its most expensive stage or sum the stages the user actually took. Write down which convention you chose; mixed conventions are the most common reason two teams report different costs for the same system.

A re-ask is the cheapest stage, which is why it is usually ignored and why it dominates the total by count. Price it as time plus compute.

Start with the time. A user who has to restate a preference, paste an account number, or reformulate a question that the system should have understood spends somewhere between a few seconds and a few minutes. Multiply minutes by a loaded hourly rate (salary plus benefits plus overhead, not the hourly wage). At a fully loaded rate of $60 per hour, one minute of extra work costs one dollar, and two minutes costs two. That sounds small until you multiply it by the miss rate and query volume, which is the whole point of an expected value.

Then add the compute. A re-ask is an extra turn, and the extra turn carries no conversation context beyond what the user retypes, so it is often a near-repeat of the first prompt at full input token cost. Multi-turn agents already accumulate context quickly: in one worked example, four turns of a research agent with tool results reached roughly 10,000 tokens of state before a follow-up question arrived (mem0). If each extra turn carries that accumulated state, the marginal token cost per re-ask is not trivial at volume.

Two measurement notes. First, instrument the turn, not just the transcript: record request start, retrieval start and end, model start, first output and completion, and trace retries as well as successes, because failed requests that return early can fake a speed improvement (Supermemory). Second, treat the latency part of the re-ask cost as a user-experience cost rather than a token cost; a re-ask that takes twelve seconds converts to escalations at a much higher rate than one that takes two, and that conversion is what you are trying to capture in the escalation probability.

Worked example. Suppose 1,000 sessions per day, 4 percent of sessions contain a missing fact, and 70 percent of those produce a re-ask of two minutes. That is 40 misses per day, 28 re-asks, and 56 user minutes, which at $60 loaded per hour is $56 per day, or about $20,000 per year. Add compute: 28 extra turns at, say, 6,000 input tokens each is 168,000 tokens per day, which is real but an order of magnitude smaller than the time cost at most retail token prices.

Escalation is where the numbers get large enough to change decisions. Published benchmarks vary widely by tier and channel, so pick the one that matches your queue and say where it came from.

A service desk benchmark cited by Maven AGI reports an average cost of $22.50 per resolved incident, with self-service averaging $15 and agent-handled tickets averaging $45 (Maven AGI). The same source reports that escalated requests cost about three times more than first-level resolutions. Moveworks, looking at IT tickets specifically, puts manual resolution of a Tier 1 ticket at around $22, Tier 2 at roughly $60 to $80, and Tier 3 above $104 (Moveworks). A cross-industry summary gives a broader range: $2.70 to $5.60 for retail e-commerce, $18 to $35 for SaaS, and $30 to $60 for B2B support (Lorikeet).

The single most expensive feature of escalation is not the first ticket. It is the repeat contact. The same cross-industry summary notes an average of 2.3 contacts per issue, which means the true cost per issue is 2.3 times the cost per contact, and first-contact resolution predicts total support cost better than handle time does (Lorikeet). This is exactly what a missing fact produces: the user opens a ticket to supply the fact, the ticket gets routed twice because it was filed under the wrong intent, and the issue resolves on the third contact. When you price escalation for a missing fact, use cost per issue, not cost per ticket, or you will halve your own estimate.

A practical allocation for an escalation triggered by a miss is one fully loaded Tier 1 ticket plus a repeat-contact multiplier between 1.5 and 2.3, adjusted to your own contact-per-issue data. At a $22 ticket and a 2.0 multiplier, each escalation costs about $44. If 12 percent of the 40 daily misses escalate, that is 4.8 escalations per day, about $211 per day, and the escalation line now exceeds the re-ask line.

Churn is the smallest probability and the largest number, so it deserves the most care. The formula is lifetime value times the incremental churn probability attributable to the failed session, and the word incremental does the work.

Get lifetime value for the affected segment first. A common trap is using an average across all accounts when the users hitting misses are concentrated in one plan. Churn rates differ sharply by plan: in a fictional but clearly constructed worked example, a Starter plan showed 6.0 percent monthly customer churn against 3.0 percent for Growth and 1.0 percent for Enterprise, with Starter contributing 78.3 percent of all losses (Cube). Small plans churning faster than large ones is a familiar pattern, and it means the customer who hits a missing-fact loop is often your highest-churn segment.

Then estimate attribution. Do not guess. A defensible method is a matched comparison: take accounts that experienced a failed session in a period and compare their retention at equal tenure against accounts that did not, holding plan and acquisition cohort constant. The Cube workflow makes the same point about attribution hygiene: a segment with high observed churn is a starting point for investigation, not a causal finding, and observed rates should be separated from hypotheses about causes (Cube). If your comparison suggests affected accounts churn 1.5 percentage points more per quarter, and lifetime value in that segment is $400, the churn term is 0.015 times $400, or $6 per affected account per quarter, allocated across the sessions that account ran in that quarter.

Behavioral signals help here at the session level. Usage-priced AI platforms see retention trouble in the usage table weeks before billing notices it: sessions thin out, agent runs stop, and the API key stays valid the entire time (Data Science Collective). That gives you a leading indicator to align with failed sessions, which is far better than waiting for the renewal to not happen.

Put the three lines together for the worked example. Daily miss volume is 40. Re-asks cost $56 per day in time. Escalations cost about $211 per day. Churn: assume 10 percent of misses come from accounts in a high-churn segment with $400 lifetime value and a 1.5 point quarterly lift, which is roughly $6 per affected account per quarter, or about a penny per session spread over a quarter. Churn lands near $1 per day at this volume. The total is about $268 per day, dominated by escalation, with re-ask second and churn nearly invisible at the per-miss level. That ranking is the insight most first-pass calculations miss: teams obsess over churn and underprice re-asks and repeat contacts, which are where the money actually goes at typical volumes.

Sanity-check the ranking against your own data before trusting it. The pattern holds when escalation rates exceed about 5 percent of misses. If almost nobody escalates, the re-ask line dominates and the fix is cheaper: better retrieval ranking and a shorter, more reliable first response. If escalation is high, the fix is likely the same but the justification is stronger, because you are removing three-times-costlier contacts rather than seconds of user patience.

Two operational rules keep the number honest. First, charge the fix against the same denominator you used for the failure: if you measured misses per session, express the fix as cost per session, not cost per query. Second, rerun the estimate on a fixed question set whenever retrieval, reranking, or context assembly changes, with the same quality criterion, because a smaller candidate set or a new reranker can save latency while quietly dropping supporting evidence (Supermemory). Routing work like this is exactly what budget-tier memory frameworks are built for: a lightweight router picks low, mid, or high tiers per query, and the measured trade-off only makes sense if you know what a miss costs (BudgetMem).

When you have the failure cost, the improvement decision usually settles quickly, and the remaining work is measurement plumbing across several models and vendors. Quanta Inference puts text, image, video, audio and embedding models behind one OpenAI-compatible API key, which makes A/B runs on the same question set cheaper to orchestrate: https://quantaintelligence.ai/inference.

A checklist to run before anyone signs off on the cost figure.

The percentile mistake is worth spelling out. You cannot add each stage's P95 and call the sum the end-to-end P95, because slow stages may fall on different requests or overlap; trace complete requests instead (Supermemory). The same logic applies to cost percentiles: a per-request cost distribution with a long tail is more informative than an average, because the costly misses cluster in the tail cases where context overflowed or a reranker picked a distractor.

If your miss analysis keeps pointing at answers that rearranged good evidence badly, the generation layer may deserve the first hour of engineering rather than the retrieval layer; one practitioner audit attributes the majority of RAG failures to the generation phase and reports that prompt fixes moved a faithfulness score far more than months of index tuning (SystemDR). Whatever you change, price the miss before and after on the same question set, or you will not be able to tell whether the fix paid.