Routing Conflicts by Latency: When a Slow Source Deserves a Human

A latency budget is a per-case-class allowance, expressed in seconds and in dollars, for reaching a decision. You set it before looking at any individual conflict, so the number is not chosen to justify a preference.

Start by timing each source end to end, not just the API call. A source's latency is retrieval plus reranking plus generation plus any queue wait. Latency accumulates across steps, and orchestration overhead alone can add hundreds of milliseconds per step (GMI Cloud on latency versus quality). A ten-step pipeline that looks fast per call can add up to several seconds.

Then write the budget down. A workable template for one case class:

Two disciplines matter here. First, use tail latency, not the average, because the deadline is a promise to the user and the tail is what breaks it. Second, measure error rates at the latency the source actually runs at in production. A source tested in isolation will look more accurate than one running under load with truncated context.

The budget also needs a review capacity line, because every escalation consumes human time. Plan human review as a metered resource: estimate review minutes per case, account for sampling and rework, and admit work only against available reviewer capacity (human review capacity for AI workflows). A budget without a capacity line is a plan to drown your reviewers.

The core calculation is one multiplication per source. Probability the source is wrong at its current latency, times the cost of acting on that wrong answer. Compare the products, not the clocks.

expected_cost(source) = P(wrong | source, latency) * cost_of_wrong_answer

Worked example. A refund decision has a wrong-call cost of $40, set by the finance team as the average cost of a misapplied refund plus rework. The fast source answers in 0.6 seconds and is wrong 12 percent of the time on this case class. The slow source answers in 3.5 seconds and is wrong 3 percent of the time. Both error rates come from labeled historical cases, not from vendor claims.

• Fast source: 0.12 × $40 = $4.80 per decision.
• Slow source: 0.03 × $40 = $1.20 per decision.

The slow source costs 2.9 seconds more and $3.60 less in expected loss per decision. At 2,000 such decisions a day, that is $7,200 of expected loss avoided daily. The decision rule writes itself: route to the slow source and let the seconds run, as long as the 3.5 second p50 and 5.2 second p95 fit inside the 8 second deadline.

Now the case where the math flips. Suppose the wrong-call cost is $2 because a wrong answer is caught downstream by a validation check that rejects it. Then the fast source costs 0.12 × $2 = $0.24 and the slow source costs 0.03 × $2 = $0.06. The slow source still wins on expected cost, but by 18 cents, and the extra latency may not be worth 2.9 seconds of a user's patience. The rule is not "always prefer the accurate source." It is "compare the products and then check whether the difference justifies the wait."

This is the same logic used to place human review in finance workflows, where the boundary between autonomous action and human review is located by pricing review against the expected value of the decisions it improves, and the required confidence rises as the impact of an error rises (Scale AI on routing work to humans).

Two refinements keep the numbers honest. First, calibrate error rates per case class, not globally, because a source can be excellent on structured records and poor on free text. Second, include the cost of delay itself in the budget line. If a slow source misses the deadline and the user abandons the flow, that abandonment cost belongs in the wrong-call cost, not in a separate column.

Between "route to fast" and "route to slow" sits a third option: hand the case to a person. The threshold for that is a number you set in advance, and it should be expressed in expected cost, not in seconds.

The rule:

1. Compute expected cost for every candidate source.
2. Take the minimum, the best available automated answer.
3. If that minimum exceeds your human review ceiling, escalate.
4. If it does not, route to the source that produced the minimum.

The ceiling should reflect the cost of the review itself. A reviewer who spends six minutes on a case at a loaded labor rate produces a known review cost. If escalation cost per case is $9, then escalating a case whose expected automated cost is $12 saves about $3 in expectation while adding certainty. Escalating a case whose expected automated cost is $2 loses money.

That last row matters most. A common failure is to let a deadline pressure the system into picking the fast source when the case should have gone to a person. Teams designing human-in-the-loop controls warn that thresholds set too tight bury reviewers in volume and thresholds set too loose let risky decisions through, and that the balance takes several rounds of adjustment (see Domo's overview of human-in-the-loop and Elementum's comparison of in-the-loop and on-the-loop models). Also define what happens when the system fails mid-review: a pending case should never default to approved.

A practical rule of thumb from the same body of practice: reserve human review for cases where the error cost exceeds the review cost, and build the escalation as a policy step with an audit trail rather than a manual habit.

Escalation also needs a feedback loop. Every escalated case is a labeled example of where your probability estimates were too optimistic or too pessimistic. Feed those labels back into the per-source error rates monthly, or the threshold will drift.

Not all conflicts are the same, and the expected-cost calculation should reflect that. Research on conflicting sources in retrieval systems introduces a taxonomy of conflict types and shows that telling the model about the conflict category improves the quality and appropriateness of its response (Google Research on conflicting sources). The practical lesson for routing: name the conflict before you price it.

Three categories cover most routing decisions:

• Ambiguity. Both answers are correct for different referents. There is no wrong answer to price, so the right move is usually to ask, present both, or escalate. RAG with Conflicting Evidence builds a dataset where ambiguity, misinformation and noise coexist and shows that handling them jointly still leaves a gap, which is a strong argument for routing ambiguous cases to a person rather than to either source.
• Staleness. The slow source holds the corrected record and the fast source holds a cached copy. The fast source is not wrong because it is bad; it is wrong because it is old. Here the slow source's expected cost is usually far lower and the escalation ceiling is rarely reached. Note that this is the case where "fast" and "accurate" genuinely trade off.
• Genuine dispute. Two sources disagree and neither is stale. If the expected cost of the best automated answer exceeds the ceiling, escalate. If it does not, route to the lower-cost source and log the disagreement for audit.

The distinction changes the default behavior, which is why a single precedence list cannot cover it. You can encode the categories in the routing policy as a conflict classifier that runs before the cost calculation, and the classifier's confidence becomes part of the probability estimate.

A cost-aware router needs the same instrumentation as any production decision system, plus a few things that are specific to latency.

First, separate the latency you control from the latency you do not. Queue wait and prefill time at a model instance depend on current load, so an estimator that simulates batch processing can predict time to first token better than a static table can (Latency-Aware LLM Query Routing). The same paper reports that jointly optimizing latency, accuracy and cost improves accuracy-cost utility by up to 40 percent while holding latencies equal to standard load balancing, which is a useful benchmark for how much a latency-blind router leaves on the table.

Second, do not confuse routing with failover. A router selects a source for expected value; failover reacts to failure. Mixing them produces policies where a timeout silently promotes the fast source, which is exactly the wrong default when the case was heading for review.

Third, keep a decision log with the inputs: source latencies, error rates, the computed expected costs, the threshold in force, and the outcome once known. This is the provenance discipline that separates grounded systems from confident-sounding ones, and the same record lets you audit whether citations in the final answer actually support the claims (provenance checks for RAG stacks).

Fourth, specify the prompt-level behavior for conflicts so the model does not improvise. A production RAG prompt should define what to do when sources disagree and when to stop instead of guessing (RAG prompting guide). The prompt should agree with the router: if the router escalates, the model should say so rather than picking a side.

A minimal policy file captures the whole rule set:

case_class: refund_over_threshold
deadline_p95_seconds: 8
sources:
  fast_cache: {latency_p50: 0.6, latency_p95: 0.9, error_rate: 0.12}
  ledger: {latency_p50: 3.5, latency_p95: 5.2, error_rate: 0.03}
wrong_call_cost_usd: 40
human_ceiling_usd: 15
review_cost_usd: 9
on_deadline_miss: escalate
on_conflict_type: {ambiguity: escalate, staleness: prefer_slow, dispute: price_and_compare}

Fifth, budget reviewer time from the escalation rate. If 2,000 cases a day cross the ceiling at an average of six review minutes, that is 200 reviewer-hours a day. A threshold that produces that volume without the capacity behind it does not route work to humans; it routes work to a backlog (review capacity planning). Tune the ceiling until escalation volume fits capacity at 80 percent utilization or below.

Finally, watch the accuracy of the probability estimates themselves. Governed, well-maintained knowledge bases show far higher retrieval accuracy than ungoverned ones, with reported accuracy in the 85 to 92 percent range on governed data against 45 to 60 percent on ungoverned sources (Atlan on RAG accuracy problems). If your fast source sits on a governed index and your slow source does not, the error rates in your table may be backwards, and the whole calculation inverts.