Routing Conflicts by Cost: Which Precedence Calls Need a Human

A wrong precedence call means the loop applied the wrong rule and something downstream inherited the error. Break C into three components so you can estimate each one from records you already have.

Downstream rework. How much labor undoes the error once it surfaces? If a wrong merge policy duplicates a customer record, someone reconciles the duplicates. Count the minutes, multiply by the loaded minute rate, and add the re-processing time. If the error reaches a batch job, add the cost of re-running it.

Duplicated effort. When the wrong source wins, work gets done twice: once against the wrong assumption, once against the corrected one. This is the quietest component because nobody files a ticket for it. Instrument it by diffing what was produced against what would have been produced under the correct precedence for a sample of resolved conflicts.

Customer-visible errors. These are the ones that show up in support queues, churn, and refunds. They are the hardest to estimate and the ones most likely to dominate C. Use your own incident history: pull the last twenty conflicts that reached a customer, total the support time, credits, and refunds each one caused, and take the median. A single number drawn from your own records beats a borrowed industry average.

A worked example makes the arithmetic concrete. Suppose a wrong precedence on an invoice-matching rule costs 40 minutes of downstream rework at a loaded $0.75 per minute, so $30, plus a duplicate payment that lands in the customer-visible bucket once in every ten errors at $200 each, so $20 expected. C is $50. If the loop is wrong 12 percent of the time on this rule, the expected cost per conflict is $6. Compare that to R and the call writes itself.

The framework for pricing human review against the expected value of the decisions it improves is set out in Scale's analysis of which work is worth routing to a human, which makes the same point: the boundary should be risk-adjusted, and as potential impact rises, the confidence required for autonomous action should rise with it.

R is where most estimates go wrong, always in the same direction: too low. The salary divided by working minutes is the starting point, not the answer.

Add these on top of base compensation: benefits and payroll overhead, the reviewer's tooling and seat cost, the queue-management time spent triaging before the review, and the context switch when a reviewer drops one task to pick up a conflict. Context switching is real and measurable; if your reviewers handle both conflict reviews and a primary queue, time a sample of interruptions and count the minutes lost on both sides.

One published anchor for review throughput comes from the systematic-review literature. Experienced screeners process roughly two abstracts per minute, and less experienced reviewers can take up to seven minutes per record, per ASReview's explanation of active learning for screening. Conflict precedence reviews are closer to the slow end because the reviewer must reconstruct context, not just classify. If your reviewer needs four minutes to load context, read both sources, decide, and record the rationale, and the loaded minute rate is $1.05, R is $4.20.

Keep a running table like this per queue. The minute rate is usually stable across queues; the minutes per review is not, and it is where the interesting variation lives.

With C and R in hand, the routing condition becomes a probability threshold. Solve p times C greater than R for p and you get p greater than R divided by C. That is the confidence floor below which a conflict needs a human, and it differs for every rule.

Work it through for three rules from an invoicing pipeline with R at $4.20:

The product description rule rarely earns a review. The loop would have to be wrong more than half the time, and if it is that bad you have a different problem. The payment routing rule earns a review nearly always, which means the right move is to route those conflicts to a human by default and stop pretending the loop owns them. The invoice rule sits in the middle and is where threshold tuning actually pays.

Per queue, the thresholds differ again because the mix of rules differs. A queue dominated by payment conflicts should have a low aggregate threshold; a queue of description merges should have a high one. Publishing a single global 0.8 undermines both.

The distinction between blocking review and watching is worth naming, because it changes the math. Human-in-the-loop means the workflow pauses until someone approves, while human-on-the-loop means a person watches dashboards and intervenes after execution, as Elementum's comparison of the two models sets out. Blocking review costs the reviewer's minute plus the queue's latency; watching costs only the minute, but you pay the error cost when the watcher misses something. For high-C rules, block. For low-C rules, watch or leave it in the loop entirely.

The formula is only as good as p. A threshold built on a guessed probability is a guess with extra steps. Three practices make p defensible.

Score with calibration, not confidence. A model saying it is 90 percent sure is a claim, not a measurement. Calibrate by binning historical conflicts by the loop's stated confidence and computing the actual error rate per bin. If the 0.8 bucket is wrong 25 percent of the time, that is your p for that bucket, whatever the model says. The detection-and-response work on conflicting sources in retrieval shows that conflict types differ and each requires tailored handling, so bin by conflict type as well as confidence; see Google Research's taxonomy of RAG conflicts.

Track p per rule, not per system. A global error rate hides the rules where the loop is genuinely weak. Log the outcome of every resolved conflict, including the automated ones, and roll up weekly.

Sample the automated calls. You cannot know p for cases you never review unless you audit a random slice. Pull a small random sample monthly, have a reviewer check the precedence decisions, and update p. This sampling is the same logic as screening prioritization in active learning, where a model ranks records and reviewers label the ones most likely to matter; the ASReview framework paper in Nature Machine Intelligence describes the approach and its efficiency gains. Your sample does not need to be large. It needs to be random relative to the automated stream.

If your loop stores summaries of past conflicts to inform later decisions, be aware that compression can lose the details reviewers need. A controlled ablation found that verbatim chunks beat extracted artifacts by 15.9 points on LoCoMo and 22.0 points on LongMemEval-S, with the authors attributing the gap to lossy distillation, as reported in the arXiv study of memory representations. For conflict review, keep the raw source spans alongside any generated artifact so the reviewer sees what the loop actually saw.

A threshold you cannot audit will drift. Every automated precedence call and every review needs the same structured record: the rule, the conflicting sources, the loop's chosen winner, its confidence, the calibrated p, the estimated C, the computed expected cost, and either the threshold comparison or the reviewer's decision. Write it as a row, not a paragraph.

{
  "conflict_id": "inv-88213",
  "rule": "invoice_line_match",
  "sources": ["erp_lines", "vendor_statement"],
  "automated_winner": "erp_lines",
  "confidence": 0.91,
  "calibrated_p": 0.11,
  "cost_of_error_usd": 50.0,
  "expected_cost_usd": 5.5,
  "review_cost_usd": 4.2,
  "routed_to_human": true,
  "decision_reason": "expected_cost_exceeds_review_cost"
}

Two details make this durable. First, record the expected cost alongside the decision, so tuning later has data instead of memory. Second, never let a failed review default to "approved"; hold items in a durable pending state, because a system crash should not silently resolve a conflict, a point the HITL implementation guidance from Domo makes explicit.

Provenance is the other half of auditability. If a reviewer cannot trace a precedence call back to the exact source span that supported it, the review is theater. The provenance checks described in this practical guide to RAG provenance apply directly: every non-trivial decision should map to a source, a chunk, and a span. Conflict handling is one of the layers a well-designed instruction set defines explicitly, per this RAG prompting guide, and keeping that layer consistent with your routing policy prevents reviewers from seeing a different notion of "winner" than the loop used.

Context quality also moves p. Industry reporting suggests retrieval is where most RAG failures originate and that governed data materially improves accuracy, per Atlan's breakdown of RAG accuracy problems. If your p values are high across many rules, the cheaper fix is often better upstream data, not more reviewers.

The formula cuts both ways, and the uncomfortable finding is usually that a queue is over-reviewed. If a rule's p sits well below R divided by C, the review is buying less than it costs. That does not mean delete the rule. It means move the conflict from blocking review to watching, or to full automation with a monthly random audit.

Watch for the second-order effect. When review gets cheaper or automation gets cheaper, the volume of work worth doing expands, so a threshold tuned last quarter may route too much or too little now. The analysis of how cheaper intelligence changes the automation business case makes this point about model routing generally: match each part of a workflow to the least costly option that meets its quality and control requirements, rather than forcing the whole workflow onto one setting.

A quarterly review of the threshold table is enough. Pull the logged expected costs and actual review costs, recompute p per rule, and move the borderline rules. Most of your thresholds will stay put; the two or three that move are where the money is.