Confidence From Span Scores: Citing Versus Escalating

Collect a labeled sample of real traffic, not a synthetic set. Synthetic question-answer pairs help bootstrap, but they carry limitations that make them a weak basis for production thresholds (Evidently AI, Anyscale). Pull a few hundred to a few thousand logged questions, capture the answer, the span, the source chunk and the score, then hand-label each answer into one of four buckets:

That fourth row is the one teams skip. Retrieval quality sets the ceiling on final answer quality, and a retriever that misses the passage cannot be rescued by a better verifier (Anyscale). If your low-score bucket is dominated by missing-evidence cases, a threshold change will not help; fix retrieval first.

Then plot the label mix by score decile. What you are looking for is where the composition flips. A worked example with illustrative numbers:

• Deciles 9 and 10: about 97 percent correct and supported.
• Deciles 6 to 8: about 88 percent correct, but only about half with a clean span match.
• Deciles 3 to 5: about 60 percent correct, and most correct ones are short factoids.
• Deciles 1 and 2: under 25 percent correct, with many "no evidence retrieved" cases.

The first three figures suggest a high cut around the boundary of decile 8 (ship unchanged), a middle band across deciles 3 to 7 (ship with the span quoted and linked), and escalation below decile 3. The exact numbers are yours to find; the shape is the reusable part.

Two measurement notes. Answer length matters: a large-scale analysis of distilBERT on SQuAD found an inverse correlation between answer length and performance (Farea and Emmert-Streib in JAIR), so a long multi-span answer with a respectable score is riskier than a two-word one with the same score. And exact match is not one metric: different definitions of exact match differ by amounts comparable to reported system improvements, so two reviewers labeling "correct" may disagree more than you expect (JAIR). Write the labeling rubric down before the first batch.

A single threshold gives you two outcomes: answer or abstain. A two-cut rule gives you three, and the middle outcome is nearly free. Quoting the supporting span and linking the source lets a reader verify a claim in seconds. It is the same contract as a search engine that highlights the answer passage with its evidence (Ontotext). For an extractive system the span is already in hand; attaching it costs a rendering change, not a model call.

The routing rule, stated plainly:

ship                 if span_score >= high_cut
ship_with_source     if low_cut <= span_score < high_cut
escalate_to_human    if span_score < low_cut

Choosing the two points is a coverage-risk decision at each boundary, and the two boundaries have different costs. Above high_cut you accept the residual error rate in that band. Between the cuts you accept a smaller risk because the reader can check. Below low_cut you pay a human.

The decision rule most teams miss: set high_cut where the answered-band error rate matches what the product tolerates without a source, and set low_cut from reviewer capacity, not from accuracy. Accuracy maximization would send far more answers to people than anyone can process; the queue then becomes the real product, and stale escalations are worse than a linked answer.

A second rule: move the cuts, do not add a fourth band. Three states map to three UI behaviors and three operational owners. Every extra band multiplies the cases your QA team has to reason about when something goes wrong.

Escalation volume is a budget line. Compute it directly from the score distribution: the share of traffic below low_cut times weekly query volume gives weekly escalations. If your labeled sample says 4 percent of traffic falls below the low cut and you serve 50,000 queries a week, that is 2,000 escalations. At a generous 30 seconds of reviewer time each, that is about 17 reviewer hours, or roughly half a full-time reviewer at 35 productive hours. Those numbers are arithmetic on your own inputs, not benchmarks.

Then decide what to do when the budget is exceeded, because it will be:

Note the shape of the middle row. When the retriever or corpus shifts, do not assume the verifier broke. The SQuAD 2.0 study found that retrieval coverage alone is insufficient to prevent unsupported answers and called for stronger evidence calibration (the evidence-calibrated RAG paper). Both layers need checking, in that order.

A capacity-safety detail: if the queue backs up, prefer showing the linked answer with a clear "unverified" marker over silent delay. A linked span that a reader can check is a defensible degraded mode. A blocked question is not.

Span scores are only comparable inside one pipeline. Change the retriever's top-k, swap the embedding model, edit the chunking, upgrade the verifier, or re-index the corpus, and the distribution moves. Cut points derived on the old pipeline no longer mean what they meant.

Practical protocol after each change:

1. Freeze a fixed evaluation set of questions with hand-scored answers, separate from the tuning sample. Reuse it across versions so movement is comparable.
2. Re-score the frozen set through the new pipeline and re-plot the label mix by score decile.
3. Recompute weekly escalation volume from the new distribution against current traffic.
4. Compare the three band error rates against the previous run. If the linked band's error rate rose past tolerance, raise high_cut before touching low_cut.
5. Hand-score a fresh slice of live answers, not just the frozen set, to catch drift the frozen set cannot see.

Sentence window retrieval is a good example of why step 2 is not optional: retrieving a sentence but passing surrounding sentences as context improves the generator's view while the matched unit stays small (Guillaume Laforge on sentence window retrieval). That change alters what the verifier sees and how spans align to sources, so scores shift even though nothing about the question set changed.

The same discipline applies to retrieval fusion. Reciprocal rank fusion merges ranked lists by position rather than raw score, because BM25 scores are unbounded and cosine similarities sit in a bounded range, so the raw numbers are not comparable (OpenSearch's RRF introduction, Abhishek Thakur's RRF explainer). Hybrid retrieval measurably changes what reaches the verifier (a hybrid search reference with benchmark figures, pgEdge on hybrid search in PostgreSQL). The governing principle is identical: never compare numbers across scales, so never reuse a cut point across pipelines.

Rerankers deserve the same caution. A reranker improves ordering, not recall, and rerankers differ widely in cost, latency and quality, so adding or swapping one moves the score distribution even when the underlying verifier is untouched (Thinking Loop's reranker comparison).

The router itself is small. What matters is that every routing decision is logged with the score, both cuts, the pipeline version and the outcome label once a human has judged it. That log is your next tuning sample.

def route(answer, span, score, cfg):
    if score >= cfg.high_cut:
        return "ship", answer
    if score >= cfg.low_cut:
        return "ship_with_source", answer, span, cfg.source_url
    return "escalate", answer, span, cfg.queue

Two operational rules for the implementation. Version the cut points alongside the pipeline, so an answer served last month can be explained by the rule that was live then. And record the middle band separately in analytics: if linked answers get clicked and then flagged, the linked band's error rate is higher than your sample estimated, and high_cut needs to rise.

For teams running multiple model types behind a verification pass (text verifier, a cross-encoder reranker, an embedding model for the retrieval side), running them through one API keeps the pipeline versioning simple: Quanta Inference exposes text, embedding and reranking-style workloads behind one OpenAI-compatible key, which makes it practical to re-score a frozen evaluation set after every change instead of deferring it.

Put the pieces together with illustrative numbers. A support knowledge base answers with extracted procedures. A labeled sample of 1,000 logged answers yields:

Check the capacity arithmetic. At 40,000 queries a week, 7 percent is 2,800 escalations. At 30 seconds each that is about 23 reviewer hours weekly, which fits one reviewer with headroom. The linked band's 12 percent error rate sounds alarming until you check the outcome: those are answers the reader can verify against the quoted span in seconds, so the residual risk is bounded by how easy the source is to read. The shipped band's 4 percent error rate is the one to argue about with the product owner, and high_cut is the lever for that conversation.

Now the decision rule in one sentence: if you can only move one cut, move high_cut when the product cares about unverified wrong answers, and move low_cut when reviewers are drowning, but never move either without re-checking the frozen set.

If the escalation queue needs real throughput rather than a shared inbox, Quanta Experts is a way to route low-score cases to an agent that reads the context and returns a finished, reviewed deliverable, which changes the capacity math in the table above.