Extractive Answers: Sending the Passage, Not the Document

Selection happens in two stages, and each stage has a cheap option.

Stage 1: find the candidate passage. Use whatever retriever you already run. If queries contain identifiers, error codes, product SKUs or parameter names, lexical matching matters: BM25 exists specifically because embeddings can miss exact strings, and combining BM25 with embeddings is the standard fix BM25 and hybrid retrieval. If you are building on PostgreSQL, you can do the hybrid search in the database itself, fusing the two ranked lists with reciprocal rank fusion hybrid search in PostgreSQL. A Python walkthrough of the same architecture, with HyDE query expansion and a reranker, is available in this retrieval pipeline tutorial.

RRF is worth understanding because it explains why you can keep the retriever cheap. It scores documents by their rank position in each list rather than by raw score, which sidesteps the fact that BM25 scores are unbounded and cosine scores are not on the same scale how RRF works. A comparison of raw hybrid scores against RRF on the same query shows RRF compressing a wide spread (0.5 down to 0.0005) into a narrow one (0.01639 down to 0.01429), which is what makes the ranking stable.

Stage 2: trim the passage to a span. Once you have a passage, split it into sentences and keep only the sentence or two that contain the answer-bearing terms. A sentence splitter is enough for most prose. For tables and code, split on rows or blocks instead, because a sentence boundary inside a table is meaningless.

A practical rule: the span must contain the entity the question names and the value the question asks for. If the question is "what was the revenue growth for ACME in Q2," the span must contain "ACME," "revenue," "Q2" and the number. If any of those are missing, widen the span before you widen the prompt.

import re

def split_sentences(text):
    # crude but predictable: split on sentence-final punctuation
    parts = re.split(r'(?<=[.!?])\s+', text.strip())
    return [p for p in parts if p]

def select_span(passage, must_contain, max_sentences=3):
    sentences = split_sentences(passage)
    hits = [i for i, s in enumerate(sentences)
            if all(term.lower() in s.lower() for term in must_contain)]
    if not hits:
        return None  # widen or fall back to the full passage
    start = max(0, hits[0] - 1)          # one sentence of lead-in
    end = min(len(sentences), hits[0] + max_sentences)
    return " ".join(sentences[start:end])

The one sentence of lead-in is not decoration. It is what resolves pronouns and carries the subject that the answer sentence assumes. If you drop it, "revenue grew 3% over the previous quarter" arrives with no company name attached.

Tokens are the unit that matters, and the ratio between document and span is usually large. Anthropic notes that a knowledge base under 200,000 tokens (about 500 pages) can simply be put in the prompt whole, and that prompt caching cuts latency by more than 2x and cost by up to 90% for repeated prompts contextual retrieval. Most corpora are far larger than that, which is why retrieval exists at all.

Here is a worked comparison for a single question. The numbers are illustrative token counts you should replace with your own tokenizer, but the arithmetic is what you will actually run.

The cost ratio is the easy part. Latency is where practitioners get surprised. Prefill work grows with the number of input tokens, so a prompt that is 30 times shorter does materially less work before the first output token. Output tokens, on the other hand, are unaffected: if you ask for a 200-token answer, you pay for 200 output tokens regardless of which context you sent. That is why extractive answers pay off most in high-volume question answering, where the prompt dominates, and least in long-form writing, where generation dominates.

A worked monthly figure makes the decision concrete. Suppose you serve 200,000 questions a month, the full-document path sends 9,000 input tokens each, and the trimmed-span path sends 300. At any per-million input token price P, the full-document path costs 200,000 × 9,000 ÷ 1,000,000 × P, which is 1,800P. The span path costs 200,000 × 300 ÷ 1,000,000 × P, which is 60P. The gap is 1,740P per month, and the verification call adds 200,000 × 400 ÷ 1,000,000 × P, or 80P. Plug in your provider's input price as P and you have the annual saving. Do not trust anyone else's P; input prices move, and the ratio survives price changes better than the absolute number does.

Cost savings are easy to measure. Quality loss is not, and it shows up in three recognizable patterns.

The pronoun problem. A sentence like "its population is 3.7 million" answers "what is the population of Berlin?" only if the preceding sentence names Berlin. Splitting by sentence alone produces chunks where the subject has been cut away, which is exactly the failure that sentence window retrieval was designed to fix: embed the sentence, but send the sentence plus its neighbors sentence window retrieval.

The chunk-boundary problem. Fixed-character splitting can cut a key number in half. The same source shows a 100-character split on a Wikipedia page where the population figure straddles two chunks, so neither chunk answers the question. Overlapping chunks or sentence-aligned chunks fix it, at the cost of a slightly larger span.

The displacement problem. Hybrid retrieval and reranking both improve ordering, but neither changes what is in the index. A reranker scores candidates with the query and improves their order; it does not add documents that were never retrieved reranker patterns and trade-offs. If your trimming step drops the one sentence that carried the answer because it lacked a keyword, no downstream model can recover it.

The decision rule: trim aggressively only when the question names its subject explicitly. When the question relies on context ("what did they report last quarter?"), send the passage, not the sentence, and let the model resolve the reference.

A verification step is what separates extractive answers from guessing. The check is short: given the question and the candidate span, does the span entail the proposed answer? Run it as a separate, cheap call with a small output budget.

The metric you are approximating is faithfulness, the share of claims in the answer that are supported by the retrieved context. It is described as the most critical RAG metric precisely because low faithfulness means hallucination RAG evaluation metrics. You can compute it at span level rather than answer level: for a one-sentence answer drawn from a three-sentence span, the check is essentially binary.

System: You check whether a passage supports an answer.
Return SUPPORTED, PARTIAL or UNSUPPORTED, then one short reason.

Question: {question}
Passage: {span}
Proposed answer: {answer}

Route the three outcomes differently. SUPPORTED goes straight to the user. PARTIAL widens the span by one paragraph and regenerates. UNSUPPORTED widens to the full passage, and if the answer still fails, return "not found in the source" rather than letting the model improvise. That last branch is the one most teams skip, and it is the one that prevents confident wrong answers on questions your corpus cannot answer.

Two implementation notes keep the check honest. First, log the span alongside every answer so you can audit later; a span is a citation, and a citation you cannot retrieve is not a citation. Second, measure retrieval and generation separately, because a bad answer can come from a wrong span or from a right span used badly, and the two require different fixes evaluating components separately. Ranking metrics such as recall@k and MRR tell you whether the right span was in the candidate set at all retrieval ranking metrics.

That last row is the quiet saving. When the retriever finds nothing that contains the named entity and value, you can return "not found" without calling the generator at all, which removes both the cost and the hallucination risk for the queries your corpus cannot serve.

The right configuration depends on how much of your traffic is answerable by a single span. Use this checklist to set defaults, then revisit it when your evaluation set changes.

• Does the question name its subject, or does it rely on context? Named subjects tolerate sentence-level spans; context-dependent questions need a paragraph.
• Does the corpus mix prose with tables, code or parameter lists? Split on structure, not on punctuation.
• Are queries full of identifiers and error codes? Keep the lexical half of hybrid search; embeddings alone will miss exact strings.
• Is your traffic high volume with short answers? Extractive spans pay off most here.
• Do you have a labeled evaluation set? Without one, you are trading a measurable cost against an unmeasurable quality risk.

If you need to run the generation and verification calls across several providers without maintaining separate integrations, Quanta Inference (https://quantaintelligence.ai/inference) puts text, embedding and reranking-style models behind one OpenAI-compatible API with prepaid US dollar pricing, which keeps the per-query cost math above readable across the whole pipeline.