A Rewrite Budget per Query: How Many Facts Before Retrieval

Conversational queries are built for humans, not indexes. A follow-up like "what about returns" carries almost no embeddable meaning on its own, and a crowded question such as "how does pricing and onboarding time compare between the basic and pro plans" folds four ideas into one sentence. Without decomposition, a vector search produces what one query rewriting explainer calls a blurry average of all four ideas, strongly none of them. The chunk that precisely answers pro plan onboarding time never speaks clearly on its own.

Decomposition fixes that, but it introduces the opposite failure. An unbudgeted rewriter told to "be thorough" will happily produce six or ten sub-queries per user question. Each sub-query costs an embedding call plus a retrieval round. Each returns top-k chunks. The generator then receives a context window stuffed with loosely related passages, and the passage that actually matters is buried or dropped by truncation. The same explainer notes that multi-query rewriting trades retrieval quality against latency, cost, and rewrite errors.

So the budget has two jobs: force enough decomposition that every required fact gets its own clear signal, and cap decomposition so the context stays focused and the bill stays predictable.

You do not need a model to classify a rewrite. You need three observable signals, and they map cleanly onto the two failure modes.

The precision and recall halves of that table come straight from standard retrieval evaluation: precision@k measures whether the top-k results are highly relevant, while recall@k measures whether the full set of relevant documents was captured. A wide rewrite hurts precision first, because diluting the query pulls in loosely related chunks. A thin rewrite hurts recall first, because the fact that never got written never gets retrieved.

Chunk attribution is the underused signal. Galileo's RAG evaluation guide describes it as a measure of how much retrieved content contributes to the final output, and notes that low attribution means the system retrieved useful documents but failed to integrate them. A rewrite that pulls forty chunks and uses four is wide, full stop, whatever the coverage number says.

A decision rule that works in practice: classify a rewrite as thin if fact coverage is below your target and precision@k is high. Classify it as wide if coverage is near 1.0 but precision@k or attribution has fallen relative to your baseline. If both coverage and precision are low, the problem is upstream of the budget (bad chunks, wrong embedding model) and raising the cap will not help.

The cap comes from data. The procedure takes an afternoon on a proper eval set.

1. Build a query set from real traffic. DataAIHub's retrieval evaluation guide recommends 50 to 200 query-document pairs drawn from your actual corpus and expected queries, and treating retrieval evaluation as a gate before you touch prompts or models.
2. For each query, have a human write down the required facts, meaning every discrete piece of information a correct answer must contain. Label them F1, F2, and so on. Ten to fifteen minutes per query for the first twenty, then it speeds up.
3. Run each query through the rewriter with no cap. Record how many required facts the rewrite names. Bucket queries by that count: 1, 2, 3 to 4, 5 to 8, 9 or more.
4. Retrieve with each rewrite and check which labeled facts appear in at least one retrieved chunk. A fact is missed if it was absent from the rewrite (so retrieval never had a chance) or present in the rewrite but absent from all retrieved chunks.
5. Compute the missed-fact rate per bucket: queries with at least one missed fact, divided by queries in the bucket.

You now have the curve the cap should follow. It usually looks like a cliff, not a slope. Buckets at 1 and 2 facts tend to show high missed rates because the rewriter under-decomposed. The rate drops sharply through 3 to 4 facts. Beyond some point, typical for many corpora at 5 to 8, the rate flattens: additional facts per rewrite stop recovering missed facts, because the remaining misses come from the corpus or the retriever, not from the rewrite.

The cap is the bucket where the curve flattens. Below it you are paying for misses in wrong answers. Above it you are paying for tokens and dilution in exchange for nothing measurable.

Here is a minimal harness for step 4, written in plain Python:

def missed_fact_rate(rows):
    # rows: list of dicts with 'query_id', 'fact_bucket', 'facts_missed' (int)
    buckets = {}
    for r in rows:
        buckets.setdefault(r["fact_bucket"], []).append(r["facts_missed"] > 0)
    return {b: sum(v) / len(v) for b, v in sorted(buckets.items())}

# Example shape of output:
# {1: 0.41, 2: 0.29, "3-4": 0.12, "5-8": 0.09, "9+": 0.08}

Read that example as an illustration of the shape, not a benchmark. Your numbers will differ, and the only number that matters is the one your own labeled set produces. Sanity-check the whole panel against the guidance that recall at k dominates retrieval quality: if the right chunk never lands in the candidate set, no amount of generation tuning recovers it.

Once you have the flattening point, enforce it in the rewrite prompt and in code. Two rules do most of the work.

First, cap at generation time. Instruct the rewriter to emit no more than N required facts and to drop lower-priority ones rather than merge them. Merging is the quiet enemy: a rewriter that squeezes four facts into one sentence has technically stayed under a word limit while still producing the blurry average.

Second, fall back rather than fail. If a query genuinely needs more facts than the cap, run a second retrieval pass for the overflow instead of stuffing one call. This is where query decomposition pays off, since multi-query rewriting for RAG exists precisely to take one messy user question and produce several retrieval-friendly ones. The budget then applies per pass, not per conversation.

A useful enforcement check, adapted from the two-pass pattern Elastic describes for atomic claim checking: never let the rewriter grade its own homework. Run a separate pass that decomposes the original query into atomic claims, then checks each claim against the rewrite and marks it present, superseded, or missing. The coverage table that comes out is your per-query budget audit, and it is far more reliable than asking the rewriter whether it did a good job.

Rebuild that audit whenever your corpus changes materially. The cap is a function of your documents and your users, and both drift. Toloka's evaluation guide recommends monitoring live performance for drift and integrating evaluation into the development workflow, which for a budget means re-running the bucket analysis on a schedule and gating prompt changes on it.

Take a support assistant with a knowledge base of plan documentation. Suppose your labeled set has 120 queries and the bucket analysis returns this pattern:

The curve falls from 0.41 at one fact to 0.13 at three to four facts, then flattens, and the small 9-or-more bucket ticks back up because over-wide rewrites start losing facts to retrieval truncation. A cap of 4 captures nearly all the recoverable gain. Setting it at 2 would leave roughly one in seven queries with a missed fact that a wider rewrite would have caught. Setting it at 12 buys three tenths of a percent and multiplies retrieval calls.

Then check the cost side. Count retrieval calls per query under a cap of 4 versus no cap. On the same 120 queries, an uncapped rewriter that averages 6.5 facts per query has roughly 60 percent more retrieval calls than a capped one averaging 4. At a few cents per grounded query, per the cost breakdown in the Digital Applied metrics panel, that gap compounds across traffic. The worked example is the whole argument in miniature: the cap is where the missed-fact curve stops bending, and everything past it is spend without recovery.

• Query set built from real traffic, not synthetic examples, sized at 50 or more per retrieval eval guidance.
• Required facts labeled per query, with a written definition of what counts as one fact.
• Buckets defined before measurement, so nobody moves the goalposts after seeing results.
• Missed-fact rate computed per bucket and plotted, with the flattening point marked.
• Precision@k and attribution tracked alongside coverage, per the signals table above.
• Cap enforced in the prompt and in code, with an overflow path for queries that legitimately exceed it.
• Coverage audit run as a separate pass, never by the rewriter itself.
• Re-measurement scheduled for corpus or traffic changes.

The evaluator side of this is where most teams stall, because scoring retrieval and generation separately takes real instrumentation. Toloka's guide makes the case for evaluating retrieval and generation separately before end-to-end scoring, since conflating them makes it impossible to tell which stage failed. If you would rather hand the measurement and the rewrite-prompt iteration to someone who does it repeatedly, Quanta Experts work inside your context and return finished deliverables rather than a chat transcript.