Mapping Retrieval Budgets to Answer Quality Targets

Ground the target in the two failure modes retrieval budgets actually trade off. Too few chunks means the evidence is not in the prompt, so the model either abstains or hallucinates; too many chunks means you pay tokens for noise, and noise has its own cost. Anthropic's Contextual Retrieval post notes that retrieval works by adding the most relevant chunks to the prompt, which makes the count of chunks a direct lever on both quality and spend. Unstructured's latency playbook puts it plainly: adjust chunk count by query intent, because a larger candidate set costs retrieval and reranking work.

Write the target as a table so it is testable:

A secondary guard matters because a high correctness score can hide unsupported claims. Groundedness evaluation checks each statement against retrieved context, and a response can be mostly grounded with one hallucinated sentence, which is enough to break trust in a support or compliance setting (Braintrust).

The budget experiment measures the effect of chunk count. If the underlying retrieval or chunking changes mid-experiment, you are measuring two things at once. So lock the retrieval pipeline first.

Two things to lock. Retrieval strategy: hybrid search (dense vectors plus lexical matching) is the common baseline because embeddings miss exact identifiers such as error codes while BM25-style matching catches them (Anthropic), and a walkthrough of a hybrid pipeline with HyDE and reranking shows the same reasoning: vector search fails for specific IDs and acronyms, so you combine keyword and semantic retrieval and then rerank. Chunk size: the Chroma evaluation found the choice of chunking strategy has a significant impact on retrieval accuracy and efficiency, and that default settings on popular splitters can perform poorly (Chroma). A practical default is recursive splitting at roughly 400 to 512 tokens with 10 to 20 percent overlap (Firecrawl's chunking comparison). Settle on one chunk size, measure its average token length, and only then start the budget sweep.

There is an important subtlety: chunk size and chunk count are not independent. Larger chunks mean fewer are needed to cover the same evidence, which is exactly why the calibration step measures both. One study introduced "Context Window Utilization" as a way to reason about how much of the window retrieved text should occupy, finding an optimum that balances sufficient context against irrelevant content (Juvekar and Purwar). That is the quantity your budget sweep estimates for your own corpus.

The sweep is deliberately boring: for each question in the labeled set, retrieve at several chunk counts and score the answers at each count.

1. Build the labeled set. 30 to 50 questions with a reference answer each. Sample from real user queries and cover the question mix in your target table.
2. Pick budgets to test. Four points are usually enough: 2, 4, 6 and 8 chunks. Include one point you expect to be too small so you can see the curve rise.
3. Hold everything else fixed. Same retrieval, same prompt template, same model, same reranker threshold. Only top_k changes.
4. Score each answer. Correctness against the reference, plus groundedness against the retrieved chunks. Score blind to which budget produced the answer.
5. Record tokens. For each budget, log the retrieved token count so you can convert chunks to tokens later.

A minimal harness looks like this:

BUDGETS = [2, 4, 6, 8]

def sweep(labeled_set, retrieve, answer, score_correct, score_grounded):
    results = {}
    for k in BUDGETS:
        correct, grounded, tokens = 0, 0, []
        for item in labeled_set:
            chunks = retrieve(item["question"], top_k=k)
            reply = answer(item["question"], chunks)
            correct += score_correct(reply, item["reference"])
            grounded += score_grounded(reply, chunks)
            tokens.append(sum(c["token_count"] for c in chunks))
        n = len(labeled_set)
        results[k] = {
            "correct": correct / n,
            "grounded": grounded / n,
            "mean_tokens": sum(tokens) / n,
        }
    return results

Thirty to fifty items keeps each sweep cheap: 40 questions times 4 budgets is 160 generations, which fits in a single afternoon and costs little even on a hosted API. The same reasoning about small fixed evaluation sets appears in Anthropic's contextual embeddings cookbook, which uses a fixed query set with a golden chunk per query so that changes can be compared like for like.

Plot correctness and groundedness against chunk count. Typical outputs fall into one of three shapes:

The third shape is the one people misread. If 8 chunks still misses the target, the problem is upstream: chunks too small to hold the answer, lexical queries that dense retrieval cannot match, or a reranker cut too aggressively. Adding chunks cannot recover evidence that retrieval never surfaced. Two systems claiming the same evidence budget can realize different actual usage rates, so a single accuracy number without the operating point is not comparable (a budget-aware Active RAG evaluation).

Once the curve does flatten, apply the safety margin. If 85 percent is the target and 4 chunks gives 87 percent while 6 gives 88 percent, pick 4: the extra chunk buys one point for a large token increase. If 4 chunks gives 86 percent and 6 gives 90 percent, pick 6, because 86 percent leaves no room for drift in production traffic. A margin of two to five points above the target is a reasonable rule of thumb, wide enough to absorb set noise, narrow enough that you are not gold-plating.

Here is the same decision with numbers. Suppose your sweep returns:

With an 85 percent target, 4 chunks clears it by one point, which is too thin. 6 chunks clears it by four points and groundedness holds at 96 percent, so 6 is the choice. Going to 8 adds 780 mean tokens per request for no measurable correctness gain. Multiply 6 by the measured average chunk size (1,560 divided by 4 is 390 tokens per chunk) to get a token budget of about 2,340 tokens of retrieved context per request. That is the number you carry into prompt assembly, cost models and latency budgets.

The multiplication is trivial; the discipline around it is not. Use the measured average chunk size from your own corpus, not a nominal setting, because overlap and document structure push actual chunk length above the configured value. Then apply the token budget in two places.

Cost. Retrieved tokens are paid on every request, so the difference between 4 and 6 chunks compounds across your traffic. Compute cost per request as retrieved tokens plus prompt and answer tokens, multiplied by price per token, then multiply by monthly request volume. That figure makes the budget conversation concrete.

Latency. Longer prompts increase time to first token because of prefill work (MLflow's latency budget guide), and a retrieval latency budget is only meaningful if each stage has a timeout and a fallback (Supermemory). A worked stage allocation for a 500 ms retrieval path can reserve roughly 50 ms for network, 80 ms for orchestration, 200 ms for search, 100 ms for reranking and assembly, and 70 ms of headroom. If your chosen budget of 6 chunks pushes reranking past its allowance, you have two honest options: cap the reranker's input, or accept a smaller chunk count and the quality it buys.

Chunk count also interacts with reranking depth. Reranking improves evidence ordering, but it adds latency and, in the shape of the curve above, may not change the answer at all; Supermemory's experiment table lists exactly this failure to watch for: added latency without better answers. Your sweep tells you whether the reranker is earning its place.

A budget chosen on the same 40 questions that produced the curve will look better than it is. Split the labeled set before the sweep: 30 to 40 items for calibration, 10 to 15 held out. After picking the budget, run the held-out items once at that budget and once at the next budget up.

The test passes if held-out correctness at the chosen budget is within a few points of the calibrated figure and still above the target. It fails if held-out correctness drops below the target, which usually means the calibration set was too easy or too small. The fix is more items in the weak question category, not a bigger budget.

Track the failure types too. If held-out failures are groundedness failures (answers make claims the chunks do not support), a larger budget may genuinely help, because the supporting sentence was outside the retrieved set. If failures are retrieval failures (the right chunk was never returned at any budget), stop tuning the budget and revisit hybrid search, reranking or chunk size. Reporting retrieval harm rates and realized usage alongside accuracy is what makes two evaluations comparable (budget-aware Active RAG).

Set a re-check trigger. Recalibrate when the corpus shifts materially, when the chunking or embedding configuration changes, or when the model changes, since a chunk size tuned for one tokenizer can silently truncate or waste context on another (ChunkRank). Each re-check is the same sweep at the same four budgets, which takes an afternoon and gives you a defensible number.

Guessing the budget from context window size. A large context window is not permission to fill it. The Chroma report notes that inserting entire documents is often inefficient and can distract the model, since every token in the window is processed for each inference (Chroma).

Comparing budgets across different retrieval setups. If one run uses hybrid search and another uses dense only, the chunk counts are not comparable. Lock retrieval first, as described above; the hybrid, HyDE and rerank pipeline walkthrough is a useful reference architecture to freeze against.

Ignoring chunk size drift. Parent-child chunking, where small child chunks are matched but larger parent chunks are returned, changes the effective token budget per retrieved item dramatically, with parents commonly in the 500 to 2,000 token range (Seahorse on parent-child chunking). If you adopt it, the multiplication factor changes and the sweep must be rerun.

Optimizing one metric. Correctness without groundedness lets a confident, unsupported answer pass. Groundedness without correctness lets a faithful answer to the wrong question pass. Score both, always.

When you are ready to run the same sweep across several candidate models without rebuilding the harness, one API key covering text and embedding models makes the comparison cheaper, and Quanta Inference is built for that.