Context Window Budget: How Many Chunks to Send and When to Stop

The first instinct is to send as much as the context window allows. That is usually a mistake for three reasons.

First, cost scales with input tokens. Every query pays for the full prompt, including instructions, the user question, conversation history, and every chunk. A prompt that is three times larger costs roughly three times more on the input side, per query, at scale.

Second, latency scales with prompt length. More tokens to read before the first output token means a slower start, and larger budgets make the slowest requests slower still. Anthropic observed that prompt caching reduced latency by more than 2x and costs by up to 90 percent for repeatedly used prompts, which is a signal of how much day-to-day cost and time sit in the prompt itself Anthropic on contextual retrieval. A generous budget is a tax on every request.

Third, more context is not monotonically better for quality. Relevant information buried in long prompts can be missed, the lost-in-the-middle problem, which is one reason to pass only the optimal amount of information rather than everything that fits Pinecone's chunking guide. Anthropic notes that for knowledge bases under about 200,000 tokens you can consider sending the whole thing with prompt caching, but that approach stops scaling as the corpus grows, which is exactly when retrieval and a deliberate budget matter Anthropic on contextual retrieval.

So the context window is an upper bound, not a target. Your target is a budget derived from latency and cost, and the chunk count is derived from that budget.

Suppose chunk size is fixed at 800 tokens with 10 percent overlap, so 80 tokens repeat at each boundary and you should count them once. You have a latency target that lets you afford about 12,000 tokens of context per request. Here is the arithmetic.

Per-chunk overhead is not zero. Each chunk arrives with a metadata header (source, section, page), separators, and the occasional citation marker. Budget 50 tokens per chunk for that. So the effective cost per chunk is:

effective_chunk_tokens = chunk_size + overhead
                       = 800 + 50
                       = 850 tokens

max_chunks = floor(budget / effective_chunk_tokens)
           = floor(12000 / 850)
           = 14 chunks

Fourteen is the hard ceiling. It is not the answer. The answer is the K where marginal recall flattens, and that is almost always well below the ceiling. To find it, run an offline evaluation on a labeled set of, say, 300 questions, each with a known correct answer, and record whether the system answered correctly at K = 1, 2, 3, and so on.

A realistic result looks like this (illustrative numbers, replace with your own):

Three things jump out. First, the steep part of the curve is K = 1 to 4; each additional chunk there buys double-digit answers. Second, from K = 4 onward the curve is flat: going from 4 chunks (3,400 tokens) to 14 chunks (11,900 tokens) adds 12 correct answers at the cost of about 3.5 times the input tokens. Third, the marginal token cost per extra correct answer explodes. At K = 4 you spend 3,400 tokens to answer 255 questions; at K = 14 you spend 11,900 tokens to answer 267. You paid about 8,500 extra tokens per query for 12 more answers.

The stop rule falls out of the table: send the smallest K after which the next chunk adds less than, say, one percentage point of additional correct answers, or costs more in added latency and tokens than it returns. In this example that is K = 4 or K = 5, not the ceiling of 14.

Two caveats. First, the numbers are illustrative; your curve will differ by corpus and question mix. NVIDIA found that factoid queries favored smaller chunks of 256 to 512 tokens while complex analytical queries benefited from larger chunks or page boundaries NVIDIA's chunking study, so segregate your evaluation by query type. Second, recall here means "the correct answer was produced," which is not purely a retrieval measurement. If you want to measure retrieval recall separately, track whether the golden chunk was present in the top K. That is the Pass@k metric Anthropic uses, where contextual embeddings improved Pass@10 from about 87 percent to about 95 percent on a dataset of codebases Anthropic's contextual retrieval cookbook.

The worked example assumed the K you retrieve is the K you send. The better pattern decouples them: retrieve a wider pool (say 40 to 50 candidates), rerank them with a stronger but more expensive model, then send only the top few. Reranking applies the expensive model to a small candidate set rather than the whole corpus, keeping latency low while surfacing the most relevant results Qdrant's hybrid search with reranking tutorial.

This directly changes your budget arithmetic. You can widen the candidate pool (cheap, since retrieval-only candidates do not enter the prompt) without inflating the token budget (only the reranked top K enters the prompt). Hybrid search, combining dense embeddings with BM25 keyword matching, is a good way to widen the net, because BM25 catches exact identifiers and technical terms that embeddings miss, such as an error code like TS-999 Anthropic on contextual retrieval. Reranking a hybrid result set is a well-trodden production pattern hybrid search and re-ranking in production RAG.

The budget decision after you add reranking becomes: retrieve 40 to 50 candidates, rerank, then send the top K you found in your marginal-recall experiment. You keep the wide recall net without paying prompt tokens for it.

A flat marginal-recall curve is not the only reason to cut K. Lower the budget deliberately in these cases.

• The query is a lookup, not a synthesis. "What is the refund window?" needs one passage. Send 2 chunks. "Compare the refund policy across three regions" may need 6.
• Your latency target is tight and interactive. Time to first token dominates perceived speed, and prefill scales with prompt length. If p95 time to first token is your SLA, the context budget is where you claw back milliseconds MLflow on latency trade-offs.
• Chunks are semantically coherent and context-rich. Parent-child chunking retrieves small child chunks but returns the enclosing parent, typically 500 to 2,000 tokens, which can carry the context without needing many separate chunks parent-child chunking in LangChain. If you use hierarchical retrieval, fewer sends may suffice.
• Your corpus is small. If the whole knowledge base fits comfortably with prompt caching, retrieve-and-send logic is overhead you may not need yet Anthropic on contextual retrieval.

A decision rule that covers most cases: set the ceiling from the budget, set the working K from marginal recall, split K by query intent (lookup vs. synthesis), and let reranking decouple how wide you retrieve from how narrow you send.

You cannot tune a dial you do not measure. Log four numbers per request: chunks retrieved, chunks sent, prompt tokens, and time to first token. Then compute a token cost per answered question and watch it against your marginal-recall curve offline. If production token cost per answer drifts upward while correct-answer rate holds, your K or your prompt template has grown without justification.

A minimal logging sketch, wired once and reused for evaluation:

record = {
    "query": q,
    "chunks_retrieved": len(candidates),
    "chunks_sent": len(sent),
    "prompt_tokens": prompt_tokens,
    "ttft_ms": ttft_ms,
    "answered_correctly": None,  # filled by evaluation harness
}

The answered-correctly field is blank in production but populated in your offline run, which is what lets you reproduce the marginal-recall table. Unstructured makes the same point from the operations side: retrieval latency covers vector search, document fetch, and any processing that must happen before the model can start generating, and you only optimize it when you can see every stage that contributes time and variation Unstructured on retrieval latency. Without per-stage measurement you will tune the wrong component. Latency budget work starts with measuring end-to-end percentiles and mapping the real request path with shared timestamps MLflow on latency trade-offs.

One lower-level point: a context budget is only as good as what it contains. If chunks genuinely lack the surrounding context needed to interpret them, and a chunk does not make sense to a human without its neighbors, it will not make sense to the model either Pinecone's chunking guide. In that case the fix is contextual preprocessing, not a bigger budget. Contextual retrieval adds chunk-specific context before embedding and reduced retrieval failures by 49 percent, or 67 percent when combined with reranking, on the Anthropic evaluation Anthropic on contextual retrieval. Spending tokens on better chunks beats spending them on more chunks.

If you are running this across several models and want one key for text, embedding, and rerank models, Quanta Inference provides OpenAI-compatible access to those model types through a single prepaid API Quanta Inference, which keeps the budget arithmetic in one place rather than spread across providers.