How Many Chunks to Send: A Context Window Budget for Answers

The cap you set on retrieved tokens is a remainder, not a starting figure. Four costs come out of the window first.

The tool and history lines are where budgets blow up quietly. A published breakdown of a moderate agent conversation totals 15,100 tokens before any retrieved documents are added: 4,500 for system prompt, tool definitions and agent instructions, 2,600 for project context, and 8,000 for twenty turns of messages in a worked context budgeting example. Retrieved rows sit on top of all of it.

A multi-turn context engineering guide makes the same point from the other direction: it lists the components competing for the window as system prompt, conversation history, tool results, retrieved context and the current input, and notes that a single tool result can return thousands of tokens in one call in that guide. Retrieved rows are one claimant among several, not the whole bill.

Then choose the retrieved cap as a fraction of what is left. Two-thirds of the remaining window is a reasonable opening bid for a question-answering product; half is a reasonable opening bid once you resend history. Pick one, write it in configuration, and set a small safety buffer below the hard limit, since production systems commonly hold back a margin to avoid hitting the cap according to the token budgeting guide.

One placement decision makes the rest of the budget readable. Put rules and role in the system message, and put document passages in the user message with the question as recommended in that walkthrough. A document welded into the system prompt changes the prefix on every call, so the provider cache never hits, and your trace no longer shows which passages the answer was grounded on. The same walkthrough prices the mistake on a small corpus: a system prompt with the document baked in ran to 225 tokens and changed every call, while a rules-only system prompt came to 18 tokens and was identical on every call.

Once rows exceed the cap, something must go. The order matters more than the mechanism.

First, drop whole low-ranked rows. If you have a reranker score per row, sort descending and fill until the cap is reached, discarding the tail. This keeps each surviving row intact, which matters because a chunk truncated mid-sentence can lose the sentence that held the answer. Drop whole rows, not halves.

Second, do not keep reranker scores in the prompt. A score is for your ranking logic. Sending it to the model spends tokens on a number the model cannot verify and may overtrust. Keep scores in your logs and your trace, and send only the passage text plus a stable identifier for citation.

Third, drop whole documents before dropping rows from a document that is already represented. If five documents each contribute four rows and only three documents are relevant, the irrelevant document's rows are the cheapest thing to remove. This is also where a deduplication pass pays: near-identical chunks from the same document crowd out distinct evidence.

The choice of retrieval strategy changes what "ranked" means. Recency is fast and returns the latest entries rather than the relevant ones; semantic similarity handles paraphrase but is brittle for rare identifiers such as an exact code or part number; keyword search is the reverse; hybrid search with a cross-encoder reranker is the configuration most production teams settle on, at the cost of an extra index lookup and a reranker call per turn per a survey of retrieval strategies. That reranker call is what gives you a trustworthy order to fill from, and the same survey puts the added latency in its own table, so you can decide whether the ordering quality is worth the wait.

If the evidence lives across many small records, sorting rows can fragment it. Systems that store observations as isolated records struggle to aggregate evidence about the same person or task, which is the fragmentation failure mode described in a memory architecture paper. When that applies, the fix upstream is grouping related rows under a topic before ranking, not a larger cap.

Order is not cosmetic. In the tasks studied in Lost in the Middle, accuracy is often highest when the relevant information sits at the beginning or the end of the input, and degrades when the model must find it in the middle of a long context, even for models built for long inputs. That finding alone changes how you place what survives the cap.

Two placements follow from it. Put your highest-ranked row first, and put the next most important evidence last, immediately before the question. Bury the marginal rows in the middle. If you use the common pattern of sorting ascending so the best chunk lands nearest the question, you have optimized for one end only; a symmetric order (best first, second-best last) uses both.

Keep a stable wrapper around each row: an identifier, the source title, and the text. Stable wrappers let the model cite and let you score faithfulness later. Send them in the same shape every call so the prompt is easy to diff when an answer goes wrong.

For long sessions, the ordering question is compounded by growth. The context window behaves like RAM rather than disk: it holds what the model reasons over on this call and disappears when the call returns as described in that multi-turn guide. If a tool result arrives mid-session, your retrieved cap has to be recomputed, not assumed.

An alternative to fitting everything in one prompt is paging evidence in over several calls, which is the operating-system analogy behind MemGPT. The paper's design treats the prompt as a small fast tier and the archive as external storage, moving material in and out as needed, and its own overview notes that enlarging the window does not guarantee the model uses all of its contents well as summarized in a paper overview. On a multi-session question-answering task the paper reports 92.5% accuracy for MemGPT against 32.1% for a fixed-context baseline, with the caveat, stated in the same overview, that the baseline received a lossy summary while MemGPT could search full history. The lesson for budgeting is modest: sometimes the right cap is small because the second hop is cheap.

Here is a concrete case. Suppose the window is 32,000 tokens, the system prompt and tool definitions come to 4,000, you reserve 4,000 for output, and you set retrieved context at half of what remains. You have 24,000 left after the first two subtractions, so the retrieved cap is 12,000 tokens.

Now suppose the reranker returns these rows, with token counts taken from your tokenizer before the call:

Four rows survive at 10,200 tokens, about 1,800 tokens under the cap. You could fill the remainder with a shorter fifth row if one exists; do not split row 5 to squeeze it in. Place row 1 first and row 2 last, with rows 3 and 4 in the middle.

Measure token counts with the tokenizer for the model you are actually calling. Counting words and multiplying is inaccurate for code, identifiers and non-English text, and the error compounds across a dozen rows. Where you send the same documents to several providers, keeping one tokenizer per call path is the difference between a budget and an estimate.

A rough shape for the fill loop, in pseudocode:

cap = retrieved_token_cap  # e.g. 12000, from the subtraction above
kept, used = [], 0
for row in sorted(rows, key=lambda r: r.rerank_score, reverse=True):
    n = count_tokens(row.text)
    if used + n <= cap and not is_near_duplicate(row, kept):
        kept.append(row)
        used += n
# order: best first, next best last, the rest in between
ordered = [kept[0]] + kept[2:] + [kept[-1]] + kept[1:-1] if len(kept) > 2 else kept

The duplicate check matters more than it looks. Near-identical chunks from the same page pass a similarity threshold in bulk, and each one spends tokens that a distinct row could have used.

A budget you have not evaluated is a guess with a config file. The test is a fixed set of questions with a written answer key, run twice: once with the cap, once with whatever you sent before the cap existed.

The measurement discipline is the same one used for retrieval latency, where the guidance is to compare changes on a fixed question set with an explicit quality criterion and to watch for evidence falling outside a smaller candidate set per a latency budgeting guide. That guide also lists a smaller candidate set as one of its experiments, with the failure to watch being supporting evidence that falls outside the set. Smaller context is an experiment with a hypothesis and a failure mode, not a cost saving.

Here is a checklist to run each variant against:

The evidence-presence line is the one most teams skip, and it is the most diagnostic. If the supporting row was dropped by the cap, no amount of prompt tuning recovers it. If it was present and the answer still missed it, the problem is placement or instruction, and a larger cap will not fix it. That distinction is the argument for keeping a trace of which rows were sent, not just which were retrieved.

Presentation in the prompt is itself a lever. One practical guide to RAG and prompt engineering frames context assembly as one of the layers separating retrieved evidence from the model's response, distinct from the system rules and from the way the question is phrased in that guide. Treat that structure as the point: separate system rules, context block and question, so you can change one without disturbing the others, and so a bad answer can be traced to the layer that caused it.

When a budget variant wins, promote it to the default and rerun the set when the corpus, the chunker or the retriever changes. A cap tuned against one corpus does not transfer automatically to another with different chunk lengths. The same warning applies to any comparison of retrieval engines: keep benchmark comparisons matched on dataset, filters, recall target and cache state, because an isolated vendor number does not establish what is fastest for your application per the latency guide.

Cutting is the default, but three situations justify a larger retrieved budget, each with a condition attached.

The answer genuinely spans multiple documents. Comparison questions need evidence from both sides. If your evaluation shows failures concentrated on multi-document questions with the supporting rows present but truncated, raising the cap is cheaper than building a second retrieval hop.

The query is a lookup for a rare identifier. Exact codes and part numbers are where semantic similarity is weakest and keyword search is strongest per the retrieval survey. Here the right move is usually a narrower query with a keyword path, not a bigger cap, because a bigger cap just adds more near-misses around the one exact row you need.

The session is long and history is eating the window. If history consumes the budget, the fix is trimming or summarizing history, not shrinking retrieval. Long-term memory holds stable facts outside the context window and pulls back only what bears on the current task, which is a different job from keeping the last twenty turns verbatim as a practical guide puts it. That guide draws the distinction sharply: memory is storage, the context window is working space, and a larger window does not create persistence by itself because the application still decides what to assemble per request. Move durable preferences and conventions into a retrieved memory entry with its own small budget, and let conversation history shrink.

Budgeting is also available at finer grain than one cap for the whole request. A runtime memory framework routes queries across memory modules at low, mid or high budget tiers, and reports better accuracy-to-cost trade-offs under tighter budgets across three benchmarks according to that paper. You do not need the routing model to borrow the idea: classify the question, then apply a small cap for lookups and a larger one for synthesis.

If you would rather not maintain separate tokenizers, caps and routing per provider, Quanta Inference puts text, image, video, audio and embedding models behind one OpenAI-compatible API key, which keeps the accounting in one place: https://quantaintelligence.ai/inference.