A Token Budget That Leaves Room for the Model to Answer

The rule is simple to state: input cap equals context length minus output reservation minus safety margin. For each task, you set the output reservation once, based on what a good answer actually needs, and then the input cap falls out of that number.

Here is a worked example with two tasks on a 128,000-token context model. The numbers are illustrative but follow the arithmetic the OpenAI documentation describes for staying within model limits the OpenAI token guide.

The insight the first search result usually misses is that the output reservation should be set per task, not per model. A classification returns a label and maybe a sentence, so 512 tokens is generous. A summarization returns several paragraphs, so 1,500 is a floor, not a ceiling. Reserving 512 for a summarization does not save you anything on input; it just truncates good summaries. Reserving 8,000 for a classification wastes window that better retrieval could use.

For budgeting, remember that a token is roughly four characters or about three-quarters of a word in English, and that 100 tokens is about 75 words the OpenAI token guide. Use those estimates to plan, but never to enforce the cap in production. The same text can produce different token counts depending on the model, its encoding, and the language, so always count with the real tokenizer the OpenAI token guide.

Context assembly is the step where your app stitches together system instructions, retrieved documents, conversation history, tool schemas and the query into one sequence before the call the Redis guide to context assembly. That is exactly where the cap belongs, because it is the last point where you can decide what goes in.

Two practical rules make this manageable:

1. Put the fixed, precious content first: system instructions and the user query. They almost never get dropped.
2. Add retrievable content in relevance order and stop at the cap. If a chunk does not fit whole, stop rather than slicing it mid-sentence.

The reader can verify the general mechanics against the OpenAI guidance on counting a complete request, which notes that message structure, tools, schemas, images and files all affect the full input count the OpenAI token guide. That means a plain-text count of your strings is a lower bound; the real count includes formatting tokens. Budget for that in your margin.

Chunking choices feed directly into this budget. Content-aware chunking significantly improves retrieval over naive fixed-length splitting, and in one large cross-domain evaluation the top strategy (paragraph grouping) reached a mean nDCG@5 of roughly 0.459 while simple fixed-size character chunking fell below 0.244 the chunking study on arXiv. Smaller, coherent chunks that fit your input budget are usually better than a few oversized ones you have to cut. A general survey of chunking strategies makes the same trade-off explicit: recursive or sentence-aware splitting at roughly 400 to 512 tokens with modest overlap is a solid default for prose Firecrawl's chunking guide.

The ordering of your assembled prompt also affects cost and quality. Static content belongs at the top where it can be cached, and dynamic content at the bottom, because a stable prefix is the cacheable one the foojay analysis of context as a budget. This is a nice side effect of building the prompt in a fixed order: the same structure that makes truncation predictable also makes caching work.

The function below does three things: it computes the input cap from the context length, output reservation and margin; it walks through messages in a fixed order and measures the running token total; and it refuses to exceed the cap, preferring to truncate a long message or drop lower-priority context than to send an oversized request.

import tiktoken

# One tokenizer per model family. Swap the encoding name to match your model.
ENC = tiktoken.get_encoding("o200k_base")

def count_tokens(text: str) -> int:
    return len(ENC.encode(text))

# Per-task budget. Set these from measurement, not guesswork.
TASK_BUDGETS = {
    "classify_ticket": {"context": 128_000, "reserve_output": 512,  "margin": 1_488},
    "summarize_doc":   {"context": 128_000, "reserve_output": 1_500, "margin": 2_500},
}

def build_messages(task: str, system: str, query: str, context_chunks: list[str]):
    b = TASK_BUDGETS[task]
    input_cap = b["context"] - b["reserve_output"] - b["margin"]

    messages = [
        {"role": "system", "content": system},
        {"role": "user", "content": query},
    ]
    used = sum(count_tokens(m["content"]) + 4 for m in messages)  # +4 per message, approximate

    kept, dropped = [], 0
    for chunk in context_chunks:  # already sorted most relevant first
        n = count_tokens(chunk) + 4
        if used + n <= input_cap:
            kept.append(chunk)
            used += n
        else:
            dropped += 1

    if kept:
        messages.insert(1, {"role": "system", "content": "\n\n".join(kept)})

    return messages, {
        "input_cap": input_cap,
        "estimated_input_tokens": used,
        "chunks_kept": len(kept),
        "chunks_dropped": dropped,
        "max_output_tokens": b["reserve_output"],
    }, b["reserve_output"]

A few details that matter when you adapt it:

• The +4 per message is an estimate for role and boundary tokens. The exact count comes from the provider's own counting tool, and the OpenAI documentation points to tiktoken.encoding_for_model for plain text and a dedicated input-token counting API for complete requests the OpenAI token guide.
• Pass the returned max_output_tokens straight into the request. The OpenAI help page confirms parameter names vary by endpoint, with Chat Completions using max_completion_tokens and Responses using max_output_tokens the OpenAI token guide.
• If a single chunk is bigger than the whole cap, truncate it to the remaining budget at a sentence boundary rather than sending it whole. The cap is a hard stop, not an aspiration.

For teams running many task types against different providers, keeping the reservation, margin and cap in one config table pays off, because the same arithmetic applies to every model even when the tokenizer differs. Quanta Inference exposes text, image, video, audio and embedding models through one OpenAI-compatible API key priced in US dollars, which means one budget table can cover your whole stack: Quanta Inference.

The hardest number in the budget is the reservation itself. A method that works: run the task without a cap, log the output token counts, and set the reservation at roughly the 95th percentile rounded up.

Worked example: you run 2,000 summarizations and find output lengths cluster between 700 and 1,300 tokens, with a tail reaching 1,900. Set the reservation at 2,000. That leaves 126,000 tokens for input on a 128,000-token model, and the rare verbose summary still fits. If instead you had guessed 800, the model would be cut off on a meaningful share of real documents, and the user would blame the prompt, not the budget.

For classification, run the same exercise and you will typically see outputs under 100 tokens. A 512 reservation is already generous; the extra is your insurance against a reasoning-heavy model. Do not shrink it to 64 just because typical outputs are short, because reasoning tokens are billed and counted as output even when the visible answer is brief the OpenAI token guide.

The margin deserves similar care. It absorbs tokenizer drift, message-format overhead, tool schemas and any provider-side framing. Roughly 1,500 to 2,500 tokens on top of the reservation is comfortable for chat-style requests without tools. If you attach tool definitions, note that they can dominate the request; five connected servers can easily contribute 5,000 to 10,000 tokens before you have typed a word the foojay analysis of context as a budget. In that case, put the tool schemas inside your margin accounting, and check them against the cap like any other content.

A hard cap does not answer every question. Sometimes the content genuinely needs to be there: a long conversation, a multi-document comparison, or an agent that has been running for twenty turns. Before you simply drop the oldest material, consider the cheaper and more accurate options.

Summarize older turns into a compact state, then keep recent turns verbatim. This is a standard move in context management, where history is trimmed, deleted, or summarized when it outgrows the window the Redis guide to context assembly. Cache the stable prefix (system prompt, schemas, skills) so it is not re-billed at full price on every call, because a cache read costs a fraction of a fresh input token the Firecrawl guide to token optimization. And route easy tasks to smaller models; the gap between top and bottom tiers is enormous, so a classification does not need the same model as a long analysis the Firecrawl guide to token optimization.

If you are building a retrieval-heavy app rather than tuning an existing pipeline, a specialist can do the setup work with you; you describe the site, app, document or plan you need and a specialist picks it up: Quanta.

One more check worth building into your tests: measure retrieval quality separately from generation quality, so a truncation bug does not look like a bad model. Recall and precision at K are the standard tools for this; precision at K measures how many of the top K results are relevant, and recall at K measures what share of relevant items appear in the top K Evidently AI on precision and recall at K. Offline evaluation against a labeled query set is repeatable and cheap, which makes it the right tool for catching a cap that is too tight Unstructured on evaluating search quality.