Estimating Token Spend Before the First Model Call

Start with the rendered template, not the template source. A template with placeholders contains fewer tokens than the string that reaches the model, because every placeholder expands to real content. The honest method is to pick a representative set of inputs, render the template, and count tokens on the final string.

Token counts are not word counts. In practice a token averages roughly 0.75 words in English, so a 1,000-word block is about 1,333 tokens, and the ratio shifts by content type: Python code is more token-dense than equivalent prose, Chinese and Arabic approach one token per character, and JSON carries bracket and key overhead (the Worklytics guide). Microsoft's documentation makes the same point in licensing terms: common words are often a single token, rarer words split into several (Microsoft's prompt token guidance).

Count with the tokenizer the model will use when you can. LangChain's splitter documentation is blunt about it: the token count you compute should use the same tokenizer as the language model, which is why its token-aware splitters accept a model name or an encoding such as cl100k_base (LangChain's splitting by token guide). Open-source tokenizers such as tiktoken are fast byte-pair encoders, and browser counters built on them can give you a quick read on a rendered prompt (the Prompt Token Counter) or compare several model families at once (the GPT for Work tokenizer).

Do not forget the parts of the prompt you did not write. Microsoft's documentation notes that its prompt builder prepends a hidden system metaprompt of roughly 1,200 tokens, which varies (Microsoft's prompt token guidance). Hosted providers do something similar with tool definitions and safety framing. Your own harness also adds chat markup around each message, plus JSON escaping when you serialize structured input. Measure a known prompt end to end once and subtract the visible text to find your per-message overhead, then treat that number as a constant in the estimator.

A practical template-sizing procedure:

1. Render the template with three representative inputs: a typical case, a large case, and your configured maximum.
2. Tokenize each rendered string with the model's tokenizer.
3. Add your measured per-message framing overhead times the number of messages.
4. Add a fixed allowance for hidden system content, using the provider's documented figure if it publishes one.
5. Record the three numbers as template plus small, template plus typical, and template plus max, and store them beside the template version.

That fifth step matters more than it looks. Braintrust recommends pinning a prompt version into request metadata so that a template change is visible in the trace, because adding context to every prompt is a cost change that ships silently (Braintrust's cost tracking playbook). If your estimate is keyed to a version, an edit that adds 400 tokens per call shows up as a diff in the estimate before it shows up in the bill.

A worked example: a 120-word instruction with a 60-word user question and a 200-word pasted email. That is 380 words, roughly 507 tokens at the 0.75 words per token ratio (the Worklytics guide). Add 1,200 tokens of hidden system content (Microsoft's prompt token guidance) and 12 tokens of chat markup for three messages, and the estimate is about 1,719 input tokens before any retrieval. The visible prompt was 507 tokens. The billable input is more than three times that, which is the single most common surprise in first-pass estimates.

Retrieved context is where estimates drift, because the number of chunks is usually a top-k setting and the size of a chunk is usually a character count that nobody converted. Convert once, deliberately.

Chunk size is the maximum number of tokens in each chunk, and it must be smaller than the model's token limit so the data fits in the context window; a 50,000-token source split at 1,000 tokens per chunk yields roughly 50 chunks, of which only the most relevant are sent (MongoDB's chunking strategies lesson). Note the phrase "roughly": overlap, separators, and sentence boundaries all move the real count, so measure the distribution over your corpus rather than trusting the configured number.

The arithmetic for a retrieval step is small:

context_tokens = k * mean_chunk_tokens
               + k * per_chunk_framing_tokens
               + query_rewrite_tokens (if any)

Here k is the number of chunks you actually pass to the model, not the number you retrieve. per_chunk_framing_tokens covers the labels and separators your harness inserts between chunks: source URL, heading, and delimiters. In practice this runs to a few dozen tokens per chunk, which on a five-chunk context is a rounding error and on a fifty-chunk context is not.

Take a support assistant with k of 6 and chunks averaging 320 tokens. Context is 1,920 tokens. Add 20 tokens of framing per chunk for 120 more, and the retrieval component is 2,040 tokens. Against the 1,719-token prompt estimate from the previous section, retrieval is more than half the input. That ratio is the reason chunk counts belong in the admission decision rather than in a footnote.

The three components of any chunking strategy are chunk size, chunk overlap, and splitting technique, and all three are adjustable (MongoDB's chunking strategies lesson). Overlap is the quiet multiplier: a 20 percent overlap on 320-token chunks effectively raises the retrieve-then-send cost per unit of source text. Model it as an input, not a constant.

Finally, sanitize before you count. If your pipeline sends raw HTML, markup overhead inflates every input token, and extracting clean text first reduces it (the Worklytics guide). Tens of kilobytes of tags can be a larger line item than the content they wrap.

A single call is a closed-form estimate. A conversation is not, because each turn resends the prior transcript. If you estimate one turn and multiply by the number of turns, you will undercount by a factor that grows with conversation length.

Model the transcript explicitly. Define prompt tokens for turn 1 as P (the template plus framing), define R as retrieval tokens per turn, and define O as expected output tokens per turn. Then:

The last row is the decision-relevant one: cumulative input grows quadratically in turn count, because each new turn pays for every prior output. Worked with numbers: P = 1,719, R = 2,040, O = 400, and 6 turns. Turn 6 sends 1,719 + 2,040 + 5 * 2,440 = 15,959 input tokens. The six-turn conversation consumes 1,719 * 6 + 2,040 * 6 + 15 * 2,440 = 10,314 + 12,240 + 36,600 = 59,154 cumulative input tokens, against a naive estimate of 6 * 3,759 = 22,554. The naive estimate is off by a factor of 2.6 at six turns, and the gap widens with every additional turn.

Two adjustments keep this honest. First, cap the transcript: when it exceeds a threshold, summarize older turns into a fixed-size block, which converts the quadratic term into a linear one and makes the estimate tractable. Second, remember that output tokens in reasoning-capable models include the chain-of-thought that never appears in the answer but is fully billed (the Worklytics guide). If your provider reports reasoning tokens separately, take the median from a sample of real traces and use that as O rather than the length of the visible answer.

Agents make this harder still. A tool-calling loop is a multi-turn conversation whose turn count you do not control in advance, so the estimate for an agent run should use the configured iteration cap as n, not the expected number of iterations. That is the same guard limit that catches runaway loops in production (Braintrust's cost tracking playbook), and using it in the estimate means the worst case is priced before the run starts.

An estimate is only useful if something compares it to a limit and acts. The cap can come from several places: a per-request ceiling you define yourself, a provider spend limit, or a gateway budget. LiteLLM, for example, supports global, team, team-member, and agent budgets with a resettable duration, enforced against spend read from a database (the LiteLLM budget documentation). OpenAI also exposes spend limits at the organization level (OpenAI's production best practices). The gap is timing: budgets and spend limits usually fire on spend already incurred, so they stop the second runaway call, not the first.

At admission time you need a rule with headroom. A workable form:

• Let cap be the token budget for this run, converted to tokens from the budget in dollars using the input and output rates for the selected model.
• Let estimate be cumulative input tokens plus expected output tokens, computed as above.
• Let margin be a safety factor, 1.25 for chat with a fixed transcript cap and 2.0 for agent runs.
• Admit if estimate * margin is at most cap; otherwise refuse, or downgrade to a cheaper route or a smaller k before re-estimating.

The estimate is not a bill. It is a gate, and a gate fails in two directions. Too tight and you refuse legitimate work; too loose and the cap never binds. Both are measurable: log the estimate next to the actual usage returned by the provider for the same run, then tune the margin until refusals are rare and under-estimates are rarer. Actual usage per request is available from provider usage reporting, and a gateway logs per-request cost alongside metadata (Bifrost's cost tracking write-up), so the comparison costs little to set up.

The choice of what to do on refusal is a policy decision, not a technical one. Soft alerts suit experimentation where visibility is the goal; constrained mode, continuing with cheaper models or smaller context, suits workloads that must keep running with degraded quality; hard caps suit non-essential workloads or pilots where unconstrained spend is itself a risk (guidance on soft alerts versus hard caps). The failure that motivated one vendor's write-up is instructive: a development worker loop on an expensive default produced $7,152.89 in charges over eight days against an $11 monthly baseline, and each control layer held for the patterns it was designed for while the spike crossed all of them at once (the hard caps versus alerts analysis). An estimate check at admission is the layer that would have caught that loop on turn one.

Where the run cannot be refused, batch processing is the pressure valve. Both major providers offer 50 percent discounts for asynchronous batch processing with results inside 24 hours, and batching suits classification, document processing, evaluation, labeling, and embedding generation, but not live chat (NeuralTrust on batching). Batch saves money, not tokens, so it changes the cap in dollars rather than the estimate in tokens. Keep the two separate in your admission logic or you will double-count savings.

Before you wire the estimate into admission, confirm each of these against your own stack. None of them requires a new system.

Teams that run several providers and model families often find the estimator becoming the hard part rather than the cap. If one API for text, image, video, audio, and embedding models fits the way you work, Quanta Inference is OpenAI-compatible and prepaid, which keeps the per-model rate table in one place: https://quantaintelligence.ai/inference. Whether you build the estimator yourself or buy it, the rule from this article holds: count the rendered prompt, count the chunks, project the transcript, multiply by a calibrated margin, and compare to the cap before the first call.