Enumerating What an Agent Sends to a Check

Attribution fails when the inventory is incomplete. Before you touch a tokenizer, write down every input that reaches the check's prompt. For a typical agent that reviews a change against acceptance criteria, the list looks like this:

The two rows that surprise people are tool schemas and serialization overhead. Tool definitions are sent on every call when tools are enabled, so a generous set of tools can dwarf the user message. Serialization overhead is invisible because it never appears in your strings: OpenAI's own documentation states that its input token counting endpoint includes formatting tokens used for request structure, such as message roles and boundaries, and that these tokens might not appear in the text you tokenize locally 4. An older community thread on counting chat tokens documents the same effect, where a single user message in chat mode consumed roughly 422 more tokens than the same text in completion mode 5.

A practical rule: any input the model reads but you did not type into a single variable belongs in the inventory. That includes anything the framework inserts.

Counting segments from your source variables gives you estimates. Counting the payload the provider actually receives gives you the truth. The way to get it is to log the request body at the boundary where it leaves your process, not to reconstruct it later from template variables.

import json, hashlib, time

def log_request(model: str, payload: dict) -> dict:
    body = json.dumps(payload, sort_keys=True, separators=(",", ":"))
    record = {
        "ts": time.time(),
        "model": model,
        "sha256": hashlib.sha256(body.encode()).hexdigest(),
        "bytes": len(body.encode()),
        "payload": payload,
    }
    # write to your trace store, one record per call
    return record

Two fields do the heavy lifting here. The payload is what you will segment later, and the hash lets you deduplicate: if the system instructions and the tool registry are unchanged between calls, their hash is unchanged too, and you can count them once and reuse the number. That matters for prompt caching, where keeping reusable prefixes stable is how cache hit rates improve and input costs fall, with savings that depend on the provider and workload 2.

Where a provider exposes an input token counting endpoint, use it on the assembled request rather than on the individual strings. OpenAI documents that the endpoint accepts the same input format as the request, including messages, files, tools, and conversations, and returns the exact count the model will receive 4. The general API guidance is blunter: a plain text token count does not necessarily include all tokens in an API request, because message structure, tools, schemas, images, and files all affect the full input count 6.

Now split the logged payload into labeled segments and count each one independently. The discipline that matters is using the same tokenizer and encoding for every segment and for the whole, so the parts reconcile with the total.

import tiktoken

enc = tiktoken.encoding_for_model("gpt-6-astra")  # use the encoding for your target model

def count(segment_text: str) -> int:
    return len(enc.encode(segment_text))

segments = {
    "system": system_instructions,
    "tools": json.dumps(tool_registry),
    "few_shot": few_shot_block,
    "history": json.dumps(conversation_history),
    "retrieved": "\n\n".join(doc["text"] for doc in retrieved_docs),
    "user": user_message,
}

counts = {name: count(text) for name, text in segments.items()}
serialized_total = request["usage"]["input_tokens"]  # from the provider response
overhead = serialized_total - sum(counts.values())
counts["serialization_overhead"] = overhead

The overhead line is the insight most teams never compute. It is the difference between the sum of your segments and what the provider billed, and it absorbs role tags, separators, and any structure the provider adds. If overhead is large and roughly constant per call, it is a fixed frame cost; if it scales with the number of messages or tools, it points at framing, not content.

One warning: do not mix tokenizers. Firecrawl's analysis of tokenization across providers reports that the same text can produce up to 2.65 times more tokens on one model than another, and that Gemini is cheapest on plain text but 46 percent dearer than OpenAI on tool definitions 7. Tool schemas are exactly the segment most sensitive to this, because they are dense JSON, and the same report notes that code and JSON tokenize far less tidily than prose 7. Count tool schemas with the target model's tokenizer every time.

For quick sanity checks, a general-purpose tokenizer tool can show how a sample string splits into tokens, though its counts apply to the model family it names, not to every provider 8. And keep prose conversions out of billing: a tokens-to-words calculator is useful for planning a prompt budget, not for estimating an invoice, since English prose runs roughly 1.3 to 1.5 tokens per word depending on the tokenizer 9.

Counting is only half the job. The other half is multiplying and presenting in a shape a team can act on. Take the input price per token for the model in use, multiply each segment, and add a share column. A worked example makes the decision rule concrete.

Assume a model priced at 2 US dollars per million input tokens, and a check that runs 100,000 times a month. The per-segment counts come from the logged payload, not from estimates.

Read the table as a set of decisions. Tool schemas at 30 percent are the largest single line, so trimming unused tools or tightening descriptions buys more than anything else. Retrieved documents at 25 percent suggest a retrieval cap or a reranker that keeps fewer chunks. Conversation history at 20 percent suggests truncation or summarization after a fixed number of turns. System instructions and user input together are 10 percent, so rewriting the prose there is wasted effort at this scale.

Glean's token efficiency guide argues for exactly this framing, tying spend to the task it served and the model that processed it rather than reporting raw consumption 10. The same source notes that output tokens cost several times more than input tokens across major providers, which is why this table covers input only: a complete cost model needs a second table for output, but input is where the attribution problem lives. A general LLM cost estimation walkthrough is a useful companion for building that second table, since it works from token usage through to total spend 11.

One operational note on cost allocation across teams: AWS's community guidance for enterprise token tracking is to treat granular cost allocation as an application-level problem when the managed service does not provide it, which in practice means your own logging and tagging, not the provider dashboard 12.

A one-off measurement decays. Segment counts drift as prompts are edited, tools are added, and retrieval settings change. Three habits keep the table current.

First, snapshot the segment counts whenever the prompt version changes, and store them with the version identifier. Braintrust recommends tracking prompt and completion tokens across prompt versions precisely because bloat arrives through small edits 1.

Second, alert on the ratio, not the total. A rule such as "retrieved documents exceed 40 percent of input tokens on any call" catches retrieval regressions before they reach the invoice, and it survives a model change that shifts every absolute count.

Third, watch the tail, not the mean. Span-level visibility is what reveals a retrying tool call or an oversized retrieval step sitting deep inside a trace and dominating usage even when the top-level numbers look normal 1. Per-call logs give you the distribution; the mean hides the 1 percent of calls that cost 20 percent of the budget.

There is a second reason to decompose rather than just count: it forces the question of whether each segment earns its tokens. A few-shot block that was added to fix one class of failure and never reviewed is a fixed cost you are paying on every call. The same logic that Berkeley researchers apply to autonomy applies to prompts: be explicit about which parts of the work are owned by the instructions and which are carried by retrieved context, because the failure modes differ 13. Write the reasoning down once, and the next person who edits the prompt knows what each block is for.

If you are running checks across several model families and want one place to compare per-segment costs without managing a key per provider, Quanta Inference exposes text, image, video, audio and embedding models through one OpenAI-compatible API with prepaid pricing in US dollars (https://quantaintelligence.ai/inference). The payload logging approach above works unchanged against it.