Measuring Token Spend from Context Files and Skills

Log at exactly one place: the code that builds the model request. Everything upstream of that is intent, and everything downstream is a network call you do not control. The assembly point is the only place where you can see the final list of context files and skills that actually made it into the payload.

Most agent frameworks give you a middleware or lifecycle hook there. LangChain, for example, describes middleware as the mechanism for hooking into any step of the agent lifecycle to update context or jump between steps (LangChain context engineering docs). If your agent calls the provider SDK directly, wrap the client call. If you use an agent runtime with a tool loop, hook the turn where the prompt is finalized, not the tool execution.

The event you emit should be one record per model request, not one per file. Per-file records explode cardinality and make joining to invoices painful. A nested array of file entries inside one request event is the shape that survives contact with a log warehouse.

{
  "event": "llm.request.assembled",
  "ts": "2026-05-12T14:03:11.482Z",
  "task_id": "T-88213",
  "session_id": "S-4471",
  "turn_index": 6,
  "request_id": "req_01HZX9",
  "model": "provider-model-a",
  "input_tokens_total": 41250,
  "output_tokens": 780,
  "cache_read_tokens": 36000,
  "context_items": [
    {"path": "AGENTS.md", "kind": "always_on", "tokens": 2140},
    {"path": "docs/style-guide.md", "kind": "always_on", "tokens": 3380},
    {"path": "skills/pr-review/SKILL.md", "kind": "on_demand", "tokens": 1240},
    {"path": "skills/pdf/SKILL.md", "kind": "on_demand", "tokens": 0},
    {"path": "tools/mcp-registry-cache.json", "kind": "tool_schema", "tokens": 9120}
  ]
}

A few practical details matter more than they look.

Count with a tokenizer that matches the provider, and count the serialized text you actually send, including the delimiters and headers you wrap around each file. A file "about 2,000 tokens" by eyeballing characters will be wrong by a meaningful margin, and the error compounds across hundreds of requests.

Record zero-token entries for skills that were advertised but not loaded. This is what lets you prove progressive disclosure is working and distinguish "expensive always-on skill" from "advertised but never triggered skill." The Microsoft documentation is explicit that the advertisement step injects names and descriptions into the system prompt at the start of each run, while the body arrives only after a load_skill call (Microsoft Learn: Agent Skills).

Keep prompt text out of the event. Full context in your log store is a privacy and cardinality problem, and the OpenTelemetry guidance for generative AI recommends putting prompt and completion content in span events rather than indexed attributes (OpenTelemetry for Generative AI). A path, a token count and a load mode give you everything the cost analysis needs.

A token log is only as useful as the dimensions you attach to it. The table below is the minimum set worth capturing, with the question each dimension answers. If you capture only two, take load_mode and path, because that pair is what drives every cutting decision later.

The cached-token dimension is the one most teams forget, and it changes the arithmetic substantially. Manus reported that in its agent the average input-to-output token ratio is around 100:1, and that cached input tokens on one provider cost a tenth of uncached input tokens (Manus: context engineering lessons). If most of your input is served from cache, a file that is present in every request is much cheaper in practice than its raw token count suggests. Instrumenting without cache numbers will make you cut the wrong thing.

There is a counterweight. Cache hits require a stable prompt prefix, and the Manus write-up notes that even a single-token difference, such as a timestamp at the top of the system prompt, invalidates the cache from that point onward (Manus). So a file placed early in the prompt is not just costly per token, it is costly because it can destroy cache reuse for everything after it. If your log shows low cache_read_tokens relative to input_tokens_total on requests that share a common prefix, look for volatile content at the front before you look anywhere else.

Suppose a team of developers runs 2,000 agent tasks in a month, and the average task issues 8 model requests, so 16,000 request events. For illustration, assume the provider charges the uncached input rate we will call $3 per million tokens, the cached input rate at $0.30 per million, and the output rate at $15 per million. These numbers are a hypothetical used for the arithmetic, not a quote.

Take two files and compute their monthly contribution.

The pattern is the decision rule in miniature. A modest always-on file that rides along on every request costs roughly 68 times what a larger on-demand skill costs when that skill fires on 4 percent of requests. Meanwhile the advertised-but-never-triggered skill still costs money through its metadata, at 100 tokens per skill per request, which matches the documentation figure (Claude Agent Skills overview).

The same arithmetic explains why tool schemas deserve their own line in the log. The SwirlAI write-up on progressive disclosure notes that connecting a handful of MCP servers without managing what loads can reach 90 or more tool definitions and over 50,000 tokens of JSON schema before the model starts reasoning (SwirlAI: progressive disclosure). If your tool_schema entries are the largest single block in the log, no amount of skill tuning will fix the bill.

One caveat on the numbers: if most of that 54 million tokens is cache reads, the real cost of style-guide.md drops toward a tenth of the figure. That is why the join in the next section matters more than the raw token totals.

Your token log gives you token counts; the provider invoice gives you money. The bridge is a rate table keyed by model and, where relevant, by token type (uncached input, cached input, output). Three steps, in order:

First, pull the invoice line items for the billing period, broken down by model where the provider offers that detail. Second, compute an effective rate per million tokens for each model and token type by dividing billed amount by billed tokens over the same window. Third, join that rate table to your request events on model and on the event timestamp, not on the invoice period alone, because rates and models change within a month.

Two checks keep the join honest. Reconcile totals: sum your logged input and output tokens per model and compare to the invoice token counts. A gap usually means unlogged traffic, such as an evaluation harness, a background summarizer, or a second service sharing the key. Then spot-check a single expensive request: pull its request_id, find it in the provider side if the provider exposes it, and confirm the token counts agree within a small margin.

A useful habit is to split every budget by attribution class: always-on context, triggered skills, tool schemas, conversation history, retrieval results. If conversation history is 60 percent of input tokens, the fix is compaction and summarization, not file trimming. Splunk's write-up on production context engineering lists compaction and caching among the core patterns and stresses an observability loop that tracks both session-level goals and step-level tool usage (Splunk: context engineering in production). Your attribution classes are the cost-side counterpart of that loop.

If you run more than one model through the system, holding several provider keys and their price sheets steady gets annoying fast. That is the case for a single OpenAI-compatible endpoint that prices in dollars across text, image and other model types, so the rate table has one source: Quanta Inference.

Rank candidate cuts by a simple product: tokens per request multiplied by requests per month. Then apply three tiebreakers that the raw product misses.

The first row deserves emphasis because it is the most common mistake. The Claude Code academy course draws the line clearly: CLAUDE.md loads into every conversation and suits always-on project standards, while skills load on demand and suit task-specific expertise such as a pull request review checklist that does not need to be in context while you write new code (Claude Academy: skills vs other features). VS Code documentation makes the same split, contrasting always-applied custom instructions with task-specific skills that load on demand (VS Code: Agent Skills).

What not to cut: small always-on files that prevent expensive rework. A 300-token block telling the agent which test command to run and which package manager to use can save multiple failed attempts per task. Practitioners report exactly this pattern, that the valuable content in an agent instruction file is the list of commands for building, testing and running the project, not prose describing the codebase (Hacker News discussion of agent instruction files). Cutting a cheap file that eliminates a whole retry loop raises cost, not lowers it. Your log will show this as tasks with a single request and no error retries, which is the signature you want to preserve.

Once the harness exists, run it on every agent you operate, not just the coding one. The same event schema works for a support agent, a research agent or a review bot, and the dimensions are the same. What changes is the mix: support agents usually carry large retrieval payloads, coding agents carry tool schemas and repository files, research agents carry search results.

That mix is exactly why fleet-level views pay off. A single dashboard that ranks all context items across all agents by monthly dollars will surface a surprise roughly once a quarter, usually a forgotten evaluation harness or a document nobody remembers adding. If different teams own different agents and you want the same instrumentation and the same review habit applied across them, a team plan is the natural fit, and Quanta Skills (https://quantaintelligence.ai/skills) is where the packaged expertise behind those agents lives, so you can install the same skill into a coding agent and measure it the same way.