Where a Check's Prompt Grows: Context Drift Across Runs

The first job is to make the prompt construction explicit. Instead of concatenating strings inline, build the prompt as a named dictionary of segments. Five segments cover most agent checks:

The names matter less than the discipline of keeping them separate all the way to the logging call. If your prompt is one big f-string, you cannot count the parts. A small builder function fixes that:

def build_prompt(task_text, docs, history, scratchpad):
    segments = {
        "system": SYSTEM_TEXT,
        "tool_schemas": json.dumps(TOOL_SCHEMAS),
        "retrieved_docs": "\n\n".join(docs),
        "history": "\n\n".join(history),
        "scratchpad": "\n\n".join(scratchpad),
        "task": task_text,
    }
    return segments

Now counting is a loop. For plain text, a local tokenizer such as tiktoken gives you a fast count, but OpenAI's own documentation warns that a plain-text count does not include everything in a real request: message structure, roles, tool schemas, images and files all add tokens. For accuracy on the full request, use a token counting endpoint that accepts the same payload you plan to send, as described in OpenAI's token counting guide. The guide notes that the count includes formatting tokens for message roles and boundaries that local tokenizers miss.

A practical pattern is to count each segment locally for the drift chart (cheap, runs on every invocation) and count the assembled request with the provider endpoint on a sample (accurate, catches structure overhead). The drift chart only needs a consistent measure per segment; the absolute total needs the accurate one.

The per-segment count is a rough proxy, and that is fine as long as you know the limits. The same text can produce different token counts on different models because vocabularies differ; Firecrawl's analysis of tokenization across models reports compression ratios varying by 20 to 50 percent on ordinary content between tokenizers, and more on emoji and numeric content. Keep the counting method for a given check fixed, and record which tokenizer produced the number.

Log one row per run with the segment counts, the run ID, and the timestamp. That schema is the whole foundation.

Every model call returns usage fields, and you should capture them alongside the segment counts. Braintrust's LLM call observability guide lists what to capture on every call: the full request payload including system message and tool definitions, the response payload, metadata such as model version and prompt version, latency signals, and cost data. Segment counts are a slice of that request payload, measured rather than stored raw.

OpenTelemetry's GenAI semantic conventions give you portable attribute names, including gen_ai.usage.input_tokens and gen_ai.usage.output_tokens, which the Braintrust token tracking guide describes. If your platform already collects OTel spans, add custom attributes for the segment counts (prompt.segment.system.tokens and so on) and they will flow into the same backend as everything else.

Groundcover's guide to AI agent observability frames the split well: metrics summarize behavior across many runs, while traces give you one run step by step. Your drift chart is a metric built from per-run fields, so it is cheap to store and fast to query. When the chart shows a spike, you open the trace for that run and look at the actual text in the segment that moved.

One gotcha: streaming responses can drop usage data unless you ask for it. The Stack Overflow thread on streaming token usage documents that usage was missing from streamed responses until providers added an option to include it, and that older SDKs silently omit the field. If your check streams output, verify the usage object is populated before you trust an empty result as zero tokens.

The chart is simple to specify: x axis is run index (or time), y axis is token count, one series per segment. Plot a rolling median rather than raw values so single outliers do not dominate, and draw two reference lines: a p50 line and a p90 line computed over a trailing window.

Most charting libraries will do this from a CSV in a few lines, but the analytical work is reading the shapes. There are four patterns worth naming:

The flat band is the goal for most segments. The task segment should be a flat band almost by definition: if your task text is growing, something upstream changed. The system segment is a flat band until someone edits it, at which point it stair-steps once and settles. Tool schemas behave the same way.

The slow ramp is the one that costs you money and breaks in production. It is also the one that is easiest to miss without a chart, because no single run looks wrong. A history segment that grows by a few hundred tokens per run is invisible in a spot check and dominant after a thousand runs.

The sawtooth is worth calling out because it often hides a bug. If a truncation rule fires only when the prompt exceeds a hard limit, the segment will climb to the limit, get cut, and climb again. That pattern means the cap exists but the source is still unbounded between cap events. Fix the source, not the cap.

Once the chart tells you which segment ramps, the fix is a number. Compute the segment's 90th percentile over a long baseline window (at least a few hundred runs, and ideally a window that covers the normal range of tasks your check handles). Then set the cap a little above it, perhaps 10 to 25 percent above, depending on how much headroom your context budget allows.

Why the 90th percentile rather than the mean or the max? The mean is dragged down by quiet runs and hides the tail that actually fills the window. The max is set by one outlier, often a pathological run you would rather not design around. The 90th percentile says: nine out of ten normal runs fit without any summarization at all, and the rare large run gets handled explicitly instead of silently dominating the prompt. If your segment is doing its job at p90 and your context window has room, capping there costs you almost nothing in quality.

Worked example. Suppose the retrieved_docs segment shows a p50 of 6,000 tokens and a p90 of 12,500 tokens across 800 runs. The context window is comfortable, your system and tool segments together use about 4,000 tokens, and you leave 2,000 tokens for output. A cap at 14,000 (roughly 12 percent above p90) means about one run in ten gets its documents summarized rather than pasted whole. Those are the big tasks, and they are exactly the ones where a summary is a reasonable trade.

Do the same math for every ramping segment, and be pessimistic about the total. If history, docs and scratchpad each ramp, their caps add up. The Glean guide to token efficiency metrics points out that output tokens carry far more price weight than input, with a median output-to-input price ratio around 4x, so squeezing input is worthwhile but not the whole budget. The cap is about predictability and headroom as much as it is about price.

If your check runs across several models, remember that a cap measured in one model's tokens does not transfer. Firecrawl's tokenization comparison shows the same text tokenizing differently across providers, so recompute the percentile when you change models. Glean's metrics guide recommends an Effective Tokens formula that weights input, cache reads and output separately, which is a useful way to compare runs when model pricing differs.

The cap itself is a branch, not a hard truncation. Truncating a document at an arbitrary token boundary loses the end, which is often the conclusion. Summarizing keeps the point and drops the bulk.

def fit_segment(text, cap_tokens, count, summarize):
    n = count(text)
    if n <= cap_tokens:
        return text, None
    summary = summarize(text, target_tokens=cap_tokens // 3)
    pointer = store_original(text)  # returns a stable reference
    return f"{summary}\n\n[full text: {pointer}]", pointer

Three details make this work in practice. First, the summary budget should be a fraction of the cap (a third is a reasonable default) so the summarized version leaves real headroom. Second, store_original must return something the model can act on: a document ID, a URL, or a retrieval handle it can pass to a tool. A pointer the model cannot follow is just decoration. Third, log that summarization happened, with the original token count and the summary token count. That log line is how you learn whether your cap is set sensibly; if 40 percent of runs are summarizing, your p90 is stale and the corpus has grown.

The pointer also protects quality on the rare run that needs everything. If the check is verifying a long contract against a policy, the summarized document may not have the clause it needs, and a tool call that fetches the original by ID lets that run succeed without making every run pay for the full text.

This is the same reasoning the OpenAI token counting documentation gives for staying within model limits: shorten or rephrase the prompt, remove repeated context, divide large inputs, or summarize and preprocess before sending. The cap plus summary is the automated version of that advice.

For history segments specifically, the cheapest fix is often not a cap but a policy: keep the last N turns verbatim and summarize everything older into one paragraph. That keeps the segment a flat band by construction, which is better than a ramp that trips a cap on every third run.

A drift chart is only useful if someone acts on it. Set two alert conditions: a ramp alert when a segment's rolling median has risen more than a set percentage above its trailing p50 over a window (a 25 percent rise over 100 runs is a reasonable starting point), and a cap alert when the share of runs that triggered summarization exceeds a threshold (20 percent is a sensible ceiling). The first catches a new unbounded source before it dominates the prompt. The second catches a cap that has gone stale.

Tie the chart to quality, not just cost. Algolia's guide to agent evaluation notes that agent quality has to be measured across full execution trajectories, because intermediate steps fail independently of the final output, and that 60 percent of AI production failures trace to data quality, context or governance rather than the model. A prompt whose history segment grew without a matching review is a context problem in that sense. If your check scores its own outputs, plot the score on the same time axis as the drift chart; a ramp that starts with a score dip is a real regression rather than a benign corpus change.

Keep the segment list short and stable. Adding a segment is easy; the discipline of keeping each one bounded is what pays. When a new source of text enters the prompt, give it its own segment on day one, so the chart shows its behavior from the first run rather than after the first incident.

If your checks already run through one gateway, Quanta Inference returns consistent usage data across text, image and embedding models behind a single OpenAI-compatible API, which keeps the segment counts comparable when a check switches models. For teams building these checks with coding agents, Quanta Skills packages the instrumentation and cap logic so a new check starts with the segments defined.