Trimming a Prefix Without Busting the Cache

Not all edits are equally destructive. Anthropic's cache follows a hierarchy: tools, then system, then messages. A change at one level invalidates that level and every level after it the cache invalidation walkthrough. That single rule lets you predict the blast radius of any edit without running anything.

The table's rows about tool edits come straight from the documented hierarchy: modifying tool definitions invalidates the entire cache, while changes to tool_choice affect only message blocks the invalidation walkthrough. That asymmetry is the first half of the decision rule. A trim to a tool definition is the most expensive edit available, because tools sit at the very front of the hash chain. A trim to a message is nearly free.

Anthropic also supports automatic caching, where one top-level cache_control field applies the breakpoint to the last cacheable block and moves it forward as the conversation grows the Claude platform docs. That mode is convenient but gives you less control over where the boundary sits, which matters when you are deliberately reshaping a prefix. For a trim exercise, explicit breakpoints are worth the extra field.

Here is the rule in one line: if the edit changes bytes before the breakpoint, price it as a reset before you ship it.

"Before the breakpoint" is broader than people expect. Prompt caching is a prefix match, not a similarity match. Systems that look completely correct can quietly never hit: a timestamp interpolated into a system prompt, a working directory, a shuffled tool list, an unstable JSON dump of configuration. One cost model put a timestamp in the system prompt and saw 7.8x higher cost on an 80-turn agent session, not because the timestamp was large but because it forfeited the discount on every turn Minh Le Duc on cache invalidation. Serialization order counts too: dumping the same config with different key order produces different bytes and a different hash.

So the first question is mechanical. Does the edited text sit before or after your breakpoint?

If it sits after, ship it whenever. Everything in the message blocks that follows the breakpoint (the current user question, retrieved chunks, per-request instructions, dates, working directory) is outside the cached span by construction. You can change it every single request and the cached prefix still hits. This is why the big structural win is relocation rather than deletion: move volatile working memory out of the system prompt and into a user message at the end, which is how one security platform took its cache hit rate from 7% to 84% and cut overall LLM cost 59%, with 9.8 billion tokens served from cache ProjectDiscovery's case study.

If it sits before the breakpoint, move on to the second question: can you batch it? A reset costs the same whether you change one character or the whole toolbar. That has a sharp practical consequence. Never ship prefix edits one at a time. Accumulate them (prompt rewordings, tool description trims, schema cleanups, tool reordering) and deploy in one change, then let the new prefix warm once. Splitting three edits across three deploys buys you three re-reads for no benefit.

The third question is arithmetic, covered in the next section.

Use a simple breakeven calculation. A reset costs you one full-price read of the cached prefix plus one cache write at the write premium. The saving is the doc

Worked example. Suppose your cached prefix is 20,000 tokens, the payload carries roughly 40 tool definitions, and you send 1,000 requests per hour against a 5-minute cache lifetime that stays warm continuously. You trim the system prompt by 2,000 tokens.

• Save per request after the trim: 2,000 tokens times the cache-read rate.
• Cost of the reset: 20,000 tokens at full input price, plus 20,000 tokens at the 5-minute write premium, paid once per warm prefix (and once per minute-to-minute gap if traffic is bursty).
• Break-even: at a read rate near a tenth of base input, the reset pays for itself within about one and a half requests Digital Applied's breakeven analysis.

Now scale the trim down. Suppose you trim 300 tokens instead of 2,000. The reset cost is unchanged (still a full read of the 20,000-token prefix), but the saving per request is 15 times smaller. On a low-traffic endpoint (say 50 requests an hour) the reset may take days to pay back, and it competes with every other prefix edit waiting in the queue. That is the decision rule in numerical form: estimate the tokens saved per request, multiply by expected requests before the next unavoidable reset, and compare against one full read plus write of the prefix. If savings do not clear the bar with margin, batch the edit with the next change.

Two caveats keep the arithmetic honest. First, the reset cost recurs when the cache expires from inactivity, not just once. A 5-minute lifetime measured from the start of the request that wrote it, with time spent generating counting against the lifetime, means bursty traffic pays the write more often than steady traffic the Claude platform docs. Trimming may be less urgent than simply keeping traffic steady. Second, track hit rate as a per-application metric rather than a fleet average. A prompt template change that breaks prefix stability can silently halve the hit rate and surface later as a cost spike Metabase's cache hit rate metric. Report prompt-cache reads (cached tokens over total prompt tokens) separately from any response cache you also run, since they measure different things.

The tool block is the most fragile part of the prefix and the most rewarding to get right. Because tools sit at the top of the hierarchy, a single reordered array or reworded description wipes out the entire cache the invalidation walkthrough. Ordering discipline fixes most of it.

ProjectDiscovery's production setup sorts tools so static tools come first in a deterministic order (alphabetical), with dynamic per-user subagents last, then marks the last static tool with a 1-hour lifetime so the shared chain survives across tasks and users ProjectDiscovery's case study. Sketching that logic in a number of lines:

# Stable tools first, deterministic order, dynamic tools last
tools.sort(key=lambda t: (t["name"] in dynamic_tool_names, t["name"]))

last_static = max(i for i, t in enumerate(tools) if t["name"] not in dynamic_tool_names)
tools[last_static]["cache_control"] = {"type": "ephemeral"}

Three practices follow from that pattern and generalize to any provider.

1. Deterministic serialization. Generate tool schemas from code with sorted keys. Any dictionary iteration order leaking into the payload produces a different byte sequence and a miss, even though the logical content is identical.
2. One breakpoint after the last tool you never edit. Keep experimental or per-tenant tools after it, and accept that they bill at full rate until promoted.
3. Descriptions written for the model, trimmed for the model. If a tool description carries stale examples, remove them, but batch that change with other prefix edits, because it costs the same as a full reorder.

The same first-stable-last-volatile discipline applies to documents. For retrieval-augmented generation, put the stable system and tool prefix first and the retrieved chunks last, since retrieved text changes per query by design. Serving papers on retrieval cache management confirm prefill tokens dominate input cost in RAG workloads, with per-query recomputation of retrieved chunks as the redundant portion Cache-Craft on chunk caches. You do not need a research system to benefit; you need stable-first ordering.

Run this before any prefix change ships.

• Identify the breakpoint. Which block carries cache_control (or sits at the automatic boundary)? Everything before it is the cached span.
• Classify the edit against the hierarchy. Tool change (worst), system change (bad), message or parameter change (free). Use the table above.
• Check for hidden volatility in the prefix. Timestamps, request IDs, user names, environment paths, non-deterministic JSON serialization, image additions before the breakpoint. Any one of these is a permanent miss regardless of what you trimmed.
• Estimate the per-request saving. Tokens removed times the cache-read rate on your model tier.
• Estimate the reset cost. One full read of the prefix plus one write premium per warm cycle, times expected cycles before the next deploy.
• Compare against break-even. If the saving does not recover the reset within the requests you expect between deploys, batch the edit with the next prefix change rather than shipping now.
• Deploy prefix edits in one batch. Warm the cache once, then verify the usage fields show read tokens rising to roughly the prefix size and write tokens falling to near zero.
• Watch for regressions. Alert on per-application hit-rate drops; a template change that breaks prefix stability shows up there before it shows up on the invoice Metabase.

One further discipline helps: treat the system prompt as a compiled artifact, not a string. If a sentence cannot earn its tokens back in output quality, cut it, but schedule the cut with the other prefix edits rather than as a separate deploy. The content-engineering view is that context is a finite resource and every token competes for the model's attention budget, so trimming is worthwhile on quality grounds too Anthropic's context engineering guidance. The cache economics merely tell you when to do it.

If you are routing the same trimmed prefix across several model families and want one key and one bill while you compare hit rates, Quanta Inference (https://quantaintelligence.ai/inference) puts text, image, video, audio and embedding models behind a single OpenAI-compatible endpoint with prepaid dollar pricing, which makes the per-request arithmetic above easy to check against real spend.