Splitting a Prompt Into a Cacheable Prefix and a Volatile Tail

Teams usually reach for retrieval tuning first: rerankers, chunk sizes, top-k. That work matters, but it operates on the region that is most likely to be volatile anyway. The layout decision determines whether any of it gets cached.

Consider what a RAG prompt actually contains. A typical request opens with the same system prompt and tool definitions, then a large block of retrieved documentation or source code, before it reaches the user's question (the FinOps breakdown). Without caching, the model processes that static content from scratch on every call and you are billed for the same tokens again and again.

Now place retrieval in the wrong slot. If retrieved chunks come before the user question and the chunk set changes per query, every request is a fresh prefix from the first differing chunk onward. The system prompt still caches, but everything behind the first variance point does not. If instead you feed a stable document set into the prefix and put the query and its query-specific snippets at the end, the whole prefix survives across requests that share it.

Anthropic's context engineering guidance frames the wider problem: context is a finite resource with diminishing marginal returns, and as token count grows the model's ability to recall from it degrades (effective context engineering). That argues against dumping a large context into every request just because it caches. Cache what is genuinely stable and genuinely useful, and keep the volatile parts small. The layout decision and the budget decision are the same conversation.

A useful working example: an internal support assistant with a 20K-token system prompt and roughly 26 steps per task. In an agentic launch case with those shapes, caching took the hit rate from 7% to 84% and cut overall LLM cost by 59%, later reaching 70% on the most optimized window (the ProjectDiscovery case study). Those numbers came from that team's workloads, not yours, but the direction is the point: the prefix did not get smaller, it got stable.

Here is the order to build toward. Every region is a separate field in your request builder, so you can measure and cap it independently.

The budgets are starting points, not law. Anthropic's caching requires a minimum prefix length before a breakpoint is worth writing, and its documentation notes that Claude caching only triggers once the cached block is over roughly 1,000 tokens (the provider issue thread). OpenAI's caching is described as automatic for prompts of 1,024 tokens or longer (the caching overview). Under those thresholds, a 600-token system prompt alone will not cache; the system instructions and tools have to be counted together as one prefix.

In code, the shape looks like this. The point is that order is fixed by construction, not by string concatenation scattered through the codebase.

def build_messages(system_block, tools_block, shared_context, snippets, question):
    # Regions 1 to 3 form the cacheable prefix.
    # Build them from versioned, byte-stable snippets, never from f-strings
    # containing timestamps, request IDs or user names.
    prefix = [
        {"role": "system", "content": system_block},
        {"role": "system", "content": render_tools(tools_block)},
        {"role": "system", "content": shared_context},
    ]
    # Region 4 and 5 are the volatile tail.
    tail = [
        {"role": "user", "content": render_snippets(snippets)},
        {"role": "user", "content": question},
    ]
    return prefix + tail

Two rules make this hold. First, tool definitions are serialized deterministically: sort tools by name, never by iteration order of a dictionary that changes between deploys, and keep static tools before any per-user dynamic tools (the ProjectDiscovery case study). Second, nothing in the prefix may interpolate a runtime value. No timestamps, no user IDs, no session names, no "current page" hints.

If you are routing across several model providers, the gateway matters too. Some gateways pin subsequent requests to the same provider endpoint after a cached request so the cache stays warm, and that stickiness expires after a period of inactivity (the prompt caching docs). A load balancer that round-robins across providers will quietly destroy your hit rate.

A cacheable prefix is not a prefix that means the same thing. It is a prefix whose bytes are the same. The classic failure is a template change that reorders a section or injects a timestamp near the top; the metabase metric guide calls this out directly, noting that reordering the system prompt or injecting a timestamp near the top invalidates every entry and shows up later as a cost spike (the hit rate metric).

Practical defenses, in order of how much they save you:

1. Build the prefix from versioned constants, not runtime strings. Treat the system and tools blocks as artifacts with a version stamp you can diff.
2. Add a test that renders the prefix twice with different request inputs and asserts the output is byte-identical. This catches interpolated timestamps before they ship.
3. Measure cached-token share per application and alert on drops. Cache savings are computed from your recorded costs, not from hits times list price, and cache writes can cost extra (the hit rate metric).
4. Audit tool ordering after every deploy. A new tool inserted alphabetically in the middle of the list changes everything after it.
5. Watch the clock. Anthropic's cache defaults to a five-minute lifetime, refreshed at no cost each time the cached content is used, and the lifetime is measured from the start of the request that writes or reads the entry, so a four-minute streamed response consumes most of the window (the Claude caching docs). For cross-user reuse, a longer write is available at a higher price per token.

That last point drives a real design choice. A short default lifetime keeps the cache warm for a burst of traffic from one user but lets it go cold for a workload that sees a given system prompt once every ten minutes. A one-hour write costs more up front and pays off when many concurrent tasks share the same agent type (the ProjectDiscovery case study). Pick based on request interarrival time, not on which option sounds cheaper.

This is where the layout decision bites. Stable prefixes want a document set that does not change. Fresh retrieval wants the opposite.

The good news is that retrieval and prefix caching can coexist if you sequence them correctly. Research on caching retrieved chunks for RAG shows that in production systems prefix caching still leaves redundant computation on the table, because chunks appear at arbitrary positions in the prompt and arbitrary contexts; a chunk-aware cache reduced redundant computation by 51% over state-of-the-art prefix caching and 75% over full recomputation on the workloads studied (the Cache-Craft paper). For most teams, the simpler move comes first: pin the stable shared corpus in the prefix and keep query-specific chunks in the tail.

A second body of work attacks the freshness problem directly. RAGCache organizes cached intermediate states of retrieved knowledge in a memory hierarchy and overlaps retrieval with generation, cutting time to first token by up to 4x over a baseline serving stack in the reported evaluation (the RAGCache paper). That is a serving-layer fix, and it helps whether or not your application-level prefix is stable.

The freshness question gets sharper at the answer level. A paper on cache routing argues that output-level semantic caching is fragile: similar prompts can map to different correct answers, retrieved evidence drifts as the corpus updates, and adversarial collisions have been demonstrated. On the mtRAG document-drift regime, a naive semantic cache served a wrong answer on 51.5% of queries, while an evidence-validated router brought that to 1.5% (the grounded cache routing paper). Read that as a boundary marker: prefix caching is safe by construction when prefixes match, but caching an answer is a different risk class.

So here is the decision rule. Ask how often the content in region 3 changes relative to your request rate.

• If the shared corpus changes hourly or less, put it in the prefix. The cached-token share will be high and the freshness cost small.
• If it changes per request, it belongs in the tail. Do not pretend it is context.
• If you need freshness inside a stable prefix, split region 3: a slowly changing shared block stays in the prefix, and a small per-request delta goes in the tail with an explicit label saying it supersedes the shared block where they conflict.

That last pattern is the one most teams miss. It gives you the cache hit and the fresh fact in the same prompt, at the cost of a few dozen tokens and one clearly worded instruction to the model about precedence.

Take a retrieval assistant that sends a 4,500-token prompt per request: 800 tokens of system, 400 of tools, 2,600 of shared reference context, a 400-token retrieved snippet and a 300-token conversation tail. It serves 200,000 requests a day.

With no caching, all 4,500 tokens are billed at the input rate on every call, which is 900 million input tokens a day. With the layout above, the 3,800-token prefix is stable and the volatile tail averages 700 tokens. If the provider discounts cache hits to a tenth of the input rate, as the caching comparison describes for prompt-cache reads against a write premium (the caching comparison), the daily cost falls by a large multiple, and the exact multiple depends on your provider's rates and your realized hit rate.

The number that matters is not that multiple. It is the hit rate you actually achieve, because a layout that is stable in theory but churns in practice produces the worst of both worlds: write premiums with few reads. Whatever the per-token math says, verify it against reported usage fields rather than estimating from list prices (the hit rate metric).

Size the tail last. If the user question is 300 tokens and retrieved snippets are 400, you have 700 volatile tokens against a 3,800-token prefix, and nearly everything is cacheable. If the snippets grow to 3,000 tokens because top-k went up, the volatility grows and the cache stops being the dominant lever. The budget table above is a forcing function for that conversation.

When you are comparing providers for this workload, the practical test is whether your prefix stays warm across a day of traffic and what a cache read costs. A single API that fronts text, image, video, audio and embedding models, with prepaid USD billing, makes it feasible to measure the same prompt layout against several models without rewriting your client. Quanta Inference (https://quantaintelligence.ai/inference) is built for that comparison.

A layout you cannot measure is a layout you will lose. Three checks, run continuously, catch nearly every regression.

Prefix identity test. Render the prefix from two different synthetic requests and assert byte equality. Run it in CI. This is the cheapest possible guard against an interpolated timestamp.

Cached-token share. For provider prompt caching, divide cached prompt tokens by total prompt tokens and track it per application, separately from any response-cache hit rate (the hit rate metric). Alert on per-app drops rather than the fleet average, which moves too slowly to catch a single bad deploy.

Tail size. Track the volatile region's token count per request. If it grows steadily, retrieval or conversation history is creeping into territory that should have been budgeted, and the prefix is paying for it.

Run the checks against the actual gateway you use, too. Some gateways route subsequent requests to the same provider after a cached one, keyed by a hashed opening message or an explicit session ID, and the stickiness expires after a period of inactivity (the prompt caching docs). If you pass an explicit session ID, make sure it is stable per conversation and not per request, or you will defeat the stickiness you just configured.

One more instrument worth having: a per-region token count printed at build time in development. When a developer adds three examples to the system prompt, they should see the system budget go from 720 to 1,180 and stop. Budgets that live only in a design document get ignored; budgets that fail a test do not.