Designing Prompts for Cacheability: A Practical Guide

The single most useful rule is also the one every provider states: put static content at the start and variable content at the end (Humanloop's guide). Here is a template that encodes it, ordered from most to least stable:

[BLOCK 1: SYSTEM ROLE AND RULES]        changes never
[BLOCK 2: TOOL DEFINITIONS]             changes per release, sorted deterministically
[BLOCK 3: FEW-SHOT EXAMPLES]            changes per release
[BLOCK 4: FIXED REFERENCE DOCUMENT]     changes per tenant or per upload, not per turn
[BLOCK 5: OUTPUT FORMAT SCHEMA]         changes per release
[BLOCK 6: USER CONTEXT]                 changes per user (name, plan, locale)
[BLOCK 7: RETRIEVED CHUNKS]             changes per query
[BLOCK 8: CURRENT QUESTION]             changes every request

The ordering principle is not "static versus dynamic" in the abstract but "how often does this content change, measured across the whole population of requests that will share a cache entry." A tenant-specific policy document is static for one tenant and variable across tenants. Depending on your traffic, it may belong at block 4 (many requests per tenant) or near the end (few requests per tenant, many tenants).

For Anthropic models, the platform exposes this structure directly through cache breakpoints, either automatic (one top-level cache_control field that moves forward as conversations grow) or explicit (a cache_control marker on individual content blocks) (Claude platform docs). Explicit breakpoints let you mark the end of each stable block so that a change in, say, retrieved documents does not invalidate the system prompt cache entry.

OpenAI's documentation frames the same benefit as compute efficiency: reuse of a shared prompt prefix avoids recalculating tokens the model has already processed (OpenAI prompt caching guide). Amazon Bedrock supports both implicit and explicit caching depending on model and API, and calls out the document-question-answering pattern as a canonical case: cache the uploaded document so later questions about it skip reprocessing (AWS Bedrock docs).

Ordering static content first is necessary but not sufficient. Variable content needs rules of its own, because a well-placed static prefix can still be ruined by a single dynamic value that leaks upward.

The first rule is the one most teams get wrong. The author of a detailed write-up on prompt caching internals describes exactly this mistake: he appended long user-specific data to the end of the system prompt, reasoning that only the prefix within a single conversation needed to stay stable. That assumption was correct for within-session hits but missed the larger win, which is that cache hits can start at the very first token across different users of the same API key (how prompt caching works). The same post lays out a message array in which the system prompt plus tool definitions sit at position zero and the conversation follows; the moment user data is welded onto the system prompt, position zero stops being shareable.

A related trap is serialization. Prefix caching depends on the serialized request, which means whitespace, JSON key order and how your SDK renders tool schemas all affect matching. Modular's docs are explicit that a cache hit still depends on the serving engine retaining the entry, routing the request to a worker that can access it, and serializing previous messages identically (Modular).

Routing matters as much as prompt shape. OpenRouter documents "provider sticky routing," which sends subsequent requests to the same provider endpoint after a cached request so the cache stays warm, and it tracks stickiness per account, per model and per conversation, deriving a conversation key from the first system message and first non-system message by default (OpenRouter prompt caching docs). The distributed equivalent sits at the inference layer: llm-d reports that standard load balancers scatter related requests across pods, destroying cache locality, and that prefix-cache-aware scheduling restores it, with benchmarks showing up to 57x faster responses and double the throughput on identical hardware (llm-d).

Consider a support assistant that answers questions about uploaded contracts. The original prompt, assembled per request, looks like this:

System: You are a contract assistant. Today is {{date}}.
User {{user_name}} (plan: {{plan}}) asked at {{timestamp}}:
{{question}}
Here are contract excerpts: {{chunks}}
Rules: cite clause numbers. Never give legal advice.

Every field here is dynamic, and two rules sit after the variable content. No two requests share a usable prefix, so cached-token share will be near zero. Here is the refactored version:

System:
  You are a contract assistant. Cite clause numbers. Never give legal advice.
  [fixed tool definitions, sorted by name]
  [three fixed worked examples]
User:
  [fixed contract text for this document ID, if the document is reused]
User:
  Question: {{question}}
  Plan: {{plan}}
  Date: {{date}}
  Excerpts: {{chunks}}

Four changes did the work: static rules moved to the front, tool definitions were sorted and frozen, the date and user metadata moved below the question boundary, and the contract text, when reused, became its own stable block rather than being interleaved with the question.

How much does this matter? A reported production case from a security-tooling company is instructive. Its agent ran an average of 26 steps with 40 tool calls and sent a 20,000-plus token system prompt on every step. A caching refactor that used three breakpoints, a one-hour TTL on the static system prompt and a five-minute TTL on the sliding conversation window moved cache hit rate from 7% to 84%, and reported cost savings of 59% overall, with later windows reaching 66% and 70% (ProjectDiscovery). The lesson is not the specific percentage. It is that agentic workloads, which re-send the entire prefix on every step, are where prompt structure has the largest financial leverage.

A second data point sets expectations for latency. On a roughly 10,000-token prompt sent twice to a single model instance with prefix caching enabled, time to first token dropped from 4.3 seconds to 0.6 seconds (llm-d). If your refactor produces no TTFT change, the prefix is not being reused and something upstream is still churning.

You cannot tune what you do not log. Start by distinguishing two metrics that teams routinely conflate. Response caching skips the provider entirely and returns a stored answer; prompt caching discounts part of a single call. They have different economics and must be tracked separately (Metabase's metric guide). Confusing the two leads to overstating savings, because a response hit is free while a prompt hit is discounted, not free.

For prompt caching, the useful ratio is cached tokens divided by total prompt tokens, taken from the usage fields your provider returns. Metabase publishes a SQL pattern that computes hit rate and estimated savings per application over a trailing window, along with a warning worth internalizing: compute savings from recorded costs, not from hits multiplied by list price (Metabase).

Measurement advice, in order of effort:

1. Log the cached-token fields on every request. Anthropic returns usage data you can print directly; the docs' Python example dumps response.usage after a call with caching enabled, which is the cheapest possible instrumentation (Claude platform docs).
2. Aggregate per prompt template, not globally. A fleet-wide average hides the template that regressed.
3. Alert on drops after every prompt deploy. A template change that reorders the system prompt or injects a timestamp near the top silently invalidates cache entries and shows up as a cost spike days later (Metabase).
4. Run an A/B test on the prompt structure itself. Keep model, temperature and retrieval constant; change only ordering. Compare cached-token share, time to first token and cost per completed task.

A worked example makes the arithmetic concrete. Suppose a workload sends a 20,000-token prefix and 1,000 variable tokens per request, 100,000 requests per month. Without caching, you pay for 2.1 billion input tokens. With the prefix cached, you pay for 100 million cached-read tokens and 100 million uncached tokens. If cached reads cost one tenth of the base input rate, as one reported pricing structure has it (llm-d), the input bill falls by roughly 86%. Recompute with your own provider's published rates before quoting any number; the ratio is what transfers between providers, not the absolute price.

One caution on cache lifetime. Anthropic's default cache lasts five minutes and refreshes on each use, and the lifetime is measured from the start of the request that writes or reads the entry, so time spent streaming a long response counts against it (Claude platform docs). If your traffic is bursty with long gaps, a structurally perfect prompt still produces misses, and the fix is traffic shaping or a longer TTL, not prompt editing.

For teams running several model families, the placement rules here are portable, but the mechanics (automatic versus explicit breakpoints, TTLs, minimum cacheable lengths) differ per provider, so verify each before assuming a hit. Quanta Inference (https://quantaintelligence.ai/inference) puts text, image, video, audio and embedding models behind one OpenAI-compatible API key with prepaid dollar pricing, which makes it straightforward to run the same refactored prompt template across model families and compare cached-token share in one place.

The minimum-length check deserves a note. Anthropic caching only triggers on prompts above a threshold on the order of 1,024 tokens, which a bug report on a provider integration demonstrates causing silent misses for users who assumed caching was on (OpenRouter SDK issue). Short prompts and prompts that technically qualify but diverge in their first few hundred tokens will look correct and cache nothing.

It is also worth knowing what is not being cached. A semantic cache stores finished response strings keyed by embedding similarity and can return a wrong answer with a 200 status, while prefix caching is exact-match and correctness-neutral (Avi Chawla's comparison). If a stakeholder asks why two identical-looking questions got different answers, that is expected: prompt caching changes computation, not sampling.