Cache Hits vs Prompt Freshness: The Trade-off

The design rule is to put only things that do not change inside the cached region, and everything that moves after the cache breakpoint. A workable structure for a business assistant looks like this.

[PREFIX  version=policies-2024-11-03 schema=quote-v3]
You are a quoting assistant for {company}. Follow the pricing policy below verbatim.
<policy text, 4,000 tokens>
<output JSON schema>
<tool definitions>
[/PREFIX]

[SUFFIX]
Current UTC time: 2024-11-03T14:02:11Z
Customer tier: gold
Inventory for SKU-4471: 12 units at 41.90 USD (source: warehouse API, fetched 14:01:58Z)
Account credit limit remaining: 8,400 USD
User question: can I ship 20 units by Friday?
[/SUFFIX]

Everything between the two markers is stable across requests and worth caching. Everything after them is per-request or per-minute and must never be cached. OpenAI's own guidance is the same shape: put static content at the beginning and variable content at the end of the prompt (Humanloop's summary of prompt caching).

Two details matter in practice.

First, the breakpoint must be placed explicitly where the provider supports it. Anthropic's cache_control marks individual content blocks, and up to four breakpoints per request are allowed, which lets you cache the system prompt, the static tool list and a sliding conversation window separately (Claude platform docs). Amazon Bedrock exposes both implicit and explicit caching, with support varying by model and API (AWS docs).

Second, watch for accidental volatility inside the "stable" region. A timestamp, a request ID, a user's name, or a reordered JSON schema will break the prefix on every call. ProjectDiscovery's engineering writeup describes walking system messages backward to find the last genuinely static one before applying the breakpoint, precisely because dynamic sections were being injected near the top (ProjectDiscovery). Reordering the system prompt or injecting a timestamp near the top is also the usual cause of a sudden hit-rate drop after a deploy (Metabase's metric guide).

The cheapest invalidation mechanism is a version string inside the cached prefix. Because the cache keys on exact token content, changing version=policies-2024-11-03 to version=policies-2024-11-04 guarantees a miss on the next request, and every subsequent request that carries the new string rebuilds the cache under it.

This gives you three properties that TTLs cannot.

• Instant invalidation. You do not wait for a five-minute or one-hour expiry. The next request is fresh.
• Deterministic rollback. Shipping a bad policy update is a one-line revert to the previous version string; the cache returns to the old, known-good content immediately.
• An audit trail. The version token in your logs tells you exactly which cached content served each response, which is what you need when someone asks why the model quoted an old price.

The version token should be derived from the content, not hand-typed, or you will eventually forget to bump it. A workable pattern hashes the stable block and takes a short slice:

import hashlib

def build_prefix(policy_text: str, schema_text: str, tools_text: str) -> tuple[str, str]:
    body = f"{policy_text}\n{schema_text}\n{tools_text}"
    version = hashlib.sha256(body.encode()).hexdigest()[:12]
    header = f"[version={version}]\n"
    return header + body, version

Any edit to the policy text changes the hash, which changes the prefix, which forces a cache miss for exactly one request per worker before the new prefix warms. That is the correct cost: one cold request instead of hours of stale answers.

Note that a version change defeats the cache entirely, including the cost savings, for the first request on each replica. If you deploy policy changes hourly, you are paying full prefill roughly hourly, which is the right trade for correctness but should be a conscious one.

The TTL is the backstop for cases the version token does not catch: data that changes without a deploy, because it arrives from a source system. Here the rule is to set the TTL from the update cadence of the slowest-moving thing you allow inside the cached region, not from convenience.

Provider cache lifetimes set the floor. Anthropic's cache defaults to a 5-minute lifetime, refreshed at no cost each time the cached content is used, with a 1-hour option that costs more to write (Claude platform docs). OpenAI's cached prefixes generally remain active for 5 to 10 minutes of inactivity, with longer persistence during off-peak periods (Humanloop). OpenRouter's response cache defaults to a 5-minute TTL and can be configured from 1 second up to 24 hours (OpenRouter response caching). A gateway-side sticky routing layer expires its session affinity after 10 minutes of inactivity (OpenRouter prompt caching).

Read that table as a hierarchy rather than a rule. Prefix caching is the cheapest layer and the safest, because what it stores cannot drift while the prompt is fixed. Your TTL should exist mainly to bound the damage from a data source you failed to model as volatile. If you find yourself lowering the TTL to minutes to keep a price list fresh, the price list belongs in the suffix.

There is a second, subtler interaction. Because a cache hit depends on the serving engine retaining the entry and routing the request to a worker that can access it (Modular), TTLs shorter than your request routing window will produce misses you did not intend. Distributed deployments scatter related requests across replicas by default and destroy cache locality unless the router is prefix-aware (llm-d).

Everything above concerns prefix caching, which reuses computation and cannot change the model's output for a given prompt. Two other caches sit at higher layers and carry more danger.

Response caching returns the stored output for an identical request, skipping the provider entirely. OpenRouter's implementation hashes the request body, model, API key and streaming mode, returns the cached response with all usage counters zeroed, and only caches successful 200 responses (OpenRouter docs). It reports hits in 80 to 300 milliseconds, with a lookup averaging 4 milliseconds (OpenRouter announcement). This is exact-match and therefore safe for a fixed input, but note the implication: if the request body contains a price and the price changes, the body changes, and the cache key changes with it. The failure mode is an application that fetches stale data upstream and then caches the answer consistently.

Semantic caching is the risky one. It embeds the request, finds a similar prior question above a similarity threshold, and returns the stored answer. The threshold governs everything: too low and you serve wrong answers, too high and the hit rate collapses. Embedding proximity is not meaning equality, since "capital of France" and "capital of Germany" sit near each other in embedding space (TrueFoundry's analysis). The companion guide on this site covers the threshold tuning in detail; the point here is that semantic caching should never hold anything account-specific or time-sensitive, and it is not a substitute for a correctly split prefix.

A useful decision rule, from cheapest and safest to most dangerous:

1. Prefix cache the instructions, schema and policy text. Correctness-neutral by construction.
2. Response cache identical requests where the input fully determines the output and the input contains no volatile fields.
3. Semantic cache only public, non-personal, slowly changing factual answers, with per-namespace keys and entity guards.
4. Never cache prices, inventory, balances, entitlements or anything a regulator would call a quote.

The dashboard most teams build shows hit rate, and hit rate will look excellent right up until the moment someone notices the model quoting last week's discount. A cache hit on stale content is a successful lookup of a wrong answer, so it needs its own counter.

Define stale-serve rate as the share of responses where the cached content's version or data timestamp was older than the maximum acceptable age for that field. Three inputs make it computable:

• Version token in every log line. You already emit it in the prefix; record it with the request.
• Data timestamps in the suffix. When the warehouse API returns inventory, include its fetch time in the suffix text. The model may even mention it, which makes staleness visible to users.
• A freshness budget per field. Inventory is stale after 60 seconds, catalog after a few hours, policy after a day. Encode these as constants next to the field definitions.

Metabase's guidance on cache metrics makes the same structural point from the other direction: keep response-cache and prompt-cache rates as separate numbers because they have different economics, compute savings from recorded costs rather than list prices, and alert on per-application drops, since a fleet average moves too slowly to catch a broken template (Metabase).

A worked example on a modest workload. Suppose 200,000 requests a day carry a 6,000 token cached prefix and 400 tokens of volatile suffix, and the vendor charges 3.00 USD per million uncached input tokens against 0.30 per million for cache reads, the ratio Anthropic's pricing table shows (Claude platform docs). Uncached, those prefixes cost 200,000 times 6,000 tokens, or 1.2 billion tokens, which is 3,600 USD a day. Cached, the same volume costs 360 USD, a saving of 3,240 USD a day, before counting the suffix. Now suppose your version token is bumped hourly and each bump costs one cold request per replica across 8 replicas: 192 cold prefills a day, roughly 1.15 million full-price tokens, about 3.45 USD. Invalidation is not the expense; the expense is any volatile field you accidentally leave in the prefix, which turns every request into a miss and quietly returns you to the 3,600 USD line.

That arithmetic is the whole argument. Make the prefix aggressively stable, put volatility in the suffix, and make invalidation cheap and explicit. If you are wiring several providers together and want one key in front of them, Quanta Inference (https://quantaintelligence.ai/inference) exposes text, image, video, audio and embedding models through a single OpenAI-compatible endpoint, which keeps prefix structure identical across models while you tune the split.

Before you ship a cached prefix into production, confirm each of these.

• Prefix contains no timestamps, request IDs, user names or reordered JSON. Any of these turns every request into a miss.
• Prefix has a content-derived version token, logged with every request.
• Volatile business fields appear only after the breakpoint, with the source fetch time written alongside them.
• TTL set per cache layer, not one value for everything, and longer than your routing stickiness window.
• Stale-serve rate instrumented and alerted, with a partial-failure path so a broken freshness check does not fail the whole request.
• Rollback rehearsed: you can flip the version string and verify a cold rebuild in staging.
• Multi-tenant isolation considered. vLLM supports per-request salting to isolate prefix cache reuse in shared environments (vLLM), and OpenRouter scopes response caches to the API key (OpenRouter docs).

If the caching work is part of a larger build and you would rather hand it to someone who has done the split before, Quanta (https://quantaintelligence.ai/) takes a description of the system you need and pairs you with a specialist who builds it with you.