Prompt Caching: How It Works and When It Pays

Generating a response happens in two phases. In the prefill phase, the model reads every input token and computes intermediate values called key-value (KV) states, which let each token refer back to earlier ones. In the decode phase, it generates output tokens one at a time using those states. Modular's inference handbook explains prefix caching in these terms: the server stores the KV states from a processed prompt, and when a request arrives it "finds the longest cached token sequence that matches the request from the beginning," loads those states and runs prefill only on the remaining tokens (Modular, Prefix caching).

OpenAI makes the same point about what is stored: the cache holds KV tensors, not the tokens themselves (OpenAI).

The match must be exact, from the first token onward. Modular's example shows why:

Prompt 1: Summarize this incident report in one paragraph.
Prompt 2: Summarize this incident report in three bullet points.
Prompt 3: Write a one-paragraph summary of this incident report.

Prompt 2 reuses the cached states for the shared opening, "Summarize this incident report in," and only processes the part that differs. Prompt 3 means almost the same thing as Prompt 1 but starts with different tokens, so it cannot reuse anything. Two prompts can share a meaning and miss the cache, while two requests with different final questions can share most of their computation.

Self-hosted inference servers implement the same idea. The vLLM documentation calls prefix caching "almost a free lunch" because it does not change model outputs, and describes hashing each block of the KV cache together with all the tokens before it, so a block can only be reused when everything before it also matches (vLLM, Automatic prefix caching).

Michael Hannecke's explainer lists the misunderstandings that cause the most trouble (Hannecke, Prompt caching explained):

Response caching is a separate, complementary technique. It returns a stored answer for an identical request and skips the model entirely, which suits deterministic lookups but not conversations where each answer should be generated fresh.

Providers expose prompt caching in two ways, and some support both.

Automatic (implicit) caching needs no code changes. The provider decides where cacheable prefixes end and reuses them when it can. OpenAI's documentation states that prompt caching is enabled by default for its supported models (OpenAI).

Explicit caching lets you mark breakpoints yourself. Anthropic's documentation describes the flow: the system checks whether the prefix up to a breakpoint is already cached, uses it if so and otherwise processes the full prompt and caches the prefix (Anthropic, Prompt caching). Explicit control is useful when part of your prompt is reusable and part is not, because you can stop the cache write before the changing part.

Details differ by provider and change over time, so check the documentation for the model you use. The questions to answer are always the same:

• Is caching automatic or do I mark breakpoints?
• What is the minimum prefix length before caching applies?
• How long does an entry live, and does each hit refresh it? Anthropic's documentation, read on 2026-10-17, gives a default lifetime of five minutes, refreshed at no cost each time the cached content is used.
• Is there a write premium, and what discount do reads get?
• Which settings are part of the prefix? Anthropic describes a hierarchy of tools, then system prompt, then messages, where a change at one level invalidates that level and everything after it. Changing tool definitions invalidates the whole cache. OpenAI lists settings such as the model, tools, output format and reasoning settings as part of what must match.

Routing matters too. OpenAI notes that cached states "live on individual machines," so a request can only reuse a cache if it reaches a machine holding a matching, unexpired entry. It offers a prompt_cache_key field to help group related requests (OpenAI). Model gateways that spread traffic across several providers face the same problem. OpenRouter, for example, uses provider sticky routing to send later requests in a conversation to the same provider endpoint, and accepts a session_id so an agent's requests stay together even when their opening messages change (OpenRouter, Prompt caching).

Prompt structure decides your hit rate. Hannecke's rule is "static content first, dynamic content last" (Hannecke):

[Tool definitions]      stable, cached
[System prompt]         stable, cached
[Reference documents]   stable per document, cached
[Conversation history]  grows by appending, cached up to the last turn
[Current user message]  always processed fresh

The practical rules:

1. Keep timestamps, request IDs and user names out of the top of the prompt. Hannecke reports that the most common reason for a poor hit rate is dynamic elements "buried in the middle of the prompt, silently invalidating the cache on every request." If the model needs the date, put it at the end.
2. Make history append-only. Do not rewrite or reorder earlier messages. Each edit invalidates everything after it.
3. Serialize deterministically. Key order in JSON, whitespace and formatting must be identical between calls. Modular stresses that even a single character difference breaks the match (Modular).
4. Keep tool definitions stable. Load the same set in the same order for the whole session instead of adding and removing tools between steps.
5. Avoid changing per-request settings that sit in the prefix, such as reasoning or output format settings, unless the provider documents a way to change them without breaking the cache.
6. Keep a session on one route. Use the provider's cache key or session identifier so related requests reach the same cache.

Agents benefit most from these rules. The system prompt, tools and growing history are re-sent on every step, so a stable, append-only context means most of each call is read from cache.

Caching pays when the same prefix is read several times before it expires. Hannecke lists the strongest cases: multi-turn conversations, question answering over the same documents, agent workflows with constant tools and instructions, and batch jobs that apply the same instructions to many inputs (Hannecke).

Where a provider charges a premium to write the cache, a cache that is never read costs more than no cache. Hannecke works through the break-even using the multipliers Anthropic's documentation listed for its standard short-lived cache when read on 2026-10-17: writes at 1.25 times the base input price and reads at 0.1 times (Anthropic).

Cost of one write plus N reads:   1.25 + 0.10 x N
Cost of N + 1 uncached requests:  1.00 x (N + 1)

N = 0  ->  1.25 vs 1.00   caching costs more
N = 1  ->  1.35 vs 2.00   caching saves about a third
N = 9  ->  2.15 vs 10.00  caching saves almost 80 percent

OpenAI's documentation gives the same arithmetic for its models that charge a write premium: one write and one full reuse cost 1.35 times the ordinary input cost, compared with twice the cost without caching (OpenAI).

Caching does not pay, or costs extra, in these cases:

• Prompts that change on every call, so there is nothing to reuse.
• Requests spaced further apart than the cache lifetime, so every request pays to write and none read.
• Prompts shorter than the provider's minimum, where caching never activates.
• One-off requests with no follow-up on providers that charge for writes.

A useful rule of thumb from Hannecke: if more than half of your input tokens repeat across requests, caching should be on.

Do not assume caching works. Read the usage data. Providers report cached tokens in each response. OpenRouter's documentation, for example, shows cached_tokens for tokens read from the cache and cache_write_tokens for tokens written on the first request, plus a cache_discount field showing what each response saved (OpenRouter).

"prompt_tokens_details": {
  "cached_tokens": 10318,
  "cache_write_tokens": 0
}

Log these fields with every request, tagged by feature, and track three numbers:

If the hit rate is low, diff two consecutive prompts for the same feature at the token level. The first difference is usually a timestamp, an ID, a reordered field or a changed tool list near the top. Fix the structure before touching any caching setting.

Gateways can help with the routing half of the problem. Quanta Inference, for example, sets session and cache keys from the system prompt so related requests land on the same cache.