Stacking Local Cache on Provider Prefix Cache

Here is the concrete path. Every request passes through the same funnel, cheapest layer first.

1. Normalize the request: resolve defaults, strip volatile fields (timestamps, request IDs), and serialize to a canonical form.
2. Compute the fingerprint: a SHA-256 hash of that canonical serialization.
3. Look up the fingerprint in your local store. A hit returns the stored response verbatim, with zero provider calls.
4. On a local miss, forward to the provider, keeping the stable content (system prompt, tool definitions, reference documents) at the front of the prompt so the provider's prefix cache can match it.
5. Record the provider's reported cache read and write token counts alongside the response.
6. Store the response under the same fingerprint with a TTL, and invalidate on any upstream change.

import hashlib, json

def fingerprint(payload, model, params):
    canonical = json.dumps(
        {"model": model, "messages": payload, "params": params},
        sort_keys=True, separators=(",", ":"),
    )
    return hashlib.sha256(canonical.encode()).hexdigest()

def handle(request, store, provider):
    key = fingerprint(request.messages, request.model, request.params)
    if (hit := store.get(key)) is not None:
        return hit, {"layer": "local"}
    response = provider.complete(request)
    store.set(key, response, ttl=request.ttl)
    return response, {"layer": "provider", "cache_read": response.usage.cache_read_tokens}

The key discipline is that step 3 and step 6 use the identical function. If your local key hashes a different serialization than the one your provider-facing code sends, the two layers will disagree about what counts as "the same request," and you get the drift described below.

The discount varies by provider and is worth reading from the source rather than assuming. Amazon Bedrock documents two styles of prompt caching, implicit and explicit, and both target long, repeated contexts such as a document a user uploads and then asks questions about in the Bedrock user guide. Anthropic's docs describe a five-minute default cache lifetime that refreshes for free each time the cached content is used, with separate write and read prices in the Claude prompt caching docs. OpenAI documents its own prompt caching behavior and diagnostics in the OpenAI API guide.

The magnitude is large enough to design around. One production write-up reports a tenfold gap between cached and uncached token prices and notes that, in a simple test, sending a roughly 10,000-token prompt a second time dropped time-to-first-token from 4.3 seconds to 0.6 seconds in the llm-d KV-cache write-up. A separate engineering playbook puts cache reads near 0.10× base input and describes a team that moved dynamic working memory out of the system prompt and raised its cache hit rate from 7% to 84% at Digital Applied. Your mileage depends on your provider's published rates, so pull the numbers from the pricing page you actually bill against.

The point for a layered design: prefix caching already captures most of the input-cost saving for requests that share a prefix. A local cache adds value only when it eliminates the completion too, which means it must match the whole prompt, not just the head.

This is the decision rule. Model it as a comparison between two savings on the same request population.

Let $N$ be requests per month, $\bar{p}$ the average prompt tokens, $\bar{c}$ the average completion tokens, $r_p$ the uncached input price per token, $r_{pw}$ the cache-write price, $r_{pr}$ the cache-read price, $r_c$ the output price, and $L$ the fraction of requests that are exact full-prompt repeats of an earlier request in the same TTL window.

With provider prefix caching only, and assuming the repeating requests share their prefix:

$$ C_{\text{prefix}} = N(1-L)(\bar p , r_{pw} + \bar c , r_c) + N L (\bar p , r_{pr} + \bar c , r_c) $$

With a local cache in front, the $N L$ repeats never reach the provider:

$$ C_{\text{local}} = N(1-L)(\bar p , r_{pw} + \bar c , r_c) $$

The local layer saves $N L (\bar p , r_{pr} + \bar c , r_c)$ in provider spend. Against that you weigh the cost of running the store (memory, request handling, on-call attention). The break-even is roughly where $L(\bar p , r_{pr} + \bar c , r_c)$ covers your per-request local overhead. A worked example with round numbers: 500,000 requests a month, 2,000 prompt tokens, 400 completion tokens, input at 3 dollars per million tokens, cache reads at 0.3, output at 15. Repeats at $L = 0.25$. The local layer saves about $0.25 \times 500{,}000 \times (2{,}000 \times 0.3 + 400 \times 15) / 1{,}000{,}000$, which is roughly 825 dollars a month. At $L = 0.02$, it saves about 66 dollars and the store is probably not worth operating. The comparison table below summarizes the tradeoff.

Correctness matters as much as cost. Prefix caching never changes the model output because it reuses the exact computation the model would have done. A local response cache can serve a stale answer, so it is only appropriate for deterministic, non-personalized requests or ones with a defensible TTL. Keep it off for anything with per-user state in the prompt.

Drift is the quiet failure mode. It happens when the local cache and the provider prefix cache are invalidated on different schedules or keyed on different strings. Three concrete cases:

• You edit the system prompt but bump only one cache version. Local hits keep serving answers built on the old prompt while the provider sees a new prefix and misses.
• You drop fields from the local hash (say, temperature) so requests with different settings collide, but the provider keys on them correctly.
• Your TTLs diverge so the local cache expires sooner or later than the provider's prefix, and your cost dashboards stop reconciling.

The fix is a single fingerprint function and a single invalidation event. Treat any change to system prompt, tool schema, model, or decoding parameters as a cache version bump that both layers observe. Include a cache-busting field in the key and the request body, the way OpenRouter supports an explicit cache-clear flag and a per-request TTL for its response cache per the docs.

There is a second drift risk: routing. Prefix caching depends on the request landing on a worker holding the cache. vLLM hashes KV blocks by the tokens in the block plus the prefix before it, and warns that hashing choices trade speed for collision safety, with a reproducible cross-language option recommended for deterministic caching per the vLLM prefix caching docs. When you spread traffic across providers, cache locality can collapse. OpenRouter addresses this with sticky routing that sends follow-up requests to the same provider endpoint and expires after 10 minutes of inactivity per the prompt caching guide. A local exact-match cache sidesteps routing entirely, because a hit never leaves your infrastructure. That is one of its quiet advantages in a multi-model setup.

The research on eviction reinforces the point: keeping the right entries resident matters, and hotness-aware admission and promotion can lift hit rates substantially, with reported latency and throughput gains up to 2.25× over a baseline in the HotPrefix paper. You do not need that machinery to start, but you do need one shared key so admission and eviction in your store line up with what the provider retains.

Instrument both layers or you will not know which one earned the hit. Log, per request: the fingerprint, whether it hit the local store, the provider's reported cache-read and cache-write token counts, and the completion token count. Then compute three numbers monthly: local hit rate, provider cache-read share, and effective cost per request. The local hit rate is the one that decides whether the second layer stays. The provider cache-read share tells you whether your prompt ordering is doing its job. If your prefix cache hit rate is low on stable prompts, the problem is prompt structure, not missing layers, and adding a local cache will not fix it.

Compare across a controlled window. Split traffic so a share bypasses the local cache, then compare cost per request and p95 latency between the two groups. This avoids the trap of attributing month-over-month savings to a change that coincided with a cheaper model or a traffic shift.

If you are routing several model families and want one key and one bill while you run this experiment, Quanta Inference (https://quantaintelligence.ai/inference) puts text, image, video, audio and embedding models behind a single OpenAI-compatible API, which keeps the fingerprint and usage accounting in one place.