Cache Invalidation Across Two Layers: A TTL Matrix

The right unit of TTL is the data source, not the endpoint. A single request often assembles several sources into one context block: a system prompt, a product catalog, a customer record, a retrieved policy document, a live status feed. Each has its own tolerable staleness.

Start with two questions per source:

1. If a user saw a version of this data that is N seconds old, would that be a support ticket?
2. How often does the source actually change, in practice?

The first answer sets the ceiling on your local TTL. The second tells you whether a longer TTL buys you anything at all. A source that changes once a day does not need a thirty second TTL, and a source that changes every few seconds cannot be cached for an hour without someone noticing.

Then set the prefix TTL. The prefix TTL is bounded by the provider's policy and by how long the prefix stays byte-identical. A prefix that embeds a customer record changes whenever that record changes, so its practical TTL is the record's local TTL, not the provider's maximum.

Two rules fall out of the table.

First, the prefix TTL should never be shorter than the local TTL for the same source. If the local layer can serve content for 60 minutes but the prefix entry only survives five, you pay for a cache write every five minutes and never read it back cheaply. Caching has a write premium; if you pay it and then let the entry die unused, you have made things worse. The Digital Applied playbook on prompt caching puts the breakeven on the shortest tier at roughly 1.4 reads per cached write, so a source that is read fewer than twice per write window should not be cached there at all.

Second, the highly volatile sources should be pushed out of the prefix entirely. The Flexera prompt caching breakdown makes the same point about static versus dynamic content: caching rewards stability, and volatile data sitting early in the prompt destroys the hit for everything after it.

Add jitter to local TTLs that expire on a shared schedule. When many workers hold the same source with the same TTL, they all miss at once and hammer the upstream. WordPress VIP's write-up on cache stampedes shows the pattern and the fix: randomize expiration slightly so load spreads out.

Take an agent that answers questions about a customer account. Its context has three parts, in this order:

• A 1,200 token system prompt and tool manifest, stable between deploys.
• A 3,000 token policy document set, republished a few times a day.
• A 400 token customer record, changed whenever the customer edits their profile.

Requests arrive for the same customer in bursts. Say a burst is five requests within two minutes, and that burst pattern repeats for many customers through the day.

A naive setup caches the whole assembled context locally for 60 minutes and relies on the provider's default five minute prefix lifetime. What happens:

• The local cache serves the burst from its first write. That part is fine.
• The prefix cache is warm for the first request after each local write, and the four follow-ups within five minutes hit it. Good.
• But the customer record changes to something else and the local key does not include a version, so the local layer keeps serving the old record for up to an hour. Users see stale account data and file tickets.
• Because the local entry outlives the prefix entry, most later requests in the hour are local hits that still pay full prefill. The prefix layer contributes almost nothing.

Now the matrix-driven setup:

The burst of five requests in two minutes now behaves like this: the first request writes the local entry and the provider prefix; requests two through five hit both layers. The customer record is fetched fresh if more than 20 seconds have passed, and a profile edit bumps the version so the next request cannot hit a stale entry. The stable 1,200 token block is shared across every customer, so its prefix entry stays warm all day and gets read thousands of times per write.

The gain is not just latency. It is that you can now state, for each source, how old the data a user sees is allowed to be, and prove it from logs.

You cannot tune a TTL matrix you cannot see. Add three fields to every response log line, at minimum:

• served_by: which layer produced the content, one of local, prefix, provider, or miss.
• age_ms: how old the served entry was when it was used.
• source_version: the version or hash of the data source that the entry was built from.

Provider APIs already give you part of this. OpenRouter's response caching docs return X-OpenRouter-Cache-Status: HIT or MISS along with X-OpenRouter-Cache-Age and X-OpenRouter-Cache-TTL, so the prefix side can be read straight from the response headers rather than inferred. The OpenRouter announcement notes the same headers and a per-request TTL override, which is what makes a per-source matrix practical rather than theoretical.

A minimal log record looks like this:

{
  "request_id": "req_8f21",
  "source": "customer_record",
  "served_by": "local",
  "age_ms": 18400,
  "local_ttl_s": 20,
  "source_version": "v42",
  "prefix_status": "HIT",
  "prefix_age_ms": 18400
}

The important habit is to log the decision, not just the outcome. If a request is served from the local layer, log why the local entry was considered valid, and log the prefix status of the request that produced it. When a stale answer reaches a user, the first question is always which layer handed it over, and the log should answer that without a reconstruction.

Invalidation drift is the slow divergence between what each layer believes is current. It shows up as a pattern in the served_by and age_ms fields long before users complain, if you look.

The decision rule that follows is short. If stale answers come from the local layer, the local TTL or its bust logic is wrong, because that layer is the one that delivers outdated content. If stale answers come from the prefix layer, the local TTL is too long relative to the prefix TTL, so the local layer is holding content that the provider has already discarded and the user is being served a version the layer below has given up on.

Watch the age distribution, not just the average. A source whose age_ms p95 sits right at its TTL is a source where a meaningful share of users see the oldest permitted version. If that is not acceptable, the TTL is too long, whatever the average looks like.

The matrix is not a rule that every source needs two TTLs. Some sources should only have one.

Skip the local cache for sources where the tolerable staleness is zero and the fetch is cheap, like a live status feed or a per-request identifier. Caching something you must always re-read just adds a way to be wrong.

Skip the prefix cache for content that is volatile or customer-specific. If a source's TTL is under a minute, the write premium will not pay back, and the content actively harms hits for everything downstream if placed before the boundary. Keep it after the cached prefix instead.

Keep both for the sources that dominate token count and change slowly: system prompts, tool definitions, document sets, and long conversation history. Those are where the local layer saves a serialization and a fetch, and the prefix layer saves the prefill.

One more control worth having: an explicit bust path that is separate from TTL expiry. A publish, a profile edit, or a deploy should invalidate the local key immediately and change the source version. OpenRouter's caching docs describe the related idea at the routing layer, where a sticky provider keeps a cache warm and a failed request does not update it. Your bust path should be just as explicit: version the key, bump the version on write, and let the old entry age out naturally rather than trying to reach into the provider.

If you are running several models behind one interface, the matrix is also the place to record that different providers have different prefix policies and lifetimes. Routing the same workload through one endpoint with a single key, such as Quanta Inference, keeps the per-source TTLs and the served_by values comparable across models, which makes the drift table above much easier to read.