A TTL Matrix for Two Cache Layers

The word "cache" hides two very different things in an LLM application.

The local cache is yours. It lives in process memory, in a sidecar, or in Redis, and holds either whole responses keyed by a hash of the request or retrieved documents keyed by a query. You control its TTL and its eviction policy completely, and you can inspect and purge it. Semantic response caches, which return a stored answer for a similar prompt, and exact-match caches, which hash the full request, both sit at this layer per the layered breakdown. On a hit they bypass the model entirely and save both input and output tokens.

The provider cache is not yours. It reuses computed attention state for a repeated prompt prefix, keyed on the literal token sequence, so a single changed character before the cache boundary breaks the hit per Modular's inference handbook. You do not set its TTL with a per-key call. You choose among the retention tiers a provider exposes and design prompts so that the prefix stays stable. Anthropic's default cache lifetime is 5 minutes from the start of the request that writes or reads the entry, refreshed at no extra cost each time the content is used, with a longer tier available at a higher write price per Anthropic's prompt caching docs. Time spent streaming counts against that lifetime, which is a trap: a response that streams for 4 minutes leaves only about a minute for the follow-up request to hit the same prefix.

This asymmetry drives the first decision rule. Your local TTL for anything derived from the provider must be shorter than or equal to the provider's effective retention. If the provider will not serve a prefix, your local layer must not pretend it still can, or you will keep answering from a snapshot whose inputs have moved.

That last row is worth stating plainly: on the provider side a TTL mistake usually costs money and latency, while on the local side a TTL mistake usually costs correctness. Different budgets, different scrutiny.

The matrix has one row per source type and one column per layer. Fill each cell by asking a single question: how stale can this source be before a user or a downstream system is harmed?

Two properties keep the table honest. First, every local TTL is at or below its provider TTL in the same row, which enforces the ordering rule from the previous section. Second, the volatile rows are never placed in a cached prefix. That is not just a cost point. It is a safety point: a cached prefix that contains one user's session data can be served to a different user if the key is not scoped properly, which is why inference engines offer per-request salts and cache isolation for shared environments per the vLLM design notes. Dynamic, user-specific content belongs at the end of the prompt, after the cache boundary.

Notice how the table encodes the prompt-order rule without a separate section. Content is ordered most stable to least stable, so every row that changes often sits downstream of every row that does not. That single ordering change is the biggest lever on provider-side hit rate. One published case study moved dynamic working memory out of the system prompt into a user message at the end and raised its cache hit rate from 7 percent to 84 percent, cutting overall LLM cost 59 percent, as reported in the Digital Applied playbook.

Numbers make the trade-off concrete. Take a checkout page that shows a live inventory count next to each item, and suppose the count is cached locally for 15 minutes because someone copied the catalog TTL. The catalog and the inventory count are different source types with different change rates, and the matrix above would never let them share a cell.

Assume the following, all of it invented for the example so you can substitute your own: 40,000 product views an hour, 6 percent of views convert, an average order value of 60 dollars, and an inventory count that drops from 10 to 0 at some point during the day. For the 15 minutes it takes the stale entry to expire, every viewer of that product sees "10 in stock."

Now compare that to what the caching was buying. If the same 15-minute TTL produced 10,000 local hits instead of 10,000 origin lookups, and an origin lookup costs 20 milliseconds of compute against 2 milliseconds for the cache, the saving is 10,000 times 18 milliseconds, or 180 seconds of compute time per window. That is trivial compared with $5,900. The stale hit was never worth it.

Set the inventory row to 10 seconds instead. Now the worst-case oversell for one product is roughly 110 requests, about 7 orders, and a few dozen dollars. You still get most of the hit rate because traffic on a hot product is high, and you have capped the damage. This is the decision rule in one line: a TTL is justified when the cost of the staleness it allows is smaller than the value of the hits it saves, and you can compute both.

Run the same arithmetic for the provider side. If a cached prefix write costs a premium and reads cost a fraction of base input, the break-even is a small number of reads per write. Published figures put Anthropic's discount at roughly 0.10 times base input for reads, with break-even around 1.4 reads per cached write on the shortest tier per the Digital Applied playbook. That is close to the write cost for a 5-minute tier and far below the cost for a 1-hour tier, which is why long-lived, stable prefixes belong on the longer tier and a burst of one-off prompts should not be cached at all.

The ordering rule is the part teams get wrong. It is intuitive to give the fast local layer a long TTL, because local hits are the cheapest thing you have. It is also backwards.

Think about what each layer can see. The local layer only knows what it stored. The provider layer sees every request across the whole organization and refreshes its entry whenever the prefix is reused per Anthropic's docs. The provider is therefore always at least as fresh as your local copy of anything derived from it. If your local TTL exceeds the provider's retention, you have built a layer whose job is to outlive the truth.

There is a second reason for short local TTLs: eviction. A generous local TTL keeps entries alive past their usefulness, and a cache full of dead-but-unexpired entries pushes live ones out under LRU. That is the classic failure where a bulk scan or a nightly job promotes two million cold rows and flushes the working set as Redisson describes. Expiring stale entries early makes eviction smarter without touching the policy.

A practical rule of thumb: local TTL is one to three times the expected interval between writes to the same key, capped below the provider tier. If inventory updates every 30 seconds, a 10 to 30 second local TTL captures most reuse without much staleness. If a catalog changes twice a day, 10 to 15 minutes is generous and still far under any provider tier.

Jitter matters here too. If ten thousand entries are written at the top of the hour with the same TTL, they all expire together, and every one of them misses at once. WordPress VIP documents this cache stampede pattern and its fix: add random variation to the TTL so entries expire spread out instead of in a wave per the WordPress VIP guide. A 10 percent jitter is cheap insurance.

A TTL you cannot defend is a guess with a number attached. Proving a value takes three measurements, and all of them should be in place before the TTL goes to production.

Maximum acceptable staleness. Ask the owner of the source, not the cache owner, how old a value can be before harm occurs. Prices at checkout need the authoritative value, while prices on a browsing page can be minutes out of date without a negative impact per AWS's caching best practices. Write that number down. It is the ceiling for the local TTL, full stop.

Observed change rate. Measure how often the source actually changes for a given key, not how often it could. A permission set that changes when a user logs in or out is fine with a 60-second TTL if sessions last hours, and terrible if sessions churn every 20 seconds. The stale-value cost from the worked example divided by the hit value gives you a break-even point you can compare against the measured rate.

Post-deploy outcomes. After you ship, watch two numbers per source type: hit rate and error rate. Measure hit rate as hits divided by total lookups, the standard definition per DebugBear, and break it down by source type rather than as one global figure, because a healthy catalog stream can hide a broken session stream. Then watch the specific error that staleness causes, such as oversell events, permission denials, or "price changed at checkout" messages.

A short instrumentation snippet makes the second metric visible. Emit it on every read:

# Pseudocode: per-source-type staleness and hit metrics
def get(source_type, key, loader, ttl, jitter):
    entry = store.get(f"{source_type}:{key}")
    metrics.incr(f"cache.{source_type}.lookup")

    if entry and entry.expires_at > now():
        metrics.incr(f"cache.{source_type}.hit")
        metrics.observe(f"cache.{source_type}.age_s", now() - entry.written_at)
        return entry.value

    value = loader()  # origin read
    ttl_actual = ttl * random.uniform(1 - jitter, 1 + jitter)
    store.set(
        f"{source_type}:{key}",
        Entry(value=value, written_at=now(), expires_at=now() + ttl_actual),
    )
    metrics.incr(f"cache.{source_type}.miss")
    return value

The age histogram is the useful part. If the 99th percentile served age for a source type approaches the maximum acceptable staleness you wrote down, the TTL is too long, regardless of how good the hit rate looks.

Provider-side proof is different. There is no per-key age to read, so monitor the cache-read token share from each response's usage metadata and the time-to-first-token distribution. A falling read share means your prefix stopped matching, usually because something upstream started varying the prefix. A rising time-to-first-token with a flat request rate means prefill is being repeated.

The matrix scales to whatever model you run. The 5-minute and 1-hour tiers, the caching discounts, and the requirement that prompt order stay stable are consistent across major providers as the cost playbook notes, and a further 24-hour extended-retention option exists for slow-burn batch agents per the same source. If you run models across more than one provider, though, the TTL values start to diverge, and keeping a matrix per provider is more work than it sounds. Routing text, image, and embedding traffic through Quanta Inference keeps one OpenAI-compatible surface and one set of rules to reason about, which makes the local-versus-provider clock ordering easier to hold in your head.

If you would rather have the matrix reviewed by someone who has run it in production, a consulting call is the shortest path from here to a table you can defend.