Choosing the Cache TTL: A Cost Model for Shared Briefs

The model needs an edit rate, a request rate and two unit costs. On the cost side, the useful unit is not dollars per token but the relationship between three published rates: uncached input, cache read and cache write. Providers publish all three. Anthropic's list prices, as tabulated by BenchLM's Claude pricing page, show cache reads at one tenth of base input for several current models, with five-minute writes at 1.25 times base and one-hour writes at twice base. OpenAI publishes a cached input column and a cache writes column per model on its API pricing page, and its extended retention option changes the write side of the equation.

The practical consequence: the read discount is large, and the write premium is small for the short tier and larger for the long tier. A worked example in this prompt caching cost analysis uses a base input rate of $3 per million tokens, a five-minute write at $3.75, a one-hour write at $6 and a cache hit at $0.30. On those numbers a single hit recovers the five-minute write premium, while the one-hour write needs at least two hits inside its window to come out ahead.

So the model has three inputs:

If your provider does not bill storage separately, the storage term is the opportunity cost of the memory: the eviction pressure the entry puts on everything else sharing the same cache. That number is real even when it does not appear on an invoice, and it is the reason a TTL that is "free" in dollars can still be wrong.

Write the cost of keeping the brief cached for one more hour and the expected cost of not keeping it.

Holding the entry for an hour costs s times B, where B is the brief's byte size. Losing the entry and rebuilding it costs one cache write plus the extra full-price input on each request that would have been a hit until the entry is rebuilt. If the brief is edited at rate e, the expected number of requests between two edits is r divided by e. That is the number of potential hits you forfeit when you choose, or are forced, to miss.

Set the two equal and solve for the request-to-edit ratio at which caching stops paying:

break-even requests per edit = (write premium + 1) / (1 - read ratio)

On the Anthropic-style numbers above (read at 0.10 of base, five-minute write at 1.25), that is (0.25 + 1) / (1 - 0.10) = 1.39 requests per edit. The Digital Applied caching guide reports the same figure as roughly 1.4 reads per cached write. For the one-hour tier at twice base input the numerator becomes 2.0, so the requirement rises to about 2.2 requests per edit.

Two readings follow. The first is the decision rule most teams miss: with a read ratio near 0.10, almost any brief that is requested even twice per edit clears the write premium, so the write side is rarely what kills a cache. The second is that the real constraint is the edit rate relative to request rate, plus storage pressure when many briefs compete for the same cache budget.

Worked example. A brief is 20,000 tokens, edited four times per hour during business hours, and requested 400 times per hour across a fleet of agents. Requests per edit is 100, far above 1.39, so caching pays. The hourly write cost at the example rates is 20,000 times $3.75 per million, about $0.075 per write, roughly $0.30 per hour across the four edits. The hourly saving on the non-edited requests is 396 requests times 20,000 tokens times ($3.00 minus $0.30) per million, about $21.40. The gap is two orders of magnitude, which is why the TTL in that setup can be set generously without risk.

Now change one input. The same brief is edited 50 times per hour by an automated job that rewrites a timestamp into the header. Requests per edit drops to 8. Still above break-even, but the write spend rises to $3.75 per hour and every edit throws away the hits that would have accrued. The fix is not a shorter TTL; it is removing the timestamp from the cached prefix, which is the same conclusion the byte-stability writeup reaches from the failure side.

Once the brief clears break-even, the TTL itself is a smaller decision than it looks. The entry lives until the earlier of two events: the provider's TTL expires it, or an edit invalidates it. Since an edit invalidates everything after the changed byte, a TTL longer than the expected gap between edits buys nothing. That gives a clean rule: set the TTL to the mean inter-edit interval, measured in requests rather than wall-clock time, and round down to the nearest tier your provider actually offers.

The last row is the short-circuit case. If a brief is edited more often than it is requested, there is no window in which a stored entry can be reused. Every write pays the premium and collects zero hits. Skip caching and accept full input price; the arithmetic is unambiguous.

There is a second reason to prefer the shorter tier even when the longer one would break even. A long TTL holds memory that other cache entries need. In a system with many tenants, the cache is a shared resource, and an entry nobody is reading pushes out entries that are being read. The cost of holding an entry is therefore the value of the evicted entry, not just the storage line item. The arXiv evaluation of prompt caching for long-horizon agents found that full-context caching can paradoxically increase latency, while strategic cache block control gave more consistent benefits. The lesson transfers: a TTL that keeps a stale or rarely read block alive measurably hurts.

The model is only as good as e and r, and both are cheap to measure once you decide to.

Instrument the request path to label every call with a prompt template id and whether the provider reported cached tokens. That gives r per template directly. For e, either read the version history of the source document, or infer it: count distinct prefix hashes seen per template per hour and divide by the number of templates. The second approach also catches accidental invalidation, because the distinct-hash count spikes when a serialization change lands.

A small check that runs on the aggregated hourly data is worth the effort:

def choose_ttl(requests_per_hour, edits_per_hour, read_ratio, write_premium,
               storage_cost_per_hour, tiers_hours):
    if edits_per_hour <= 0:
        return tiers_hours[-1]          # nothing invalidates it
    if requests_per_hour / edits_per_hour <= (write_premium + 1) / (1 - read_ratio):
        return 0                        # skip caching
    interval = 1 / edits_per_hour       # mean hours between edits
    eligible = [t for t in tiers_hours if t <= interval]
    return max(eligible) if eligible else tiers_hours[0]

Two cautions on measurement. Hit rate alone is a misleading success metric, because the same number means different things in different caches and a high rate can still be the wrong outcome for the workload (Aerospike on hit rate, DebugBear on how to measure and improve it). Track dollars saved per template instead. And keep the cache key stable: a key that changes on every deploy produces a written entry nobody ever reads, which is the failure mode OpenAI's prompt cache diagnostics exist to expose.

The same accounting applies to batching and service tiers. If your traffic can tolerate asynchronous processing, batch pricing changes the recomputation cost so much that the whole break-even shifts; the pricing pages list batch rates alongside standard ones (OpenAI, Claude rates).

The rule, in one pass:

1. Estimate e in edits per hour and r in requests per hour for the brief. If r divided by e is at or below (W + 1) divided by (1 minus R), do not cache.
2. Compute the mean inter-edit interval in hours, which is 1 divided by e.
3. Pick the largest TTL tier that does not exceed that interval, and round down.
4. Recompute quarterly, or whenever the prefix changes shape: a new tool, a new model that uses a different cache pool, or a whitespace change in the template all move e.

A worked check on the numbers from the longer tier: the one-hour write needs about 2.2 requests per edit just to pay its premium, but if the edit rate is one per day and the request rate is 200 per day, requests per edit is 200 and the TTL should be the longest tier that ends before the next edit, which is the medium or one-hour tier. The storage charge over that hour is tiny compared with the recomputation it avoids.

Two failure modes are worth naming because the model catches them and intuition does not. The first is caching a brief that is rebuilt more often than it is read, which the break-even test rejects outright. The second is setting a long TTL to "be safe" on a brief that is edited every fifteen minutes, where the TTL expires no laster than the edit would have invalidated the entry anyway; the only effect is that the entry occupies cache during the gaps. The arithmetic points the same way both times.

For teams running many briefs across several model providers, the price tables are part of the input, so a single gateway that normalizes billing makes the comparison tractable. Quanta Inference exposes text, image, video, audio and embedding models through one OpenAI-compatible API key with prepaid pricing in dollars (https://quantaintelligence.ai/inference), which removes the per-provider pricing table from the estimator and lets the TTL model use one set of read and write ratios across the fleet.