When to Cache Agent Memory: Trade-offs and Strategies

The mechanics matter because they determine what you can safely put where. Serving engines hash the KV cache in blocks, and each block's hash includes the tokens of its parent block plus its own tokens, so a block is only reusable if every block before it matched exactly per the vLLM design notes. In practice that means one edited character near the top of the prompt invalidates everything after it.

This is why the author of a detailed write-up describes putting long user-specific data at the end of the system prompt as his mistake: the cache can start at position zero across all users of the same API key organization if the system prompt is genuinely shared, but a per-user blob near the front kills that shared prefix as he explains here. The fix is the same one that helps everywhere else: static content first, variable content last.

[ shared system prompt + tool definitions ]   <- cacheable across users
[ stable reference documents ]                <- cacheable per tenant
[ retrieved context for this task ]           <- cacheable per task
[ conversation history so far ]               <- grows, cacheable
[ the new user turn ]                         <- never cached

Two gotchas are worth designing around. First, dating information ("today is Tuesday...") near the top of a prompt guarantees a miss on every request that crosses midnight, or sooner. Second, prefix caching depends on the serving engine keeping the entry and routing the request to a worker that can see it, and the messages must serialize identically according to this handbook. A load balancer that scatters a conversation across pods throws the cache away without telling you.

If you call many different providers, the routing problem multiplies. Quanta Inference (https://quantaintelligence.ai/inference) puts text, image, video, audio and embedding models behind one OpenAI-compatible key, which reduces the number of routing layers you have to reason about.

The decision rule is arithmetic, not instinct. Caching a value pays off when the expected savings from hits exceed the cost of storing it plus the expected cost of a stale answer. You can estimate the first part before you ship.

Take a support agent that retrieves the same 8,000-token product manual for every question in a session. Say the cacheable prefix is 10,000 tokens, you serve 20,000 requests a day, and 70 percent of them repeat a prefix you already wrote. With input priced at $3 per million tokens and cache reads at a tenth of that, uncached input alone costs 10,000 × 20,000 ÷ 1,000,000 × $3 = $600 a day. At a 70 percent hit rate the same traffic costs $180 for the misses plus about $42 for the hits, roughly $222 a day. The saving is about $378 a day, and it arrives without touching answer quality because prompt caching changes no output. The 10x gap between cached and uncached token prices is the number behind most of these calculations, as stated in this engineering write-up.

Now the freshness side. Suppose 2 percent of answers built on that manual would be wrong if the document is more than a day old, and a wrong answer costs you a $15 credit. You want the manual to be invalidated whenever a new revision is published, which is a write-triggered invalidation, not a TTL. The cache is still worth it, but the invalidation design is now load bearing.

A short checklist before you cache anything:

• Can you express the invalidation trigger as a signal your system actually receives (a write event, a revision ID, a content hash)?
• Is the cost of one stale answer lower than the savings from the hits it buys?
• Does the cache key include every input that changes the output, including model choice and temperature?
• Can you observe hit rate and staleness separately in production?

If any answer is no, cache a smaller thing. Storing a compiled retrieval result is safer than storing the final answer.

Treat memory as layers with different refresh rules rather than one cache with one timer.

The provider defaults set a floor on what is feasible. Anthropic's cache has a five-minute lifetime by default and refreshes at no cost each time the content is used, measured from the start of the request that wrote or read the entry per the Claude docs. That detail matters: a slow response eats into the window, so a four-minute stream leaves about a minute for the follow-up to reuse the same prefix. If your agent thinks for a long time between turns, an explicit longer-lived cache breakpoint is the right call.

Routing layers add their own clocks. A gateway may keep subsequent requests on the same provider to preserve cache locality, and sticky sessions can expire after a period of inactivity per these routing notes. When you debug an unexpected miss, check the routing layer before you doubt your prompt.

The mistake that bites teams is using one TTL for everything. A single hour on a mixed cache means volatile facts go stale and stable facts get evicted early. Split the store by tier and you can tune each one against its own risk.

TTL is a fallback, not a strategy. Any memory derived from a record should carry a version stamp, and any write to that record should bump the stamp so dependent entries stop being served.

A pattern that holds up well:

def cache_key(kind, record_id, version, model, extra=""):
    return f"{kind}:{record_id}:v{version}:{model}:{hash(extra)}"

def get_plan(record, version, model, store):
    key = cache_key("plan", record.id, version, model)
    hit = store.get(key)
    if hit is not None:
        return hit
    plan = build_plan(record, model)
    store.set(key, plan, ttl=3600)
    return plan

Because the version is inside the key, a write does not need to hunt down and delete old entries: the new version simply produces different keys and the old ones age out. This is the same reasoning vLLM uses when it hashes a block together with its parent hash, so a changed prefix cannot collide with an old one per the vLLM docs.

Versioning also covers the parts of a prompt people forget. Change a tool definition, a JSON schema, or the system prompt, and every plan cached against the old one is suspect. Fold a hash of the whole static block into the key and those caches invalidate themselves the next time you deploy.

For user-facing answers, the cheap insurance is to cache the plan and not the final prose. A research paper on agentic plan caching reports reducing cost by 50.31 percent and latency by 27.28 percent on average by reusing structured plan templates across semantically similar tasks while re-deriving task-specific details per the paper. You keep most of the savings and re-read current data on every run.

Once you see prompt caching as a memory layout problem, the design choices stop being separate. Both ask the same question: which bytes are shared, which are unique, and where do they sit relative to a boundary?

Cache-aware memory management comes down to four habits.

Order by stability. Most stable content first, most volatile last. A memory record written into the middle of the system prompt costs you every downstream hit.

Never mutate a cached prefix. Append to history rather than editing it. If compaction has to rewrite older turns, do it rarely and accept the miss when you do.

Keep tenant boundaries explicit. Shared prefixes should contain nothing that belongs to one customer. Serving engines offer per-request salting to isolate caches in multitenant environments per the vLLM docs, and you should use it rather than trusting that two tenants happen to have identical text.

Measure what you think you measured. Hit rate alone is not enough. Track time to first token, billed input tokens, cache reads and, separately, how often a served value was older than its tolerance. The advice in the Claude docs on automatic versus explicit cache breakpoints is a good design prompt here: automatic caching tracks the end of a growing conversation, explicit breakpoints let you pin exactly what should be stable per the Claude docs.

The payoff shows up in latency numbers you can feel. A recorded example sending a roughly 10,000-token prompt to a 32B model a second time drops time to first token from 4.3 seconds to 0.6 seconds per this benchmark write-up. Streaming speed stays the same, because caching shortens the wait before output, not the output itself.

For teams running agents across several modalities and providers, the operational surface grows quickly, and managing cache behavior per vendor is easy to get wrong. That is where a single endpoint and a specialist who can sit with your team helps: Quanta (https://quantaintelligence.ai/) takes a described app or agent and builds it with you.