When Agents Run Out of Context: Memory Stores and Failure Modes

Give each failure mode a signature so an on-call engineer can recognize it at 3 a.m.

Silent wrong answers. The store returns success with empty or weakly relevant results. The agent answers with confidence because nothing in the prompt says context is missing. Research on vector retrieval quantifies how quickly quality degrades with scale: one study cited in an analysis of retrieval fragility found vector search accuracy dropping by up to 12% when the corpus grew to 100,000 pages, while a non-vector approach lost about 2% (The Hidden Fragility of AI Agents). A 12% miss rate on relevant information is not a cosmetic issue for an agent that decides things.

Stale memory. The store returns old facts because a write landed late or a revision never propagated. A benchmark built for exactly this problem, STALE, has 400 expert-validated conflict scenarios and found that even the best evaluated model reached only 55.2% overall accuracy at detecting and acting on outdated state (STALE benchmark). Retrieving updated evidence and acting on it are different skills, and the gap between them is where bad decisions live.

Retry storms and corrupted writes. The store is slow, every worker retries, and retries multiply the load that caused the slowness. Worse, some agents respond to a slow read by writing their own summary back into memory, which duplicates records and pollutes future retrieval. Default retry behavior plus agent autonomy is a compounding-failure generator.

Empty-context action. The read fails outright, the framework swallows the exception and substitutes an empty string, and the agent proceeds. If the task is "refund based on order history," an empty history may look like "no prior refunds" and the agent issues a duplicate.

One reason teams assume memory is safe is that the access path looks modern. Workload identity federation replaces static service account keys with short-lived tokens exchanged through OIDC, so a workload proves who it is without a secret sitting in a config file, as described in this walkthrough of replacing GCP service account keys and in a broader federated workload identity guide. AWS applies the same pattern to CI pipelines, where the workflow presents a signed token, AWS verifies it, and the credentials expire in minutes (identity federation on AWS). That eliminates secret sprawl and manual rotation.

It does not give the store an emergency mode. A token exchange proves the caller is the pipeline. It says nothing about whether the index behind the endpoint is populated, current, or answering. Credential hygiene and availability are different controls, and conflating them is how a team ends up with impeccable secret management and no answer for what to do when retrieval returns nothing.

The same logic applies to how agents handle secrets in general. AWS advises AI coding agents not to call get-secret-value directly and receive plaintext, and to go through guarded retrieval instead (AWS secrets guidance for agents). If you accept that a secret should be read through a guarded, time-bounded path, accept the same for memory: the agent's read should be time-bounded, freshness-checked, and impossible to confuse with a real answer.

The useful test is not "did the read succeed" but "what happens if this answer is wrong." Sort every agent turn into one of three classes before you write the memory layer.

• Read-only, low stakes (search, summarize, draft). Degrade. Answer with the partial context you have, state plainly that memory was unavailable, and offer to retry. A degraded answer that names the missing dependency is better than a stall.
• Answer depends on personal or account state (balance, order history, prior agreement). Fail closed on any statement of fact. If the store times out or returns zero records, the agent says it cannot confirm and stops.
• Consequential write or irreversible action (payments, permissions, deletions, outbound messages). Fail closed, full stop. No write the agent cannot verify, and no action based on memory it could not read.

A worked example makes the arithmetic concrete. Say the store's p99 read latency is 400 ms and you allow four retries with a 2 s timeout. Under a load spike where the store responds at 3 s, each turn burns 2 s of timeout, up to roughly 8 s of retries, and then either fails or proceeds with nothing, while other workers retry into the same overloaded store. Reduce to one retry with a 1 s timeout and a circuit breaker at a 20% error rate, and the worst case per turn drops to about 2 s, the breaker sheds load after the threshold, and the agent gets a clean "context unavailable" signal. That signal is what your code needs to choose degradation versus refusal.

CONTEXT_TIMEOUT_S = 1.0
MAX_FRESHNESS_S = 3600  # how old a record may be for state questions

def read_memory(query, *, allow_stale=False):
    try:
        hits = store.search(query, timeout=CONTEXT_TIMEOUT_S)
    except TimeoutError:
        return {"status": "unavailable", "hits": []}
    if not hits:
        return {"status": "empty", "hits": []}
    age = now() - max(h.updated_at for h in hits)
    if age > MAX_FRESHNESS_S and not allow_stale:
        return {"status": "stale", "hits": hits, "age_s": age}
    return {"status": "ok", "hits": hits}

The three statuses map directly to the classes above: unavailable and empty trigger refusal on state questions and writes, stale allows a read-only answer only with the age stated to the user, and ok proceeds normally. Never map any of these to an empty string.

Treat the store's availability as an SLI in its own right. Useful measurements, in the framing a federated workload identity guide uses for token services, include successful reads per minute, read latency percentiles, the share of reads returning zero results, and the age distribution of the records returned. That last one is the metric most teams are missing: if the median answer age on a state question climbs past your TTL, the store is quietly serving history.

A minimum checklist, borrowed from the structure of break glass procedures because memory needs the same treatment:

The tested no-memory mode is the item most likely to be skipped and the one that pays off. Run it in staging on a schedule. If the agent cannot complete its core task without memory, you learn that now, not during an outage. If it can, you have a fallback and you know what quality to expect from it.

Read failures get the attention; write failures cause the lasting damage. Three rules cover most of it.

First, version records rather than overwriting them. When a new fact arrives, write it with a timestamp and a revision link, and let reads resolve to the most recent valid revision. This is roughly how the STALE authors' prototype, CUPMem, approaches the problem, with explicit state consolidation and propagation-aware search at write time (STALE benchmark). Second, deduplicate on write using a stable key, so a retried write does not create two near-identical memories that retrieval later alternates between. Third, block agent-initiated writes while the store is unhealthy. An agent that "helps" by summarizing into a sick store is creating the stale records it will later trust.

Forgetfulness should be a feature with a policy attached, not an accident. That means TTLs per memory type, freshness metadata on every record, and a distinction between facts that decay (preferences, locations, task state) and facts that do not (a signed contract's terms). The design guide that frames forgetting this way treats the three decisions of what to store, when to load, and when to forget as the whole design problem (AI Agent Memory Design Guide). Most teams consciously make only the first.

Full unavailability is the easiest case to handle well if you prepared. The agent should stop consequential tools, return a response that names the dependency and what was last known, and offer a path forward that does not require memory: a human handoff or a retry later. What it must not do is proceed on the model's priors.

There is also a retrieval-design decision hiding here. Files-and-agent retrieval can beat vector retrieval on answer quality for small, complex corpora, scoring 8.4 versus 6.4 on correctness in one benchmark, but it cost about 3.8 seconds in latency (7.36 s versus 11.17 s), and the advantage inverted as the corpus grew, with RAG pulling ahead on speed and slightly on correctness at scale (Did Agents Kill Vector Search?). The same analysis reports retrieval accounting for roughly 41% of end-to-end RAG latency in typical workloads (same analysis). Read that as a resilience argument, not just a performance one: if retrieval is a large share of your latency, it is a large share of what breaks, and it needs the same timeout discipline as any other critical call. A hybrid design that tolerates losing either path is more robust than a single retrieval mechanism you cannot substitute.

If you are still choosing where memory lives and how the agent reaches it, this is the kind of architecture decision worth talking through with someone who has shipped it. Quanta Consulting offers a free 30-minute AI discovery call (book a call): bring your current retrieval path and failure plan, and leave with a recommendation.