Retrieval Budgets for Agent Memory: How Much to Fetch Per Turn

A retrieval budget has two dimensions, and teams usually tune only one of them.

The row budget (top-k) is how many memory entries you pull per call. It controls recall: the chance the one useful fact is in the set.

The token budget is how many tokens those rows may occupy in the assembled prompt. It controls whether the result fits alongside everything else.

The two are related but not identical, because rows vary wildly in length. A distilled preference ("prefers snake_case") might be 15 tokens; a stored episode summary might be 300. Ten rows of the first kind cost 150 tokens; ten rows of the second cost 3,000.

Here is a worked example of how the budget competes. The component estimates below follow a published account of a multi-turn agent's token usage (context budget breakdown):

That baseline is roughly 15% of a 100K window (context budget breakdown). The good news is that a modest read budget leaves plenty of headroom. The bad news is what happens without one: the same source notes that appending everything fails in predictable ways, with budget exhaustion, latency spikes and degraded coherence as early history crowds the signal (Mem0 context engineering).

So the budget question is not "can the window fit it?" It is "does this row earn its slice against the 12,500 tokens already committed?" A read budget of 1,500 to 2,500 tokens is 10% to 17% of that baseline, which is a defensible ceiling for a per-turn memory injection.

Top-k looks like a knob you can set by taste. It is not. It has a measurable shape: quality rises as you add rows, flattens, then can fall as irrelevant rows dilute attention. Your job is to find the flatten point for your workload, then set k just above it.

The procedure:

1. Build a fixed evaluation set. Take 40 to 100 real questions from your agent's traffic. For each, write the gold answer and the specific memory row or rows that should be retrieved. This is the only way to separate "retrieval missed" from "model ignored it."
2. Log every retrieval call. Record the query, the returned rows with their IDs and scores, the row and token counts, retrieval latency in milliseconds, and the final answer. Trace the full request, not just the search step (latency budgets).
3. Run the sweep. For each k in [1, 2, 4, 8, 12, 20, 30], recompute answers on the eval set. Judge each answer against the gold answer, either by rubric or by a second model with the rubric in its prompt.
4. Plot two lines. Quality (share of judged-correct answers) against k, and p95 retrieval latency against k. The crossing point tells you what extra rows cost you.
5. Pick k at the knee. The knee is where the quality curve's slope goes to about zero. Everything past it is latency and tokens spent for nothing.

A typical curve in a personalization agent looks like this (illustrative, from a simulated run in the manner of the benchmark harness described by BudgetMem):

Read it this way. Going from 4 to 8 rows buys 5 points of quality for about 700 tokens. Going from 8 to 20 buys 1 point for roughly 2,100 tokens and doubles the tail latency. Going past 20 adds cost and can subtract quality. That is the argument for a budget of 8 to 12 rows: it sits on the flat part of the curve, before the tail cost.

The larger lesson from the research is that the same performance can be reached on very different budgets: a runtime routing framework reports that explicit budget tiers (low, mid, high) produce meaningfully different cost-quality trade-offs across memory benchmarks, and that query-aware routing beats a single fixed setting (BudgetMem). You do not need a learned router to apply that finding. You do need to measure more than one tier.

One caution on latency: do not sum stage percentiles. A slow vector search and a slow reranker may occur on different requests, so add up p95 figures and you will overstate the end-to-end cost (latency budgets). Trace whole requests.

The budget is a ceiling, not an allocation. Different retrieval strategies use the same rows with different efficiency, so your strategy choice changes the right k. The comparison below draws on the strategy table published by Mem0:

Two implications for budgeting. First, a hybrid pipeline can fetch a wide candidate set (20 is common) while keeping the injected count small, because the reranker reorders before anything enters the prompt (Mem0 retrieval strategies). Your row budget applies to the post-rerank set, not the candidate set. Second, the reranker itself may add 250 to 450 ms over a simple vector search, so the latency half of your budget has to account for it.

This is also where you decide whether retrieval happens in one shot or across several rounds. An agentic, iterative memory read (the model calls memory multiple times before answering) can reconstruct answers that a single fetch cannot, and topic-structured memory built around iterative inspection scored 64.7% overall on MemoryAgentBench (Infini Memory). But each extra round spends another read budget. If you allow three rounds, divide the per-turn budget across them (for example 500 tokens per round) or the ceiling silently triples.

The read budget is a promise about how much useful memory exists. The write policy decides whether that promise can be kept. If the write side stores hundreds of near-duplicate observations, a top-10 read returns ten versions of the same fact. If it stores almost nothing, a top-10 read returns noise.

The documented failure modes are structural, and the report that introduced them is worth reading alongside your retrieval logs (Infini Memory): memory fragmentation, where evidence about one user or event scatters across many small records; memory conflict, where an append-only store keeps an old fact next to the new one and both are retrievable; and difficulty revising stale facts over time. The authors address these by grouping observations into topic documents and consolidating them periodically, with iterative retrieval at inference time (Infini Memory).

OS-inspired designs tackle the same problem on the write side. MemGPT keeps a small main context, pages out content when the prompt passes a warning threshold, and flushes and summarizes at a higher threshold, so what lands in external memory is a compressed summary rather than raw chatter (alphaXiv overview).

For a read budget, the practical settings are:

• Write threshold. Store a memory only if it would change an answer later. Preferences, decisions, durable facts, constraints.
• Deduplication and supersession. When the same key is written again, mark the older row superseded at write time so it cannot occupy a read slot.
• Consolidation cadence. Merge session notes into long-term records on a schedule, not on every turn.
• Recency field. Any recency-based retrieval needs a trustworthy timestamp on every row.
• Quality gates. Keep a small eval set that runs after every write-policy change, not just after retrieval changes.

The write side also determines what "nothing relevant in N turns" means. If the write policy consolidates a session into memory at session end, a fresh session genuinely has nothing recent and the retrieval skip rule below should fire until the consolidation lands. If it writes per turn, recent memory always exists, and the skip rule mainly protects against fetching for turns that do not need it.

The cheapest retrieval is the one you do not make. A skip rule is not laziness; it is how you keep the average cost of a session down so you can afford a real read on the turns that matter.

Four conditions are worth encoding, roughly in order of how often they fire:

The answer is in the current context. If the relevant fact appears in the last few turns or in a tool result already in the window, retrieving it again burns budget for a duplicate. A cheap check (exact or fuzzy match of the query against the current working context) catches most such turns.

The write policy has nothing recent or relevant. If no memory covering this topic was written in the last N turns, retrieval returns low-score rows that are more likely to mislead than help. Score a skip when the top candidate is below your threshold.

The question is a task step, not a recall request. Tool calls, formatting requests and confirmations rarely need memory. Classify intent first, then route only the recall-shaped queries to retrieval.

The request is latency-critical. A voice turn has a much tighter budget than a background research run (latency budgets). For the first output in a live interaction, prefer no retrieval over a slow one, and let later turns retrieve if needed.

A simple, testable implementation:

def read_memory(query, ctx, budget):
    if budget.tokens_left < 400:
        return []                       # budget exhausted for this turn
    if query_matches_working_context(query, ctx.window):
        return []                       # answer already visible
    if not ctx.recent_writes(within_turns=budget.n_turns):
        return []                       # nothing fresh to find
    rows = retrieve(query, k=budget.candidate_k)
    rows = [r for r in rows if r.score >= budget.min_score]
    rows = rows[:budget.max_rows]
    rows = fit_to_tokens(rows, budget.max_tokens)
    return rows

Skipping has to be measured too. Add a "skipped" flag to your retrieval logs, and in your eval set include questions that should return nothing. If skipped turns show up in low-quality answers, your skip rule is too aggressive; if skip share is under a few percent, it is not firing at all and you are paying for retrievals you did not need.

A useful framing for the whole decision comes from the memory architecture literature: retrieval decides what the agent knows in the moment of a turn (Mem0 retrieval strategies). The decision rule is therefore: retrieve when there is a plausible row you would bet on, skip when there is not.

Putting it together, here is a rule you can write on the wall:

Retrieve by default at k equal to the knee you measured (start at 8 to 12 rows, 1,500 to 2,500 tokens). Skip when the answer is in the current context, when nothing relevant was written in the last N turns, when the top candidate scores below threshold, or when the turn is latency-critical. Spend extra rounds only on turns that are visibly recall-heavy, and divide the budget across them.

Rollout order matters more than the specific numbers:

1. Log before you tune. Two weeks of retrieval traces with row counts, token counts, latency and judged answers gives you the curve.
2. Fix the write side first if duplicates dominate. A deduplication pass often improves quality more than any k change.
3. Set the budget as configuration, not code. Per-agent budgets let a recall-heavy agent keep 12 rows while a fast one uses 5.
4. Re-run the sweep after every write-policy change, because the two are coupled.
5. Watch the tail. Track p95 latency and skip share weekly; both drift as memory grows (latency budgets).

Two reseeding checks prove it is working. First, the knee should be stable across two eval runs. Second, removing the budget entirely should measurably worsen either latency or judged quality; if it does not, you are under-reading and can raise k.

If you are running several agent types and want one place to route model calls, embeddings and rerankers under a single key with prepaid dollar pricing, Quanta Inference offers an OpenAI-compatible API for text, embedding and rerank workloads at https://quantaintelligence.ai/inference. For teams that want the whole memory pipeline (write policy, eval harness, budget tuning) set up with you, a consulting call is the faster path.