Replay Questions: A Test Set That Proves Memory Kept Facts

The trap in memory evaluation is scoring answers on whether they look right. A general judge, human or model, will accept a fluent answer that paraphrases the allergy line as "the patient has a penicillin sensitivity" and then recommend amoxicillin anyway, because the answer reads fine.

The published numbers on this are blunt. A study of long-running agent memory found that Claude Code's compact prompt on Sonnet 4.6 preserved 53 percent of safety rules after one compaction round and 10 percent after five per the Compaction Cliff paper. Hierarchical truncation, the common summarize-and-retain pattern, preserved only 50 percent of safety constraints across 50 real agent configurations in the same work per the same paper. None of those losses show up as a crash. They show up as an answer that is 90 percent correct.

Strict matching means the scorer checks for the presence of named facts, not overall plausibility. If the required fact is "no NSAIDs, gastritis history", an answer that omits the condition fails even if it names the drug. This is the same discipline retrieval evaluation uses: separate retrieval from generation so a failing score points at an owner as Evidently's RAG guide explains.

Two scoring modes work in practice:

The decision rule: use exact or normalized match for anything a regulated process would quote back, and model-judged matching only for the rest, with the judge prompt and model version pinned in the replay file so a judge change is itself a change you test.

Start from logs, not from imagination. Pull a sample of real user questions from the last few weeks, then keep the ones whose correct answer depends on a fact stored in memory. A sample of 10 to 30 is enough to start; wider coverage comes later.

A worked example. You pull 200 logged questions. Triage:

1. 120 need no memory at all (general how-tos). Drop them.
2. 50 need a fact that is captured in the pre-fold baseline run. Keep 10 of these as canaries.
3. 30 reference facts your current memory system cannot answer. Drop them for now; they are backlog, not regression.
4. The final set is 10 memory-dependent questions plus the 10 canaries, 20 total.

For each kept question, write the expected facts as a list, not a paragraph. One question might read: "Which database should I use for the billing service?" with expected facts ["Postgres 15", "read replica in eu-west-1", "no ORM, raw SQL"]. Anything the answer must contain goes in the list. Anything that is nice to have stays out, because it will create false failures.

Capture the baseline. Run all 20 questions against the current memory configuration with the current model, temperature zero, and the same retrieval settings, and store the full answers. This snapshot is the pre-fold baseline. Every later run is compared to it, not to a moving target. The MemGPT line of work uses exactly this idea of paging between a small working context and persistent external storage, and the deep memory retrieval test it introduced asks a narrow question about one of the five preceding sessions and counts an answer correct only when it matches the gold answer per the alphaXiv summary of the paper. Narrow, checkable questions are the whole point.

A fold is any change that reshapes memory: switching from a rolling window to summarization, enabling compaction, editing the extraction prompt, changing chunk size, or moving to a new model for the memory pipeline.

Run the replay three times per candidate change:

• Before: on the current production configuration, to confirm the baseline still reproduces.
• After: on the candidate configuration, same model, same judge prompt, same retrieval parameters.
• Repeat: run the after pass twice more if any result is borderline, because model outputs at non-zero temperature drift.

What you are looking for is per-question pass and fail, not an average. One question failing on a safety fact is worth more than three flaky failures on a preference. Score each answer against its expected facts, record pass or fail, and keep the raw answers as artifacts so a disagreement can be read rather than argued.

# replay.py (illustrative scoring loop)
import json, re

def normalize(text):
    text = text.lower()
    text = re.sub(r"[^a-z0-9 ]", " ", text)
    return " ".join(t for t in text.split() if t not in {"the", "a", "an"})

def passes(expected, answer):
    ans = normalize(answer)
    return all(normalize(fact) in ans for fact in expected)

replay = json.load(open("replay_set.json"))
results = []
for case in replay["cases"]:
    answer = run_agent(case["question"], case["memory_config"])
    results.append({"id": case["id"], "passed": passes(case["expected_facts"], answer)})

passed = sum(r["passed"] for r in results)
print(f"{passed}/{len(results)} passed")

The exact matcher is deliberately dumb. That is a feature: a dumb scorer cannot drift, and when it fires a false failure you fix the expected fact, which is a reviewed change to a versioned file.

The threshold has to be decided in advance, or it will be negotiated after a disappointing run. The rule the article recommends: a fixed count, not a percentage of a moving set, and no waivers for safety facts.

A defensible starting rule:

In the 20-question example above, a candidate change passes only if all safety, identity and canary questions pass and the preference and episodic groups hit their counts. Anything else blocks the fold. The numbers are a starting policy you calibrate to your users and risk, the same way quality gates are treated in RAG testing, where thresholds in examples are explicitly illustrative and not claims about acceptable production quality per this QA Skills RAG testing guide.

Two design choices follow from this. First, keep the set small enough to run on every fold. SentinelLABS measured a compaction change that cut input tokens 86 percent with no measurable change to aggregate evaluation score per their context engineering writeup, which is exactly the kind of change your replay set should be able to bless or block in minutes. Second, once the set stabilizes, run it in CI on every change to memory code, not only memory policy.

The set earns its keep by catching specific, named ways memory changes lose information.

The first is uniform compaction. When a safety rule and an episodic log compete for the same tokens, both get summarized at the same rate, and only the rule needed exact wording. The Compaction Cliff authors formalize this as Knowledge Triage, classifying each line by type and routing each type through its own retention policy, and report their approach preserves 2 to 4 times more safety rules than the strongest single-shot LLM compactor at every ratio, with 96 percent recall over five rounds per the paper. A replay set with a hard safety-rule question measures whether your system is in that 2 to 4 times band or not.

The second is silent loss at the retrieval step, when the fact exists in storage but does not rank into the prompt. This is the top-k omission failure: the answer exists in the corpus but never reaches the model as Splunk's failure-mode guide lists. A replay question whose expected fact sits far down the ranking will fail, and that failure is a retrieval owner's ticket, not a prompt tweak.

The third is the wrong-kind-of-memory bug: an episodic fact ("the user said they prefer Python on March 4") got stored where the system expects semantic facts ("the user prefers Python"), so retrieval for the general question misses it. Tulving's 1972 distinction is worth keeping straight here: semantic memory is what is true in general, episodic memory is what happened to a specific user at a specific time as Mem0's semantic memory guide explains. A replay set with one question per class catches cross-class confusion.

The fourth is drift from replacing summarization with memory formation or the reverse. Both are legitimate strategies with different trade-offs, and summarization of raw history behaves differently from extracting facts into memory per Mem0's chat history guide. The replay set does not care which you choose. It only records whether the required facts survived the switch.

If you are deciding which memory strategy to adopt in the first place, running the same replay set against two candidate configurations, with the same questions and expected facts, gives you a comparison you can show to someone. For teams that need several model options under one interface while running these comparisons, Quanta Inference exposes text, image, video, audio and embedding models through a single OpenAI-compatible API key, which keeps the replay harness to one client.

A replay set decays in three ways, and each has a cheap fix.

Questions go stale when the underlying fact changes. The customer renewed, the plan upgraded, the schema changed. Version the file, and when a fact changes, update the expected fact in the same commit that changes the stored memory, so the diff shows both.

The set overfits when you tune the memory system against the same 10 questions repeatedly. The practical answer is a holdout: keep a second set of questions you do not look at during development, and run it only at release. The RAG evaluation literature makes the same point, recommending test sets built from real user logs with ground-truth annotations plus synthetic edge cases, so performance regressions are tracked rather than discovered in production per Splunk's guide.

Judges drift when a model-judged fact is scored by a judge whose model or prompt changed. Pin both in the replay file, and treat a judge change as a fold: run the old and new judge on the baseline answers and reconcile disagreements before trusting new results.

One more discipline: a replay pass is not a quality claim. It says the required facts survived. It does not say the answer was helpful, well formatted or fast. Keep it narrow and pair it with whatever session-level or user-facing checks you already run, as Evidently's guide separates retrieval evaluation from generation evaluation for the same reason per the guide.

If you would rather not build the harness from scratch, a specialist can set up the replay set, the scorer and the gate with you through Quanta, which handles site, app and tooling builds with a person on the other end rather than a prompt box.