Testing a Memory Write Policy: Replay, Fixtures and Drift Checks

A fixture is a conversation plus the metadata the policy will see: user id, tenant, session id, timestamp, channel, and any scope tags your policy reads. Keep it as plain data in the repo, not in a database, so a reviewer can read it in a pull request.

fixtures/
  pref_update/
    meta.json          # user, tenant, session, tz
    turns.jsonl        # one turn per line
    expected.json      # snapshot of memory rows after replay
  correction/
  contradiction/
  pii_rejection/
  chit_chat_no_write/

What to include matters more than how much. Six fixture families cover most write-policy behavior:

Aim for roughly 8 to 15 fixtures with 3 to 8 turns each. That is small enough to run in seconds and large enough that a careless change to a similarity threshold or a dedup rule will break something.

Be deliberate about turn metadata. If your policy uses recency to decide which of two conflicting facts wins, put explicit timestamps in the fixture rather than relying on wall-clock time during replay. Fix the clock in the harness. Otherwise the same fixture passes on Monday and fails on Friday, and you will spend an afternoon debugging the calendar.

Write the expected rows by hand for the first pass. Do not generate them from the current policy, because whatever bug exists will be baked into the snapshot. Hand-writing five fixtures forces you to state what the policy should do, which is the whole point of the exercise. As one engineering guide puts it, memory is a pipeline covering what to remember, how to inject it, and how to merge session notes without stale data, and evaluation and safety is an explicit layer of that pipeline per this framework-agnostic memory guide.

The harness feeds turns one at a time, in order, into the real write policy against a throwaway store, then dumps the resulting rows. The policy function should take a single turn and the current memory state and return a list of operations. That signature is what makes it testable.

def replay(fixture, policy, store):
    meta = fixture["meta"]           # frozen clock, user, tenant
    ops = []
    for turn in fixture["turns"]:
        result = policy.decide(turn=turn, meta=meta, memory=store.snapshot())
        for op in result.operations:
            store.apply(op)          # insert | update | delete
            ops.append(op)
    return ops, store.snapshot()

Use an in-memory store that implements the same interface as production: insert, update, delete, and a scoped read. If your production store is a vector database with metadata filters, the fake should enforce the same filter semantics, or your scope-isolation fixture will pass for the wrong reason. A practical rule: for anything a user might later retrieve on, memory should be a durable record chosen deliberately, not a transcript as JetBrains' guide frames it.

Determinism is the hard part. If your policy calls an LLM to decide what to store, that call is a source of nondeterminism. Three options, in order of preference for testing:

1. Record and replay model responses. Capture the extractor output once per fixture turn, commit it as a JSON file, and have the harness return the recorded response instead of calling the model. The policy logic is then fully deterministic.
2. Stub the extractor. Replace the model call with a hand-written extraction for each fixture turn. Fastest, but it stops testing the prompt.
3. Allow limited live calls with a temperature of zero and a strict response schema. Keep this for a nightly job, not the pull-request gate.

A mix works well: recorded responses in CI, a nightly run against live models to catch extractor drift. When you need a live model for the nightly job, routing text and embedding calls through one OpenAI-compatible endpoint keeps the harness code short; Quanta Inference (https://quantaintelligence.ai/inference) exposes text, embedding and other model types behind a single key, which means the same harness can swap extractor models without touching its transport layer.

Record the operation log alongside the final rows. Two policies can produce the same final state through very different paths, and a delete-then-insert is not the same as an update when history matters. Assert on the final rows by default and on the operation log for fixtures where the path is the point, such as the contradiction case.

Store expected.json as a normalized list of rows sorted by a stable key, with volatile fields excluded: created timestamps, row ids, and embedding vectors. Compare on meaning, not on bytes.

[
  {
    "scope": "user:42",
    "kind": "preference",
    "key": "deploy_day",
    "value": "Friday",
    "status": "active",
    "source_turn": 3
  }
]

Two rules keep snapshots useful. First, drop anything the policy is allowed to change freely (vector values, TTL counters) from the snapshot, or every run will fail. Second, keep a source_turn field so a reviewer can see which message produced the row. When a diff appears, that field answers the first question anyone asks.

Treat a snapshot failure as a review prompt, not an automatic pass. The workflow is: the test fails, you read the diff, you decide whether the new rows are better or worse, and only then do you update the snapshot. If you regenerate snapshots without reading, you have built a test that documents behavior rather than constraining it. A useful habit is to require snapshot updates in a separate commit with a one-line justification, so the decision is visible in the history.

Normalize aggressively for order and format, not for content. Sort keys, sort rows by scope and key, round any numeric scores to a fixed precision, and strip whitespace. Never strip a value field, even if it looks trivial: the trivial field is usually the one that changes when someone edits the extraction prompt.

Snapshots catch changes you made. Drift checks catch changes you did not make: a model upstream of your policy gets updated, an embedding model is swapped, a tokenizer changes, and the extractor starts emitting slightly different facts. The write policy is unchanged and the rows are not.

Two schedules handle this. On pull requests, run the full replay with recorded responses. On a nightly or weekly schedule, run the same fixtures against the live extractor and compare its output to the recorded responses before the policy even runs. That truncates the debugging problem: you learn whether the drift is in extraction or in policy logic, which are different teams and different fixes.

Measure drift as a number, not a feeling. For each fixture, compute the share of turns where the extracted candidate facts differ from the recording (exact match on key and normalized value), then report a single percentage across the suite. If extraction agreement drops from 100% to, say, 80% overnight, you have a concrete trigger to investigate. The threshold is yours to set per fixture family, and rejection fixtures should be held to a stricter bar, since a false positive there is a compliance problem.

Also check the retrieval side after write changes. A fixture that writes the correct row can still break the product if the new row no longer matches a query the agent relies on, which is the classic evidence-aggregation problem where related observations end up scattered across records per the Infini Memory discussion of fragmentation. Keep a small set of retrieval assertions (query plus expected top row) in the same suite, so a write change that breaks recall fails in the same run.

Decision rule: if a proposed policy change improves a fixture family you care about but breaks any rejection or scope-isolation fixture, it does not merge. Correctness on refusals and isolation outranks coverage of new memory.

Rows are necessary but not sufficient. Add these assertions to the harness, each cheap to compute:

• Idempotence. Replay the same turn twice. The second pass should produce no new rows or at most a bumped counter. Duplicate rows from a retried request are one of the most common production bugs in memory systems.
• Scope isolation. A fact written under user A must never appear in a read scoped to user B, even when the text is identical.
• Deletion completeness. When a user requests deletion, every row tagged with that user goes, including summarized or consolidated rows. Tie this to your retention rules.
• Latency budget. Time the replay. A policy that makes three sequential model calls per turn may be correct and too slow.
• Token cost per turn. Estimate tokens for the extraction prompt and the candidate facts. It is the number that decides whether you can afford to run the extractor on every message.

On that last point, write-level tests are also where you find waste. If chit-chat fixtures trigger an extraction call that returns nothing, that is a signal to add a cheap pre-filter. A keyword or embedding-similarity screen before the extractor can cut calls substantially with no change to stored rows, and the replay proves it.

Keep the suite honest by pruning. Every quarter, delete fixtures that no longer reflect real usage and add ones drawn from recent incidents. A thousand fixtures nobody reads is worse than twelve that map to bugs you actually shipped. And keep the suite runnable by a single command with no network access, so it can sit in the pull-request gate where it does the most good.