Replaying the Checker: Fixtures From Promoted and Dropped Rows

The row log is the raw material. Most promotion queues already write enough to reconstruct a decision even if nobody designed the log for testing. The work is selection, not collection.

Start with the decision record. You need the row identifier, the verdict (promoted or dropped), the reason code if the checker emits one, and a timestamp. These four fields are almost always present because operators need them for support. The timestamp matters twice: it dates the decision, and it is the input the staleness rule reads.

Then the evidence the checker actually consumed. The Memory Atlas describes the general shape well: evidence (messages, files, tool output) at the bottom, derived belief in the middle, context at the top. The checker sits in the extraction step, so you need the exact inputs it saw, not the summarized form downstream. If the checker scored a claim against a source document, freeze the extracted claim text, the source document identifier, and any scope keys (user, project, workspace) that filtered the retrieval.

Then the computed signals. Whatever the checker uses to make its decision, and the staleness rule is the obvious candidate, needs to be frozen as an input rather than recomputed. If the staleness rule reads a clock, do not freeze the clock and expect stable results. Freeze the age in days that the rule would have computed at decision time. This is the single most important selection decision in the whole exercise, and it is the one most replay sets get wrong.

A practical field checklist, organized by what each field is for:

Two fields in that table deserve a note. claim_hash is not decoration: it lets you detect when a "fixture" has silently changed because someone edited the text during extraction refactoring. acceptance_time is separate from event time on purpose. The distinction between when a source says something happened and when the application accepted it is what stops a recency-based conflict resolution from being gamed by a fake later timestamp in memory systems that rank by recency. If your checker uses recency anywhere, and most do, you need both.

Selection across the two populations matters more than volume. Promoted rows tell you what the checker accepts; dropped rows tell you what it refuses. A replay set drawn only from promotions will happily pass a change that drops everything new. Sample both populations, and stratify the drops by reason_code so a change that only affects staleness drops cannot hide behind a mass of duplicate drops. A useful starting split is roughly balanced by reason code, not by raw count, because reason codes are where the checker's logic actually branches.

One more selection rule. Deduplicate by claim_hash plus scope. Ten copies of the same claim from one source are one fixture, not ten. Rather than easily repeated observations counting as independent verification, the replay set should reflect the number of distinct claims, or you will over-weight whichever claim happened to be written most often.

A fixture is only useful if it is immutable. Database testing practice is blunt about this: seed data should be deterministic and should not change unless explicitly modified, so that a failing test tells you the code changed, not the data. The same discipline applies here, with one addition: the fixture file should be reviewable by a human in a pull request.

The format can be plain JSON, one file per fixture or one file per reason-code bucket. Plain JSON keeps diffs readable and avoids depending on a test framework's serialization. A minimal fixture looks like this:

{
  "fixture_id": "f-00412",
  "source_row_id": "pq_88123",
  "verdict": "dropped",
  "reason_code": "stale",
  "decided_at": "2026-02-14T09:12:00Z",
  "scope": { "user_id": "u-3391", "workspace_id": "w-7" },
  "claim_text": "Customer prefers email contact.",
  "claim_hash": "b3f1...",
  "source_event_id": "evt_55012",
  "source_type": "email",
  "age_days_at_decision": 41,
  "occurrence_count": 1,
  "dependency_group": null
}

Note what is absent: no clock, no live database reference, no embedding computed at test time. Everything the checker needs is already in the file. If the checker requires an embedding or a similarity score, precompute it once and store it as a field, otherwise the fixture stops being deterministic the moment the embedding model changes.

There is a genuine tradeoff here, and it is worth naming. Precomputing derived values makes the fixture hermetic but means the fixture does not exercise the derivation step. If your change touches the embedding path, precomputed scores will hide it. The resolution is to keep two fixture groups: one hermetic set for checker logic and rules, and a smaller set that recomputes derived values end to end. The hermetic set is the release gate; the second set is a slower, noisier signal you run less often.

Version the fixtures with a schema field and a checksum over the sorted fixture set. When a fixture is intentionally updated, the checksum change appears in the diff and a reviewer must approve it. That single line of friction catches the most common failure mode, which is someone quietly "fixing" a fixture to make a release pass. If that happens often, the fixtures have become a rubber stamp, and the audit value is gone. Logs only prove execution if they are tamper-evident and complete, and a fixture set is a log of decisions in the same sense.

The point of the replay is to test your checker, not the world. That means the thing you run fixtures through must return the same verdict for the same input, always, with no network calls and no sampling. Call it the fake checker.

Two properties define it. First, it is deterministic: identical input produces identical output, including the reason code, not just the promote/drop bit. Second, it is honest about what it fakes. If the real checker calls an external model, the fake checker must return fixed verdicts for fixture inputs rather than calling the model, or you are back to testing the model's mood. The property-based testing literature makes the general point well: randomly generated or nondeterministic inputs are not reproducible and cannot be replayed when a failure appears, which is exactly the failure mode a fake checker exists to prevent.

The cleanest design is a table lookup keyed by fixture_id that returns a verdict and reason code, with an explicit failure for unknown keys. That sounds like it tests nothing, and taken alone it would. The fake checker's job is to pin the expected outputs, not to compute them. The compute comes from running the real checker logic against the frozen inputs, where "real checker logic" means the deterministic parts: the rule evaluator, the threshold comparison, the staleness arithmetic. If some portion of the checker is an LLM call, that portion must be replaced by a stub that returns a fixed value per fixture, and that stub is part of the fixture contract.

There is a subtle version of this that trips people up. Suppose the checker calls a model to decide whether a claim is a preference or a fact, and the rest of the logic is rules. The stub replaces only the classification. The rules around it (does the scope match, is the claim stale, has it been promoted before) should run for real against frozen inputs. This is where the replay earns its keep: it tests the rules that people actually change on Tuesdays at 4pm.

A worked example makes the arithmetic concrete. Take 200 fixtures: 100 promoted, 100 dropped. Run the current checker and confirm 200 of 200 agree with the recorded verdicts. That confirmation step is required, and if it fails, stop. A set that does not replay cleanly against the current code cannot detect future changes. Now change the staleness rule from 30 days to 14. Re-run. Suppose 17 fixtures flip: 15 previously promoted rows now drop as stale, and 2 previously dropped rows are unaffected in the other direction. The rule change is now quantified, and you can decide whether 15 promotions were wrong before or whether the new rule is too tight. Without the replay set, the same change would have gone out with a code review comment saying "seems fine."

The threshold is a policy decision, and it belongs in the repository next to the fixtures. Three common settings:

Full agreement is the right default for a promotion checker. The reason is asymmetric cost. A false promotion persists and is read back later by unrelated queries, which is what makes memory poisoning dangerous rather than merely annoying. A false drop is annoying but recoverable. Given that asymmetry, a release gate that fails on any flip is proportionate. It is also simple enough that nobody argues about its interpretation at 6pm on a Friday.

The allowlist deserves more caution than it usually gets. It is where releases go to die quietly. If you adopt it, require that each entry name the fixture, state the old and new verdict, and give a reason. Rotate the list. A list with 40 entries is not an allowlist, it is the new baseline.

Implementing the gate is small. The important part is that the failure message names every flipped fixture and shows the old and new verdict side by side, because that message is what the engineer will read at the moment they need to decide.

def gate(fixtures, run_checker, allow=()):
    flips = []
    for fx in fixtures:
        got = run_checker(fx)
        if got != (fx["verdict"], fx["reason_code"]):
            flips.append((fx["fixture_id"], fx["verdict"], got))
    unexpected = [f for f in flips if f[0] not in allow]
    if unexpected:
        for fid, was, now in unexpected:
            print(f"{fid}: {was} -> {now}")
        raise SystemExit(f"release blocked: {len(unexpected)} fixtures flipped")

Run it on every change to the checker, the staleness rule, or the scope filter, and run the confirmation pass against unchanged code as part of the same job. A fixture set that passes only on the new code is worthless; the confirmation is what proves the baseline is real.

A replay set ages. New reason codes appear, the checker grows branches, and the fixtures describing the old behavior start to look quaint. The maintenance rule is simple: add a fixture whenever the checker's behavior changes for a reason you can name, and never edit an existing fixture's expected verdict without recording why.

Growth should be bounded. Regression test suites have to fit a time and compute budget, and the research on selecting regression tests from logs exists because unbounded suites stop being run. For a promotion checker, a few hundred well-chosen fixtures will catch most rule changes; thousands add cost and obscure the diff. Prune fixtures whose reason code has been retired, and retire duplicates on the same rule.

Provenance is the other thing that decays. Every fixture should carry source_row_id so a human can trace the decision back into the row log. Without it, a flipped verdict is a number, and with it, it is a story about a specific row promoted on a specific day. The same argument applies to agent memory generally: retrieval without provenance makes correction guesswork. A replay set without provenance makes a blocked release impossible to adjudicate.

Finally, treat the fixture set as a release artifact, not a test directory. It should be tagged with the queue's configuration it was built against, checked into version control, and referenced in the release notes when it changes. That turns a pile of JSON into a record of what your checker believed, which is the thing you will want six months from now when someone asks why a particular row was promoted.

If you are building this alongside a larger agent stack and want the checker, the fixtures and the surrounding services speaking to each other without stitching five providers together, Quanta Inference runs text, image, video, audio and embedding models through one OpenAI-compatible API key, which keeps the frozen-input design intact when a model call is part of the path.