Triaging a Failed Replay: Fold, Retrieval or Prompt

Each artifact answers a different question, and confusing the questions is what produces wasted debugging time.

The folded summary answers: did the system carry the fact forward at all? It is the state after compaction, recursive summarization or eviction. If the runtime uses paging between a small in-context working area and external storage, the summary is the main context, the part the model can currently read (MemGPT on arXiv). If your fact is not there, check whether it was ever written to external storage. MemGPT-style systems move data between tiers through explicit store, retrieve, summarize and update operations, and a fact that was never stored cannot be retrieved later (hierarchical memory overview).

The retrieved chunks answer: did search surface the evidence? This is a pure information retrieval question, and it has standard metrics. Recall at k asks whether a known relevant document appears in the top k results, and precision at k asks how much of what you returned was relevant (Evidently AI RAG evaluation guide). For triage you mostly care about recall: is the chunk with the fact anywhere in the set that reached the model?

The answer answers: did the model use what it was given? Groundedness and faithfulness metrics exist precisely for this, and they compare claims in the output against the retrieved context rather than against the world (Confident AI RAG metrics).

A short checklist for reading an artifact:

Run this table against every red replay. The first row that fails is your class.

Fold loss means the fact is missing from the folded summary. There are two sub-cases, and they have different fixes.

The first sub-case is that the summarizer never saw the fact. Maybe it was in a tool output that got cleared early, or in a turn that fell outside the window before the fact mattered. Harnesses differ here: some use a sliding window that keeps only the most recent turns, some use an LLM summarization pass, some clear large tool results while keeping the reasoning about them, and some layer these strategies (memory compaction in agent harnesses). A rolling window that drops older turns will lose any fact that was stated once early and needed later, and no amount of prompt engineering downstream will restore it.

The second sub-case is that the summarizer saw the fact and dropped or paraphrased it away. This is the compaction cliff in miniature. The failure is not random: it is uniform treatment of heterogeneous knowledge. A safety rule needs exact wording to stay enforceable, while a background observation tolerates heavy paraphrase. When both compete for the same token budget under a single summary policy, the rule loses just as easily as the trivia (the compaction study).

Fixes for fold loss:

1. Add an explicit, writable working-context block for facts that must never be summarized, and have the runtime write to it on pressure warnings rather than on eviction. MemGPT pages this way: the queue manager alerts the model at a warning threshold so it can save important details before the flush threshold evicts anything (alphaXiv MemGPT overview).
2. Classify lines by type before compaction and give each type its own retention policy. The compaction study calls this Knowledge Triage, and reports that typed compaction preserved two to four times more safety rules than the strongest single-shot LLM compactor at every ratio, with 96 percent recall over five rounds.
3. Pin critical facts outside the summarizer's scope entirely, so a summary round cannot touch them.

The diagnostic signal that tells you which sub-case you are in: check the raw transcript for the fact. If it appears before the fold and not after, the summarizer dropped it. If it never appeared, the conversation never carried it and you have a missing-input bug rather than a fold bug.

Retrieval loss means the fact is not in the summary and not in the retrieved chunks, but it does exist somewhere in your source material. This is the most common class in document-heavy systems, and it splits into three causes.

The first cause is query construction. The user's wording and the document's wording do not overlap, so the embedding search misses. Experienced debuggers fix this before touching anything else. In a study of engineers debugging RAG pipelines, the observed pattern was to enlarge chunks when the retrieved information looked insufficient, then to add a query rewriting step when the retrieved documents were right but the query was ambiguous (RAG Without the Lag). That two-step sequence, widen first, rewrite second, is a good default order.

The second cause is chunk boundaries. A sentence that says "the limit is 40, except in cases covered by section 12" is unusable if the chunk ends after "40." Chunking too large can bury a pertinent fact in noise, and chunking too fine can separate a fact from the qualifier that makes it actionable (Snorkel AI on RAG failure modes). Overlap helps, but structural chunking by section is usually better than fixed token windows.

The third cause is coverage and ranking. The fact may not be in the index at all, or it may be ranked below the top k cutoff. Top-k omission is a named failure mode in RAG evaluation: the definitive answer exists in the collection but does not rank highly enough to reach the context (Splunk on RAG failure modes). Coverage gaps produce the same symptom through a different door, and they are worth auditing separately because the fix is ingestion, not ranking.

A decision rule for retrieval loss: first check whether the source document exists in the index. If not, you have a coverage problem, and no retriever tuning helps. If it does, check whether the exact fact is in any chunk; if not, you have a chunking problem. If the chunk exists and still does not rank, raise k, add a reranker, or rewrite the query. Only then does embedding model choice come up.

Prompt loss means the fact is present in the retrieved chunks, correctly, and the answer is still wrong or silent. The model had what it needed and did not use it. This is the class prompt edits actually fix, and it is the one people habitually assume they have.

Three patterns recur. The first is synthesis drift: the model retrieves three chunks that are each correct, and combines them into a claim none of them supports. Inventory numbers, dates and entity names are the usual casualties (Towards AI on RAG in production). The second is silent substitution: the model fills the gap with prior knowledge because the instructions did not forbid it. The third is dilution: the fact is buried in the middle of a long context, and the model attends to the edges.

Fixes for prompt loss:

• Require a quote-then-answer structure, so the model must reproduce the supporting sentence before it speaks.
• State an explicit abstain rule: if the context does not contain the answer, say so. This is the behavior a faithfulness metric scores, and it is cheap to enforce in instructions.
• Put the most relevant chunk first or last rather than in the middle, and deduplicate near-identical chunks so one fact does not crowd out another.
• Add a citation requirement per claim, so an unsupported claim becomes visible in the output rather than buried in fluent prose.

Suppose your replay suite has 200 cases, and 20 go red on a release candidate. You resist the urge to edit the prompt and run triage artifact by artifact.

You find the fact absent from the folded summary in 9 of the 20. Of those, 6 have the fact in the raw transcript, so the summarizer dropped it: fold loss. The other 3 never carried the fact at all, which points at an upstream tool call that did not run. You check the retrieved chunks for the remaining 11 and find the fact missing in 7. Of those 7, 2 have the source document absent from the index entirely (coverage), 3 have the fact split across two chunks by a boundary (chunking), and 2 have the right chunk ranked below the cutoff (ranking). That leaves 4 cases with the fact in context and a wrong answer: prompt loss.

So the distribution is 9 fold, 7 retrieval, 4 prompt. Had you started with prompt edits, you would have been working on 20 percent of the problem. The arithmetic generalizes: the upstream classes are usually the majority, which is why the prompt editor is the most expensive place to start.

There is a second payoff. Because retrieval and generation fail independently, evaluating them separately with their own metrics is what makes a failing score point at an owner rather than a mystery (component-level RAG evaluation). Fold, retrieval and prompt map cleanly onto memory engineering, search engineering and application engineering. Assign the ticket accordingly.

Triage goes fast only if the artifacts are already in hand. Four instrumentation habits pay for themselves.

Log the artifact set with every run: summary text, chunk IDs with scores and ranks, and the answer with its citations. Send the same request to two configurations side by side when you can, because the contrast makes memory differences visible in a way single runs do not; this is how OpenAI's own memory demo compares trimming and compaction settings (memory strategies demo). Record token counts per artifact, since compaction that cuts input tokens heavily can be nearly free in quality: one evaluation of native compaction reported roughly 86 percent fewer input tokens with no measurable change to the aggregate score in a malware analysis harness (SentinelLABS). And keep a labeled set of queries with known answers, because without ground truth you cannot tell recall failure from relevance judgment.

One caution: keep the fold policy and the retrieval policy in separate configuration. When memory compaction and search tuning live in the same file, a change meant to fix fold loss quietly alters what gets retrieved, and your next triage is contaminated.

If you are running replays across several model families, one API key through Quanta Inference keeps the routing constant while you isolate the memory and retrieval variables.