A Quarantine Release Checklist: Proving a Chunk Is Safe

Each test below states its query, threshold and expected result. Record the query text and the returned value in the evidence field, not a verdict word.

Test 1: the source hash matches the current page.

• Query: fetch the recorded source URL, extract the main content, normalize it, hash it.
• Threshold: the new hash equals the stored hash exactly.
• Expected result: pass, or mismatch.

A mismatch is not automatically a drop. It means the page changed since the chunk was captured. The reviewer asks the source owner one question: does the chunk's text still reflect the page on the topic it covers? If the owner confirms the text is accurate, the reviewer updates the stored hash and marks test 1 as passed with the owner named in the evidence field. If no owner confirms within the review window, the chunk is dropped on the first hard stop.

This follows the reasoning behind freshness rules in production retrieval, where the operational checklist treats clear freshness rules as a first-class requirement alongside backups and access controls, because a retrieval service can be perfectly healthy while returning stale content (production vector search checklist). A chunk whose source has moved on is stale by definition until someone with authority says otherwise.

Test 2: a sample query returns the chunk in the top five.

• Query: one natural question the chunk should answer, in the words a user would use, run against a shadow collection or staging segment with the chunk re-enabled.
• Threshold: the chunk appears at rank 5 or better.
• Expected result: pass, or not retrieved.

Top five is the cutoff because that is where retrieval context usually stops being useful when results feed a generation step. Coding agents repeat large stable context, and the cache-hit behavior that keeps those bills sane depends on prompts sharing the same long prefix (OpenAI's prompt caching guide, how prompt caching works under the hood). A chunk that never lands near the top of a result list will not reliably enter that stable prefix either. It occupies index space and returns nothing.

Do not test only with the query in the quarantine trigger. That query is adversarial by construction. Write a second, honest query that a real user would type. If the chunk passes the honest query but fails the trigger query, the content is legitimate but retrievable by the wrong path, and that still counts as a pass for test 2 with a note.

Vector search alone is often not enough on a filtered or sharded collection. Practical deployments combine a vector index with payload indexes for the filters that are actually used, and the guidance is to index the field that limits the result set the most (Qdrant's indexing documentation). A chunk that only appears when a filter is wrong is not retrievable in practice.

Test 3: the text contains no stale figures.

• Query: run a small extractor over the chunk text that pulls every number with a unit, date or proper noun implying a changing value (prices, headcounts, benchmark scores, version labels, retention windows), then check each one against the current source.
• Threshold: every extracted figure is either present in the current source or explicitly non-varying (a mathematical constant, a fixed policy date, a statutory deadline).
• Expected result: pass, or a list of figures that cannot be reconciled.

This test catches the most quietly harmful chunks, because a stale figure produces a confidently wrong answer rather than a missing one. The check is mechanical on purpose. A reviewer reading prose will skim past a number; a reviewer comparing a list of numbers to a source will not.

Flag these patterns:

• Currency amounts and percentage figures
• Counts of people, teams or customers
• Dates described as deadlines, holidays or effective dates
• Model or product version labels
• Limits described as "up to" or "no more than"

If a figure cannot be reconciled, the chunk is rewritten by its owner and re-enters review from field one. If the owner is unreachable, the hard stop applies.

Test 4: the original failing query passes.

• Query: the exact query or rule that triggered the quarantine, rerun unchanged.
• Threshold: the chunk is returned and the specific failure condition that fired before no longer reproduces.
• Expected result: pass, or the same failure.

This test prevents a reviewer from declaring victory because a different query now works. The trigger query is fixed evidence, and it must pass on its own terms. If the trigger was a poisoned instruction embedded in the text, the chunk must come back with that instruction removed. If the trigger was a label flip on a classification query, the chunk must not flip the label.

Defenses that combine consistency checks with sanitization do much better than blunt filtering. Research on retrieval-augmented intrusion detection found that a label-embedding consistency check was the strongest contributor to robustness under retrieval-layer attack, and that soft trust demotion outperformed hard filtering (RAG-IDS paper). For a single chunk under review, that translates into: do not just delete the suspect sentence. Confirm that what remains is consistent with its neighbors, then rerun the trigger.

Here is a record a reviewer can check number by number.

Filed record. Chunk id kb-8841-c03, from the page on refund handling for a subscription product. Source hash a41f..., captured 40 days before review. Trigger: the query "how long do refunds take" returned this chunk at rank 2, and the chunk text said refunds are processed "within 3 business days," while the current page says "within 10 business days." Filed by the ingestion monitor as a stale-figure suspect. Reviewer named on the ticket.

Test 1, source hash. Mismatch. The page was edited 12 days after capture, when the refund window changed. The page owner confirms by ticket comment that the current text governs and the chunk is out of date. Hash updated, test 1 recorded as passed on owner confirmation. One hard stop avoided by a single owner reply.

Test 2, top-five retrieval. The reviewer writes the honest query "when will my refund arrive after I cancel." Against the staging segment the chunk returns at rank 9. Nine is outside the top five. Fail.

Test 3, stale figures. The extractor pulls "3 business days," which does not appear in the current source. Fail, same root cause.

Test 4, trigger query. The chunk still returns at rank 2 for "how long do refunds take" and still states 3 business days. The original failure reproduces. Fail.

Decision. Three tests fail, two for the same reason, and the root cause is the same as the hash mismatch: the chunk describes an old process. The reviewer records a drop.

A second reviewer pulls the same record a week later and reaches the same conclusion in under four minutes because every field is filled and every test result has its query attached. That reproducibility is the point. The catch hidden in this example: test 2 would have passed if the reviewer had tested only the trigger query. The honest user query is what exposed that the chunk is not merely stale, it is also poorly ranked even for its own topic, which suggests the surrounding content moved on without it.

The far more dangerous failure is a single poisoned document, not a flood of them. Research on CorruptRAG shows that one injected text can raise attack success over baselines that assume many poisoned documents per query (practical poisoning attacks against RAG). Reviewers should therefore not skip test 4 because "only one chunk was flagged."

The table below resolves every combination of field and test state. Pin it above the review queue.

Two rows deserve emphasis because teams get them wrong. The penultimate row prevents a cheap workaround: a chunk that is correct but unfindable should not be restored just because it is factually accurate, because it will waste a retrieval slot forever. The final row exists because a chunk is a copy of source content moved into a different access-control environment. Source metadata and permissions must be carried through every stage, and copied chunks in a retrieval store need the same access rules as the original system (RAG pipeline best practices for enterprise systems). A chunk that pulls a salary figure out of a restricted document into a general collection is a data incident, whatever its test results say.

Treat the review window as part of the policy too. Predefine how long a chunk may sit in quarantine before it is dropped for lack of a decision, and apply that same window to the owner confirmation in test 1. Data retained beyond its useful window carries cost and exposure without benefit, which is the core argument for any retention policy, and the logic applies to a quarantined chunk with no reviewer assigned (data retention policy guidance). A fixed window converts "someone should look at this" into a decision.

Mechanical checks beat judgment calls when volume rises. Two small scripts carry most of the load.

The first pulls the current source and compares hashes using the pipeline's own normalization:

import hashlib, re, urllib.request

def normalize(text):
    text = re.sub(r"\s+", " ", text)
    return text.strip().lower()

def current_hash(url):
    body = urllib.request.urlopen(url).read().decode("utf-8", "ignore")
    return hashlib.sha256(normalize(body).encode()).hexdigest()

match = current_hash(record["source_url"]) == record["source_hash"]

The second runs the honest query against the shadow collection and returns the rank, so the reviewer records a number rather than an impression. Keep the result JSON next to the ticket, so two reviewers compare evidence instead of opinions.

Two habits keep the queue honest. First, review records oldest first, because age is the strongest predictor of a stale chunk and an old record has had the most chances to mislead a user. Second, never let the person who filed the quarantine also sign the decision. The filer's trigger is evidence, not a verdict, and a separate reviewer is what makes the four tests meaningful.

If your team is reviewing hundreds of chunks a week and the bottleneck is writing honest queries and reconciling figures, that work is a good candidate for automation behind a single API. Quanta Inference routes text and embedding models through one OpenAI-compatible key, enough to generate candidate queries and run the figure extractor without maintaining separate provider integrations. The decision stays with the named reviewer; the mechanical steps get faster.