A Quarantine Table for Suspect Chunks: Schema and Retention

The short answer names the columns. Here is why each one exists.

Two design choices are worth defending. First, store score and threshold together. Storing only the reason code loses the information you need to tune the filter: a chunk that scored just under the threshold is a very different case from one that scored at the floor, and without both numbers you cannot tell. Second, store both chunk_offset and the character span. Chunk offsets survive small changes in chunking strategy, character spans let you reproduce the exact text even after the chunker is re-tuned.

On PostgreSQL, a partial index on live states keeps the queue index small:

CREATE TABLE quarantine_chunk (
  id              bigserial PRIMARY KEY,
  source_id       text NOT NULL,
  chunk_offset    integer NOT NULL,
  char_start      integer NOT NULL,
  char_end        integer NOT NULL,
  chunk_text      text,
  reason_code     text NOT NULL,
  score           numeric,
  threshold       numeric,
  filter_version  text NOT NULL,
  state           text NOT NULL DEFAULT 'held'
                  CHECK (state IN ('held','in_review','restored','dropped')),
  held_at         timestamptz NOT NULL DEFAULT now(),
  expires_at      timestamptz NOT NULL,
  reviewer_id     text,
  decided_at      timestamptz
);

CREATE INDEX idx_quarantine_queue
  ON quarantine_chunk (state, expires_at)
  WHERE state IN ('held','in_review');

CREATE INDEX idx_quarantine_source
  ON quarantine_chunk (source_id, chunk_offset);

Keeping chunk_text nullable is what lets the retention job redact it without losing the audit trail. The column becomes NULL while the row and its metadata stay in place.

The queue is "what should a reviewer look at next", ordered so that the most urgent items surface first. Urgency here means nearest expiry, then age.

SELECT id, source_id, reason_code, score, threshold,
       held_at, expires_at, state
FROM quarantine_chunk
WHERE state IN ('held','in_review')
ORDER BY expires_at ASC, held_at ASC
LIMIT 50;

The partial index on (state, expires_at) matches that WHERE and ORDER BY exactly, so the planner reads an already ordered range. Filtering by reason code is the second most common access pattern (a reviewer who only handles injection flags), and a composite index on (reason_code, state, expires_at) covers it. Resist the urge to add more indexes up front; every index is a write cost on a table that receives a row per rejected chunk.

Claiming work needs to be atomic or two reviewers will open the same row. A single statement with FOR UPDATE SKIP LOCKED does it:

UPDATE quarantine_chunk
SET state = 'in_review', reviewer_id = $1
WHERE id = (
  SELECT id FROM quarantine_chunk
  WHERE state = 'held'
  ORDER BY expires_at ASC
  FOR UPDATE SKIP LOCKED
  LIMIT 1
)
RETURNING id, source_id, chunk_text, reason_code;

SKIP LOCKED is what makes this a queue rather than a lock fight: a second reviewer's query skips the row the first one is claiming and takes the next. The same pattern works whether you have two reviewers or twenty.

The source index serves a different question entirely: "is this document broken?" When a reviewer sees ten rejections from the same source, the right action is usually to fix or drop the source document, not to adjudicate ten chunks individually. A GROUP BY source_id over the source index answers that in one pass. If your reviewers spend most of their time on bad-source triage rather than per-chunk decisions, the workflow itself may be worth handing to Quanta Experts, which reads the context and returns a finished deliverable rather than a queue of items.

Four states are enough for most teams, and every transition should be a single UPDATE that also sets decided_at.

• held. The default. The chunk is invisible to retrieval, its text is stored, and the clock is running.
• in_review. A reviewer has claimed it. Nothing else changes; the clock keeps running. An in-review item that reaches expiry still gets redacted, because the retention clock is a policy commitment, not a task timer. If you want review to pause the clock, say so explicitly and set a maximum pause, or a forgotten item lives forever.
• restored. A reviewer decided the chunk is safe. The application re-ingests the text through the normal path into the main store, and the quarantine row keeps its text until expiry for audit. Restoration should never be a flag flip on the quarantine row that a retrieval query reads; it should be a re-ingest, so the main pipeline's normal filters and chunking apply.
• dropped. A reviewer or the retention job decided the chunk will not be used. Text is redacted immediately and the row stays for the audit window.

Watch for two shortcuts that look reasonable and are not. First, allowing restored to be set by the ingestion job when a filter version changes; that turns a human decision into an automated one and defeats the point. Second, letting reviewers write directly to the main store instead of going through re-ingest; that bypasses the very filters that rejected the chunk.

The state machine also gives you a clean metrics surface. Count of rows per state per day tells you whether the queue is keeping up. Median time from held to decided_at tells you whether reviewers are the bottleneck. Rate of restored relative to dropped per reason code tells you whether a filter is miscalibrated: a filter whose rejections are restored most of the time is rejecting things it should not.

Retention has three clocks, and conflating them is the most common mistake.

A worked example. Suppose a pipeline ingests 20,000 chunks a day and rejects 0.5 percent of them, which is 100 rows a day. At an average chunk of 800 characters, that is about 80 KB of text per day, roughly 2.4 MB a month. Storage is trivial; the risk is not. Under the 14/90 rules, the steady state is about 1,400 rows with live text and about 9,000 rows of metadata, a table small enough that the queue query stays fast and the audit query stays honest. If your rejection rate is 10 percent instead of 0.5, you have a filter calibration problem, and the quarantine table is where you find that out.

The redaction job is one statement against the redaction clock:

UPDATE quarantine_chunk
SET chunk_text = NULL
WHERE chunk_text IS NOT NULL
  AND state IN ('held','in_review','restored','dropped')
  AND expires_at < now();

Note that dropped rows are included. Redacting a dropped row early is fine (the text was already supposed to be gone), and including every state means a bug in the reviewer workflow cannot keep text alive past its expiry. The only exception should be a row under legal hold, which is a separate column (hold_until timestamptz) and a separate, explicit query. Do not encode a legal hold as a state value; it is orthogonal to the review decision.

The deletion job then removes rows whose audit window has closed:

DELETE FROM quarantine_chunk
WHERE state IN ('restored','dropped')
  AND COALESCE(decided_at, expires_at) < now() - interval '90 days';

Two rules keep this safe. First, run redaction more frequently than deletion (hourly versus nightly, say), so a text row never survives much past its expiry. Second, make both jobs idempotent and bounded (LIMIT 5000 in a loop) so a backlog drains gradually instead of taking a lock on the whole table during business hours.

After redaction, the row still carries source_id, chunk_offset, char_start, char_end, reason_code, score, threshold, filter_version, state, and the timestamps. That is enough to answer the questions a regulator, an auditor, or your own future self will ask: which document produced a rejection, which filter version did it, did a human look at it, and when did the text stop existing in this system.

What it cannot answer is what the text said, and that is the point. If you need to prove content-level decisions later (for example, that a specific injection marker was present), store a hash of the redacted text and the filter's matched span in a separate, non-reversible column, not the text itself. A chunk_sha256 column plus matched_span gives you the evidence without the exposure.

This is where clean separation from the main retrieval store pays off. The main store's chunks carry their own embeddings and are served to the model; quarantine rows never are. Context engineering practice is unequivocal that the context window is a scarce resource and every token in it must be high signal (Anthropic context engineering), and a quarantined chunk by definition failed that test. Retrieval-augmented pipelines also inherit poisoning risk from their knowledge base, with prompt injection and retrieval hijacking among the documented attack classes (Promptfoo RAG poisoning), which means the restore path is a security decision, not a convenience feature. A single poisoned document can succeed against undefended systems (CorruptRAG), so restored should require the same scrutiny as the original ingestion.

One more operational detail: give reviewers a stored view that excludes chunk_text by default and fetches it by id on demand. That way a screenshot, a log dump, or a dashboard export of the queue does not leak quarantined text, and access to the raw text is a deliberate, logged action.