Detecting Poisoned Chunks Before They Reach the Prompt

Start with where the chunk came from, before you look at what it says. A chunk has a source URI, an ingestion path, an owner and an ingestion timestamp. Each of those is a check.

A useful decision rule: score provenance on a three-level scale. Trusted sources are ones your team controls and reviews (product docs, published policies, signed content). Semi-trusted sources are internal but open-write (wikis, tickets, shared folders). Untrusted sources are anything external or anonymous. Then set your retrieval threshold by level: for untrusted sources, require a higher similarity score before the chunk is even a candidate, and never let an untrusted chunk be the only support for a factual claim.

Content hashes catch a specific attack that source checks miss: the edit after ingestion. Store a hash of each chunk at ingest time and re-hash before use. An unchanged chunk that suddenly contains a new instruction, or a document whose hash changed without a corresponding review event, is a quarantine candidate regardless of its score. This is cheap and it closes the window where an attacker with write access to a wiki edits a previously safe page.

Ingestion timestamps deserve a flag of their own. A chunk ingested in the last few minutes that ranks into a live prompt is a legitimate signal for a review queue, not an automatic reject, because it also catches normal fresh content.

Provenance also has a reporting use. When a bad answer reaches a user, the first question in review is always which source it came from, and a pipeline that never recorded source tiers has no answer. Writing the tier into the chunk metadata at ingestion costs one lookup and saves an hour of guessing LangChain's context engineering docs.

Embedding similarity is a number, and numbers have distributions. Most legitimate retrieval sets have a smooth score curve: a few strong matches, then a gradual tail. A poisoned chunk engineered to win retrieval tends to break that shape.

Compute the gap between the top score and the median score of the returned set, and between the top score and the second score. A top result that sits far above its neighbors, especially when it is also from a low-trust source, is the classic hijack signature. A worked rule that holds up in practice: flag when the top score exceeds the second-highest by a large margin and the top chunk's source is untrusted. Both conditions together cut false positives, because a genuinely dominant document from a trusted source should pass.

There's a second anomaly worth logging: the chunk that scores well on many different queries. Poisoned documents crafted for several attack-targeted questions tend to be retrieved repeatedly across unrelated user intents. A per-chunk retrieval counter, sampled over time, surfaces these documents before they do damage. Keep the counter on the chunk ID, not the query text.

The table is also a budget guide. The three low-cost rows should run on every chunk, every request, because they are cheap enough that skipping them is indefensible. The medium-cost rows belong on the retrieved set after the first pass has already removed the obvious junk, so you pay for a classifier on ten chunks rather than on a thousand candidates.

If two trusted sources say different things, the model cannot resolve it, and neither can a keyword filter. Disagreement is a signal in its own right: it tells you the context contains a contradiction that will produce an unreliable answer.

The practical version is a consistency check. Take the retrieved set, ask a cheap model to extract the factual claim each chunk makes about the user's question, and compare. When one chunk asserts a fact that the rest contradict, demote it, not reject it. The RAG-IDS defense work found that soft trust-based demotion outperformed hard filtering while label-embedding consistency checking was the main contributor to robustness under poisoning the RAG-IDS paper. That result matches operational intuition: an aggressive filter throws away good context, while a demotion keeps recall intact and lowers the poison's influence.

There is a second structural version of this check. CleanBase exploits the fact that malicious documents crafted for the same questions tend to be highly similar to each other, so they form cliques in a similarity graph over the knowledge base the CleanBase paper. You can approximate it on each query: run a pairwise similarity check within the retrieved set and flag tight clusters from different, low-trust sources that agree with each other but disagree with your trusted sources. That pattern is coordination, and coordination is rarely an accident in user-generated content.

One operational note: run the consistency check on claims, not on wording. Two chunks can describe the same refund policy in completely different sentences without disagreeing, and a naive text comparison will flag both as suspect. Extracting "what does this chunk assert" first is what keeps the signal clean.

Some poisoned text is not subtle. It repeats a template, stuffs keywords, or ends with a directive. You can detect a lot of this with cheap text statistics before any model is involved.

Rank chunks by compression ratio (a proxy for low entropy and high repetition) and flag those at the extreme low end of your corpus distribution. Legal boilerplate and product terms of service will show up, which is why this signal quarantines for review rather than blocks. Pair it with a near-duplicate check against a corpus of known-good chunks: an exact or near-exact copy of a trusted chunk with one sentence appended is a strong indicator of an edit attack.

Then scan for instruction-shaped content inside data. Retrieved chunks are data. Text like "ignore previous instructions," "system:" prefixes, or markup that mimics your prompt delimiters belongs in a filter. This connects to a discipline most teams already know: keep data and instructions separated with clear delimiters, because sloppy structure invites injection LaunchDarkly's prompt engineering guide. The retrieved chunk should sit in a labeled data block, and anything inside it that looks like an attempt to close that block should be quarantined.

Delimiters do double duty here. Wrapping retrieved text in XML-style tags gives the model an unambiguous boundary between instructions and data, which is the same reason structured prompts parse more reliably in general IBM's prompt engineering techniques. When a chunk contains a closing tag it should never contain, the parser can flag it before the model ever reads it.

The lowest-effort attack in the playbook is invisible text: zero-width characters, bidirectional control codes, and Unicode tag characters that render as nothing but tokenize as instructions. A human reviewer reading the chunk sees clean prose. The model sees the payload.

A practical scan normalizes Unicode, then flags any chunk containing characters from a small blocklist of invisible and control format characters, plus any sudden spike in non-ASCII ratio. Log the codepoints found, because the specific characters tell you whether this is an attack or a corrupted file. Pair the scan with a strip step at ingestion, and keep the original hash so you can prove what was ingested.

Sanitizing retrieved text before assembly is the same idea the RAG-IDS work applies at the retrieval boundary, where prompt sanitization sits alongside trust scoring and consistency checks the RAG-IDS paper.

A classifier pass catches the cases your rules miss. A cheap safety or instruction-detection classifier run over the retrieved set can score chunks for embedded directives and harmful content, and it belongs on the retrieved set rather than the whole corpus because that is where the volume is already small. Treat its verdict as one signal among several rather than a final answer, since quoted examples in legitimate documentation will trip it honestly.

Detection without a gate is theater. Put a quarantine function between retrieval and prompt assembly. Every chunk gets a verdict: pass, demote (included but ranked lower and labeled), or quarantine (excluded). The prompt builder never sees quarantined chunks, and a query whose surviving context is too thin returns a grounded refusal rather than a confident answer built on poison.

def assemble(query, retrieved, trust, classifier):
    passed, demoted, quarantined = [], [], []
    for chunk in retrieved:
        reasons = []
        if trust[chunk.source] == "untrusted" and chunk.score_gap > GAP_LIMIT:
            reasons.append("score_anomaly")
        if chunk.hash != chunk.ingest_hash:
            reasons.append("hash_changed")
        if contains_hidden_unicode(chunk.text):
            reasons.append("hidden_unicode")
        if classifier.flag(chunk.text):
            reasons.append("classifier_flag")
        if len(reasons) >= 2 or "hidden_unicode" in reasons:
            quarantined.append((chunk, reasons))
        elif reasons:
            demoted.append((chunk, reasons))
        else:
            passed.append(chunk)
    log_decision(query.id, passed, demoted, quarantined)
    return build_prompt(query, passed, demoted)

The two-reason threshold is the decision rule that matters. One soft signal demotes; two independent signals quarantine. Hidden Unicode quarantines alone because it has almost no legitimate cause in prose. Tune the thresholds against your own corpus, but keep the rule shape: independent signals, combined, with a hard override for the unambiguous one.

Log enough that an incident can be reconstructed later. For each chunk, write the chunk ID, source URI, retrieval score, each signal's value, the classifier verdict, the final decision (pass, demote, quarantine) and the prompt assembly ID. For each request, write the full set of decisions so you can replay it. This is the same observability loop production agent teams describe as necessary to catch non-deterministic behavior at scale Splunk. Without it, a bad answer is a mystery; with it, it is a chunk ID and a timestamp.

One caution on caching. If you cache prompt prefixes, and you should for cost, a quarantined chunk must never enter a cached prefix, and a cache hit must never bypass the gate. Keep the static, gate-free content first in the prompt and assemble retrieved chunks after it the OpenAI prompt caching guide. Caching reuses computation for identical prefixes; it does not re-run your safety checks, so the checks belong at assembly time, not in the cache. The prefix mechanics matter for the same reason: any change early in the prompt invalidates everything after it, which is one more argument for putting variable retrieved chunks late how prompt caching works under the hood.

You cannot tune thresholds you never see fail. Generate poisoned documents and run them through your pipeline. Promptfoo ships a red-team plugin that poisons documents toward a stated goal, such as extracting an API key, and then runs a scan against the configured system Promptfoo. Use it to build a regression suite: one poisoned PDF, one wiki page with hidden Unicode, one near-duplicate of a trusted policy, one coordinated cluster of three mutually consistent fake docs. Measure what fraction reaches the prompt and what fraction your quarantine catches. Track the two numbers separately, because a demotion that still reaches the model is not a pass.

Then close the loop on the product side. Build the evaluation harness in the same context-engineering discipline the rest of the pipeline uses, where the right context, formatted correctly, is treated as the core engineering job LangChain's context engineering guide and Weaviate's guide. If your team would rather operate one endpoint across the models you test with, Quanta Inference puts text, embedding and classifier calls behind a single OpenAI-compatible key Quanta Inference, which keeps the harness simple when you compare detectors across providers.