How to Eval Whether Your Pipeline Honors Exclusions

A planted excluded document needs three properties to be a useful probe.

It must be semantically dense relative to the queries. If the planted document is a near-duplicate of the query, it proves only that the filter works in the easiest case. Write the planted text so it would rank in the top three on embedding similarity alone: use the same vocabulary, the same product names, the same phrasing a real user would type.

It must have a unique, greppable identifier that cannot appear by accident. Use a distinct token in the document body and in its metadata, for example a field such as doc_id: exclusion-probe-0007 plus a nonsense phrase like "zebra protocol annex". Grepping for a generic phrase such as "internal memo" will produce false failures.

It must be representative of the exclusion classes you actually maintain. Most teams hold several kinds of exclusion at once, and they are enforced by different mechanisms, which fail in different ways. The table below is the checklist I use.

Plant at least one document per row, because the enforcement points differ. A test suite that plants only a takedown case will never catch a tenant-boundary leak, and a suite that plants only one per class will miss the sibling-endpoint problem described later.

The query set is where most exclusion evals are too weak. A query that would never have retrieved the document proves nothing. You need queries where the planted document is a strong match, plus queries where it is a marginal match, because a marginal match is the case a post-ranking filter is most likely to mishandle.

Build the set in four slices.

First, direct hit queries. Take sentences from the planted document itself and paraphrase them lightly. These should return the document with the filter switched off. Verify that: before you trust the eval, run it in a deliberately unfiltered mode and confirm the planted identifier appears. A test that cannot fail is worthless, and this is the step that separates a real exclusion eval from a green tick.

Second, neighbor queries. Take real user queries from your logs that are adjacent in topic. If the excluded document is an internal pricing sheet, use the queries that currently return the public pricing page. These catch filters that are scoped too narrowly, for example one keyed on an exact product name.

Third, adversarial phrasing. Rephrase the same information request in the way a curious or careless user would, including misspellings and ambiguous acronyms. Evidently's guide treats stress and adversarial testing as a distinct layer for exactly this reason: normal traffic does not exercise the edge of the retriever's behavior.

Fourth, cross-path queries. For each query, call every endpoint that serves retrieval, not just the main one. Chat, autocomplete, "related documents" panels, summary tools and any agent tool that performs a search are separate code paths, and each needs its own assertion. In practice this is where the leak lives.

A workable size is small. Twenty to forty queries across the four slices, with at least one planted document per exclusion class, runs in seconds and fits in a pre-merge check. Generated question-answer pairs can bootstrap the set, but Evidently's guide frames synthetic data as a starting point rather than a substitute, so hand-check the direct-hit slice yourself.

The mechanics are simple and deterministic: no LLM judge is needed for the core assertion, which is a string search over returned identifiers.

PLANTED = "doc_id:exclusion-probe-0007"
K_SERVED = 8   # the largest k any endpoint returns

def run_exclusion_eval(retrievers, queries):
    failures = []
    for name, retrieve in retrievers.items():
        for q in queries:
            hits = retrieve(q, k=K_SERVED)
            ids = [h.doc_id for h in hits]
            if PLANTED in ids:
                failures.append({
                    "path": name,
                    "query": q,
                    "rank": ids.index(PLANTED) + 1,
                })
    return failures

Three details in that snippet are load-bearing. K_SERVED must be the largest k your product exposes, not the k of the endpoint under test, because a document at rank 40 is still reachable by a user who asks for more or by an upstream reranker that expands the candidate set. The check reads the doc_id field rather than the text, so chunking and reflow cannot hide a leak. And the function returns the rank, which turns a failure into a diagnosis: rank one means no filter at all, rank seven means the filter is applied after ranking with a small window, and a rank that varies between identical runs means the filter is applied to a subset of shards.

The pass threshold is worth stating formally, since it is the part teams get wrong.

That last row is the one that protects you. Treat uncovered paths as failures of the eval, not passes of the pipeline, and add a planted document and query for each.

To keep the identifier check honest, store the mapping from probe to expected result in version control alongside the query set. CASRAI's guide to audit trails makes the general point well for research records: a trail of decisions is what lets someone else follow how a conclusion was reached. The same discipline applies to an eval suite, where the "why" behind each planted document is the thing a new engineer needs when the test starts failing.

When the check fails, the rank and the path usually identify the cause before you read any code.

Filter after rank. The retriever returns the nearest N vectors and the application removes excluded identifiers from the list afterwards. This passes when the document sits at rank 30 and fails when it climbs to rank 3, which is why marginal queries matter. KX Systems' write-up on metadata filtering makes the architectural distinction clear: metadata attached to each vector can be used as a filter during the similarity search, rather than as a cleanup step applied to the results. The fix is to push the exclusion into the query itself, as a metadata predicate, so the candidate set never contains the document.

Sibling path. One service enforces the exclusion and another does not. This is the most common cause in mature systems and the one a single-endpoint test never catches. Compare the retrieval code between paths; the leaking one is usually missing the predicate, or has it and drops it when a request includes an override flag.

Stale index. The exclusion was applied at ingestion, but the identifier name changed or the rebuild covered only part of the corpus. The failure signature is sporadic: the document appears for some queries and not others, correlating with which shard served the request. Rebuilding is the fix, but the durable fix is a check that runs after every index build and compares the count of excluded identifiers in the index against the count in the exclusion source of truth.

Answer-level mention without retrieval. Occasionally the document identifier is absent from retrieval but the generated answer mentions the sealed fact anyway, because the model memorized it or because a different chunk restates it. This is why the exclusion eval belongs at the retrieval layer, and why the generation-layer check, if you add one, should use a judge calibrated against human labels rather than a keyword rule. Arize's guide to human-LLM judge alignment sets out the calibration approach: compare the judge against multiple human annotations, report agreement alongside a chance-adjusted statistic, and treat the judge as a classifier whose precision and recall you can state. Keep that judge out of the blocking gate and let the deterministic identifier check decide pass or fail.

An exclusion eval decays in three predictable ways, and each has a maintenance rule.

The corpus drifts and the planted documents stop being plausible matches. Queries that once ranked the probe at position two now rank it at forty, so the test passes for the wrong reason. Re-run the unfiltered verification monthly, or whenever the embedding model or chunking changes, and confirm each probe still ranks highly without the filter. If it no longer does, rewrite the probe text or the query.

New exclusion classes appear and no probe covers them. Every time the product team adds a new kind of exclusion, the eval suite gains a row. Make that part of the definition of done for the feature, not a follow-up ticket.

The plugin of exclusion grows and nobody audits it. Exclusion lists accumulate entries and each one is a chance for a typo that silently excludes nothing. Periodically reconcile the list against the systems of record. Snorkel's write-up on rubrics makes the broader argument that structured, written criteria beat ad hoc checks because they can be audited and improved. Your exclusion list is the same kind of artifact: if it is not written down in a form a reviewer can check, it is not enforced.

One more maintenance point is worth more than it looks. Keep a short decision log next to the suite recording why each probe exists, who owns it, and when it was last verified. Atlas's example of a decision log shows the useful shape: date, decision, reason, affected artifact, and a review point. Six lines per probe is enough to stop the next engineer from deleting a test they do not understand.

The eval itself is cheap once you commit to the binary threshold. What tends to be expensive is the surrounding work: standing up a reproducible index, running the same suite against every retrieval path, and wiring the result into the merge gate. If you would rather not build that plumbing from scratch, Quanta is a place to hand the job over: describe the pipeline and the exclusion behavior you need, and a specialist builds it with you.

If the retrieval layer is not the only part you are testing, note that the same discipline extends upward. A pipeline can pass the retrieval assertion and still leak through a generated summary, which is a generation-layer problem with its own eval design. Keeping the two layers separate from the start saves a painful refactor later.