How to Test Whether a Disclosure Reviewer Catches a Planted Leak

The short answer is 20 to 50 drafts, each carrying at least one planted disclosure. Below about 20 you cannot say much about a low-frequency miss; above 50 the setup cost usually exceeds the value until you have fixed obvious gaps.

Coverage matters more than raw count. A useful structure is 30 planted drafts and 20 clean ones for a first run, with the plants spread like this:

Two design choices keep this honest. First, keep plants realistic in placement: bury one inside a table, one in an appendix, one in a sentence that also contains public information. Upguard's write-up on shadow AI data leaks is a reminder that most leakage is mundane and convenience-driven rather than dramatic, so the plants should look like ordinary work product. Second, plant near-misses deliberately. If your reviewer flags every nine-digit number, it will look excellent on planted Social Security numbers and useless in production, which is the point Netwrix's tuning playbook makes about pattern-only matching.

Write the ground truth before you run anything. For each draft, record the exact span that counts as the plant, its category, its severity, and the single sentence that would satisfy you as a catch. Store this in a file the reviewer never sees, keyed by draft ID.

{
  "draft_id": "D-014",
  "plants": [
    {
      "plant_id": "P-014-a",
      "category": "commercial_terms",
      "severity": "critical",
      "span": "floor discount of 18% for volumes above 500 seats",
      "pass_if": "flags the discount floor as unreleased commercial terms"
    }
  ],
  "expect_clean": false
}

Sanitize and fictionalize: use invented company names and structurally valid but fake identifiers, and keep the whole set in a separate repository with the same access controls you would put on the real thing. A test corpus built from real customer data becomes its own disclosure problem, and the Principles of Transparency and Best Practice are a useful reference for how policy documents should be clear about process and ownership.

This is where most internal evals quietly become theatre. If a reviewer output says "this document may contain sensitive information" and your scorer counts that as a hit, you will measure politeness, not detection.

Write a scoring rule per plant and apply it mechanically:

• Catch: the output identifies the specific span or an unambiguous paraphrase, names the category correctly, and cites the location. For P-014-a above, "unreleased discount floor" with the section reference is a catch; "commercial content present" is not.
• Partial: identifies the right area but the wrong category or no location. Count partials as misses in the headline number and track them separately, because a reviewer that only produces partials is telling you its prompts or context window are wrong.
• Miss: nothing, or a flag on a different part of the document.
• False positive: any flag on a clean draft, or a flag on a near-miss that is genuinely not sensitive.

The vocabulary here is borrowed from classification metrics. Recall, also called the true positive rate, is the share of actual positives that were flagged, and precision is the share of flags that were correct, as Google's machine learning crash course sets out. Your catch rate is recall at the plant level. Accuracy alone is a bad headline metric for this task because most drafts in a real queue are clean, so a reviewer that flags nothing scores well on accuracy and misses everything that matters.

The DISCO guide to generative AI document review offers a matching discipline: run an alignment sample, check results against human reviewers, analyze discrepancies, then run a second sample. Adopt that for your scorer too. Have a second person score 10 drafts independently and compare. If the two of you disagree on more than about a tenth of plants, your definitions are too loose and the catch rate is not yet a number you can act on.

There is a second reason to write the pass condition down. Different review tools phrase their instructions as explicit pass and fail criteria (for example "pass only if the exact disclosure text appears fully visible on page 1"), and that phrasing is what makes automated suggestions checkable, as Ziflow's prompt documentation shows. Your scoring rules should be written to the same standard: specific, testable, and identical in shape to the instruction you would give the reviewer.

Blindness has three parts: the reviewer must not know which drafts are planted, must not see the ground-truth file, and must not see the scoring rules in a form that lets it reverse-engineer the test. Strip draft IDs that hint at category, shuffle order, and run the reviewer in a fresh session per draft or per batch so context from a previous draft does not leak in. That last point has empirical support: a controlled study of 30 artifacts with 150 planted errors reported that removing the generation conversation before review improved defect detection, and that repeated review rounds increased false positives by 62 percent without improving recall, per the cross-model review paper on arXiv.

Then compute:

catch_rate_overall = plants_caught / plants_total
catch_rate_by_type = plants_caught_in_type / plants_total_in_type
false_positive_rate = clean_drafts_flagged / clean_drafts_total
precision = plants_caught / (plants_caught + clean_drafts_flagged)

Work a small example. Suppose you plant 30 disclosures across 30 drafts and add 20 clean drafts. The reviewer catches 22 plants and flags 3 clean drafts. Overall catch rate is 22/30, about 73 percent. False positive rate is 3/20, 15 percent. Precision is 22/25, 88 percent. Suppose further that the 8 misses split as 6 critical and 2 medium, so only 8 of 14 critical plants were caught, which is 57 percent. A blended 73 percent hides a 57 percent critical number, and the critical figure is the one that should stop the rollout.

Two cautions on the arithmetic. Small samples make each plant worth a lot: with 14 critical plants, one miss moves the critical rate by 7 points, so treat differences under about 10 points as noise. And if you have several reviewer configurations, do not pick the one with the best number on the same test set you used to tune it. Hold back a validation set of 10 to 15 drafts, run once, and report that.

Report the results as a short table by category, severity, and reviewer version, plus the false positive rate and a one-line note on how partials were counted. That is the artifact your team can argue about, and it is reusable for every prompt change. Version the prompt alongside the results: when a prompt description is edited, a new version is stored and can be re-run against the same checklist, as Ziflow describes for its own review tooling, and mirroring that discipline makes a regression visible immediately.

There is one more leak to close. If the person who wrote the plants also runs the reviewer and decides what counts as a catch, you have a single point of judgment with a stake in the outcome. Swap roles for the scoring pass: whoever wrote the drafts scores the output, and whoever runs the reviewer reviews the scores. It costs an hour and removes an obvious way for a weak reviewer to look strong.

A bare threshold ("must catch 90 percent") invites gaming. Use two numbers plus a stop rule, and tie both to the cost of a miss.

The reasoning behind the false positive column: alert fatigue is what kills review programs, not misses alone. Cyberhaven's analysis cites a survey of 300 security leaders finding that 51 percent of DLP alerts are false positives on average, and Netwrix cites ESG research finding teams remediate just 8 percent of generated DLP alerts as true positives. A reviewer that floods a queue trains people to dismiss it, and Prophet Security's piece on uninvestigated DLP alerts makes the same point from the queue side. Your eval exists to catch that before it happens.

The stop rule is simpler than the table: any missed critical plant is a named failure, written down, with the draft ID. Recurring misses in one category mean the reviewer's instructions or its context are wrong for that category, not that the number needs a friendlier threshold. OAD Technologies' guide to reducing false positives makes the same recommendation for DLP tuning generally: trace each alert to its cause and change the rule with evidence, rather than lowering enforcement.

Two more practical rules. Re-run the same test set after any prompt, model, or pipeline change, and keep the old results so you can see regressions. And retire and replace roughly a quarter of the test drafts each quarter to reduce the odds that the reviewer has been tuned into a fixed set; keep the category mix stable so the numbers stay comparable.

Finally, decide in advance what happens on a fail at the workflow level, not just the model level. The disposition options are: block publication until a human clears the draft, ship with the reviewer's flags attached for the editor to address, or ship clean and audit a sample later. Tying each disposition to a tier in the table above means nobody has to negotiate in the middle of a live review.

If your reviewer is a model behind an API, the test set doubles as your acceptance test for any provider change. Running 50 drafts through one endpoint is cheap and takes minutes, which is what makes a blinded eval practical rather than a quarterly project. Quanta Inference offers text, image, and embedding models through one OpenAI-compatible API key, so you can hold the eval fixed and swap the model underneath it to see which reviewer configuration clears your bar.

If you would rather have the review pipeline, the scorer, and the test harness built and versioned with you, Quanta takes a description of what you need and pairs you with a specialist, and a free 30-minute discovery call is a sensible place to decide whether the test set or the reviewer should be rebuilt first.