When a Judge Cites a Passage That Does Not Back the Claim

The unit of work is a pair. Left side: a single factual claim, stripped of hedging and split so it asserts one thing. Right side: the exact text of the passage the author cited, copied verbatim, not the whole document and not a summary. Ask the verifier to compare the two and return structured output.

A workable input shape:

{
  "claim": "The intervention reduced readmissions by 18% among adults over 65.",
  "passage": "Among participants aged 65 and older, the readmission rate fell from 22.4% to 18.4% over twelve months (n=412).",
  "task": "Decide whether the passage supports the claim. Return supports, partially_supports, or does_not_support, plus a reason under 30 words that quotes the passage."
}

The output contract matters as much as the input. Ask for a closed set of three labels and a short reason that must contain a quoted span. A reason without a quote is unfalsifiable and therefore useless in review. Promptfoo's rubric grader shows the same pattern: the judge returns a reason, a score and a pass flag, and the rubric text defines what each level means (the llm-rubric docs).

Define the three labels in the prompt using the language a reviewer would use:

Two rules make the check sharper. First, give the verifier only the passage, never the surrounding document, so it cannot wander off and find support somewhere the author did not cite. Second, instruct it that silence is a failure. Without that instruction, judges tend to be generous: an adjacent paragraph mentioning the same topic reads as relevant, and relevance gets confused with support. Evidently's guidance on grading scales makes the same point from the other direction, recommending a few named categories over numeric ranges because a human reviewer should be able to apply the rubric consistently (the LLM judge FAQ).

A verdict you cannot reproduce is not a measurement. Two things drive reproducibility: anchored rubric language and a judge that is not the model under test. Anchors pin each label to observable criteria, and rubric quality moves judge to human agreement far more than swapping to a bigger judge does, with reported Cohen's kappa rising from roughly 0.4 with a one line rubric to about 0.78 with anchored levels (the rubric glossary). That is the highest leverage edit available to you, and it is free.

On judge selection: using the same model that wrote the text to grade it invites self preference and shared blind spots (Evidently). A different model family is a reasonable default. Running the check through one interface that reaches several model families makes the swap a configuration change rather than a rewrite, which is what Quanta Inference provides: text, image, video, audio and embedding models behind one OpenAI compatible key (Quanta Inference).

Then validate before you trust. Take thirty claim and passage pairs, label them by hand, and compare your verifier's verdicts. Report agreement, not accuracy, when the mix of labels is skewed, because a judge that says "supports" for everything scores well on a clean corpus and tells you nothing (Galileo on calibration). Keep those thirty pairs as a regression set and rerun them whenever you change the prompt, the judge model or the rubric. Providers update models without notice, and the same prompts can then produce shifted scoring distributions while your dashboard stays green.

A short calibration loop:

1. Label 30 pairs by hand, oversampling failures so the set is not 28 supports and 2 fails.
2. Run the verifier, record agreement per label, and note which label it confuses.
3. Add one anchor sentence to the rubric that separates the confused pair, for example a sentence stating that a passage reporting a subgroup result only partially supports a claim about the full population.
4. Rerun and keep the change only if agreement holds or improves.

Keep the three labels separate. The headline number is supports divided by total claims, but the two failure counts carry the diagnosis. A batch at 91% supports with 3% partial and 6% does not support is a different problem from 91% with 8% partial and 1% does not support: the first is mostly clean with a few misses, the second has a structural gap where the generator consistently overstates what sources say.

Track the rate per document and per source, not just per batch. If one retrieved document accounts for half the failures, your retrieval is fine and your summarizer is overreaching on that document. If one claim type (percentages, dates, causal language) dominates failures, the prompt that produces claims needs a constraint, not the verifier.

Scaling to a large corpus needs a threshold and an owner, not a perfect judge. A useful decision rule: ship a document if supports is at or above your threshold and does not support is zero; hold it if partial exceeds a small allowance; quarantine it if any does not support verdict stands after re-judging. Pick the threshold from labeled data, not from ambition. If your hand labeled set shows the verifier agrees with you on 8 of 10 failures, a 90% supports threshold leaves roughly a tenth of real failures in the shipped pile, and that residual is what the human queue is for.

verdicts = run_verifier(claim_passage_pairs)
total = len(verdicts)
supports = sum(v.label == "supports" for v in verdicts)
partial = sum(v.label == "partially_supports" for v in verdicts)
fails = sum(v.label == "does_not_support" for v in verdicts)
support_rate = supports / total
block = fails > 0 or partial / total > PARTIAL_ALLOWANCE

Log every verdict with the claim hash, the passage hash, the judge model identity, the rubric version and the reason string. The passage hash is what makes auditing possible later: when a source page changes underneath you, you still hold the exact text the judge saw.

Every failure gets a disposition, decided by rule, not by whoever is on call. The policy below is deliberately boring so it can run unattended.

Three details keep this policy honest. Quarantine means the claim does not appear in the shipped output at all, not that it appears with a warning badge; a warned claim is still read as true by most readers. Re-judging happens exactly once, with a change to either prompt or judge: repeating the same call with the same inputs and hoping for a different answer is a coin flip, and the within-question noise in judge scores is large enough that you will sometimes get the flip (Galileo). And the human queue is a sampled, prioritized stream: sort by claim importance first, then by how close the reason came to quoting supportive text. A failure where the reason quotes a sentence that almost works is the highest value review, because it tells you whether the rubric or the writer is wrong.

Keep re-judge results in the log alongside the first verdict. If the second judge disagrees with the first on a stable set of pairs, you have a disagreement rate worth measuring, and rubric ensembles with majority vote are a documented way to reduce single judge noise (the Autorubric framework).

Take a report pipeline that emits 200 cited claims across 20 sections. The verifier runs as 200 independent calls, each with one claim and one passage. The batch comes back: 171 supports, 18 partially supports, 11 does not support. Support rate 85.5%, and the two failure counts together are 14.5%.

Apply the policy. The 18 partial verdicts get one re-judge with a stricter prompt that removes partial credit for narrower populations. Suppose 7 come back as supports (the original claim was in fact scoped correctly and the first judge was strict) and 11 stay partial. The 11 does not support verdicts get one re-judge with a different model; 2 come back supports, 9 stay does not support. The 9 join the 11 remaining partials in the human queue, which is 20 claims from 200, or 10%.

Now compute what that costs and what it bought. The pipeline made 229 verifier calls (200 first pass plus 29 re-judges) and then handed a reviewer 20 pairs instead of 200 claims. That is the whole economic argument for this step: it does not remove human review, it concentrates it. Without the check, a reviewer reading all 200 claims might skim, and the 9 genuinely unsupported claims are exactly the ones that read fluently enough to pass a skim.

The second worked number is the one worth watching over time. If the support rate sits at 85% across batches but the does not support count is zero while partial climbs to 15%, your rubric is drifting toward leniency, not your writing getting worse. Track the two failure counts as separate series and you can tell which.

The check belongs between generation and publication, in the same place a build step sits between commit and deploy. It runs per document, writes a verdict file, and blocks shipment on the policy above. The verdict file is also the audit trail. When a reader disputes a sentence, you can pull the exact passage, the verdict, the judge identity and the reason, and show that either the check passed and the rubric was too loose, or the check failed and the policy was overridden.

Two operational habits keep it alive. First, refresh the calibration set every time the retrieval corpus or the writer prompt changes; a prompt change shifts the distribution of outputs the verifier sees even when the rubric is untouched, and a verifier calibrated against last quarter's response shape can start scoring valid outputs as failures (Galileo on drift). Second, keep the verifier's rubric in version control next to the prompt that generates claims, and record both versions in every log line.

For teams that would rather not assemble the judge harness from scratch, the same split applies: deterministic checks where the answer is objective, a rubric judge for the judgment calls, and human review for calibration and edge cases (Masood's rubric survey). The claim and passage check is a rubric judge with a very narrow rubric, which makes it one of the easier pieces to get right and one of the easiest to leave out.

If the volume is the obstacle, the parallel version of this work is a coding agent that runs the check on every pull request that touches citation logic, with the rubric pinned as a skill. Quanta Skills is the library behind the Quanta experts, and a skill can be installed into a coding agent or run as an expert job (Quanta Skills).