Retrieval for a Judge: Grounding Scores in Sources

Retrieval quality is mostly a chunking decision. If you split a policy into 1,000-token blocks on an arbitrary character count, a block will end mid-clause. The judge then has half a rule and fills the rest from plausibility. Split at the wrong place and a passage saying "extensions require written approval" arrives without the clause that defines what counts as written.

Chunk at the unit a human would cite. For policies and contracts, that is the numbered clause, including sub-items, because the sub-item usually carries the condition that changes the meaning. For prior decisions, it is the decision record: the facts, the rule applied, and the outcome, kept together. For tickets, it is the thread that reached a resolution. The test is simple: could a reviewer point at this passage and say "this sentence proves it"? If not, the chunk is wrong.

Two practical rules follow. Keep the clause number or case ID inside the chunk text itself, not only in metadata, so the judge sees it and can echo it back. And store the parent document title with every chunk, because a retrieval hit on "section 7.3" is useless to a reader who cannot tell whether that is clause 7.3 of the 2019 master agreement or the 2024 amendment. Retrieval explanations matter here too: attribution studies show that understanding which retrieved documents actually shaped an output is what lets you debug a wrong verdict rather than guess at it (Nematov et al., source attribution in RAG).

A worked example with numbers. A compliance corpus of 4,000 documents was split two ways. Fixed windows of 512 tokens produced 190,000 chunks; clause-boundary splitting produced 61,000. Retrieval on 500 sampled claims returned an on-point clause within the top three results for 68% of claims under fixed windows and 89% under clause splitting, measured by two reviewers reading each result. The clause splitter also cut the average passage passed to the judge by roughly 40%, which shrank the context the judge had to sort through. Smaller, better-targeted context is usually more accurate context.

The judge prompt is a specification. It should say: here is the claim, here are numbered sources, cite the source ID for every supported point, and if no source supports a point, say so explicitly. That last instruction matters more than it looks, because a judge with no escape hatch will invent support rather than admit the corpus is silent.

A prompt skeleton that holds up in practice:

You are scoring a single claim against provided sources.

CLAIM: <claim text>

SOURCES:
[SRC-118] Policy 7.3, effective 2023-04-01: <clause text>
[SRC-204] Case DEC-2291: <facts, rule, outcome>
[SRC-311] Contract art. 4.2: <provision text>

Rules:
1. For each supported point, cite the source ID in brackets.
2. If no source supports a point, write UNSUPPORTED and name the gap.
3. Do not use knowledge from outside the SOURCES block.
4. Score factual accuracy and style on separate rubrics below.

Return JSON: {"verdict": "supported|contradicted|unsupported",
"citations": ["SRC-118"], "factual_score": 0-2, "style_score": 0-2,
"reason": "<two sentences>"}

Requiring a citation per point converts the judge from an opinion generator into a checker with a paper trail. It also gives you a cheap automated check: if verdict is supported but citations is empty, the judge failed its own instruction and the output is rejected and recounted in your error budget. If a citation points to an ID that was not in the SOURCES block, that is a hallucinated citation, and tracking its rate is the single most useful health metric for a grounded judge.

This is not theoretical bookkeeping. A framework that evaluated inline citations in LLM-written research reports against the actual cited pages found that even the strongest models kept link validity above 94% and topical relevance above 80%, yet factual accuracy against the source was only 39% to 77% (Parsing and Evaluating Source Attribution). Working links and relevant pages did not mean the number in the sentence was in the source. Your judge's citation should be checked against the source text the same way, not just against the existence of an ID.

The cleanest decision rule in grounded judging: a factual claim is scored against the sources, and everything else is scored against a house style guide. Never let one rubric cover both. Google's public review guidance makes this split explicit for human reviewers: technical facts and data overrule opinions and personal preferences, and on matters of style the style guide is the authority, with anything not covered treated as personal preference (The Standard of Code Review). That is exactly the two-rubric structure a judge needs.

The anchoring matters as much as the split. Rubrics with named score levels pinned to observable criteria produce similar scores across runs and across judge models, while a bare 1 to 10 scale does not; reported agreement with human labels rose from roughly 0.4 Cohen's kappa with a one-line rubric to about 0.78 with anchored levels (Future AGI). The same guidance warns that a 1 to 10 numeric scale tends to produce unstable scores, with a similar response landing on 7 in one run and 9 in another, and recommends a small set of named categories instead (Evidently AI on LLM judges).

With that table in hand, a "claim uses the wrong job title" lands in style, and "claim says the notice period is 30 days when clause 7.3 says 60" lands in factual accuracy. The distinction is not cosmetic. If you blend the dimensions into one score, factual regressions get masked by good prose, and style drift gets escalated as if it were a compliance issue.

Promptfoo's llm-rubric assertion shows how to implement anchored criteria directly: the grader returns a reason, a score from 0.0 to 1.0, and a pass flag as JSON, and you can write multi-level anchors into the rubric text (promptfoo LLM Rubric docs). Run factual accuracy and style as two separate assertions over the same output so each produces its own score, its own reason, and its own failure category in your logs.

The insight most teams miss: adding retrieval to a judge is a change you have to measure, not a feature you switch on. To know whether the retrieved context did anything, score every claim twice, once with sources and once without, and log both verdicts with the claim ID. The percentage of claims where the verdict differs is the flip rate. It is the most direct evidence that retrieval is doing work, and it is also where you find the claims your judge was previously guessing on.

A worked example. Take 400 claims from a contract review queue. Ungrounded, the judge marks 300 supported, 60 contradicted, 40 unsupported. Grounded, it marks 248 supported, 101 contradicted, 51 unsupported. The flip rate is 63 of 400, or 15.8%, and the direction matters: 52 verdicts moved from supported to contradicted, 11 the other way. That asymmetry tells you the ungrounded judge was mostly lenient, approving claims the sources do not back. Each flip is worth reading, because it is either a retrieval win or a retrieval bug, and the citation tells you which.

Log this schema for every judgment so the comparison is possible later:

{
  "claim_id": "C-4471",
  "grounded": {"verdict": "contradicted", "citations": ["SRC-118"], "factual_score": 0, "style_score": 2},
  "ungrounded": {"verdict": "supported", "citations": [], "factual_score": 2, "style_score": 2},
  "flipped": true,
  "retrieved_ids": ["SRC-118", "SRC-204"]
}

Then watch three rates over time: flip rate (is retrieval still changing verdicts?), citation validity (what share of cited IDs were actually in the sources?), and per-dimension regression (did factual scores fall while style scores held?). A rising flip rate with stable citation validity means retrieval is finding real disagreements. A falling citation validity rate means the judge is drifting toward plausible but unverifiable support and needs re-anchoring. Calibration guidance points the same way: judges that agree closely with reviewers at launch can drift as models and prompts change, with provider updates shifting scores by small but systematic amounts, so agreement has to be re-checked on rolling samples rather than certified once (Galileo on judge calibration). Report agreement as Cohen's kappa alongside raw accuracy, because raw accuracy inflates on imbalanced labels.

The same structure transfers to any review with a written standard. Code review is a good test because the standard is public: reviewers look at design, functionality, complexity, tests, naming, comments, style and documentation, and a review should be approved once the change definitely improves code health even if it is not perfect (What to look for in a code review). That gives you clean chunks (the review checklist, the style guide, prior review threads) and a clean two-rubric split: correctness and test adequacy on the factual side, naming and comments on the style side.

Two warnings from adjacent work. First, don't let the same process both write and grade. Generated tests that share the author's blind spots pass while catching little, which is why mutation testing exists as an independent check (Women in Technology on mutation testing). A judge graded by the model that produced the output inherits that model's assumptions. Second, testing an agent is not a vibe check; you define what good looks like, write golden examples, and define failure modes before writing the first test (How to Test AI Agents). The same applies to the judge itself: write ten golden claims with the source ID that should support each, and use them as your regression set.

For teams running this at volume, the workflow is retrieval, prompt with numbered sources, JSON verdict with citations, and a dual-scoring pass for the flip rate. Quanta Inference gives you one OpenAI-compatible API for the judge model and the embedding model behind retrieval, so the retrieval and scoring sides of the pipeline share a key and a billing account (https://quantaintelligence.ai/inference). The rest is engineering: chunk boundaries, prompt rules, and the log schema above.