From Verdicts to a Queue: A Schema for Judge Scores and Reasons

The schema below is the whole design. One table, one row per checked item, no joins required to route work.

Three of these deserve comment because they are the ones teams skip.

passage must be what the judge read, not what you meant to send. If a retrieval bug hands the judge the wrong chunk, the stored passage is the only evidence that explains a strange verdict. Papers on citation evaluation show why this matters at scale: a framework that parses Markdown reports and retrieves the actual cited content to judge each claim against its source closes a loop that claim-in-isolation checks cannot, as described in this source attribution evaluation. And even strong frontier models with link validity above 94% reached only 39 to 77 percent factual accuracy in that benchmark, which is exactly the gap you cannot see from a pass/fail column.

verdict stays categorical even though score exists. Named categories carry more meaning than arbitrary numbers. Evidently's judge FAQ recommends binary or a few clearly defined classes and warns that a similar response may get a 7 in one run and a 9 in another on a 1-10 scale (Evidently AI). The numeric score is still useful for thresholding and trend charts, but the categorical verdict is what the routing rule and human reviewers read.

judge_id is not optional. Judge upgrades are not interchangeable: one audit across four judgment datasets found that scaling a judge from 1.7B to 32B parameters produced a robust adjacent gain while adjacent API releases did not, and that stronger judges reduce but do not remove position and verbosity bias (auditing LLM-as-judge reliability). If you cannot group rows by judge, you cannot tell a quality regression from an instrument change.

The claim_text and passage split also matches how claim-level benchmarks are built. PARCEL labels 3,396 legal parenthetical claims as Supported, Refuted or Not Found, and finds that missing support is harder for models to detect than direct contradiction (PARCEL benchmark). A three-way verdict enum is the practical consequence: you need unclear as a first-class outcome, not an error state.

Promptfoo's llm-rubric assertion documents the shape most judges converge on: a JSON object with reason, score between 0.0 and 1.0, and pass as a boolean (promptfoo docs). That is a good starting point, but it lacks the two fields routing needs. Ask for confidence and a null-safe verdict explicitly.

{
  "verdict": "pass | fail | unclear",
  "score": 0.0,
  "confidence": 0.0,
  "reason": "one or two sentences citing the passage's wording",
  "passage_quote": "the exact span relied on, or null"
}

Then validate before insert, and treat validation failure as unclear with confidence 0.0 so it lands in the human queue. A parser that accepts true for one call and "true" for the next silently breaks your routing rule; the audit paper on judge reliability notes that shifts produced by deliberation protocols cannot be attributed without parser and fallback logs (the same audit).

Keep the score range fixed per rubric and record it. Mixing a 0-1 faithfulness judge with a 0-100 quality judge in one column makes thresholds meaningless. The rubric anchors are what turn "1-5" into something reproducible across runs and across judge models, per Future AGI's description of anchors.

Routing is a pure function of two numbers and one label. Write it down once so reviewers, dashboards and backfills agree.

PASS_SCORE = 0.80
FLOOR = 0.60
BORDER = 0.45

def route(row):
    if row["verdict"] == "pass" and row["score"] >= PASS_SCORE and row["confidence"] >= FLOOR:
        return "done"
    if row["verdict"] == "fail" and row["confidence"] < FLOOR:
        return "human_review"
    if BORDER <= row["score"] < PASS_SCORE:
        return "second_judge"
    return "human_review"

The four thresholds are not arbitrary but they are not universal either. A confidence threshold is a business decision layered on a probability estimate, and the default 0.5 is one of the most common production mistakes (LlamaIndex on confidence thresholds). Their table of domain ranges gives a starting calibration: document extraction runs 0.85 to 0.95, medical diagnosis 0.90 to 0.98, spam filtering 0.60 to 0.80. A faithfulness judge on marketing copy sits nearer the spam end; a legal or medical claim judge sits near the top.

Here is the decision rule in plain words, which is also the rule to paste into your runbook:

Two properties matter. First, done requires both a categorical pass and a numeric one, so a judge that says "pass" with a 0.5 score lands in the borderline band instead of disappearing. Second, low-confidence failures go to humans rather than a second judge, because a false accusation is more expensive than a missed one in most review workflows, and a second model sharing the first model's bias will not catch it. Repeated-sample juries add little when errors are correlated, per the reliability audit.

Suppose a report generator emits 1,000 claims in a nightly batch. The first judge scores every row. Assume the distribution lands like this (illustrative numbers you should replace with your own after two weeks of logging):

The routing saves the interesting number. Without the rule, all 380 non-pass rows reach a person at roughly 3 minutes each, about 19 hours of review. With the rule, 180 rows go to a second judge for a second API call, 90 confident failures get a 10 percent spot sample of 9 rows, and the human queue is about 120 rows, under 7 hours. You have cut reviewer load by roughly two thirds and, more importantly, every row that reaches a person arrives with a passage and a reason attached.

The 10 percent sample on confident failures is the calibration loop. When a sampled row shows the judge was wrong, you have a labeled example, and those examples are what Galileo's calibration guide argues should feed back into the judge's prompt, anchors and rubric continuously, because provider-side model updates shift score distributions without warning. The guide cites mean score shifts of +0.022 to +0.072 on HelpSteer2 when re-judging with an updated model version, and a documented case where a model update caused a 9.6 percent accuracy drop for one prompt and a 5.1 percent increase for another on the same task.

That is the argument for judge_id in one sentence: the same fixed candidates can score differently because the instrument changed.

The table exists partly so that four questions are one query each.

-- 1. Which sources produce the most unsupported claims?
SELECT source_id, count(*) AS fails
FROM checks WHERE verdict = 'fail'
GROUP BY source_id ORDER BY fails DESC LIMIT 20;

-- 2. Is the current judge drifting against human labels?
SELECT judge_id,
       avg(CASE WHEN verdict = 'fail' AND score >= 0.80 THEN 1.0 ELSE 0 END) AS contradiction_rate,
       avg(confidence) AS mean_confidence
FROM checks GROUP BY judge_id;

-- 3. How much work is the second judge absorbing?
SELECT date_trunc('day', created_at) AS day,
       count(*) FILTER (WHERE route = 'second_judge') AS to_second_judge,
       count(*) FILTER (WHERE route = 'human_review') AS to_human
FROM checks GROUP BY 1 ORDER BY 1;

-- 4. Are reviews changing verdicts?
SELECT c.judge_id,
       avg(CASE WHEN r.final_verdict <> c.verdict THEN 1.0 ELSE 0 END) AS overturn_rate
FROM checks c JOIN reviews r USING (claim_id)
GROUP BY c.judge_id;

Query 2 is the contradiction check: a row that says fail with a high score means the rubric anchors or the prompt are internally inconsistent, and those rows should be rare. Query 4 is the cheapest calibration signal you can build, because it needs no extra labeling work, only that reviewers record a final verdict. Cohen's kappa is the metric to report rather than raw accuracy, since accuracy inflates on imbalanced labels, per Galileo's calibration guide.

One caution on aggregation. Multi-judge ensembles are supported in research frameworks with majority, weighted, unanimous and any-vote aggregation, along with a CANNOT_ASSESS verdict for explicit uncertainty handling (Autorubric). If you add a second judge to the borderline band, store its verdict in the same row with its own judge_id suffix, and define the tie rule before you run the batch. Otherwise the second judge becomes a second opinion nobody can act on.

Keep the table boring and the routing explicit. Any eval harness can write these rows; the classification work is the judge call itself. If you are running judges across text, image and audio inputs, routing them through one endpoint with an OpenAI-compatible interface keeps the client code identical and the judge_id honest, which is what Quanta Inference is for. If the work is better handed off than wired up, Quanta Experts read your context and return the finished rows in this shape.

The design survives the two changes that break naive eval tables: a new judge model and a new rubric. Both change judge_id, both are visible in query 2, and neither requires a migration.