Scoring an Automated Judge on the Golden Set First

Before you write a single judge prompt, the humans need a rubric. If two experienced support leads cannot apply your criterion consistently to the same twenty tickets, no model will. The advice from calibration practitioners is blunt: if human reviewers cannot apply the rubric consistently, an LLM will not repair the underlying ambiguity (OneUptime).

Write the rubric as explicit label definitions plus boundary examples. For a support judge scoring whether an agent response correctly resolved the customer's stated problem, a three-point scale is usually enough:

{
  "labels": ["pass", "partial", "fail"],
  "definitions": {
    "pass": "Response states the correct resolution or correct next action, and every factual claim about policy, price or timing matches the knowledge base.",
    "partial": "Response is directionally correct but omits a required detail, or asks for information already present in the ticket.",
    "fail": "Response states an incorrect policy, promises something the knowledge base forbids, or does not address the stated problem."
  },
  "boundary_examples": [
    {"label": "partial", "why": "Correct refund window but no instruction on how to start the return."},
    {"label": "fail", "why": "Quotes a 60-day window when policy says 30."}
  ],
  "output_schema": {"label": "pass|partial|fail", "reason": "string, max 40 words"}
}

Three labels rather than five is a deliberate trade. Nominal scales with many levels scatter human labels and depress kappa even when the judge is reasonable. If the underlying property is genuinely ordered (severity, satisfaction), use weighted kappa so near-misses count as near-misses (OneUptime). If the labels are nominal categories such as pass and fail, use unweighted kappa.

Two design rules pay off immediately. First, constrain the output to a schema and treat anything outside it as an explicit error outcome, not as a missing value. Second, cap the reason field. A judge that writes three paragraphs is harder to audit and slower to run than one that writes one sentence.

Teams that skip the rubric step often discover the real problem was never the model. The labels disagreed because the policy itself was ambiguous, and the judge was faithfully reproducing the ambiguity.

Accuracy is the wrong metric for this job, and the math shows why. If 80% of your agent responses are correct and your judge answers "pass" every time, it scores 80% accuracy while contributing nothing. The same judge's kappa is 0.00, because it agrees with humans no better than chance on the cases that matter (GEDD). Kappa is computed as observed agreement minus chance agreement, divided by one minus chance agreement. A worked example makes it concrete: 100 items, two raters, agreement matrix with 40 true agreements and 35 true disagreements, gives observed agreement of 0.75 and expected chance agreement of 0.50, so kappa is 0.50, or moderate (Galileo).

The research literature confirms the practical point. In a study of nine judge models, judges with high percent agreement still assigned wildly different scores, and the authors explicitly rediscovered the importance of Cohen's kappa over simple percent agreement (arXiv). A judge can look aligned and rank systems poorly.

Kappa is not a universal quality number, though. It must be read alongside the confusion matrix, class prevalence, and the actual disagreements, because a kappa of 0.7 on an intent where 95% of tickets are passes says something different from the same kappa on a balanced intent (OneUptime).

For a release gate, the widely used interpretation bands from Landis and Koch give you defensible cutoffs. Substantial agreement (0.61 to 0.79) is acceptable for low-stakes automation, and 0.80 and above is the band adopted by much of the NLP evaluation community for inter-annotator agreement (GEDD). Support ticket grading with customer-visible consequences is not low stakes, so set the autonomous-grading bar at 0.7 and require human sign-off above that only if the intent has real financial or legal exposure.

Also track a second, simpler number: coverage. Calibration is a trade between precision (agreement with humans) and coverage (the share of cases scored without abstention) (Galileo). A judge that abstains on half of a tricky intent and gets the rest right has not earned the right to grade that intent alone.

Pin everything before you run. Model version, full prompt, few-shot examples, output schema, decoding settings, and any reference documents supplied to the judge all belong in the configuration record. Then run the judge on the untouched golden set and normalize only harmless formatting, mapping anything unexpected to an explicit judge error label (OneUptime). Do not drop errors from the denominator. A judge that labels 90% of cases and errors on the remaining 10% has, at best, 81% end-to-end agreement.

Here is a compact scoring script that computes per-intent kappa and the release decision:

from sklearn.metrics import cohen_kappa_score, confusion_matrix
import pandas as pd

MIN_N = 30
KAPPA_BAR = 0.70
LABELS = ["pass", "partial", "fail", "judge_error"]

def gate(rows: pd.DataFrame) -> pd.DataFrame:
    out = []
    for intent, group in rows.groupby("intent"):
        human = group["human_label"].tolist()
        judge = group["judge_label"].tolist()
        n = len(group)
        raw = sum(a == b for a, b in zip(human, judge)) / n
        kappa = cohen_kappa_score(human, judge, labels=LABELS)
        # one-sided conservative read: treat negative kappa as zero
        kappa_adj = max(kappa, 0.0)
        decision = "autonomous" if (n >= MIN_N and kappa_adj >= KAPPA_BAR) else "human_review"
        out.append({
            "intent": intent, "n": n, "raw_agreement": round(raw, 3),
            "kappa": round(kappa, 3), "decision": decision
        })
    return pd.DataFrame(out).sort_values("kappa", ascending=False)

A worked example with numbers shows why the per-intent read matters. Suppose you have four intents in the golden set:

The aggregate across all 147 cases would land near 0.87 raw agreement, an impressive-looking number that hides a failing intent. Refund eligibility has the same kind of tickets that generate the costliest mistakes, and its kappa says the judge is only moderately aligned. Billing dispute looks better on kappa but has too few cases, so the threshold cannot be trusted. This is the decision rule in miniature: two intents earn autonomy, two do not, and the judge still adds value by pre-labeling the human queue for the intents it cannot own.

Keep errors visible in that table even where they are rare. If judge errors cluster on one intent, that is usually a signal about ticket format or length rather than about the rubric.

The gate is a simple conjunction: at least 30 human-labeled cases for that intent, and kappa at or above 0.7, and judge error rate under a small ceiling you choose (for example 2%). Fail any condition and the intent stays with humans plus a sampled spot-check.

The spot-check is not a formality. For intents below the bar, let the judge draft a label and route it to a human for confirmation, then measure the human override rate weekly. If overrides on Refund eligibility fall below, say, 5% for several weeks, that is evidence to expand the golden set for that intent and re-run the gate.

This is also the point where the business case for automation becomes concrete. Escalated tickets cost substantially more than first-level resolutions, with benchmark figures putting Tier 3 handling at roughly three times the cost of Tier 1 (Maven AGI). A judge that reliably grades an easy, high-volume intent frees senior reviewers for exactly the intents it cannot handle. The gate protects the savings from being eaten by quietly misgraded disputes.

Write the gate into your pipeline as code, not a document. The judge run should emit the per-intent table above, and your CI job should fail the release if any intent currently marked autonomous drops below the bar. That converts calibration from a one-off study into a regression test.

A one-time calibration captures judge behavior at a single moment, while production distributions keep moving (Galileo). Three things break a passing gate.

First, provider-side model updates. Re-judging with an updated model version produced uniformly more lenient scores, with mean score gaps shifting in the range of a few hundredths of a point across two datasets (Galileo). Small shifts matter when your gate sits at 0.70. Freeze the judge model version where the API allows it, and re-run the gate before adopting a newer one.

Second, your own prompt edits. Changing one line in the judge prompt can move kappa by more than a model swap. Treat the prompt as versioned code with the golden run attached.

Third, drift in the input distribution. If a new returns tool changes how agents phrase confirmations, the judge, calibrated on conversational answers, may start scoring valid flows as low quality. Historical drift analysis is a useful way to set monitoring triggers: run drift checks on past data at different thresholds and windows to see how much change is normal before you pick an alert level (Evidently AI). For a support judge, the practical version is simpler: compare the intent mix of live traffic to the golden set mix monthly, and re-run the gate on any intent whose live volume or phrasing has clearly shifted.

The operational loop is: any change to judge model, prompt, schema, knowledge base, or support policy triggers a full golden-set run; the per-intent table is attached to the change; and any intent that fails drops back to human review automatically.

If you are running several judges (correctness, tone, escalation fit) across multiple model providers and want one place to route prompts and keep model versions pinned, Quanta Inference gives you text, image, and embedding models through a single OpenAI-compatible API key, so swapping or freezing a judge model is a config change rather than an integration project. You can see the model lineup at Quanta Inference.