When the Judge Rejects a Cheap Model: Distinguishing Faults

You cannot attribute what you did not keep. Most pipelines log the post-parse object, which is exactly where the evidence is destroyed. Log the raw string, the finish reason, the token counts, and the prompt version, and attach request metadata at the call site so the record is joinable later. The minimum useful set is the user, the feature, the deployment, the prompt version, and a run identifier (Braintrust on cost tracking).

A practical logging record looks like this:

{
  "request_id": "req_8f21",
  "prompt_version": "judge_v3_hash_9a2c",
  "model": "cheap-tier",
  "finish_reason": "length",
  "raw_output": "{\"verdict\": \"reject\", \"reason\": \"the claim about refund window is not supported by",
  "input_tokens": 1840,
  "output_tokens": 64,
  "max_output_tokens": 64
}

The finish_reason: "length" with output tokens equal to the cap is an instant format diagnosis: the model was truncated. That one field will explain a large share of your "the cheap model is bad" complaints. Without it, truncation looks exactly like a refusal or a hallucination.

Also record latency, because routing decisions that ignore responsiveness can silently degrade the product even when quality holds. Time to first token is the metric that most affects perceived speed in interactive paths, while end-to-end latency matters when the full response must arrive before the next step (Anyscale on latency metrics). If your cheap judge is slower than the escalation target, you have a latency bug masquerading as a quality problem.

Run every raw output through a validator that fails loudly and specifically. The goal is not a boolean pass or fail but a machine-readable reason. A JSON Schema validator that reports the failing keyword and path turns "unparseable" into "missing required property plan_tier at root" or "value Enterprise does not match enum at /plan_tier."

This is the cheapest step in the whole process and it catches the format bucket almost completely. It also separates escaping problems (a quote inside a string value breaking the payload) from truncation (a payload that ends mid-token) from commentary (prose before or after the object). Each of those has a different fix.

For the escaping and commentary cases, a tolerant extraction layer often resolves the issue without touching the model: strip code fences, find the outermost balanced braces, then validate. If that alone drops your format rejections sharply, you have proven the model was fine and the parser was the fault. That is a successful diagnosis, not a workaround to be embarrassed about. Where available, prefer constrained decoding so the model cannot emit invalid structure in the first place, but do not let a clean parse lull you into skipping the factuality check (Song et al.).

The heart of the method is a fixed set of cases where you already know the correct answer and the intended failure mode. A golden dataset is exactly this: a stable set of test cases you run before every meaningful change, so you can tell whether a change helped or hurt (Langfuse on golden datasets). Evaluation datasets can hold inputs alone or inputs with expected outputs, and they pair with an evaluator that either checks properties directly or compares against ground truth (Evidently on test datasets).

Build the set deliberately so each case is designed to elicit one fault:

Keep the set small enough to run on every prompt change, large enough that a single flaky case does not move the number. Thirty to fifty well-chosen cases usually beat a thousand random ones for this purpose, because you are not measuring general quality, you are distinguishing causes.

For expected answers, define evaluation criteria as concrete, checkable statements rather than vibes. A worked pattern from an evaluation write-up shows the criteria as a list of yes-or-no questions about coverage, specifics, practical detail, tone, and forbidden content, paired with a dataset of input and expected output pairs (Seya on evaluation criteria). That mirrors the binary-verdict advice from judge tuning: ask "does this response contain any factually incorrect statements, yes or no" rather than "rate accuracy from 0 to 100" (Galileo).

Now tally. For each case, record the designed fault and the observed fault. The matrix tells you whether your judge fails in the way you think it does.

Read it by column and row together. Here, long context and adversarial phrasing collapse into format: the fix is a token budget and an extraction layer, not a better model. Missing field splits between factuality (the model guess-fills) and instruction (it skips the field), which tells you to add an explicit abstain instruction and a required-field check. The clean-control column is your alarm: two instruction failures on cases that should trivially pass mean the judge prompt itself is over-constrained.

The off-diagonal cells are the interesting ones. A designed factuality case that shows up as format usually means the model started fabricating and then broke the schema trying to fit the fabrication in. Treat that as factuality with a format side effect, because fixing the schema will not remove the invention.

The decision rule itself is simple enough to write on a sticky note. Swap the model only when the same cause repeats across varied prompts. Otherwise:

• Format dominates and finish reasons show truncation: raise the output cap, add constrained decoding, keep the model.
• Format dominates and finish reasons look normal: fix the parser or the prompt's output instructions, keep the model.
• Factuality dominates: the task needs recall or grounding the cheap model does not have on this data. Either improve retrieval and require source spans, or escalate this task class to a stronger model. This is the one bucket where a swap is often the correct answer.
• Instruction-following dominates: decompose the prompt. One job per call, constraints moved into the schema, a short checklist eval. A bigger model will paper over an overloaded prompt but you will pay for it on every request.

A caution on the swap: automatic selection by price or benchmark is risky when tool behavior, data policy, or quality requirements are strict, and a generic benchmark is a poor guide for a specialized task (Inworld on routing). Your confusion matrix is a better routing signal than any public leaderboard, because it is built on your data and your fault distribution.

Track a small set of routing-health metrics alongside the matrix: fallback rate, cache hit rate, and cost per successful outcome. A routing layer that looks accurate can still be inefficient, and silent drift in these numbers is the usual culprit (MLflow on routing). Cost per successful outcome is the number that actually decides whether your cheap model is cheap, and it is the number the matrix exists to protect.

A support team runs a cheap model as an extractor over policy documents. Every extracted claim is tagged with a refund window in days. The rejection rate on the downstream validator is 22 percent and the team is about to move to a frontier model.

Attribution on 200 sampled rejections, run through the steps above:

1. Raw logs show finish_reason: "length" on 61 of the 200, all on documents over roughly 3,000 input tokens. Format, truncation.
2. A schema diff on the remaining 139 finds 38 with an enum mismatch ("30 Days" versus 30) and 12 with an unescaped quote inside a reason string. Format, parser and prompt.
3. Of the 89 that parse, 54 cite a window not present in the source span. Factuality.
4. The last 35 parse correctly, cite a real span, but miss the secondary constraint ("also report the currency"). Instruction-following.

So format is 111 of 200, factuality 54, instruction 35. The proposed swap would have addressed about a quarter of the problem and left the dominant cause untouched. The fixes: raise the output cap for long documents, coerce enum types in the parser, add a currency field to the schema with an explicit example, and for the 54 factuality cases require the extractor to return the exact source sentence or abstain. Only if factuality stays high after grounding do you escalate that task class, and then you escalate it selectively rather than moving all traffic.

Replaying the labeled set after these fixes is what proves the diagnosis. If format rejections drop to near zero and factuality holds, you saved a model migration. If factuality stays high with grounding in place, you have earned the right to swap, and you can show the evidence.

Be honest about judge reliability throughout. Judges are unstable across scenarios, and rubrics or references help but are not always sufficient (LongJudgeBench). If your confusion matrix has a lot of mass in "observed pass" for cases you designed to fail, the problem may be the judge's rubric, not the model under test. Judges agree with human reviewers often enough to be useful for scoring open-ended text, but the wrong scoring method or rubric makes the scores just as flaky as the model being tested (Confident AI on judges).

When the taxonomy points to a task class that genuinely needs a stronger model, the natural next step is a single gateway that reaches text, image, video, audio, and embedding models through one OpenAI-compatible key with prepaid US dollar pricing, which is what Quanta Inference provides. That lets you escalate one task class without rebuilding each provider integration.