Keeping a Scoring Prompt in Sync Across Two Models

Before you write a single test, know where the seams are. These are the failure modes that show up first when you run the same scoring prompt on two models.

Tokenization. A prompt that fits comfortably in model A's context can be truncated or split differently in model B. Long drafts, code samples and tables are the usual casualties. Truncation rarely throws an error; it just quietly removes the part of the draft that carried the evidence for criterion three. Always log the token count of the assembled prompt and the completion, and assert that the completion did not stop for length reasons.

Instruction following. Some models obey "return only JSON" and some wrap it in a sentence of preamble. Some follow a numbered list of criteria in order; others merge criteria or skip one they consider redundant. Braintrust's prompt testing guide treats format compliance as a separate, deterministic check precisely because it fails independently of answer quality. Score format validity in code, not with a model.

Output formatting. Even when both models return JSON, key order, number types (3 versus "3" versus 3.0) and extra fields vary. A schema validator catches this, but only if you actually run one. The Autorubric framework paper notes that rubric evaluation needs explicit handling for criterion types (binary, ordinal, nominal), because a model that returns a float for an ordinal scale has silently changed the scale.

Refusal behavior. A draft about a sensitive topic may trigger a refusal in one model and a score in the other. Refusals are not scores and must never be coerced into them. Braintrust calls these behavioral boundaries: whether the model should refuse, escalate or proceed is a testable expectation, and a missed or spurious refusal is a regression like any other.

Numeric interpretation. Ask for a 1 to 5 score and one model clusters at 4 while the other spreads across 2 to 5, even when both agree on which drafts are better. This is a scale-calibration difference, not a quality difference. LangChain's write-up lists known judge biases, including verbosity and position, and the Autorubric paper adds length penalties and per-criterion evaluation as mitigations. The practical consequence: compare criterion-by-criterion agreement and rank correlation, not raw averages.

The table doubles as a checklist. If you cannot name which row caught a given discrepancy, you have not localized it yet.

The single most effective habit is to treat the scoring prompt as source code. Put the prompt text, the rubric definitions and the output schema in one file, commit it, and have every model call load from that file. Nothing is pasted into a chat window; nothing is edited per provider.

# judge_rubric.v3.yaml
version: 3
criteria:
  - id: correctness
    scale: ordinal_1_5
    anchors:
      1: "Central recommendation is unsafe or factually incompatible with the source"
      3: "Core diagnosis is sound but an important limitation or step is missing"
      5: "Meets level 4 and handles relevant edge conditions precisely"
schema:
  type: object
  required: [criterion_id, score, evidence]
  properties:
    criterion_id: {type: string}
    score: {type: integer, minimum: 1, maximum: 5}
    evidence: {type: string, minLength: 10}

Three design choices matter here. First, one criterion per call. The QASkills guide warns against combining correctness, style, completeness and safety into a single holistic number, because it forces the judge to negotiate weights the rubric never stated. Per-criterion calls are more expensive but far easier to compare across models.

Second, anchor every level to evidence that can be found in the draft, not to adjectives. The example above borrows the anchoring style from QASkills, where level 4 permits immaterial blemishes and level 5 requires useful treatment of edge conditions. A judge deciding between 3 and 4 can cite the missing limitation, which makes disagreements diagnosable.

Third, require an evidence field. It gives you something to read when two models disagree, and it discourages the judge from returning a number with no grounding. LangChain makes the same point about chain-of-thought in evaluation prompts: a rationale makes disagreements easier to diagnose.

When you record each run, store the rubric version, the model identifier, the temperature, the output schema version and the full prompt token count alongside the score. Braintrust insists that these settings travel with the prompt version, so a score difference can be traced to a model swap as readily as to an edited instruction. Without that record, a diff between two runs is guesswork.

You cannot regression-test a swap on drafts you generate during the test. You need frozen inputs: real drafts saved with their metadata, scored once under the current model to establish a baseline.

Start with a corpus large enough to be informative but small enough to inspect by hand. The QASkills guide recommends building a diagnostic corpus that deliberately includes near-boundary cases, because that is where score variance concentrates. A useful split:

• About 60 percent ordinary drafts that the current judge scores without drama.
• About 25 percent near-boundary drafts where a human reviewer hesitated between two adjacent scores.
• About 15 percent adversarial drafts: empty content, contradictory instructions, sensitive topics that might trigger refusal, and very long inputs that stress the token limit.

Then split the corpus into a tuning set and a held-out set. Write the acceptance threshold down before you run anything. A defensible starting rule: the new model must match the current model's criterion score exactly on at least 80 percent of held-out items, or differ by at most one point on at least 95 percent, and must not increase refusals or schema failures. The 80 percent figure is not arbitrary: LangChain reports that strong LLM judges reach about 80 percent agreement with human evaluators, roughly human-to-human agreement, so demanding more than that between two models is asking for more consistency than the task supports.

If you want the corpus to double as a human calibration set, have one domain expert score the held-out items independently. Label Studio's overview of evaluation methods calls human judgment the gold standard and recommends cross-validating automated judges against human-reviewed examples. A model that agrees with itself but not with your expert is stable and wrong.

The test procedure is deliberately boring. Run the current model and the candidate model over the same saved drafts, same rubric file, same schema, same temperature, and store every raw response before parsing.

results = []
for draft in heldout_set:
    for model in [current_model, candidate_model]:
        raw = call(model, rubric_file="judge_rubric.v3.yaml", draft=draft)
        results.append(parse_and_validate(raw, model, draft.id))

Then compute, per criterion:

1. Exact agreement rate between the two models.
2. Adjacent agreement rate (differ by at most one point).
3. Weighted kappa, which corrects for agreement that would happen by chance. Autorubric reports weighted kappa for ordinal scales alongside accuracy and distribution comparisons, and notes that kappa must be computed with the ordinal scale in mind, since an unweighted kappa on an ordinal scale understates real disagreement.
4. Refusal rate and schema failure rate for each model.
5. Score distribution per model, to expose the clustering-at-4 problem.

A worked example makes the decision concrete. Suppose you have 200 held-out drafts and four criteria, so 800 judgments per model. The current model and the candidate agree exactly on 612 (76.5 percent), within one point on 771 (96.4 percent), weighted kappa is 0.71, and the candidate refuses on 4 drafts where the current model scored. Your pre-registered rule said 80 percent exact or 95 percent adjacent. Exact agreement misses the first bar but the adjacent bar clears, so the question becomes whether 96.4 percent adjacent plus a kappa of 0.71 is acceptable for your use. If this judge gates deploys, the four new refusals are a hard fail regardless of the aggregate, because a refusal is a missing decision, not a borderline one.

That is the decision rule in practice: aggregate agreement sets the overall verdict, but a veto on any criterion where safety or compliance is at stake overrides the average. QASkills makes the same recommendation: if any safety failure must block release, express it as a veto rather than hoping it lowers an average enough.

Expect some variance even from the same model on repeated runs. QASkills suggests repeating judgments under identical conditions and measuring within-item dispersion before you ever compare two models, so you know the noise floor. If the current model alone varies by one point on 10 percent of items, a 96 percent adjacent agreement between models is close to the ceiling you can expect.

Two models agreeing is not the same as two models being right. The comparison inherits whatever biases both share, and the literature names several.

Verbosity bias is the best documented: judges tend to favor longer answers. Autorubric implements length penalties as a mitigation, and the QASkills guide points out that a level 5 anchor should require useful edge-case handling, not more words. Position bias matters for pairwise comparisons, which is why LangChain recommends option shuffling when a judge sees two outputs. For a single-draft rubric, position bias shows up as criterion order effects, another argument for one criterion per call.

There is also the risk of treating a stable judge as a correct one. A judge that always returns 5 is perfectly repeatable and useless. Label Studio is blunt about this: LLM-as-judge is fast and scalable but introduces model biases, and works best when cross-validated with human-reviewed examples. Sebastian Raschka's survey of evaluation approaches groups LLM judges alongside multiple-choice benchmarks, verifiers and leaderboards, noting that each answers a different question. A rubric judge answers "does this draft meet our standard," not "is this model better."

The practical takeaway for a swap: keep a small human-scored calibration set and re-check both models against it whenever you change the rubric. If the candidate model agrees with the expert at the same rate as the incumbent, the swap is safe on calibration grounds even if exact agreement between models is imperfect. If the candidate agrees with the expert better, the swap is an improvement, not a regression.

Once you have a frozen rubric, a saved corpus and an acceptance threshold, comparing models becomes routine, and the friction shifts to plumbing: different SDKs, different authentication, different response shapes for the same JSON schema. That is the point where a single OpenAI-compatible endpoint pays for itself. Quanta Inference routes text, image, video, audio and embedding models through one API key, prepaid and priced in US dollars, so the swap test above is a configuration change rather than an integration project. Keep the rubric file and the test harness; change the model identifier.

If the scoring work sits inside a larger review process (editorial gates, support ticket triage, code review) and you would rather have a specialist run the calibration and hand back the regression report, Quanta Experts reads your context, does the work and returns finished deliverables.