Pinning a Rubric to a Judge Call: Versioning and Rerun Protocol

A rubric version should communicate intent to a human, and only after that identify bytes. Use semantic versioning with a clear policy:

• Patch (1.1.0 to 1.1.1): typo fixes, formatting, whitespace, anything that should not change judgments. If the hash changes on a patch, you must still rerun, because the judge reads bytes, not intent.
• Minor (1.1.1 to 1.2.0): additive change, such as an extra edge-case example at score 5 or a clarified anchor. Scores remain comparable in direction but not strictly.
• Major (1.2.0 to 2.0.0): weights changed, criteria added or removed, scale changed. Old and new scores must never be averaged.

Store the rubric as a single canonical file in source control, and make the version label an attribute of that file, not a string typed into code. A rubric that lives half in a prompt template and half in a spreadsheet will drift. Keep it in one place, review changes like code, and record who approved each bump. The discipline of version control for evaluation assets is the same one that makes evaluation harnesses trustworthy at all the MLflow evaluation harness article.

One trap to avoid: adjectives without anchors. A rubric becomes unstable when score labels are adjectives rather than decision rules, because labels like excellent, good and adequate leave the judge to invent boundaries on every call the rubric consistency guide. Anchor each level to observable evidence. That makes the rubric easier to hash meaningfully, because every byte corresponds to a claim.

The hash must be computed over the exact string handed to the judge, after all templating, whitespace normalization and any per-model formatting. Not over the file on disk. Not over the version label. The bytes that matter are the bytes the judge read.

import hashlib

def rubric_hash(rubric_text: str) -> str:
    # Hash the exact string sent to the judge, UTF-8 encoded.
    return hashlib.sha256(rubric_text.encode("utf-8")).hexdigest()

def judge_call_record(sample_id, model_id, rubric_text, rubric_version, score, raw):
    return {
        "sample_id": sample_id,
        "model_id": model_id,
        "rubric_version": rubric_version,           # rubric-v3, human readable
        "rubric_sha256": rubric_hash(rubric_text),  # source of truth
        "score": score,
        "raw_response": raw,
        "created_at": "2026-01-01T00:00:00Z",
    }

Two fields, two jobs. The version tells a reviewer what changed and why. The hash tells the pipeline whether two calls are comparable, without trusting any human to have relabeled correctly. If someone edits the rubric and forgets to bump the version, the hash catches it. If someone bumps the version without touching a byte, the hash forgives it: same bytes, same comparability, and you avoid a pointless rerun.

Ask the judge for a structured output so scores are parseable and comparable across calls. Structured Outputs guarantees the response adheres to a supplied JSON Schema, so you do not have to worry about the model omitting a required key or hallucinating an invalid enum value the OpenAI structured outputs guide. That matters here because a judge that occasionally returns prose instead of a number creates the very parse failures that look like rubric drift. Providers differ in their support for schema-constrained responses, and one comparison of function calling, JSON mode and structured outputs notes that Gemini only supports structured outputs through function calling rather than a standalone final-response format the Vellum guide. Your hash protocol is provider-agnostic; your output parsing needs to be.

A stronger option, when you run open models, is grammar-constrained generation. Constrained decoding intervenes at the token level so invalid tokens are masked and the output cannot structurally deviate from the schema, and the masking step filters tokens before sampling rather than validating afterward the constrained decoding survey. It eliminates the entire category of structural failures, leaving semantic judgment as the only variable, which is precisely what you want when defending a comparison.

The gate is a single query. For a comparison run, group calls by rubric hash and refuse to produce a result unless the group contains exactly one hash.

SELECT rubric_sha256, COUNT(*) AS calls, COUNT(DISTINCT model_id) AS models
FROM judge_calls
WHERE run_id = :run_id
GROUP BY rubric_sha256
ORDER BY calls DESC;

If that returns one row, you are fair. If it returns more, you are not, and no amount of statistical cleverness rescues the number. Exclude stale rows from aggregate scores and treat the surviving hash as the run's definition.

The detection question is worth separating from the noise question. Within-item repeatability asks whether the judge returns the same or adjacent score across repeated calls with an identical rubric; temporal drift asks whether the score distribution has moved after a model or prompt change the rubric consistency guide. The hash answers drift by construction. It cannot tell you whether the judge is noisy, only whether you are comparing like with like. Keep a small frozen corpus that you rerun at every rubric bump so you can see how much the change moved scores on samples whose answers did not change.

Rubric instability also interacts with sample size. If you plan to claim one model beats another, decide the corpus size before you run, and use a two-proportion calculation to set it. As a reference point, detecting a move from 15% to 25% at 95% confidence and 80% power requires 248 samples per group the Select sample size calculator. Underpowered comparisons flip on noise, and rubric churn turns noise into an apparently meaningful result.

The rerun protocol is a join, not a reset. Every call that already carries the current hash is valid evidence. Only the calls on superseded hashes are invalid.

1. Freeze the current rubric. Compute its hash and record it as the run's target.
2. Select the affected calls: all rows in the run whose rubric_sha256 differs from the target.
3. Collect the distinct (sample_id, model_id) pairs from those rows. That set is the rerun worklist.
4. Reissue calls only for that worklist, with the frozen rubric bytes.
5. Replace, do not append, the superseded rows so aggregates cannot double count.
6. Rerun the drift query and confirm exactly one hash.

That last row is the one teams forget. Hash stability protects the rubric, not the judge. A judge model swap is a separate experimental condition and must be recorded and rerun across the full corpus, because a change in the judge is a change in the instrument. Constrained decoding and logit masking ensure the schema holds, but they do not constrain which evidence the judge weighs.

Suppose you are comparing four models on 200 samples, so 800 judge calls. You start with rubric v3, hash a1b2..., and complete 600 calls. Mid run, someone tightens the correct-but-incomplete anchor and saves the file without bumping the version.

The drift query returns two rows: a1b2... with 600 calls and c3d4... with 200 calls. Without the hash you would report a full comparison from mixed instruments.

With the protocol, you exclude the 200 stale rows, collect the distinct sample and model pairs behind them, and reissue only those calls against the frozen current bytes. Depending on how the 600 were distributed, you rerun 200 to 800 calls, not 1,200. Suppose the average call costs a fraction of a cent and takes two seconds. Rerunning 200 calls instead of the whole set is a fifth of the work and a fifth of the wall clock, and it fixes the exact rows that were compromised.

Now suppose a second change adds an edge-case example, so you bump to v4. If you had already published a leaderboard on v3, do not quietly replace the numbers. Rerun the affected samples, publish v4 with the v3 figure attached as a note, and keep the hash in the report. Anyone reading the table can verify that the model ranking rests on one instrument. That habit also protects the result from a subtler threat: access providers that serve mislabeled or downgraded models. An audit of shadow APIs found performance divergence reaching up to 47.21% against official endpoints and identity verification failures in 45.83% of fingerprint tests the shadow API study. A pinned rubric plus a recorded endpoint is the minimum evidence trail for any number you cite.

For rerunning across several providers, it helps to route every judge call through one interface so the rubric bytes and record fields are assembled by a single code path. Quanta Inference exposes text, image, video, audio and embedding models through one OpenAI-compatible key with prepaid USD pricing, which keeps the call path uniform at https://quantaintelligence.ai/inference.

Make rubric pinning a build step, not a reviewer's promise. A pre-merge check can compute the hash of the rubric file, compare it to the version's registered hash, and fail if a patch or minor version changed bytes without a corresponding rerun entry. The same harness pattern that catches model regressions in CI applies here: a runner, provider adapters and a scorer, all version controlled so the same task and the same scorer produce numbers you can compare later the MLflow evaluation harness article.

Two failure modes are worth explicit tests. First, the forgotten bump: assert that every change to rubric bytes is accompanied by a version change. Second, the phantom bump: assert that a version change with no byte change is flagged as a label error, not a rerun trigger. Together they keep the label and the hash aligned without forcing anyone to remember which one matters. When a comparison crosses teams, hand over the report with the hash, the version, the judge model ID and the corpus version. Four fields, and the result becomes reproducible.