A Test Set for Model Swaps: Label Once, Reuse Across Versions

The short answer is about 30 to 50 examples per task. That number is a starting point, not a law, and it follows from two constraints.

The first is stratification. You want the set to contain the easy majority, the awkward middle and the rare failures, in proportions that reflect production or that deliberately oversample risk. With fewer than about 30 examples, splitting into even three strata gives you cells of ten or fewer, and a single flipped label moves a stratum score by ten points or more.

The second is stability. The same prompt can produce different outputs across calls, so a passing test suite on one run does not guarantee consistent behavior (MLflow). If a task is high variance, either run each example more than once and average, or raise the count. A useful rule: if two runs of the same model on the same set disagree on more than a handful of examples, the set is too small or the rubric is too loose.

A workable allocation per task looks like this:

Two practical rules keep the set durable. First, never tune a prompt against the whole set; keep a dev slice for iteration and a sealed slice you only touch when comparing models. Second, version the set in the same repository as the code, so a change to an expected output is a reviewed commit with a diff. If you already run checks in CI, a nightly job that scores the current model against the set costs almost nothing, and the workflow file can be as small as the examples in this step-by-step GitHub Actions guide.

The rubric is where most test sets quietly fail. Criteria like "clear and helpful" or "professional tone" cannot be checked against text, so two reviewers apply them differently and the resulting labels encode the disagreement. The fix is to make every criterion observable: a reviewer should be able to point at the span of output that satisfies it.

Three design choices do most of the work.

Use a fixed, small scale. Three or four levels per dimension, each with a written description of what that level looks like. Avoid a 1 to 10 scale, which invites reviewers to invent distinctions the descriptions do not support.

Separate dimensions from the overall verdict. Score things like instruction adherence, factual correctness, format compliance and safety independently, then let the overall pass or fail follow from an explicit rule rather than a holistic feeling.

Anchor every level with an example. Not an abstract description, an actual output fragment that sits at that level. Anchors are what make calibration converge quickly.

Here is a compact example for a task that turns a support ticket into a structured reply:

Dimension: instruction_adherence (0-2)
  2 = Every instruction in the prompt is followed, including format and length limits.
  1 = All requested content is present, but one formatting or length constraint is missed.
  0 = A requested element is missing or contradicted.
Dimension: factual_correctness (0-2)
  2 = Every factual statement in the reference is present and no contradicting statement appears.
  1 = No contradictions, but one required fact is omitted.
  0 = Any statement contradicts the reference or invents a fact not in the reference.
Dimension: format_compliance (0-1)
  1 = Output parses against the expected schema.
  0 = It does not parse.
Overall pass = instruction_adherence >= 1 AND factual_correctness >= 1 AND format_compliance = 1

Notice that the correctness dimension is defined against a reference, not against reviewer intuition. That is the bridge to free text, covered below.

Calibrating two reviewers

Calibration is a loop with a stopping condition, not a meeting. The mechanics:

1. Both reviewers independently score the same 20 to 30 examples with the draft rubric.
2. Compute agreement and read every disagreement out loud.
3. Classify each disagreement as a rubric defect (the criterion was ambiguous), a reference defect (the stored answer was wrong or incomplete) or a reviewer error (the criterion was clear and one reviewer misapplied it).
4. Fix rubric and reference defects, rewrite the anchors, and repeat on fresh examples.

For the agreement statistic, use a chance-corrected coefficient rather than percent agreement. Percent agreement looks high when one class dominates the label distribution even if reviewers agree on the rare classes no better than chance (Appen). Krippendorff's alpha is a reasonable default because it handles any number of raters, missing labels and nominal through ratio data, with 1 meaning perfect agreement and 0 meaning chance (Appen, Label Studio's walkthrough, worked formulas).

Two cautions. First, pick the measurement level deliberately: nominal weights penalize every mismatch equally, while ordinal weights penalize adjacent-level disagreements less, and the same data can produce quite different scores depending on which you choose (Appen's worked example). Second, if you only have two raters and a binary or nominal label, Cohen's kappa is the conventional measure, but it is defined for exactly two raters, while alpha generalizes (comparison of coefficients, review of nominal IRR coefficients). Whichever you use, report it next to the test set version so a later reader knows how much to trust the scores.

A rough decision rule from the same source: treat low alpha as a signal to fix the guideline with boundary examples rather than to add more annotators (Appen). In practice, that means your first calibration round should expect low agreement and treat it as normal.

Exact match fails on free text, and similarity metrics like BLEU or ROUGE reward surface overlap rather than correctness. There are four practical options, and the right one depends on how deterministic the task is.

The survey of evaluation methods by Sebastian Raschka groups the field into multiple choice, verifiers, leaderboards and LLM judges, and it is worth reading for the trade-offs before you commit (Understanding the 4 Main Approaches to LLM Evaluation). The relevant point for a swap is that judge-based and rubric-based scoring both depend on a stable definition of good, which is why the rubric has to exist before you reach for a judge.

For free text specifically, the recommended combination is reference plus tolerance plus rubric. Store the reference answer, store an explicit list of acceptable variations (a different but valid summary, a reordered list, an equivalent phrasing of a required fact), and score with the dimensions above. Write the tolerance list at labeling time, when the context is fresh, not later when a model has already produced something surprising.

Using a judge without fooling yourself

An LLM judge is a reasonable way to scale rubric scoring, provided you validate it against human labels on the calibration subset and report the agreement. Two habits keep it honest.

First, judge per dimension, one call per criterion or one call returning the structured dimensions, never a single overall score. Second, rotate the order of candidate outputs when comparing, so position does not decide the winner.

Here is the shape of a scoring script that keeps the prompts and the rubric in one place, so the same file runs against whatever provider you point it at:

import json, statistics

def score_all(cases, call_model, rubric_text, k=3):
    results = []
    for case in cases:
        runs = [call_model(case["prompt"]) for _ in range(k)]
        scores = [grade(rubric_text, case["reference"], r) for r in runs]
        results.append({
            "id": case["id"],
            "stratum": case["stratum"],
            "score": statistics.mean(s["total"] for s in scores),
            "dims": {d: statistics.mean(s["dims"][d] for s in scores)
                     for d in scores[0]["dims"]},
        })
    return results

Run that against the incumbent, store the per-example scores as a baseline file, and every future candidate is compared to that file rather than to a memory of how the old model felt.

A model swap is a release, and releases benefit from the same staging discipline as deploys. Canary releases exist precisely so a new version serves a fraction of traffic while the rest stays on the known-good version; in Cloudera's model serving API, for example, a traffic field diverts a chosen percentage of requests to a new model version while the remainder stays on the previous one, and promoting means setting that value to 100 (Cloudera canary deployment docs). Amazon API Gateway documents the same construct for API deployments (canary release deployment).

Your held-out set plays the role of the pre-traffic check. A workable sequence:

1. Seal the set. Freeze the prompts, references, tolerances and rubric, and record the test set version.
2. Baseline the incumbent. Score the current model or provider on the sealed set, k runs per example. Save the per-example results, not just the mean.
3. Score the candidate. Same prompts, same rubric, same k. Do not edit the rubric between the two runs.
4. Compare per stratum, not just overall. An overall mean can hide a regression in the failure-mode stratum, which is exactly the stratum you built the set to protect.
5. Decide against a pre-registered rule. For example: ship if overall pass rate is no worse and no stratum drops by more than a stated margin. Write the rule before you see the numbers.
6. Canary the winner. Send a small traffic share to the new model behind a flag, watch latency, error rate, token consumption and eval scores, and revert by flag if the numbers move the wrong way (MLflow).

The rollback half of this deserves the same care as the forward path. A rollback is only useful if it has been tested; a strategy that has run once in four years is not a strategy (Octopus on modern rollback strategies). Harness's pipeline rollback, which triggers a failure strategy on defined error types and rolls the pipeline back, is a concrete example of wiring the reversal into the same system that does the forward run (Harness rollback pipelines).

When the test set itself has to change

It will. New intents appear, the reference answers age, and a model may legitimately produce a correct output your tolerance list did not anticipate. Treat edits with the same discipline as schema migrations: add before you remove. The expand and contract pattern introduces the new structure alongside the old, migrates consumers, and only then drops the old one, keeping every intermediate step reversible (Tim Wellhausen's paper, Prisma's guide).

Applied to a test set: add the new case or wider tolerance, mark the old version deprecated but keep it scoring, run both in parallel for a period, then retire the old one. The reason is the same as in schema work: application code and schema changes cannot deploy atomically, so coupling them multiplies risk (PlanetScale on backward compatible changes). A rubric change mid-comparison invalidates the comparison the same way, which is why the rubric is versioned with the set.

A worked example with numbers makes the value concrete. Suppose a task has 40 examples, baseline pass rate 33 of 40, and a candidate scores 30 of 40. Overall that is a drop of 7.5 percentage points. If the failure-mode stratum went from 5 of 6 to 2 of 6, the overall number understates the problem: the candidate broke exactly the cases you care about, and the right call is to reject or investigate, not to average it away.

You do not need a large system. The minimum viable version is a directory in the repository holding prompt files, a JSONL file of cases with id, stratum, prompt, reference and tolerance, a rubric document with a version string, a scoring script, and a CI job that runs on a schedule against the current production model. The GitHub Actions docs cover the workflow file, the events that trigger it and the syntax for jobs and steps (GitHub Actions documentation, workflow syntax, step-by-step guide).

Two engineering details save time later. Keep the model call behind a thin interface that takes a prompt and returns text, so swapping providers is a configuration change rather than an edit to every test. And log the provider, model identifier, prompt version and test set version with every run, because a comparison is only meaningful if you can reconstruct what was compared.

If the swap involves several modalities or several providers, routing every call through one interface removes most of the per-provider glue. Quanta Inference exposes text, image, video, audio and embedding models through one OpenAI-compatible API key, which is the natural next step once your test set is provider-independent: https://quantaintelligence.ai/inference.