A Quality Gate for a Cheaper Model: Scoring Before You Switch

The most common mistake is testing on clean, synthetic prompts that look nothing like real usage. A test set should reflect the distribution of requests your system actually receives, including the ugly ones. LLM evaluation guides are consistent on this: an evaluation dataset is a fixed set of test cases representing real-world usage, and it should include happy paths, edge cases and adversarial inputs (Evidently AI's guide to LLM test datasets).

Two sources of examples, in order of preference:

1. Real production prompts, sampled and cleaned. Pull a stratified sample from logs: proportional to your traffic mix by task type, language, length bucket and user tier. Redact anything sensitive, but keep the raw structure (typos, missing context, mixed language).
2. Synthetic variations on real examples, when logs are thin. Generate paraphrase and edge-case variants from a seed set when you lack volume, but keep the seeds real (Evidently AI).

Define the test set as a versioned artifact before you start. A "golden dataset" is the fixed set you run against before any meaningful change, and the discipline is to version it and keep it stable (Langfuse's guide to golden datasets). A practical file format keeps the inputs, the expected behavior and the subset tag together:

[
  {
    "id": "ex-0042",
    "input": "Summarize this support thread in 3 bullets.",
    "context": "...",
    "subset": "summarization_critical",
    "expected": "Three bullets covering refund, timeline, next step.",
    "metric": "rubric_1_5"
  }
]

Tag each example with a subset. At minimum: one critical subset (things that must not regress, like legal disclaimers, JSON validity, safety refusals) and one non-critical subset (nice-to-have style and length adjustments). The gate applies different rules to each.

How many examples do you need? At least 100 for the overall set. For a per-subset decision, aim for 50 or more in that subset, and be honest about what a small subset can tell you. See the next section for the arithmetic.

You cannot compare two models on "vibes". Pick one primary metric and make it mechanical enough to run repeatedly. For a quality gate, choose the primary metric that matches your dominant evaluation approach:

• Exact match, schema validity or pass/fail checks for structured outputs (JSON, extraction, code that runs). Cheap, objective, no judge required.
• Rubric scoring, 1 to 5 per criterion, for open-ended output where an automated judge or a human scores against a written rubric.
• Pairwise comparison where a judge or human picks the better of two outputs for the same input, useful when absolute scores are noisy.

The systematic approach many teams use is a layered suite: unit tests for deterministic behaviors, then LLM-as-a-judge or rubrics for the open-ended ones (a walkthrough of LLM evals with unit tests and judges). Whichever you pick, write the rubric before you look at any outputs, and hold the judge constant across both models. If a judge model scores the current model and a human scores the candidate, the comparison is meaningless.

One practical rule: the primary metric should be the one that maps to user harm. If a wrong answer means a refund gone wrong, your critical subset metric is primary. Style is secondary. Prioritize accordingly.

Run both models on the same examples, same prompt template, same parameters, and the same scoring pipeline. The only variable is the model. Record per-example scores so you can compute subset results and paired differences rather than just overall averages.

A small runner makes this repeatable:

import json, statistics

def score_all(model_fn, examples, judge):
    results = []
    for ex in examples:
        out = model_fn(ex["input"], ex.get("context", ""))
        results.append({"id": ex["id"], "subset": ex["subset"],
                        "score": judge(out, ex)})
    return results

def subset_means(results):
    by = {}
    for r in results:
        by.setdefault(r["subset"], []).append(r["score"])
    return {k: statistics.mean(v) for k, v in by.items()}

Run it against the candidate and the incumbent. Then write the comparison down as a table before you decide anything, so the numbers are visible and arguable:

The overall row shows a tiny difference, but the critical subset and the schema-valid check both fail. That is the gate working. A cost saving never justifies a schema failure on a critical path.

This is the step most teams skip, and it is the decision rule that makes the gate more than a vibe check. A 1-point difference on 100 examples is not automatically real. If 92 and 91 out of 100 users get the same answer, the difference could easily be sampling noise.

Use a simple significance test on the paired results. For pass/fail per example, a paired comparison over the same examples is right: count the cases where the candidate wins and where the incumbent wins, and only the cases where they disagree matter. A rough 95% confidence interval on the difference of two proportions (with n around 100) is about plus or minus 10 percentage points at worst, and tighter near the middle of the range. That is why a 1-point gap is not by itself proof, and why the gate asks two questions:

1. Is the candidate within tolerance on every subset? (Within 1 pp on critical subsets, within a wider band on non-critical.)
2. Is the observed gap distinguishable from noise at 95% confidence, or is the candidate actually better?

The rule that makes sense in practice: do not switch if the candidate is worse than the incumbent by more than the tolerance on any critical subset, even if the overall numbers look close. If the candidate is indistinguishable from the incumbent, treat that as acceptable only when cost savings are meaningful and you plan a canary. If the candidate is better, switch. The point is to refuse the swap when the evidence does not support it, not to demand statistical purity every time.

A small script for the paired test:

# Only disagreements matter for a paired sign test.
def sign_test(candidate_scores, incumbent_scores):
    wins = sum(c > i for c, i in zip(candidate_scores, incumbent_scores))
    losses = sum(c < i for c, i in zip(candidate_scores, incumbent_scores))
    n = wins + losses
    # Rough two-sided 95% interval on the win rate.
    p = wins / n if n else 0.5
    margin = 1.96 * (p * (1 - p) / n) ** 0.5 if n else 0
    return p - margin, p + margin  # interval around the win share

If the interval straddles 0.5, the cheaper model is not clearly better or worse; combine that with the subset check to decide.

Even a passing gate is a prediction, not a guarantee. Production has inputs your test set never saw. So treat the swap as a progressive release: use a feature flag or an A/B split so the candidate sees a small share of traffic first, then widen exposure as metrics hold (LaunchDarkly on progressive rollout and kill switches). Traffic splitting and sticky assignment keep a single user on one model during the test, so behavior does not bounce between models mid-session (Inworld's routing strategies guide).

Write the rollback rule before you flip the flag, in the form of a numeric checkpoint:

Then for the first week, watch the live signals against that same test set, replayed daily. Log the model and route on every request so you can join quality outcomes back to the decision (merge.dev on LLM routing strategies). A pass in the gate plus a clean week in the canary is the signal to complete the rollout. A 2% degradation is the signal to revert instantly, not to open a discussion.

Suppose you have 100 labeled examples: 40 critical (JSON extraction for a billing flow) and 60 non-critical (marketing copy variants). The incumbent scores 96% on the critical subset and 85% on the rest. The candidate scores 94% on critical and 84% on the rest. A 1 point difference on critical and a 1 point difference on non-critical.

Here is how the gate reads it:

• Critical subset: 2 pp below the incumbent, outside the 1 pp tolerance. Fail, regardless of cost. The primary metric is the one tied to user harm, and billing extraction is that metric.
• Non-critical subset: 1 pp below, inside the 3 pp tolerance. Pass.
• Cost: the candidate may be materially cheaper, but cost does not override a critical-subset failure.

Decision: do not switch on this evidence. Options, in order: improve the prompt or schema handling to recover the critical subset, expand the test set to see whether the 2 pp gap is a small-sample artifact, or route only the non-critical task to the cheaper model on a conditional basis. That last option is exactly the pattern of routing cheap tasks to a cheaper model and reserving the capable model for the rest (merge.dev on minimizing spend per request). The decision rule produced a smaller, safer switch instead of a blanket one.

This is also where a single-API control plane helps. If both models sit behind one API, running the comparison, shipping behind a flag and swapping back is a configuration change rather than a migration; Quanta Inference offers text, image, video, audio and embedding models through one OpenAI-compatible key, which makes candidate-and-incumbent scoring a straight comparison.

Testing on toy prompts. The set must mirror traffic. A gate that passes on clean prompts and fails on real ones wastes a release cycle.

Using a different judge for each model. Hold the scoring method constant. If you change the judge, you are comparing judges, not models.

Averaging over subsets. An overall score of 91 versus 92 hides a critical subset at 98 versus 94. Report per subset, and gate per subset.

Skipping the significance check. Small deltas on small sets are noise. Acknowledge it and widen the set or accept only "distinguishable or within tolerance".

No rollback rule. A swap without a numeric revert condition is a gamble. Write the threshold down before you flip it.

Treating the gate as a one-time event. Re-run it whenever the candidate model is updated, the prompt template changes, or your traffic shifts. The same golden dataset discipline applies every time (Langfuse).

For teams that want the scoring, comparison and flag rollout maintained as a standing workflow rather than a one-off script, Quanta Skills packages the evaluation pattern as a skill you can install into a coding agent.