Scoring Outputs Before You Switch to a Cheaper Model

A threshold is a minimum pass rate for one intent on the golden set. Deciding it before you see results is the point. If you read the candidate model's scores first and then choose a bar, you are not gating, you are rationalizing.

Several inputs belong in the threshold decision.

Start with the incumbent model's own results on the same set. That number is your floor, and comparing a candidate against a real baseline avoids the classic trap of judging against an imagined standard. Run the current model and record its pass rate per intent before the candidate ever fires.

Then adjust for traffic weight. An intent that accounts for 2 percent of traffic deserves less strictness than one that accounts for 25 percent, all else equal. You can see that weighting in production traces, which is one reason span-level tracing pays for itself (Arize).

Then adjust for the cost of a miss. A wrong refund amount costs money. A wrong greeting tone costs a review. The threshold should reflect the consequence, not just the frequency.

A worked example, with invented numbers for illustration only: suppose your incumbent passes 98 percent on refund, 94 percent on cancellation, 96 percent on technical fault, and 99 percent on out of scope. You might set refund at 98, cancellation at 94, technical fault at 94, and out of scope at 98. Those numbers encode three decisions: no regression allowed on the money intent, a small allowance on cancellation because the judge itself is noisy, and no tightening on out of scope because the incumbent has near-zero headroom.

Small intents need special handling. If an intent has fewer than twenty or thirty cases, a single flipped case moves the pass rate by several points, so the threshold crossing is noise rather than signal. Options are to pool small intents into a related parent intent for gating purposes, to treat them as advisory only, or to grow the set for those intents before you rely on them. A set in the range of 50 to 100 well-chosen cases is a reasonable starting floor for the whole evaluation, which tells you how careful to be about tiny slices (qaskills.sh).

The single most common mistake in a model swap is measuring two changes at once. If the candidate model runs with a shorter prompt, a different temperature, or a trimmed context window, and the scores move, you cannot attribute the movement to the model. Change one variable at a time, a rule that shows up in prompt versioning guidance for exactly this reason (Microsoft Q&A).

Concretely, freeze the following across both runs: the exact system prompt text, retrieval settings (same index, same top-k), tool definitions, temperature and top-p, the judge model, and the judge prompt. Capture them in a run manifest so the comparison is reproducible later. A versioned prompt asset should snapshot the template, provider, model ID, hyperparameters, input schema, tools, and metadata, because any one of those changing alters system behavior (dev.to).

Non-determinism is the reason a single run per case is not enough. The same input can diverge across runs even with fixed settings, so use multiple trials and report pass rates rather than a single sample. The variation is exactly why you compare distributions by slice instead of eyeballing individual replies (Statsig). If your budget permits three trials per case, run three, take the pass rate per case, then aggregate to the intent.

For the judge, keep the same evaluator for both models. If the judge is an LLM, its prompt must be byte-identical between runs, including any rubric wording. A judge change between runs invalidates the comparison as surely as a prompt change does.

A minimal runner skeleton:

run(case, model, temperature=0.0, prompt_version=PINNED, top_k=5)
# for each case: 3 trials, judge each output, pass = 2 of 3
# aggregate: pass_rate = passes / cases, grouped by intent tag

Keep the raw outputs. Scores without outputs are unauditable, and the first question after a surprising drop will be "show me three failing examples in that intent."

The aggregate number is the least useful one you will compute. Aggregate pass rate hides redistribution: an intent that gains two points can mask another that loses fifteen, and the total stays flattering. Compare slices, not single replies; what matters is the distribution of outcomes by intent and segment (Statsig).

The comparison table is the deliverable. Build it as a gate artifact, with one row per intent, incumbent and candidate pass rates, the delta, and a verdict.

Consider the worked example continued: the candidate comes back at 99 percent on refund, 94 on cancellation, 95 on technical fault, and 91 on out of scope. Three of four intents clear. Out of scope drops 8 points against a threshold of 98 and an incumbent of 99. The aggregate might sit near 95, which sounds fine, and the swap is still blocked. That single row is the decision.

Now look at what that row means in practice. Out of scope is usually the refusal behavior: off-topic requests, attempts to extract the system prompt, questions about competitors. A model that handles those poorly is not a cheaper version of your assistant, it is a different and riskier product. Feature flag practice has a name for this pattern. An ops toggle exists so you can turn behavior off instantly without a redeploy, and the same instinct applies to model routing: the swap should be reversible by configuration, not by rollback of an artifact (Harness, Martin Fowler).

Two more subtle readings of the table are worth building into the review.

First, look for intents where the candidate is better. Those are candidates for partial routing: send that intent to the cheaper model and keep the incumbent for the rest. This is a real option, not a consolation prize, and it often captures most of the savings. A lower-priced model does not automatically make a cheaper system, because extra retries, extra turns, or extra context can erase the per-token gain, so partial routing should be validated the same way the full swap is (Arize, Braintrust).

Second, look at variance across trials within an intent. If the candidate swings widely on the same cases, a passing average can still be a fragile deployment. Record the spread alongside the mean.

A gate that does not block is a dashboard. Write the rule down before the run and enforce it mechanically: if any intent falls below its threshold, the swap does not ship, no exceptions granted in the moment.

The gate belongs in the same pipeline as your other regression checks, and ideally runs without a human in the loop. LLM regression tests run automatically against a fixed dataset, comparing quality metrics before and after a change, and flagging degradation before release (Evidently AI). If you already run prompt evaluations in CI, this is a second invocation with a different model string and a different threshold file.

Layer the release. Even a passing eval is evidence, not proof, so ship the candidate behind a flag to a small traffic slice first, watch the same intent metrics live, and expand only if the live slices agree with the offline table. Feature flags decouple deployment from release, which lets you expose a change to a subset and widen gradually rather than all at once (LaunchDarkly, Unleash). A cheap-model rollout is a textbook release toggle: small exposure, kill switch ready, measured against the same intent slices.

The log entry should be durable and boring. Record the date, both model identifiers, the prompt version hash, the dataset version, the judge version, the per-intent table, the verdict, and a link to the CI run. Two practical reasons: the next person asked "why are we still on the expensive model" needs the answer, and the same record becomes the baseline for the next candidate. Immutable versioning exists so a log can be tied back to an exact artifact rather than a fuzzy memory (dev.to, Braintrust).

Keep the gate artifact small and disposable. When the swap completes, retire the temporary flag rather than leaving it to rot, because unfixed flags accumulate debt and can silently evaluate to the wrong state later (LaunchDarkly).

A blocked swap is not a permanent answer. Models change under you, prompts drift, and the case set ages, so define the events that force a re-run.

Useful triggers:

• The incumbent's price, rate limits, or availability changes enough to alter the economics.
• A new candidate appears that you have not tested on this set.
• The golden set gains new cases from production failures, since a changed dataset invalidates the old comparison.
• The judge or rubric changes.
• The route, prompt, or retrieval configuration for an affected intent changes.
• A calendar checkpoint, quarterly or semiannual, to catch quiet drift.

That last one deserves emphasis. Prompts and model APIs can shift behavior over time without any code change, which is why regression suites are scheduled rather than event-only (Statsig, Evidently AI). If your last golden-set run is older than the oldest case you have added since, the comparison you are about to trust is stale.

Two disciplines make the triggers work. Keep thresholds in a file under version control, so a threshold change is a reviewable diff, and log every threshold change with its reason. And when a re-run flips a verdict, treat that the same as a code regression: open a ticket, fix or re-block, and confirm the fix with a fresh run.

For teams running several models across text, images, and embeddings, the practical burden of re-runs is mostly plumbing. Pointing every candidate at the same endpoint and the same judge reduces the runner to a configuration change; Quanta Inference exposes text, image, video, audio, and embedding models through one OpenAI-compatible API key (https://quantaintelligence.ai/inference), which keeps the manifest identical across candidates while only the model string varies.