A/B Testing Model Routing Rules Without Breaking Production

Before you can compare two policies, name the decision each one makes. A router is a function from request features to a model or provider, and most production systems stack several patterns: direct calls, provider fallback, conditional rules, weighted traffic splits and sticky assignment, as catalogued in these LLM routing strategies.

Useful inputs for a conditional rule include language, geography, subscription tier, interaction type, safety level, context length, modality and session depth, and the condition should have a product reason and an observable outcome. That last point is the one teams skip. A rule such as "long prompt equals expensive model" can be wrong for a simple summarization task, and the only way to know is to log the matched condition alongside cost, latency and the product result.

A practical policy definition has four parts:

1. The condition that fires, expressed over request metadata.
2. The model tier or specific endpoint the condition selects.
3. The fallback behavior when the first attempt fails or is slow.
4. The logging fields that let a reviewer reconstruct why this request went where it did.

If you cannot express the change as a diff on those four parts, the shadow test will not produce a readable result. "We lowered the threshold from X to Y" is testable. "We improved the router" is not.

The harness mirrors a representative slice of live requests to both policies. Representative is doing the work in that sentence. If the candidate policy only ever sees your easiest traffic, its quality scores will look great and mean nothing.

Three design choices decide whether the test is worth running.

Sampling. Start with a percentage of traffic rather than everything, because you are paying twice for each mirrored request. A five percent mirror on a steady workload gives you a large sample within days and bounds your worst-case bill to a known multiple of current spend. Exclude requests that must not be duplicated: anything with side effects such as sending an email, writing a record, charging a card, or calling a tool that mutates state.

Synchrony. A synchronous mirror adds the candidate's latency to the live path and can take production down. Send the mirror asynchronously, after the live response is returned, and accept that the two policies see the same input but not the same timing.

Logging. Every decision needs a record that joins the two policies on the same request ID.

{
  "request_id": "req_9f2c",
  "policy": "candidate",
  "matched_condition": "tier=free AND intent=summary",
  "selected_model": "reported by the gateway",
  "input_tokens": 3120,
  "cached_input_tokens": 0,
  "output_tokens": 410,
  "reasoning_tokens": 0,
  "retries": 0,
  "cost_usd": 0.0,
  "latency_ms": 1840,
  "quality_score": null,
  "scored_at": null
}

The token buckets matter as much as the total. A bill is the sum of uncached input, cache writes, cache reads, output and any separate tool charges across every billed attempt, and reasoning tokens are counted inside output usage on some endpoints even when the text is hidden, per this pricing guide. Logging output as a single number hides the two things most likely to have changed when you edited a threshold.

There is an observability argument here too. LLMs break traditional monitoring because billing is per token and the same request can vary wildly in consumption, so cost and usage tracking has to be instrumented rather than inferred from invoices, as this OpenTelemetry guide for LLM systems explains. A shadow harness is a small on-ramp to that instrumentation, and the fields you add for the test are the fields you will want permanently.

The most common way a shadow test fails is not a technical bug. It is that the team looks at the results, decides the numbers are "probably fine," and ships. Pre-agreeing the gates removes that discussion.

A gate has three parts: the metric, the threshold, and the rule for when the evidence is sufficient.

Two of these deserve emphasis. First, compare cost per request, not total spend. Total spend moves with traffic volume and will make a good policy look bad during a busy week. Second, quality must be measured by scoring the output, not by trusting the router's own signal. As Braintrust's routing guide puts it, pre-response signals select a model before any output exists, so prompt complexity and pricing can estimate cost but cannot show whether the selected model returned an accurate, relevant and correctly formatted response.

The scale of the risk is not hypothetical. A product team shipped an AI feature tied to a major release and finance found a $44,000 line item increase tied to the feature, well beyond what anyone modeled, because nothing in the stack attributed cost at the level of detail needed to catch it earlier, per this account of AI ROI measurement. Shadow logging at the request level is that level of detail, applied before the change instead of after.

Worked example with numbers

Take a support assistant handling 200,000 requests a month. The current policy costs $0.011 per request on average, so $2,200 a month. The candidate policy sends short, low-risk tickets to a cheaper model and keeps the rest where they are. You mirror 5 percent of traffic, which is 10,000 requests a month, and you pay for each mirrored request at the candidate's rate.

Suppose the candidate comes in at $0.006 per request on the mirrored sample, a saving of $0.005, which projects to $1,000 a month across all traffic. The mirror itself costs about $60 a month at the candidate rate, so the test pays for itself roughly twice over in the first month if the policy is promoted, and costs $60 if it is not.

Now the quality side. The automated check passes 94.0 percent of current-policy responses and 93.2 percent on the candidate, on the same 10,000 requests. With roughly 10,000 paired observations, the standard error on a proportion near 0.93 is about 0.26 percentage points, so a 0.8 point drop is roughly three standard errors and is not noise. That is a real regression, and the decision rule should say what happens next: either narrow the condition so the cheaper route covers fewer request classes, or keep the current policy. Widen the sample only when the gap is smaller than noise, not when it is unfavorable.

Note how the arithmetic is done. Cost difference times volume gives the benefit. The paired quality difference divided by its standard error tells you whether the difference is real. Both numbers come from the same mirrored requests, which is why the pairing matters.

Cost is arithmetic. Quality is a judgment, and the shadow harness will happily collect a judgment-free number if you let it.

The safe pattern is to classify requests into classes and give each class its own check. A routing threshold should apply to one query class because each task has a different definition of quality and cost of failure, and combining an easy task with a hard one lets strong results on the easy task conceal a serious regression in the other, per Braintrust's classification guidance.

Then choose the cheapest check that can actually fail:

• Deterministic checks for anything with a right answer: schema validity, required fields present, numbers that must reconcile, links that must resolve.
• Model-graded checks with a fixed rubric for open text, applied identically to both policies with the same grader so the grader's bias cancels.
• Human spot checks on a small random subsample, used to calibrate the automated checks rather than to score every request.

Run the automated checks against both policies on the same request IDs. A grader that sees only the candidate's output has no baseline and will drift toward "looks fine."

One more trap: do not let the shadow test double as a load test unless you have sized it. Mirroring a large share of traffic to a secondary provider can hit rate limits you did not know about, and the resulting errors look like quality failures. Cap concurrency on the mirror and log throttling separately from content failures.

Promotion is a sequence, not a switch.

1. Confirm both gates were met on the pre-agreed metrics with the planned sample.
2. Promote in steps: the same small percentage of live traffic first, then wider, holding each step long enough to see the tail metrics.
3. Keep the old policy configurable and one edit away, and define the rollback trigger in advance, for example a quality pass rate more than a set margin below the current policy for a full hour.
4. Re-check cost after promotion, because a live route sees a different traffic mix than the mirror did.

That last point is why post-promotion monitoring is not optional. AI spend rarely shows up as one clean attributable number: a gateway can aggregate traffic and strip the caller's identity before it reaches the provider, and a shared cluster can serve several products at once, which is why attribution is the hard half of AI ROI measurement, per DoiT's analysis. The gateway configuration that made your shadow test readable is what makes the live route readable afterward.

Reliability also deserves its own gate, because routing changes touch failover. Error fallback and latency fallback solve different problems, and a latency fallback can create simultaneous or sequential spend, so the timeout should come from the application budget and the primary route's normal p95, per Inworld's routing guide. If your threshold change alters when fallback fires, that belongs in the test.

If you are routing across several providers and model families, the mirror is easier when every model sits behind one key and one billing line. Quanta Inference exposes text, image, video, audio and embedding models through one OpenAI-compatible API with prepaid pricing in US dollars, which means the candidate and current policies can be logged against one cost ledger instead of reconciling several invoices.