Testing Output Against a New Model Without Burning the Budget

Shadow traffic testing duplicates live requests to the candidate model, logs the candidate's responses, and compares them offline. Users only see outputs from the current production model, so there is zero risk to user experience (CodeAnt). The mechanics are straightforward: duplicate requests at the API gateway, application layer, or load balancer; route the copy asynchronously to the candidate; capture both responses with metadata such as latency, token count and timestamp; compare offline.

Four practical rules separate a clean shadow run from a mess:

Keep the candidate out of the rendering path entirely. The candidate call should be fire-and-forget from the user's perspective. Timeouts on the candidate should never block or slow the production response, because a slow candidate that drags down your p99 latency has already contaminated the experiment.

Send the identical input, including the schema. The comparison is only valid if both models receive the same prompt, the same system message and the same JSON Schema. If your baseline uses a provider-native structured output mode with a Pydantic or Zod schema, the candidate must receive the equivalent constraint expressed the same way, not a prose description of the shape (OpenAI). Passing the schema formally rather than describing it in prose reduces field-name drift, because the model copies names rather than reconstructing them (GMI Cloud).

Handle sensitive data deliberately. Your logs now contain two copies of real user input and model output. Apply the same data handling and encryption policies you use for production data (CodeAnt). If the protocol you use for context or tool access includes consent requirements, the shadow copy inherits them: hosts must obtain explicit user consent before exposing user data to external servers (MCP specification).

Record cost per request, not just per token. Parse failures trigger retries, and each retry is a full inference call at full cost and latency (tianpan.co). Log attempt counts so the cost comparison reflects the retry tax, not the sticker price per token.

A minimal recorder looks like this:

import json, time, uuid

def shadow_call(candidate_client, request, schema, endpoint_label):
    record = {
        "id": str(uuid.uuid4()),
        "endpoint": endpoint_label,
        "prompt_hash": hash(request["prompt"]),
        "candidate_raw": None,
        "parse_ok": False,
        "schema_ok": False,
        "latency_ms": None,
        "attempts": 0,
        "tokens": None,
    }
    start = time.monotonic()
    try:
        resp = candidate_client.responses.create(
            input=request["prompt"],
            response_format={"type": "json_schema", "json_schema": schema},
        )
        record["attempts"] = 1
        record["candidate_raw"] = resp.output_text
        record["tokens"] = resp.usage.total_tokens
        record["parse_ok"] = True  # json.loads below confirms
        json.loads(resp.output_text)
    except json.JSONDecodeError:
        record["parse_ok"] = False
    except Exception as exc:
        record["error"] = type(exc).__name__
    finally:
        record["latency_ms"] = int((time.monotonic() - start) * 1000)
    return record

The point of the recorder is that every shadow response lands in a table with the same fields, so scoring is mechanical.

Picking 50 requests because it is Friday afternoon is the most common way these evaluations produce noise instead of answers. Use a two-proportion power calculation. The standard formula sizes each group as n = (Zα/2 + Zβ)² × (p1(1−p1) + p2(1−p2)) / (p1−p2)², where p1 is the baseline parse-failure rate, p2 is the candidate's, and typical settings are 95 percent confidence (Zα/2 = 1.96) and 80 percent power (Zβ = 0.84) (Select Statistical Services).

Worked example with realistic numbers. Suppose your baseline model fails to parse on 5 percent of requests and you expect the candidate to fail at 3 percent, a two-point improvement. Plugging in: (1.96 + 0.84)² × (0.05 × 0.95 + 0.03 × 0.97) / (0.02)² = 7.84 × (0.0475 + 0.0291) / 0.0004 = 7.84 × 0.0766 / 0.0004, which is roughly 1,501 requests per model. That is the honest number for detecting a two-point difference. If the expected gap is larger, the sample shrinks fast.

The difference between the calculator's numbers and your gut matters. The most common mistake is treating 300 to 500 requests as sufficient for a two-point difference. It is sufficient for a five-point difference at moderate baseline rates, not for a two-point one. Use a pilot of 100 to 200 requests first to estimate both rates, then size the full run from the formula. A quick truth check: if the observed candidate parse-failure rate is zero in the pilot and the baseline is 5 percent, you are looking at a large gap and a few hundred requests will resolve it; if the candidate is 4 percent, plan for a bigger run.

Stratify the sample by endpoint and prompt type. A model that parses cleanly on short classification prompts can fail on long extraction requests with array fields, because truncation is the most common failure for schemas with variable-length arrays (GMI Cloud). Stratification ensures every request shape gets adequate representation rather than the endpoint that happens to fire most often.

If you route shadow calls through a single API for multiple candidate models, model swapping for comparison runs is a configuration change rather than an integration project; Quanta Inference exposes text, image, video, audio and embedding models behind one OpenAI-compatible key, which is the practical way to hold the request constant while varying the model (https://quantaintelligence.ai/inference). Just keep the shadow copy out of the render path, as above.

A single 1-to-5 quality score hides the failure that costs money. Score every shadow response against a fixed rubric with three gates, applied in order, so a response that fails gate 1 is never credited with good content.

Gate 1 is measured mechanically: run json.loads on the raw output only, with no fence stripping. Regex extraction "succeeds but is fragile" (GMI Cloud), so a response that only parses after cleanup is a parse failure for scoring purposes even if your production code would rescue it. The reliability decision uses the strict number because the retry tax is real either way.

Gate 2 catches the class of errors the parser misses. Valid JSON is not a contract: JSON mode can produce parseable output without required fields, correct enums or the expected object shape (TrueFoundry). Validate with a real JSON Schema validator or a typed parser (Pydantic, Zod) that enforces types and local invariants (OpenAI).

Gate 3 is where teams skip work and pay for it later. Shape is not truth: a schema-valid customer ID, amount or date can still violate domain rules (TrueFoundry). For gate 3, either sample 50 to 100 passing outputs for human review or use a stronger model as a judge with the original input attached. Constrained decoding solves structural validity, not semantic correctness, and teams that stop at structure trade a visible failure mode for an invisible one (GMI Cloud).

One rubric detail saves arguments later: score the raw output exactly as the candidate returned it. If the candidate uses a strict schema mode that the baseline lacks (or vice versa), note it in the methodology, because structured output support varies by provider and by mode (Vellum). You are comparing configurations you would actually ship, schema enforcement included.

With shadow calls live and the rubric fixed, the run itself is a table exercise. For each stratum and in aggregate, compute the parse-failure rate, schema-violation rate and content-error rate for both models, plus the confidence interval on each difference. Then apply the decision rule.

A practical decision rule with four outcomes:

• Candidate is clearly better (difference significant, candidate lower, content parity): schedule the cutover, keep the shadow pipeline running for one week after to catch drift.
• No detectable difference: switch only for cost, latency or operational reasons, not quality. If the candidate is much cheaper per successful task, the business case may still be strong, but say so explicitly.
• Candidate worse on reliability: stop. Do not run a canary "to see." A canary or A/B test exposes real users to the candidate, which is appropriate only after shadow validation passes (CodeAnt).
• Mixed per stratum: the candidate may be fine for some endpoints and not others. Route per endpoint rather than all-or-nothing.

This two-stage discipline matters. Shadow testing is observational and catches regressions in accuracy, formatting, latency and cost; A/B testing is experimental and captures user behavior, but it carries direct user exposure (CodeAnt). Running the shadow stage first is how you make the canary stage low-risk.

Factor in what each failure costs when reading the numbers. A parse failure means a retry at full inference cost and latency, and on a high-volume pipeline the retries compound quickly (tianpan.co). If you have measured your own cost per successful task, the comparison becomes arithmetic: rate difference multiplied by volume multiplied by cost per attempt. Teams that have not measured per-task cost typically end up guessing; a usage-based cost-per-million-tokens figure is the minimum you need to make the switch decision defensible (OpenCost). Using wording such as "usage-based cost" rather than "price per token" matters here, because the two are distinct and self-hosting comparisons quietly conflate them.

Here is the full sequence, with the numbers for a hypothetical service that handles 200,000 structured-output requests per day and currently parses cleanly on 95 percent of them.

Week 1, instrument. Add the shadow recorder to the top three endpoints by volume. Capture raw candidate output, parse status, schema validation status, latency, tokens and attempt counts. No scoring yet. Confirm the baseline model reproduces its known 5 percent parse-failure rate in the logs.

Week 2, pilot. Send 200 requests per endpoint per model. Score gates 1 and 2 mechanically. Estimate p1 and p2 from the pilot. If the pilot shows a two-point gap, the full run needs roughly 1,500 requests per model; if it shows five points, a few hundred will do (Select Statistical Services).

Weeks 3 to 4, full run. Collect the sized sample stratified by endpoint and prompt type. Score gate 3 on a subsample of passing outputs. Note that if the candidate receies a formal schema while the baseline receives prose instructions, you are measuring the enforcement mechanism as well as the model (OpenAI); that is often exactly the comparison you want, but state it.

Decision. Bring the confidence interval on the parse-failure difference to the review. If the upper bound of the candidate-minus-baseline difference is below your tolerance (for example, under one point at 200,000 requests per day), proceed. If the interval straddles zero or the sign is unfavorable, extend the sample or stop.

Post-cutover. Keep the old model as the shadow for one week. Distinct failure rates after cutover usually mean schema drift, truncation from token limits or model-specific formatting weaknesses, so track parse failure as its own metric separate from general error rate (GMI Cloud).

Two cautions on interpretation. First, performance differences between endpoints that claim to serve the same model can reach up to 47.21 percentage points, with identity verification failing in 45.83 percent of fingerprint tests in one audit of third-party services (arXiv). If you test a candidate model through a reseller, your result may describe the endpoint, not the model. Second, if your pipeline includes tool calls, a structurally perfect call is still an intended external action, and authorization decides whether it may proceed (TrueFoundry), which is why gate 3 belongs in the rubric even when gates 1 and 2 look clean.

For teams that run these comparisons across many endpoints or business lines, the natural next step is putting the evaluation work in the hands of a specialist rather than a generalist engineer, or having a Quanta expert produce the scoring sheet and decision memo for a line of business (https://quantaintelligence.ai/agents).