How to Build a Shadow Test for a New Model Before You Switch

There are two ways to feed the candidate. Duplication sends a sanitized copy of a live request to the candidate at roughly the same moment the current model handles it. Replay sends already-captured requests to the candidate later, from storage. Yotta Labs' comparison frames the tradeoff: duplication tracks current traffic more closely, while replay is easier to control, review and repeat.

Most teams do both in sequence. Replay a curated sample of a few thousand requests to get quality scores you trust, then run duplication for a day or two to see how the candidate behaves on p95 latency, error rates and long-tail inputs that never made it into the curated set.

The mechanics of capture are well established. Mokapi's record and replay approach forwards requests to the real backend, records request and response pairs as JSON, then swaps in a replay script that matches incoming requests against the recordings. Keploy's kernel-level capture goes further, recording at the system call layer with roughly 1 to 3 percent overhead, so the application does not need to know it is being recorded. For an LLM shadow test you rarely need kernel-level capture: logging at your application's model gateway gives you the prompt, the settings and the response in a form you can replay directly.

One caution about replay realism. A production request may depend on retrieval context, tool results or conversation history that changes between capture and replay. Record the full assembled input, not just the user message, or your replayed prompt will differ from what production actually sent.

Log enough that you can reconstruct the decision a reviewer would make. The field list below is the minimum for a useful shadow run.

The Siteimprove migration QA framework makes an argument that transfers directly: a single pre-launch sign-off is structurally blind to errors that appear at different stages, so validation should be continuous rather than a gate. A shadow run spanning a full week of traffic, including whatever peak load and unusual inputs that week contains, is the model-level version of that idea.

A practical harness has four moving parts: a sampler that pulls recorded requests, a runner that calls both models with identical inputs, a scorer that computes metrics, and a store that keeps every record.

Keep the candidate call strictly out of the response path. The DevOpsBoys shadow mode example shows the pattern: fire both calls in parallel, return the production response to the user, and send the paired records to a logging service asynchronously. Their code also shows the failure mode to avoid, where an exception in the shadow call could propagate into the user request; wrap the candidate call so a timeout or error is logged as a data point, never raised.

Two operational rules make the harness trustworthy:

• Set the same timeout on both models and record timeouts as a category, not as missing data. A candidate that finishes slowly is different from one that fails.
• Run the models in isolation from production dependencies. If the candidate is served from a different provider, a rate limit or outage should never touch the current model's availability. Mirror the environment: same retrieval index, same tools, same guardrails.

If you need a single endpoint across many candidate models, Quanta Inference exposes text, image, video, audio and embedding models through one OpenAI-compatible key, which simplifies pointing the same harness at several candidates without rewriting client code for each provider.

Average quality alone is a poor switch signal. Compute four families of metric, each with a distribution rather than a mean.

Task quality. Define the success criterion for your task. For structured extraction, that is schema validity plus field-level agreement with a ground-truth sample. For support replies, it is a rubric score from human reviewers on a sample plus a check that required facts (policy numbers, refund amounts) match the retrieved context. For classification, it is exact match against a labeled sample. Yotta Labs recommends judging structured tasks by schema validity and field-level agreement, and conversational tasks by tone, completeness, citation behavior and escalation decisions.

Agreement with the current model. Compute how often the candidate's output matches the current model's on the same input, using a calibrated judge or exact match depending on task. This is not a quality measure, but a change measure. A 40 percent divergence on a stable task is a signal to look at the diff.

Cost per task, not per token. Token counts differ between models on the same input, so per-token pricing overstates or understates the real comparison. Pertama Partners' TCO guide notes that license fees typically represent only 20 to 40 percent of true AI implementation costs, and that comparison requires normalizing vendor pricing to actual usage patterns. The CodeAnt guide makes the same point in shadow-test terms: a cheaper model can cost more per completed task because it uses more tokens to finish the job.

Worked example. Suppose your current model averages 1,200 input and 400 output tokens per task at a blended rate, and a candidate averages 1,200 input and 290 output tokens at a rate one-third lower. Per task:

• Current: 1,600 tokens at rate R.
• Candidate: 1,490 tokens at rate R/3, so roughly 497 units against 1,600.
• That is about a 69 percent reduction per task, before counting retries. If the candidate needs a retry on 8 percent of tasks and the current model needs one on 3 percent, the gap narrows but the candidate still wins on cost.

Latency and errors. Track p50 and p95 latency, timeout rate and error rate per model. A candidate with equal quality and half the cost but a p95 latency above your user-facing budget fails the threshold regardless.

The most common failure in these tests is moving the goalposts after seeing results. Write the rule down first, in a short document that names the approval owner.

A workable rule:

• Quality: candidate success rate at least equal to baseline on the labeled sample, with a confidence interval that excludes a regression larger than your tolerance (for instance, 2 percentage points).
• Agreement: divergence rate on the stable task within an expected band, with every cluster of disagreement reviewed by a human.
• Cost: candidate cost per successful task at or below baseline.
• Latency: candidate p95 within the user-facing budget and no worse than 15 percent above baseline.
• Reliability: candidate error plus timeout rate at or below baseline.
• Safety: no increase in policy violations on the sensitive-prompt subset.

If any metric fails, the outcome is no-go, with the failing metric named and a follow-up action (prompt change, different candidate, more data). A mixed result is not a reason to switch and watch. It is a reason to fix the specific failure and rerun.

The CodeAnt comparison of shadow and A/B testing is a useful reminder of sequence: shadow testing validates model behavior with zero user exposure, while A/B testing measures user behavior and satisfaction. Shadow first to catch regressions, then a small canary or A/B step to confirm users respond no worse. The DevOpsBoys canary stages show a reasonable progression, starting at 1 percent of traffic and requiring manual approval at each step before full rollout.

Migration deadlines make this sequencing matter. Airia's deprecation guide argues that a 90-day window is often insufficient, because inventory, behavioral validation, compliance documentation and stakeholder communication each take weeks, and teams that start at the notice are already behind. The cross-vendor lifecycle calendar is a reminder that retirement dates differ by platform, so your shadow test should be complete well before the date on the vendor page you actually use.

Shadow traffic is production traffic, which means it carries the same personal and regulated data. Yotta Labs recommends sending a sanitized copy of the request to the candidate, and the CodeAnt guide advises applying the same data handling and encryption policies to shadow logs as to production data. KPMG's survey data underlines why this is not optional: 55 percent of companies cite data quality as a major barrier to AI adoption, and organizations report spending up to 80 percent of project time just preparing data (KPMG).

A practical sequence:

1. Classify before copying. Run the request through the same PII detector you use for production logging. Redact or tokenize names, account numbers, addresses and free-text identifiers.
2. Approve the batch for replay. For recorded replay, a human or classifier reviews the sample before the candidate model sees it. For live duplication, the scrubber runs inline and a rejected request is simply not shadowed.
3. Separate the stores. Candidate outputs go to a table or bucket with its own access controls and retention period, not into your production response logs.
4. Keep provider data terms in view. Whether the candidate provider can train on or retain your inputs is a contract question; query it before the first replay, not after.

If you would rather have this review done for you, Quanta Experts work from your context and hand back finished deliverables, which can include a scrubbed replay sample and a written scoring rubric.

Pull it together with a concrete plan for a support assistant migration.

• Sample size. 2,000 recorded requests from a two-week window, stratified by intent category so low-volume intents are not drowned out. 200 of those have human-written reference answers.
• Curated replay. Candidate runs on all 2,000. A judge model (calibrated against 100 human scores first) grades relevance, factual grounding and tone on a 1 to 5 scale. Schema validity and required-fact presence are checked programmatically.
• Live duplication. Two days, all traffic, sanitized copies only. You measure p50 and p95 latency, error and timeout rates, and cost per request under real load.
• Scoring. Compare success rate (score of 4 or above plus all required facts present) against baseline. Hold the sample fixed across both runs.
• Decision. If the candidate meets the pre-written rule on all metrics, schedule a canary at 1 percent. If not, the failing metric names the fix.

The whole run is a few engineer-days plus evaluation cost. That is far less than the cost of a bad switch. The Pertama TCO analysis notes that exit costs from a vendor relationship are rarely considered and can be substantial, which is another reason to test thoroughly before committing rather than after.

If your team has never run a shadow test and wants a plan tailored to your stack, Quanta Consulting offers a free 30-minute discovery call to scope the work.