Shadow Mode for a Candidate Prompt: Running It Beside Live

A candidate prompt that passed a static eval set can still fail on live traffic, and the reason is structural. LLM outputs are not deterministic in practice even at temperature zero, with accuracy variations reported across runs on identical inputs, driven by floating-point arithmetic and batch-size variability during inference the gradual rollout writeup. A unit test cannot settle the question, and a handful of hand-typed questions proves very little, because real customers misspell product names, paste error logs and bury the question in paragraph three IrisAgent's test plan.

The published numbers explain why teams keep over-trusting demos. Gartner surveyed 5,728 customers and found only 14% of service issues are fully resolved in self-service, and 36% even for issues customers themselves called very simple, while vendor pages routinely claim 70% or 80% Richpanel's evaluation method. Both numbers can be honest and still measure different things on different tickets. The only evidence that settles it is a test built from your own cases.

Shadow mode is the bridge between a sealed offline eval and a canary. Your eval set is a sample you chose; shadow traffic is the population. If your candidate is a rewrite of a support triage prompt that decides refund, exchange, escalate or answer, the eval set tells you it handles the cases you thought of. Shadow tells you what it does with the 3 a.m. message that chains three unrelated intents together. That is the gap between "looked good in the demo" and "safe in production."

The mechanics are mundane and that is the point. You fork each request at the application layer where your prompt is invoked, send the original down the normal path, and send a copy to the candidate. The candidate path must terminate in a logger, never in your action code. The standard implementation routes all production traffic to the current version while a background process sends the same requests to the candidate, with candidate responses going to a logging system for evaluation rather than to users the gradual rollout writeup.

Four constraints keep it safe:

1. No write actions from the candidate. If the candidate's parsed decision is refund, you record it. You never call the refund tool. Tool-correctness and argument-correctness checks matter precisely because the right thing to measure is what the candidate would have done, including any side effects, not whether it actually did it Confident AI's metric guide.
2. No user-visible latency added. If your HTTP handler awaits the candidate, you have shipped a slower product. Fire the candidate call asynchronously or off a queue; the live path must not wait on it.
3. Structured output on both sides. Both prompts should return the same parsed shape, so a diff compares decisions, not prose. Prompt experiments with structured output need the schema passed through to the call, which is exactly the complication teams hit when they try to run experiments against tool-using prompts the Langfuse discussion.
4. Immutable prompt versions. Each call logs which version produced the decision. Once a prompt version is created it should stay unchanged, and any edit should create a new version, which is what makes traces and rollbacks dependable later Braintrust's prompt versioning guide.

A minimal fork in the handler looks like this, and the shape matters more than the language:

def handle(request):
    trace_id = request.trace_id
    result = live_prompt.run(request)          # the only decision that acts
    enqueue(shadow_job, {
        "trace_id": trace_id,
        "input": request.input,
        "prompt_version": CANDIDATE_VERSION,
    })
    return result

def shadow_job(job):
    out = candidate_prompt.run(job["input"])   # parsed decision only
    log_shadow_decision(
        trace_id=job["trace_id"],
        prompt_version=job["prompt_version"],
        input_hash=sha256(job["input"]),
        parsed_decision=out.decision,
        latency_ms=out.latency_ms,
    )                                          # no side effects, ever

Note what the shadow job returns: nothing. Its entire output is one log row and one metric. If you use a multi-provider gateway, keep the candidate on the same model family and parameters as production so the comparison isolates the prompt. When you later want to compare across models as well as prompts, one key and one OpenAI-compatible endpoint turns the model swap into a config change rather than a re-integration, which is the case Quanta Inference is built for: https://quantaintelligence.ai/inference.

Log too little and you cannot debug a disagreement. Log too much and nobody reads the table. The schema below is the minimum that lets you answer the question you actually care about: where did the two prompts differ, and does the difference matter?

Store the raw output text too, but keep it out of the comparison table and behind a retention policy. You need it when a disagreement requires a human read; you do not need it in your daily rollup.

Here is a worked example with numbers. You run 4,000 shadow requests a day and see 312 decisions where the live prompt escalated and the candidate answered, plus 41 where the live prompt answered and the candidate escalated. That is a disagreement rate of about 8.8% on 4,000 requests. Before you read individual rows, check that the 41 are genuine policy violations and the 312 are genuinely safe answers. One direction of disagreement is almost always far more expensive than the other.

Evaluation frameworks take the same shape: you hand the agent a dataset of questions and review whether each response resolved the question, then act on the misses by updating knowledge or refining the agent's setup Atlassian's evaluation docs. Shadow mode is the same loop with live data streaming in.

Do not review every disagreement. Sample and stratify. A pattern that holds up: review all money-adjacent disagreements (refunds, credits, charges) every day, review a random sample of the remainder, and review every case where the candidate errored or returned an unparseable decision. Money and policy edges are where a wrong decision creates a liability, and they are usually where the disagreement rate is highest.

When you read a pair, you are not asking which output is prettier. You are asking three questions in order:

1. Did the candidate take an action the policy does not allow? That is a blocker, full stop.
2. Did the candidate miss an escalation the live prompt caught? Escalation precision and recall matter most here, because a missed escalation is a customer trapped in a bot Richpanel's evaluation method.
3. Did the candidate produce a better answer than the live prompt, or merely a different one? "Different" is not "better" and gets no credit in the tally.

For volume a human cannot read, an automated judge is the standard answer: a model that scores both outputs against criteria relevant to your use case, plus a token and cost diff and latency measurements under realistic load the gradual rollout writeup. Judge scores are a triage signal, not a verdict. Every changed decision the judge flags gets a human read before it influences the switch.

Log your reasoning against each reviewed pair in one line, because it is the audit trail you will want later. "Candidate refunded a 45-day-old order; policy window is 30 days" is enough. When someone asks three weeks later why the candidate was or was not promoted, that line is the answer.

Cadence should be explicit before the run starts, because otherwise the run drifts until someone gets bored and promotes on vibes. A schedule that holds up in practice: daily review for the first week, then weekly until you either switch or stop the run. The first week is where routing bugs, schema mismatches and logging gaps surface; the weekly cadence is for the steady state of disagreements.

The promotion rule needs to be a number, written before you look at the data. Something like: over at least 1,000 shadowed requests, the candidate must have zero policy-violating actions, escalation recall at or above the live prompt's, and a lower disagreement rate in the direction that matters. Write it down, then let the data answer.

Two things catch teams out. First, the real post-launch risk is a test case that used to pass and quietly stops, which is why the whole suite gets re-run on every change that could move an answer Richpanel's evaluation method. Shadow traffic is a great each-test but a poor regression guard on its own, so keep the offline suite alongside it. Second, shadow tells you the candidate is not obviously broken, not that users like it. When you switch, go to a canary next: a small share of real users, widened over time, so early users absorb a limited blast radius rather than everyone at once Portainer's guide.

The most common failure in shadow runs is not a bad candidate prompt. It is an unread log. If nobody owns the daily review in the first week, the run produces a database of disagreements and the team ships the candidate anyway. Name the reviewer and the calendar slot before the first shadow request fires. If your team would rather hand the review to someone who reads context and returns a finished comparison, that is a job shaped for Quanta Experts: https://quantaintelligence.ai/agents.

Take a support prompt that reads a customer message plus order data and decides one of four things: answer from the knowledge base, approve a refund, escalate to a human, or decline. You have written a candidate that adds three policy examples and a stricter rule about refund windows. Here is how the run reads.

Assume 12,000 requests over a week. The live prompt approved 1,900 refunds. The candidate, on the same 12,000 inputs, approved 1,640 and escalated 210 more. Human review of the 260 approval disagreements finds 190 are orders inside the policy window that the candidate escalated, a recall miss but a safe one, and 70 are orders outside the window that the live prompt approved, a live bug the candidate fixed. The candidate is slower by 380 ms on average, which your queue absorbs because it never sits on the user path.

That is a switch. The candidate is not perfect, but it is safer on the money edge and the latency cost is off-path. Flip it as a canary at a small share of traffic, watch the cohort metrics, then widen. Note that the live-bug finding is often the most valuable output of a shadow run, and it is worth filing back to the team that owns the live prompt.

Cost is part of the decision too. A new approach that is meaningfully better on quality might cost far more per call, and a gradual rollout lets you discover the cost profile at small scale before it becomes your entire budget the gradual rollout writeup. Pull token_in and token_out from the schema, multiply by your rates, and put the number in the promotion note next to the disagreement rate. A candidate that fixes 70 policy bugs but triples spend is a different conversation than one that fixes them for free.