Shadow Mode: Running an Agent Alongside Staff Before Launch

The routing rule that keeps this safe is simple: the shadow path forks off after the work order is accepted, and it can never write back. Draw the line at the queue, not inside the handler.

A workable sequence:

1. The work order arrives as normal and is persisted by the existing system.
2. Staff claim it and work it, exactly as they do now.
3. A copy of the payload is published to a separate queue that only the agent consumes.
4. The agent runs against a read-only copy of the tools it will use in production: a snapshot of the order database, a sandboxed refund API, a mailbox that sends nowhere.
5. The agent's output, its full transcript and the tool calls it attempted are written to a comparison store keyed by the same order ID.
6. Staff record their own outcome on the same order in the same store.

The correlation ID step is not optional bookkeeping. Practitioners running shadow deployments for models recommend a correlation ID linking the production and shadow results so analysis stays manageable (Qwak). For agents you want the link at the order level and the trace level, because the same order can produce a 40-step transcript.

Two rules protect you from the most common failure. First, the agent's tools must be incapable of acting, not merely configured not to act. A refund tool that points at the live payments API is a live refund tool, whatever the flag says. Second, put a kill switch on the shadow queue itself so you can stop consuming copies without touching the main queue; if something looks too good to be true in the logs, it is most likely wrong (Qwak).

One privacy note that catches teams: the shadow queue now holds full copies of customer work. If the agent calls an external model API, those copies leave your boundary. Decide that before the first copy is sent, and route the shadow queue through whatever gateway your other model traffic already uses.

The mistake here is picking a metric because it is easy to compute. "Agent answered" is easy. It is also useless, because the question the business is asking is whether the agent gets the outcome right often enough that staff time shifts from doing the work to checking it.

Start from a success statement specific enough to become a number. Google's evaluation guidance contrasts "the agent should be helpful" with a measurable version such as a RAG agent providing a factually correct, concise summary fully grounded in known documents (Google Cloud). Do the same for your work order.

Then pick one primary metric and a small set of guardrails. A practical set for a work-order agent:

Outcome match is the primary. Everything else is a guardrail that can veto a launch even when outcome match passes. That ordering matters, because a fast agent with one harmful error in a hundred orders is worse than a slow one with none.

Getting the outcome comparison right is where most of the work sits. Staff resolutions are inconsistent: two people close identical tickets with different codes. Before you compare anything, have a reviewer read 50 staff-resolved orders and check how often the codes agree with each other. That number is your ceiling. If staff agree 88 percent of the time, an agent scoring 85 percent against staff codes is roughly at the level of a second experienced human, and an agent scoring 95 percent is probably being graded on a rule that staff do not follow.

A shadow run produces one number (outcome match) and one long list (disagreements). The number is for the decision meeting. The list is where the insight is.

Sort disagreements into four buckets, and read at least 20 per bucket by hand:

• Agent right, staff wrong. These are the ones worth studying first. If the agent consistently catches something staff miss, you may be looking at a process fix that has nothing to do with the agent. Check the transcript before you believe it; what looks like a catch is sometimes a lucky guess over a hallucinated document lookup, which is exactly the hidden failure that output-only scoring cannot see (LangChain).
• Agent wrong, staff right. Read the transcript step by step. Anthropic's evaluation guide defines the transcript as the complete record of a trial, including tool calls, reasoning and intermediate results, and recommends grading the outcome separately from the path (Anthropic). Ask whether the agent retrieved the wrong document, called the right tool with the wrong argument, or reasoned correctly from bad context. Each has a different fix.
• Both wrong, same way. Usually a policy ambiguity. Fix the policy, not the agent.
• Both wrong, different ways. Often the most expensive cases, and the ones where a human reviewer is still needed in production.

This bucket analysis is the reason to run shadow mode at all. An offline test set of 300 cases tells you the agent handles the scenarios you imagined. Live shadow traffic tells you the distribution you actually get, including the order types that appear twice a month and never made it into the test set. Treat the shadow logs as a source for your golden dataset: teams that maintain evaluation sets curate them from real logs and representative scenarios so the tests predict in-field performance (Maxim AI). Any order that produced an interesting disagreement belongs in that set.

You will need help reading the transcripts at volume. Judge models are the usual tool: a judge can score a transcript against a rubric and return a score with a written justification (MLflow). Judges can operate pairwise, comparing two outputs directly, which fits shadow mode well because you always have two outputs to compare (Evidently AI). Calibrate the judge against your own hand-read sample first. Judges have known biases, including showing preference for certain writing styles, so treat a judge score as a triage signal and your hand-read sample as the truth (arXiv).

Running the same order through a judge with the two outputs swapped is a cheap way to catch position bias, since a judge that prefers whichever output it reads first will flip its verdict when you swap them (Confident AI).

Suppose a support team handles 4,000 work orders a month, and the agent is being considered for one category: refund requests. That category is 600 orders a month. You run shadow mode on all 600 for three weeks, which gives about 450 completed orders once you drop ones still open.

The team agrees the thresholds in advance: outcome match at least 85 percent, zero harmful errors in the hand-read sample, median handling time under the staff median of 14 minutes.

Results at the end of the run:

The headline is a failure: 82 percent is under the 85 percent bar. But the buckets change the decision. Of the 78 disagreements, 41 are the agent approving refunds that staff denied under a policy that changed two months ago, and the agent's knowledge document still carries the old wording. That is a retrieval problem with a single fix, not a reasoning problem. Another 22 are orders where the agent's output was cut off before the final message, which the transcript shows clearly.

The escalation row is the one that should worry you most. The agent escalated a third as often as staff, and a chunk of its errors are exactly the cases staff would have passed to a supervisor. Low escalation plus confident errors is the pattern that produces customer complaints, and it is invisible in the outcome match number alone.

The decision rule that follows: adjust and rerun on the refund category only, with the policy document corrected and an escalation rule that forces a human when the order amount exceeds a set figure. Do not launch. Do not stop either, because the 3-minute median and the small number of harmful errors say the agent is close.

Write the decision rule before the run starts, or the meeting becomes a debate about whether 82 percent "feels fine." A rule that survives contact with a real team looks like this:

LAUNCH if:
  outcome match >= threshold, AND
  harmful errors in hand-read sample = 0, AND
  escalation rate within 10 points of staff, AND
  cost per order below the labor cost it replaces

ADJUST if:
  one or more thresholds missed, AND
  the disagreements concentrate in <= 2 fixable causes, AND
  the fix does not require new data the team cannot get

STOP if:
  harmful errors appear in a pattern that needs a judgment you
  cannot encode (policy exceptions, legal review), OR
  two adjust cycles in a row fail to move outcome match

Two things make this rule honest. First, the harmful-error condition is absolute, not a rate. One unapproved refund in the sample that a customer would have received is a launch blocker regardless of the average. Second, the adjust path has a limit: two cycles, then stop. Without that limit, "adjust" becomes an indefinite shadow run that costs money and attention every month.

When you do launch, shadow mode is not the last step. The standard progression is shadow, then canary, then full production, with the old version kept around briefly so you can roll back (Qwak). Canary starts on a small share of live traffic and widens as you confirm nothing breaks, which is where you finally learn what real customers do with the agent's answers as opposed to how the answers compare on paper.

Keep the comparison harness after launch. Route a small share of orders through the old configuration as a control so you notice drift, and move every interesting shadow disagreement into your evaluation set so the next change is tested against it (Langfuse). If you are running several model providers during the shadow period to see which handles your transcripts best, a single endpoint for every model type removes the integration work from the critical path; Quanta Inference is built for exactly that, text, image, video, audio and embedding models through one OpenAI-compatible key. And if the harder question is what the agent should be allowed to do without a human, a consulting call with people who have run this before is cheaper than learning it from a customer complaint.