Shadowing a Candidate Checker Against Live Promotions

The most important property of a shadow deployment is that the candidate cannot act. In practice that means three rules.

First, the candidate runs on a copy of the row, not on the live row. Whatever the candidate writes to its own state stays in its own namespace. Second, every outbound call the candidate makes (another service, a model endpoint, a database lookup) goes to a shadow target or a logged stub. If it would have written to the promotions table, it writes to a shadow table or only logs the intent. Third, the candidate never reads the incumbent's verdict before producing its own. If it does, you have a feedback loop that contaminates the comparison.

The same caution applies to any memory or cache the candidate keeps. Persistent agent state can be poisoned by content from one session that is later served back into another, and a poisoned write can propagate through summarization and retrieval before anyone notices. For a checker, the equivalent risk is a candidate that quietly caches a "confirmed promotion" from one row and applies it to a later, unrelated one. Run the candidate with memory scoped per row and cleared between batches, and check that the cleared state is actually cleared.

A decisive detail: separate the event time from the acceptance time in every log line. A promotion row may have been created hours before your candidate saw it, and an artifact you blame on "stale data" may simply be a clock difference. Recording event time and acceptance time separately is the standard fix, and it also lets you replay the shadow window later without re-querying source systems.

You cannot triage what you did not record. Each shadow evaluation should emit one record per row with at least the fields below. The payload hash is the anchor: it lets you prove two runs saw the same input, and it lets you join the shadow log to the replay set and the incumbent's audit trail.

Two fields are easy to skip and expensive to omit. candidate_exception matters because a checker that crashes on 2 percent of rows will look like a checker that disagrees on 2 percent of rows if you only store verdicts. And candidate_reasons is what turns a queue into a debuggable list: without it, every disagreement needs a manual re-run before anyone can say why.

A minimal emission step looks like this. The point is the shape, not the language:

def shadow_check(row, incumbent_verdict):
    payload = canonical_json(row)          # sorted keys, stable datetime format
    record = {
        "row_id": row.id,
        "payload_hash": sha256(payload),
        "incumbent_verdict": incumbent_verdict,
        "event_time": row.created_at,
        "accepted_at": now_utc(),
        "checker_version": CANDIDATE_VERSION,
    }
    try:
        result = candidate.evaluate(parse(payload))   # shadow client, no writes
        record["candidate_verdict"] = result.verdict
        record["candidate_reasons"] = result.reasons
        record["candidate_latency_ms"] = result.elapsed_ms
    except Exception as exc:
        record["candidate_verdict"] = "ERROR"
        record["candidate_exception"] = repr(exc)
    shadow_log.write(record)               # append-only, never blocks the incumbent
    return incumbent_verdict               # incumbent still owns the decision

Note that the shadow call must not block the incumbent's path. If your gateway mirroring adds tail latency to production, you have traded a correctness risk for a performance one. Mirror asynchronously, and if the queue backs up, drop shadow rows rather than delay real ones.

The disagreement queue is the operational heart of the week. It is a filtered view over the shadow log where candidate_verdict != incumbent_verdict or the candidate errored. Build it as a scheduled job, not a dashboard someone has to remember to open, and route it to the person who owns the checker.

Sort it by risk, not by time. A practical ordering:

1. Rows where the candidate errored or timed out. These are bugs by default until proven otherwise.
2. Rows where the candidate would promote and the incumbent held. These are the expensive direction: a false promotion puts a bad entity into downstream systems, and false promotion is exactly the failure mode the memory governance literature treats as the one to control.
3. Rows where the candidate would hold and the incumbent promoted. These are the "we just made the system stricter" diffs, often acceptable but sometimes a quiet recall loss.
4. Everything else, grouped by candidate_reasons so you see clusters rather than individual rows.

Track two numbers every day: the disagreement rate (disagreements divided by shadowed rows) and the bug count per class. On a healthy rollout both trend down as you fix and redeploy. If the disagreement rate is flat while the bug count falls, you are probably converting bugs into improvements, which is the point.

Give the queue a fixed size discipline. If a day produces more than a few dozen disagreements, do not triage all of them; sample, classify the sample, and use the mix to decide what to look at next. A queue nobody finishes is worse than a sample everybody finishes.

Every diff gets exactly one label, and the label decides what happens next.

A bug is a diff where the incumbent is right and the candidate is wrong on the merits, or where the candidate crashed, timed out, or produced a verdict outside its allowed set. Bugs get a ticket, a fix, and a re-run on the same row. The re-run should be stored alongside the original so you can see the fix confirmed.

An improvement is a diff where the candidate is right and the incumbent was wrong or too coarse. Improvements get logged with a short justification and an owner who is willing to accept the behavior change. A candidate can be perfectly correct and still not cut over, because a deployment state is a decision about appetite as much as about correctness; the improvement diffs are where you record that appetite.

An environment artifact is a diff caused by something other than checker logic. Typical causes: the row changed between when the incumbent saw it and when the candidate did, a lookup service returned a different answer, a time-dependent rule fired differently because the two evaluations straddled a boundary, or a feature is present in one environment and not the other. Artifacts are not bugs, but they are not free either: a high artifact rate means your shadow window is not measuring what you think.

This third class is where most disagreements quietly go to die, and it is where the discipline pays. An audit trail that records who or what changed a value, and when, is what lets you distinguish a real behavior change from an artifact. If you cannot answer "did the input change between the two evaluations?" for a given row, your queue will drift toward blaming the candidate for the environment.

A worked example helps here. Suppose a week of shadowing produces 8,000 rows and 640 disagreements, an 8 percent rate. On review, 390 are artifacts (the row was modified midway through the day by a support fix), 180 are improvements (the incumbent was applying an outdated scope), and 70 are bugs, of which 55 are one parser defect. Fix the parser, re-run those 55 rows, and the true bug count falls to 15. Then fix the artifact source, which in this case is a race between two jobs, and the disagreement rate drops to about 3 percent on day five. That is the shape of a week that ends in a cutover decision, and the only way to see it is to record the four categories separately.

The "undecided" row matters. Ambiguous matches are a normal part of any matching problem, and recognizing them as a category rather than forcing them into pass or fail is how mature resolution systems are built. If undecided cases are more than a few percent of disagreements, your specification is underspecified, not your checker.

The duration is a floor, not a target. Seven days is a reasonable minimum because it captures a full weekly traffic cycle, including the day your batch jobs run and the day support works on escalations. But the real gate is coverage: you want several thousand shadowed rows and evidence that each high-stakes edge case appeared at least once. Check coverage daily rather than at the end, because if a category never appeared you may need to wait, or to seed a shadow subset deliberately.

Use these criteria as the cutover gate:

The bug-rate-to-zero rule on the last two days is the one people argue with, and it is the one worth defending. If you cut over with a known bug rate, the first days of production become your shadow window, except now the mistakes are real. Two clean days is a low bar that nonetheless forces you to finish the queue rather than carry it forward.

There is a second condition people underweight: the improvement diffs. It is entirely possible to reach zero bugs and still not be ready, because the candidate promotes rows you actually wanted held. Read those diffs as a policy decision, not a test result.

Shadow logs are audit logs, and audit logs are useless if they are not queryable when somebody asks. Retain the shadow window, the replay set, and the disagreement queue with their labels for at least as long as your change management policy requires. Auditors treat logs as primary evidence and expect retention beyond the lifetime of the system that produced them, which is a good default even if you are not in scope for any particular framework.

Two habits make the record durable. First, store the labels with a link to the row hash so a later reader can reconstruct why a decision was made. Second, when the candidate goes live, keep a residual shadow: run the old incumbent in shadow for a short period after cutover so you can compare live behavior against the baseline you validated against. The direction reverses, but the mechanics are the same, and it catches the class of bug where the candidate behaves differently once it is allowed to write.

If the shadowing harness itself needs to run at volume across several models, one API key for text, image and embedding models removes a chunk of credential and routing work; Quanta Inference is one option, and the logging schema above is model-agnostic either way. And if your team would rather hand the harness and the triage queue to someone who has built this before, a consulting call is the shorter path.

The one thing not to do is treat shadowing as a formality to schedule after the real decision has been made. The value is in the queue and the labels, and those only exist if someone reads them daily.