Shadow Runs: Comparing the Gate's Verdict to a Human Reviewer

The implementation goal is a code path that computes the verdict from the same features the enforcement path would use, writes a record, and returns a neutral result to the caller. Keep the two paths sharing one feature function, or the comparison you do later measures a different rule than the one you will promote.

# shadow.py
import json, time, uuid

def score_event(event, rule):
    features = rule.extract_features(event)   # same function used in enforce mode
    verdict = rule.evaluate(features)         # {"verdict": "block"/"allow", "score": float}
    return features, verdict

def handle(event, rule, sink):
    features, verdict = score_event(event, rule)
    sink.write(json.dumps({
        "event_id": event["id"],
        "rule_id": rule.id,
        "rule_version": rule.version,
        "shadow_verdict": verdict["verdict"],
        "score": verdict["score"],
        "features": features,
        "evaluated_at": time.time(),
        "trace_id": str(uuid.uuid4()),
    }))
    return {"action": "allowed", "reason": "shadow"}  # caller unchanged

Three properties matter more than the code itself.

Record the rule version and the feature values, not just the verdict. When disagreement clusters, you need to know whether two similar events took different paths because of a threshold change or because a feature was missing. Versioning the rule keeps a rolling shadow run interpretable when you tune mid-flight.

Sample deliberately, not uniformly. If the rule fires on 0.5 percent of traffic, uniform sampling of the log gives you almost no positives. Oversample the events the rule flags and keep a uniform background sample so you can still estimate the base rate. Store the sampling weights alongside each record.

Attach a stable event identifier that reviewers can join on. The whole calibration depends on pairing a shadow verdict with a human decision for the same event. If your review tool records decisions against a different key than your event log, you will spend the labeling pass doing fuzzy joins.

Agent and model outputs raise the same question in a different form: an in-session check catches problems while the producer can still fix them, while a commit-time gate is the final barrier against invalid state per this pre-commit validation write-up. A shadow run is the third layer, the one that tells you whether the barrier would have been correct before you install it as blocking.

The comparison set is the heart of the exercise. For each sampled event you need three fields: the shadow verdict, the reviewer's decision, and ideally the reason the reviewer chose it.

A labeling pass that works looks like this:

1. Export the sampled shadow records with the verdict hidden from the reviewer.
2. Have reviewers label the event as they would have if no rule existed (block or allow, plus a short reason).
3. Blind the second pass: if you can afford two reviewers on a subsample, do it. Disagreement between two humans is the floor on agreement between a human and a rule, and it is easy to forget that.
4. Join on the event identifier and compute the matrix.

Step 3 is the one teams skip and later regret. Cohen's kappa exists precisely because raw percentage agreement can be inflated by chance, and the statistic's originators introduced it to correct for raters guessing per this review of the kappa statistic. If two reviewers only agree with each other at kappa 0.55 on a sample, a rule that agrees with a single reviewer at kappa 0.60 is not obviously worse, and your promotion target needs to reflect that.

Keep the labeling workload proportionate. A sample of a few hundred labeled events is usually enough to see whether a rule is grossly miscalibrated; a small sample is not enough to certify a rule at 99 percent precision. Work out the sample size from the precision you need before you start labeling, not after you see a number you like.

Start with the 2x2 confusion matrix, using the reviewer as the reference. This is the standard layout a diagnostic test is evaluated with, and the vocabulary (true positive, false positive, false negative, true negative) transfers cleanly to a detection rule.

Raw agreement is (TP + TN) / N. Kappa is (observed agreement minus expected agreement) divided by (1 minus expected agreement), where expected agreement is computed from the row and column totals with a worked calculation here. A compact implementation:

from sklearn.metrics import cohen_kappa_score

rule     = [1, 1, 0, 0, 1, 0, 1, 1, 0, 0]   # 1 = block
reviewer = [1, 1, 0, 1, 1, 0, 0, 1, 0, 0]

obs = sum(r == v for r, v in zip(rule, reviewer)) / len(rule)
kappa = cohen_kappa_score(reviewer, rule)   # reviewer first = reference
print(f"raw agreement {obs:.2f}  kappa {kappa:.2f}")

Kappa carries a standard error, and libraries such as statsmodels expose it directly per the implementation notes here. Report it. If you get kappa 0.62 with a standard error of 0.04, you know the agreement is probably substantial but you do not know it is above 0.61 with any confidence; a wider sample buys you the certainty.

Interpret the magnitude against a stated convention, not vibes: 0.81 to 1.00 almost perfect, 0.61 to 0.80 substantial, 0.41 to 0.60 moderate, below 0.21 slight to poor on the standard scale. The medical literature warns that those bands were suggested for general use and may be too lenient for high-stakes decisions per the kappa review, which is a good reason to set your own bar rather than borrow one.

Worked example: is this rule precise enough to enforce?

Suppose a shadow run over three days produced 12,400 events, and the rule flagged 310 of them. You sampled 300 flagged events and 200 unflagged events for labeling (with known weights, though for a first pass it is fair to read the flagged sample directly).

Of the 300 flagged events, reviewers agreed it was a block in 246 cases, so precision is 246 / 300 = 82.0 percent. The 95 percent confidence interval for a proportion that close to 0.8 on n=300 is roughly plus or minus 4.3 points, so the plausible range is about 77.7 to 86.3 percent.

Of the 200 unflagged events, reviewers wanted a block in 6 cases, so the false negative rate on that sample is 3.0 percent.

Now the decision rule. If your pre-registered threshold was "ship only if the lower bound of precision on flagged events is at least 85 percent," this rule does not ship. It is not far off, and the disagreement analysis in the next section tells you whether a threshold tweak or a feature change is the right fix. Had the lower bound been 85 percent, you would have had grounds to enforce on the flagged class while keeping the unflagged class in shadow for another pass, which is a legitimate intermediate state.

If this example reads as a lot of statistics for an afternoon's work, it is: the arithmetic is trivial, and the hard part is the labeling discipline. Where the rule involves model inference over text, images or events at volume, running those calls in shadow mode through a single API is one less moving part; Quanta Inference provides text, image, video, audio and embedding models through one OpenAI-compatible key, which keeps the shadow path cheap and the feature extraction consistent. The broader point stands regardless of vendor: the evaluate-only path should be boring and cheap, because you will run it a lot.

A confusion matrix tells you how often you are wrong. Grouping the off-diagonal cases tells you why.

Split false positives by the features that drove the score. If most false positives share a feature value (a new signup with no history, an unusual region, a burst of activity after a marketing email), you have found a segmentation problem, not a bad rule. Options then are feature-level: exclude that segment, add a feature that distinguishes it, or raise the threshold for it only.

Split false negatives the same way. A cluster of missed blocks on one feature pattern usually means either the feature is absent for those events or the threshold is too high for that class. Pull ten of each and read them side by side. Reviewer notes are worth their weight here; a reviewer who wrote "legitimate retry, upstream timeout" saves you a week.

Two failure modes are worth naming.

Reviewer disagreement masquerading as rule error. Run the double-labeled subsample described earlier. If two humans disagree on 12 percent of the sample, a rule that disagrees with one human on 18 percent is closer to the human baseline than it looks, and your promotion target should account for that.

Feature leakage. If the shadow record includes a field that would only be populated after the action the rule is meant to prevent, the rule will look excellent in shadow and fail in enforce. Audit the feature list against the event timeline before you compute anything.

There is a useful parallel in code review checklists, which exist to make reviews consistent rather than to catch everything. Reviewers are told to consider the return on investment of each comment and to separate blocking issues from nits as this PR checklist argues. The same framing applies to tuning: each candidate threshold change should be judged by how many true positives it adds and how many false positives it costs, and changes that only move noise should be left alone.

A tuning loop that stays honest

1. Fix the evaluation sample before tuning. Tune on half, hold out the other half.
2. For each candidate threshold, recompute the confusion matrix and the precision interval on the tuning half.
3. Carry the two best candidates to the holdout half and report precision there.
4. If a change helps on tuning and not on holdout, you are overfitting to your sample; stop.

Promotion is a release, so treat it like one. The mechanics:

• Freeze the rule version. The version you promote is the version you shadowed, with the same feature code and the same threshold.
• Keep a kill switch. Shadow mode already gives you a logged, non-blocking path; keep it reachable so that rollback is a flag change rather than a deploy.
• Stage the rollout. Start at a small fraction of traffic and watch the disagreement rate against a continuing reviewer sample, because live enforcement surfaces cases your shadow sample underrepresented.
• Re-run the shadow comparison on a schedule. Rules drift as traffic and reviewers change. A periodic shadow pass on a reassigned rule is cheap insurance.

For teams whose gates guard machine-generated output rather than user events, the same discipline applies: embed the gate in the workflow, require independent acceptance evidence rather than the producer's own self-check, and keep the protected check separate from the actor being checked as argued in this piece on independent acceptance tests. The shadow run is the step that produces that independent evidence before the gate starts blocking.

Where the rule is one of several gates across a release process, a platform that lets you run the evaluate-only pass, store verdicts and reviewers' labels, and swap models without rewriting the pipeline shortens the loop; Quanta Skills covers installing an evaluation or review skill into a coding agent or putting an expert on the labeling and triage work. The essential discipline is unchanged whether you build it yourself or not.