How to Detect Reviewer Drift Without a Golden Set

Approval rate tells you whether reviewers accept the model. It does not tell you how far apart they are. For that you need a fixed scale that both sides are scored on, and it has to be frozen before monitoring starts, because a scale you redefine mid-stream makes every historical point incomparable.

Three design choices decide whether the series is usable.

First, score the reviewer against the model's own stated confidence or category, not against a hidden label you do not have. A simple version: on each item, the reviewer records a 1 to 5 confidence that the machine decision was correct, the model's stored score is mapped to the same 1 to 5, and the disagreement score is the absolute difference. The daily series is the mean of those differences, so it is continuous and can be charted as an individuals chart rather than a p-chart.

Second, keep the sample fixed per day. Take a random sample of reviewed items, say 300, so the standard deviation of the daily mean is stable and the control limits do not stair-step. If a day produces fewer than a threshold count of reviewed items, mark it as an excluded period rather than plotting a noisy point.

Third, record the reason codes that produce the score. In an alert-review setting, the failure mode to watch for is dismissal without evidence: research on LLM alert triage found that across 1,247 alerts in a multi-day attack scenario every tested approach missed at least 40.4% of attack-related alerts, and trace analysis showed alerts were more likely to be dismissed when searches returned no records the alert triage study. Translate that into reviewer terms: a reviewer who rejects the model after a failed lookup is not the same as one who rejects it after reading the evidence. If your disagreement score rises while the share of "no evidence found" rejections also rises, your process changed, not your model.

The Shewhart chart uses limits at plus or minus three standard deviations from the center line the p-chart guide. That catches large step changes fast and misses slow ones almost entirely. A drift of a few points a week, spread over a quarter, can sit inside three-sigma limits on every single day.

So run two charts on the same series.

The run and trend rules are the standard Western Electric patterns, and the guidance is to treat points beyond the limits, runs on one side, trends of six or more, and any non-random pattern as signals the p-chart formula walkthrough. The run rule is listed as seven consecutive points above or below the center line the p-chart guide.

The worked decision rule: a single point beyond three sigma on a 250-item day at p_bar of 0.90 happens by chance roughly 0.3% of the time per chart if the process is stable, so with two charts and about 250 working days you should expect a false alarm on the order of once or twice a year. That is an acceptable false alarm rate for a signal that also catches real step changes within a day. Seven-point runs are far more frequent under the null than intuition suggests, which is why a run alone should trigger a look, not a rollback.

The insight the single-metric dashboard misses is that the pair of series localizes the cause. Plot the day's movement in approval rate against the day's movement in mean disagreement, both in standard deviations from baseline, and read the quadrant.

The mechanism behind the first row is documented in compliance operations: industry estimates put false positives in transaction monitoring alerts between 95% and 99%, and when alert volume doubles with flat headcount, investigation quality drops, analysts start pattern-matching shortcuts and context gets skipped the AML rules tuning guide. Shortcuts show up as approvals, so approval rate rises while disagreement between reviewers and the model rises too. The same source notes the cost per alert review falls in a range of $30 to $70, which is the number you use to size the impact of a shift the AML rules tuning guide.

A second documented mechanism: monitoring rules calibrated for a different customer base, transaction volume or risk environment keep running while the customers change the AML rules tuning guide. That is your third row from the top, and it presents as an approval drop with flat disagreement, because reviewers are consistently rejecting a new kind of item they have never seen.

The no-golden-set problem is old in machine learning operations. A model's ability to generalize rests on the assumption that training and live data are independent and identically distributed, and that assumption is routinely violated because human behavior changes; one documented case saw a forecasting model's accuracy fall from 93% to 61% when shopping behavior shifted the Cloudera Fast Forward report on concept drift. The report catalogs methods for inferring drift without labels, including statistical tests on the feature space, on the response variable, and on margin density the Cloudera Fast Forward report on concept drift.

Reviewer signals are a cheap proxy for the response-variable test. You cannot see ground truth, but you can see whether the humans who do see the item disagree more with the model than they used to. That is a monitoring signal, not a label, and it should be treated as evidence of change rather than evidence of error. The distinction matters for governance: a reviewer approval is a human judgement, and a system is not there to make the decision for the analyst the AML monitoring overview. So the chart flags the process, and a person decides what it means.

Once you have a signal, you still have to know whether the model, the prompt or the configuration moved. That is why the same push that changes a model should change a versioned artifact you can trace a point back to. Prompt management practice treats each saved version as immutable with a unique identifier, so loading a version always returns the same text and parameters and production behavior stays reproducible the prompt management guide. Without that, a chart point at day 40 cannot be attributed to anything.

For rollout itself, the same logic as a canary applies: send a small slice of traffic to the new version, monitor both concurrently, and route back automatically if quality degrades; AI canaries watch not only error rate and latency but automated quality scores and business signals like task completion the canary deployment guide. A reviewer approval series is exactly such a business signal, which means your control chart doubles as the canary's stopping rule.

Take a review queue of roughly 2,000 items a day with a sample of 300 scored items per day. The baseline is built from 25 stable days.

Step 1, build the approval chart. Total approvals across the baseline divided by total items reviewed gives p_bar. Suppose it is 0.88. Then the standard deviation for a 300-item sample is sqrt(0.88 x 0.12 / 300), which is 0.0188, so the limits are 0.88 plus or minus 0.0563, giving an upper limit of 0.936 and a lower limit of 0.824. Day 26 comes in at 76% approvals on 300 items. That is 12 points below the center line, more than two standard deviations, and outside the lower limit. Signal one.

Step 2, check the disagreement series for the same day. Suppose the baseline mean disagreement is 0.62 on the 0 to 4 scale with a day-to-day standard deviation of 0.08, and day 26 reads 0.94. That is four standard deviations above the center line. Approval down and disagreement up, so the quadrant points to the model or the input population, not to reviewer behavior.

Step 3, attribute before acting. Pull the version manifest for day 26. If a prompt version or model endpoint changed the previous evening, you have your cause. If nothing changed, compare the score distribution of incoming items against baseline: a shift toward the low-confidence band produces exactly this signature with no model change at all.

Step 4, cost the delay. If the shift persists and review time per item rises from 4 minutes to 7 because reviewers now need to hunt for evidence, at 2,000 items a day that is roughly 100 extra reviewer-hours a day, and at the $30 to $70 per-review range cited earlier the AML rules tuning guide, the daily impact is measured in thousands of dollars. That number is what justifies the investigation, and it is also what gets budget for the fix.

Step 5, verify. After the fix, do not immediately reset the limits. Keep the old center line for a defined period so you can see the return to baseline, then recompute the limits from the post-change stable period. Recomputing too early hides the recovery and inflates your false alarm rate afterwards.

The charts only work if the series themselves are stable in definition. A few controls prevent most self-inflicted alarms.

• Freeze the disagreement scale, the sampling method and the sample size for a full baseline period before you publish any limits.
• Log a change event for every model endpoint, prompt version and scoring-logic update, and overlay those events on the chart as vertical markers. A control chart without an annotated change log turns every real signal into a guess.
• Exclude days below the minimum sample size rather than plotting them, and count them so a rising trend of excluded days becomes its own signal.
• Recompute limits on a fixed cadence (quarterly is common) rather than after every signal, so the chart does not chase itself.
• Keep a sample audit of approvals, because the approval rate cannot distinguish careful review from rubber-stamping on its own.

On tooling, the practical route is to log every reviewer decision with the model score, the item identifier, the reviewer identifier and the version manifest in one table, then compute both series in a scheduled job and push them to whatever dashboard your team already reads. If you are running several model providers behind one interface, a unified inference layer keeps the request-side metadata consistent enough for the version manifest to be reliable; Quanta Inference exposes text, image, video, audio and embedding models through one OpenAI-compatible API with prepaid US dollar pricing, which simplifies attributing a chart point to a single endpoint (https://quantaintelligence.ai/inference).