Auditing Human Decisions in the Task Log

You cannot audit an unbounded stream. Define two populations before you draw anything.

1. Approved diffs. Diffs that a named human reviewer approved for merge or deployment. Exclude anything merged automatically by a policy with no reviewer in the loop, because there is no human decision to audit.
2. Rejected prompts. Cases where a reviewer overrode or declined what an agent proposed. Here you are testing whether rejections were justified, which means sampling rejections more heavily than their raw frequency might suggest.

Draw the sample on a schedule rather than when someone thinks there might be a problem. A defensible default is 20 approved diffs and 20 rejected prompts per reviewer per month, with a minimum of 30 total comparisons per reviewer so the agreement estimate is not noise; a published heuristic on kappa studies suggests no fewer than 30 comparisons and notes that samples in the thousands produce tight confidence intervals (McHugh on the kappa statistic and the discussion of sample size heuristics in this review of agreement sample sizes).

The accounting analogy is useful here: audit sampling applies a procedure to less than 100 percent of items to evaluate a characteristic of the whole population, and both statistical and nonstatistical approaches require judgment in planning and evaluating the sample (PCAOB AS 2315 on audit sampling). Twenty items per reviewer per month is not a statistical inference about defect rates. It is a consistency probe, and it is enough to detect a reviewer applying a different rule from everyone else.

Record the draw in a reproducible way so the same period can be re-scored later. The principle is the same one that governs reproducible model outputs: fix the inputs, the random seeds, and the parameters so results can be rechecked (strategies for reproducible AI outputs).

import random, json

def draw_sample(entries, n_approved=20, n_rejected=20, seed=42):
    rng = random.Random(seed)
    approved = [e for e in entries if e["decision"] == "approved"]
    rejected = [e for e in entries if e["decision"] == "rejected"]
    return {
        "approved": rng.sample(approved, min(n_approved, len(approved))),
        "rejected": rng.sample(rejected, min(n_rejected, len(rejected))),
    }

A rubric that says "quality looks good" will produce disagreement that teaches you nothing. Each criterion needs a binary test and a worked example of a failure. The rubric should mirror what you expect the reviewer to check, which means it should be short enough to apply in minutes.

Two things about this table matter more than its contents. First, every criterion is checkable from the diff plus the linked task; none of them requires knowing the reviewer's mind. Second, the reason codes are the unit of analysis later. When you compare two reviewers, you want to say "we disagree about TESTS twice and EVIDENCE five times," not "one of us is stricter."

A useful external anchor for the critic-era failure modes: an analysis of AI-co-authored pull requests in a 470-PR sample found 75 percent more logic and correctness issues alongside elevated security and error-handling flaws, and AI-assisted changes frequently exceed 400 changed lines, a point past which defect detection drops (CodeRabbit's guide to reviewing AI-generated diffs). If your rubric has no criterion for error handling and size, it is not testing the failure modes that actually appear.

Writing the criterion down also improves the agent side. The spec-driven approach to coding with an agent pushes requirements into a requirements.md file and a plan into plan.md before any code exists, precisely so the work can be checked against a stated target (JetBrains on the spec-driven approach). A rubric criterion like INTENT is easier to pass when the original task has a written statement of what the agent was supposed to achieve.

Take a random subset of the sampled items, ideally 20 to 30 percent, and have a second reviewer score them without seeing the first reviewer's score or the final decision. Blind is the important word: an unblinded second score mostly measures deference.

Then compute agreement. Raw percent agreement is the wrong headline number for two reasons: it does not correct for chance, and chance agreement is high when one decision dominates the population. If 90 percent of sampled diffs are approved, two reviewers guessing from that base rate would agree on roughly 82 percent of items before any real consistency. Cohen's kappa corrects for that, so a kappa of 0.8 means 80 percent of the possible agreement beyond chance has been achieved (the sample size review discussing kappa under imbalance).

Those band labels come from the standard interpretation guidelines reproduced in sample size guidance for kappa agreement studies. Use them as a shared vocabulary for calibration decisions, not as a grade.

def cohen_kappa(a, b, labels=("pass", "fail")):
    n = len(a)
    observed = sum(x == y for x, y in zip(a, b)) / n
    expected = sum((a.count(l) / n) * (b.count(l) / n) for l in labels)
    return (observed - expected) / (1 - expected)

A worked example makes the value of kappa concrete. Suppose two reviewers score 30 items and agree on 26 of them, an agreement rate of 86.7 percent, which sounds fine. Suppose 22 of the 30 were scored pass by both scales' marginal rates. The expected agreement by chance is (22/30 × 23/30) + (8/30 × 7/30), which is 0.562 + 0.062, or about 0.62. Kappa is (0.867 − 0.62) / (1 − 0.62), roughly 0.65, which is substantial but well short of the number the raw percentage implied. If you instead had 24 agreements out of 30 with the same base rates, kappa lands near 0.55 and the calibration conversation changes.

For sample planning, kappa precision depends heavily on how close your confidence limit sits to the estimate. Tightening the nearer limit is expensive: distances of 0.20, 0.05 and 0.01 require roughly 118, 1,625 and 38,203 subjects respectively (the kappasizeci documentation on precision-driven sample size). That is why monthly samples of 20 plus a rolling pool across quarters is the practical design. You are not trying to publish a kappa with a narrow interval, you are trying to catch a reviewer drifting away from the rubric.

Every disagreement gets a recorded reason before it gets a discussion. Tag it with the rubric criterion, not with a person. At the end of a month you should have something like: eight EVIDENCE disagreements, five TESTS, two SCOPE, one INTENT. That distribution tells you exactly what to fix.

The meeting itself should be 30 minutes and should never include a leaderboard. The moment reviewers learn that disagreement counts against them, they will agree with whatever the other score says, and kappa will rise while quality does not. Keep the metric at the level of the rubric criterion and the team, and use individual data only in a one-to-one conversation about a specific criterion the reviewer asked to understand better.

There is a good model for this in how agent review comments get triaged. When unresolved agent comments were analyzed, the most common reasons a comment went unaddressed were incorrect suggestions and intentional design decisions, and short comments with an inline code suggestion were the strongest predictor of being acted on (the same study of agent review comments). Your rubric criteria work the same way: short, specific, attached to a concrete artifact. A criterion that requires a paragraph of interpretation will produce disagreement even among careful reviewers.

Three outputs belong on the team dashboard: kappa per rubric criterion, reason-code mix over time, and the count of rubric revisions. Three things do not belong there: individual reviewer rankings, approval rate as a target, and the number of agent changes reverted.

The reason approval rate is a trap deserves a sentence. If reviewers know their approval rate is watched, the cheapest way to improve the number is to reject more, which pushes the cost onto the people re-doing the work. A log-based audit that rewards rejection volume teaches the same behavior. Anchor the audit to consistency instead: are reviewers applying the written rule, and does the written rule predict a good merge?

When the rubric turns out to be wrong, the audit has done its job. If one criterion produces most of the disagreements and most of the failures the team later regrets, rewrite it, re-score the last two periods against the new version, and note the change. That is the loop. The humans in the loop are the thing being improved, and the log is just the substrate.

If you are running several teams with different reviewers and want the audit to compare like with like, one choice is a single model interface so the agent side stays constant while you calibrate the human side. Quanta Inference routes text, image, video, audio and embedding models through one OpenAI-compatible API, which keeps the agent behavior stable enough that reviewer disagreement is not confounded by model swaps: https://quantaintelligence.ai/inference.