AI coding

Calibrating automated reviewer decisions

• Designing Reviewer Abstention: A Band Between Pass and Fail
• Shadow Runs: Comparing the Gate's Verdict to a Human Reviewer
• Two Judges and a Human: Three-Way Agreement as a Gate
• Two LLM Reviewers Disagree: A Tie-Breaker Pass
• When the Tie-Breaker and the Human Disagree: An Override Log
• A Calibration Log for Pass Criteria
• A Calibration Set for the Abstention Band
• Cohen's Kappa Per Criterion: A Spreadsheet and Its Limits
• Counting Claims That Never Reached a Judge
• Designing an Abstention Band Between Rule and Reviewer
• Measuring Reviewer Consistency Across a Team
• Measuring Reviewer Consistency Before Comparing a Rule
• Pairing Rule and Reviewer Verdicts in One Log Row
• Recording a Pass or Fail for Every Model Call: A Log Schema
• Resolving Disagreements Between an Agent and a Human Reviewer
• Sampling Judge Calls: How Few Labels Prove Agreement
• Sampling Real Requests for a Rule Comparison
• Seeding Defects to Calibrate an Automated Reviewer
• Shadow Traffic for Model Migrations: Comparing Old and New
• Splitting Retrieval and Generation in the Eval: Two Scores
• Turning Abstention Logs into Rubric Edits
• Two Thresholds or a Four-Level Action Table
• Watching the Abstention Rate for Calibration Drift
• When the Rubric Moves: A Rerun Protocol for Agreement
• When the Self-Review Passes and the Human Gate Still Fails
• When to Trust an Agent's Self-Reported Confidence
• Where Agent Code Reviews Catch Bugs: Sampling Merged Diffs