AI coding

Using LLM judges for code review

• A Second-Pass Agent Reviewer Before the Human Gate
• A Tie-Breaker Prompt That Requires Evidence: Fields and Examples
• From Verdicts to a Queue: A Schema for Judge Scores and Reasons
• Grading Agent Output That Has No Assert: Rubrics and Judges
• Pinning a Rubric to a Judge Call: Versioning and Rerun Protocol
• Retrieval for a Judge: Grounding Scores in Sources
• Setting a Minimum Agreement Rate Before an LLM Judge Gates
• When a Judge Cites a Passage That Does Not Back the Claim
• Writing a Review Rubric an LLM Judge Can Apply
• A Disagreement Log for Two Judges and a Human
• A Disagreement Log for Two Reviewers and a Tie-Breaker
• A Log Schema for Automated Review Verdicts
• A Rubric Change Log: Who Edited Which Criterion and Why
• A Second Judge Pass on Sources: Cost and Score Flips
• An Override Log for Judge and Reviewer Disagreements
• Calibrating an LLM Judge Against Human Labels: A Cookbook
• Costing a Three-Way Gate: Judges, Human Minutes, Throughput
• Fields a Tie-Breaker Must Return: Verdict, Reasons, Evidence
• From Verdicts to a Queue: A Schema for Scores and Reasons
• Holding Prompts Fixed While You Compare Models
• Pinning a Rubric Version to Every Judge Call
• Prompts That Make a Judge Name the Criterion It Applied
• Reading a Package Changelog with a Judge: Fields and Criteria
• Routing Judge Disagreements to a Human Queue
• Sampling the Unanimous Cells for Shared Blind Spots
• Seeding an Eval for a Contradiction Prompt
• Splitting a Rubric into Deterministic Checks and Judge Calls
• Splitting a Rubric into Fact and Style Checks
• Storing a Judge's Reason Next to the Evidence It Quotes
• Storing the Judge's Raw Output Alongside the Hash
• The Cost of a Human Review: Setting a Maximum Override Rate
• Tracking False Passes and False Fails in a Test Gate
• Turning a Context File Edit into a Judge-Scored Diff
• What a Tie-Breaker Must Return When the Evidence Is Two Lines
• When a Judge Field Drifts: Pinning Rubrics and Versions
• When a Model Passes on Average but Fails a Slice
• When a Reviewer Should Abstain Rather Than Score
• When the Judge Model Changes: A Rebaseline Protocol
• When the Tie-Breaker and the Human Reviewer Disagree
• When the Upstream Signal Is a Judge: Automated Review
• Writing Judge Anchors That Survive a Model Upgrade
• Writing the Rubric a Tie-Breaker Reads