AI agents

LLM judges and human baselines

• Calibrating a Verification Score Against Reviewer Labels
• How to Calibrate an LLM Judge Against Human Labels
• Pricing a Judge's False Pass and False Fail
• Reviewer Decisions as Training Data
• Reviewing a Judge Against a Reviewer: Disagreement and Its Cost
• Scoring Agent Output Against a Human Baseline: A Rubric
• What a False Pass Costs After It Ships
• A Decision Log for Equivalence Verdicts
• A Judge That Reads Only Pseudonyms: Keeping Blind Review Blind
• A Judge's Confidence Score: When a Verdict Skips the Human
• A Rubric Change Log: Versioning Dimensions and Anchors
• Charging a False Pass Back to the Caller: A Chargeback Schema
• Correcting a Judge That Skews Against the Human Label
• Detecting a Precedence Rule That Memorized One Reviewer
• Detecting Judge Drift When the Grader Model Changes
• From Bins to Budget: Turning Calibration Into a Release Rule
• Grading a Summary Against Its Source With a Judge
• Judge and Reviewer Disagree: A Tie-Break Table
• Moving a Tier's Rate From Sampling to a Live Judge
• Picking an Independent Checker API and Holding Its Credentials
• Precision, Recall and Threshold Choice When Failures Are Rare
• Pricing a Tolerance Band: False Passes vs False Fails
• Reason Codes for Reviewer Disagreement
• Reviewer Minutes, Loaded Rate and the Break-Even Volume
• Rubrics for Grading Multi-Step Agent Trajectories
• The Human Baseline: Labeling Coders Before You Trust the Agent
• The Human Baseline: What Correct Means Before a Judge Is Built
• Training a Judge on Approver Decisions: When a Model Can Sign Off
• Training a Judge on Reviewer Decisions
• Turning Judge Disagreements Into a Human Review Queue
• Turning Reviewer Overrides Into a Held-Out Precedence Set
• What a Failed Sample Costs After It Reaches a Customer
• What a False Pass Costs After It Reaches a Customer
• When to Move From Review Sampling to a Live Judge
• Writing an Answer Rubric a Rater Can Apply in a Minute