Evaluating AI output and human review
Review rubrics and anchors
Writing review rubrics and anchors
- A Checklist a Reviewer Can Apply in Under Five Minutes
- Confidence Anchors That Survive a Reviewer Change
- Confidence Boxes for Human Reviewers: What the Anchors Should Say
- Grading Retrieval Relevance: A Rubric with Anchors
- How to Score a Pair When the Human Did the Task Differently
- How to Write a Scoring Anchor Example a Reviewer Can Copy
- Reviewing a Cluster Label: A Second-Pass Checklist with Anchors
- Writing the Review Rubric So Two Reviewers Land in the Same Cell
Sampling AI output
Sampling AI output for review
- Sizing the Review Sample for a Changed Process
- When the Review Sample Is Too Small to Trust
- A Fidelity Sample: Scoring Summaries Against Their Passages
- A Sampling Plan for Pair Reviews: How Many and How Often
- Estimating Reviewer Hours from Ticket Complexity
- How Many Labeled Pairs You Need Before Trusting a Classifier
- How to Sample AI Outputs for Review Without Reading Everything
- How to Sample Paired Tasks Without Re-Reading Everything
- All of Sampling AI output
Catching rubber stamps
Catching rubber-stamp approvals
- Detecting Rubber-Stamp Approvals in an Agent Decision Log
- Detecting Rubber-Stamp Approvals in an AI Decision Log
- How to Audit Your Own Reviewer: Spotting Rubber-Stamp Approvals
- How to Audit Your Own Reviewers: Spotting Rubber-Stamp Scores
- Spot-Checking the Approvers: A Sampling Plan for Review Quality
- The Reviewer's Reading Time: A Proxy for Attention
- When a Second Reviewer Is Worth the Hour
- When the Agent Is Faster but the Reviewer Is Slower
- All of Catching rubber stamps
Running a review queue
Running a review queue
- How to Approve AI Decisions About People: A Review Log
- How to Compare an AI Agent's Output to the Work It Replaced
- The Escalation Note a Reviewer Can Act On in One Page
- What the Person on an Assisted Step Needs to See
- When the Gate Says No: The Agent's Failed-Verify Escalation Note
- A Review Queue for Sampled AI Support Answers
- A Two-Step Review: Content Check Before Source Check
- Building the Correction Ledger an Assisted Step Writes To
- All of Running a review queue
Appeals and disagreements
Handling appeals and disagreements
- Explaining an AI Decision to the Person It Affected
- The Arbitration Log: What to Record When Reviewers Clash
- When Two Reviewers Disagree: A Tie-Breaking Rule for Scored Pairs
- How to Handle an Appeal of an AI-Assisted Decision
- Retention for the Arbitration Log: Who Reads a Clash
- Sampling the Arbitration Log: How Many Clashes to Read
- The Tie-Breaker's Third Option: When Reviewers Clash on a Fact
- When a Reviewer Disagrees with the Process Map
- All of Appeals and disagreements
Reasoning and citations
Scoring reasoning and citations
- Auditing an Agent's Stated Reason for a Handoff
- Calibrating a Reason-Check Classifier Against Human Labels
- Training Staff to Spot a Hallucinated Number
- When the Agent's Summary Replaces the Source
- When the Reviewer Cannot Check the Source: A Fallback Screen
- Writing the Reason Code Taxonomy Before the Model Runs
- Logging a Citation You Cannot Open
- Logging a Handoff Reason Without Blocking the Agent
- All of Reasoning and citations