Human in the loop and review workflows
Approval flows and prompts
Approval flows and confirmation prompts
- How to Add a Human in the Loop to an AI Workflow
- A Promotion Queue: Routing Draft Writes to a Second Source Check
- Approval Prompt Fields That Map to a Schema
- Human-in-the-Loop for Tool Calls: When to Require Approval
- Structured Output for Approval: A Schema Reviewers Read
- Versioning an Approval Schema Without Breaking Queues
- What to Record in an Agent Approval Log
- Writing Approval Prompts a Reviewer Can Decide Fast
- All of Approval flows and prompts
Review queues and routing
Review queues, routing, and escalation
- A Reviewer Queue for Regression Fixtures That Keep Failing
- Approval Fatigue: Measuring Reviewer Load and Tuning the Gate
- Designing Reason Codes for Agent Refusals: A Taxonomy That Scales
- Routing Agent Work: Auto-Approve, Queue or Block
- Routing Reason Codes to Queues: A Table and Fallbacks
- Sign-Off Ownership for Held Chunks: Who Approves Which Source
- Teaching an Agent to Pick Its Own Reason Code
- A Hold Reason Taxonomy That Routes to the Right Fix
- All of Review queues and routing
Sampling and reviewer load
Sampling, calibration, and reviewer load
- How Many Outputs to Label: Sizing a Human Review Sample
- Recording Rater Scores Without Rater Names
- Sample Rates by Risk Tier: Proving the Audit Was Met
- Sampling Approvals: How Often to Review Agent Output by Task Risk
- Sampling Subagent Instructions: When a Delegation Needs Review
- Sizing a Review Sample: Confidence, Error Rate and Volume
- Turning a Review Sample Into a Stop or Go Signal
- A Decision Rule Template: Threshold, Confidence, Cost
- All of Sampling and reviewer load
LLM judges and human baselines
LLM judges and human baselines
- Calibrating a Verification Score Against Reviewer Labels
- How to Calibrate an LLM Judge Against Human Labels
- Pricing a Judge's False Pass and False Fail
- Reviewer Decisions as Training Data
- Reviewing a Judge Against a Reviewer: Disagreement and Its Cost
- Scoring Agent Output Against a Human Baseline: A Rubric
- What a False Pass Costs After It Ships
- A Decision Log for Equivalence Verdicts
- All of LLM judges and human baselines
Shadow mode and rollout gates
Shadow mode and rollout gates
- From Shadow Score to Rollout Decision: Rubric to Go/No-Go
- Shadow Mode: Running an Agent Alongside Staff Before Launch
- Shadowing a Candidate Checker Against Live Promotions
- Sizing a Shadow Run from Rubric Variance
- A Canary Process for Rolling Out a Model or Prompt Change
- A Replay Set That Labels Each Disagreement Class
- Canary Releases for LLM Features: Steps, Metrics and Rollback
- Designing a Review Queue When an Agent and a Person Disagree
- All of Shadow mode and rollout gates
Escalation and handoff
Escalation, handoff, and work routing
- A Reason Code Taxonomy for Agent Escalations
- Confidence From Span Scores: Citing Versus Escalating
- Feeding a Stopped Review Back Into the Eval Set
- What a Support Agent Should Say When It Does Not Know
- A Conflict Taxonomy for Precedence: Escalate, Merge or Drop
- A Cost Table for a Second Model Call Versus a Human
- A Second-Hop Queue: When the First Owner Does Not Answer
- Confidence Scores as Review Triggers: Calibrating the Threshold
- All of Escalation and handoff