Testing and evaluating AI behavior
Golden sets and labels
Golden sets and labeled samples
- A Golden-Set Check That Blocks a Gateway Rule Change
- Anonymized Data for Model Evaluation: Building a Held-Out Set
- Evaluating a RAG Pipeline End to End: Golden Sets, Judges, Drift
- Golden Sets for Support Agents: Intents and Escalation Paths
- A Golden Set of Context Tuples for Key Regression
- A Golden Set That Ranks Models by Cost Per Accepted
- A Second Golden Set for the Cheap Model's Strengths
- Building a Golden Dataset for Retrieval Evals
- All of Golden sets and labels
Scoring model output
Scoring model output
- Writing Assertions on Model Output That Catch Real Regressions
- Writing Checklist Lines a Model Can Score Automatically
- A Severity Ladder for Model Output Failures
- Asserting Business Invariants, Not Just Shapes
- Which Checklist Lines Ever Fire: Counting Failures
- Writing a Rubric Line a Model Can Score the Same Way Twice
Judges and calibration
Automated judges and calibration
- Evaluating Prompt Candidates Without a Human Judge
- Scoring an Automated Judge on the Golden Set First
- A Judge's Agreement Score: When to Trust It and When to Retrain
- A Relevance Rubric an Automated Judge Can Apply
- Calibrating a Judge Against a Cheap Model: Agreement and Bias
- Calibrating a Judge Against Human Labels: Size and Agreement
- Calibrating an Automated Judge Against Human Labels
- Calibration Before Thresholds: Making a Score Mean What It Says
- All of Judges and calibration
Stubbing and replay
Stubbing and replaying model calls
- Choosing a Re-Record Cadence from Vendor Change Volume
- Recording Vendor Responses for Replay in CI
- Splitting Tests by Model Dependency: Stub, Sandbox, Live
- Stubbing an Embedding API in CI Without Losing Fidelity
- Stubbing Model APIs So CI Quickstarts Stay Deterministic
- A Determinism Contract With Every Model Vendor
- A Fixture Inventory That Names Every Owner
- A Fixture Linter: Catching Drift Before It Breaks CI
- All of Stubbing and replay
Model swaps and gates
Model swaps and quality gates
- A Cost Per Accepted Output: The Metric That Decides a Model Swap
- A Quality Gate for a Cheaper Model: Scoring Before You Switch
- A Test Set for Model Swaps: Label Once, Reuse Across Versions
- Scoring Outputs Before You Switch to a Cheaper Model
- Thresholds That Fire Too Often: Tuning a Gate
- When the Judge Rejects a Cheap Model: Distinguishing Faults
- A Fault-Injection Suite for Model Swaps
- A Golden Set Where the Cheap Model Should Win
- All of Model swaps and gates
Drift detection