Golden tasks and harnesses

Golden tasks and evaluation harnesses

• How to Evaluate an AI Agent Before Launch
• How to Build a Golden Task Set for a Support Agent
• How to Evaluate a RAG Pipeline: Retrieval and Answer Metrics
• Setting the Tolerance Band for an Agent Equivalence Rule
• A Budget Sweep Harness: Fixtures, Metrics, a Rerun Curve
• A Fact List per Query: Turning Coverage Checks into a Test
• A Generator Version Field in the Task Contract
• Building a Prompt Evaluation Harness with Golden Tasks
• All of Golden tasks and harnesses

Fixtures and replay testing

Fixtures, mocks, and replay testing

• A Repair Pass When a Checksum Mismatches: Find, Fix or Drop
• Mocking a Memory Store for Fast Policy Tests
• Mocking a Model Endpoint Without Losing Fidelity
• Mocking Tool Responses for Fast Agent Tests
• Pinning Model Versions in Tests: Aliases and Snapshots
• Replaying the Checker: Fixtures From Promoted and Dropped Rows
• Running One Spec With the Cache On and Off
• Simulating Refusals in Staging: Fixtures for Every Code
• All of Fixtures and replay testing

Fault injection testing

Fault injection and error-class testing

• A Fake Tool Server That Returns Every Error Class
• A Fault Injection Playbook for Tool Calls in Staging
• Fault Injection in Agent Tests: Timeouts, 500s and Bad Payloads
• Simulating Tool Failures in Staging: A Fault Injection Playbook

Regression gates

Regression gates and release checks

• Feeding a Failed Sample Back Into the Tier's Prompt or Tool
• Turning a Retrieval Curve Into a Go/No-Go Release Rule
• A Decision Log for Waived Gates: Fields, Owners and Expiry
• A Gate That Blocks a Retrieval Change Before Release
• A Retrieval Regression Gate for Release
• A Waiver Log for Regression Fixtures a Team Chooses to Skip
• Canary Process for Prompt Changes After a Fixture Passes
• Choosing a Pass Threshold When Failures Are Asymmetric
• All of Regression gates

Synthetic test cases

Synthetic cases and test data generation

• A Synthetic Task Is a Contract: Fields a Generator Must Fill
• Generating the Fact List Automatically Before a Coverage Check
• Synthetic Tasks: Generating Cases You Have No Logs For
• Mining Support Logs for the Slices Your Golden Set Misses
• Sampling Generated Tasks for Human Review
• Scoring the Fact Rewriter Against a Human List
• Verifying Generated Cases: Sampling, Judges and a Human Gate
• When a Generated Task Is Wrong: Detection and Retirement
• All of Synthetic test cases

Tool-call scoring

Tool-call and trajectory scoring

• Scoring Tool-Call Accuracy, Not Just Final Answers
• Scoring Argument Fields Separately: A Scheme for Tool Calls
• Scoring Tool Selection: Metrics Beyond Pass or Fail
• When the Tool Call Is Wrong but the Answer Is Right