Evals and gold sets
Founders
Evals and gold sets
- How Many Evals Are Enough? A Sample Size Rule for Model Testing
- How to Build an Eval Set from Your Support Tickets
- How to Compare Two Codebook Versions on the Same Batch
- How to Eval Whether Your Pipeline Honors Exclusions
- How to Set a Minimum Accuracy per Ticket Category
- How to Test a Schema Card Against a Set of Planted Questions
- How to Test Whether a Disclosure Reviewer Catches a Planted Leak
- How to Turn Adherence Scores into a Batch Go/No-Go Rule
- How to Version a Clause Library Without Breaking the Screen
- How to Attribute a Theme Tag Shift to the Codebook or the Batch
- How to Audit a Go/No-Go Rule Against the Outcomes That Followed
- How to Audit a Model's Clause Extraction Against a Human Sample
- How to Audit a Per-Category Accuracy Bar Against Real Outcomes
- How to Audit a Router's Escalation Log Against What It Cleared
- How to Audit an LLM's Sentiment Labels Against a Human Sample
- How to Brief a Second Model to Re-Code the Same Transcripts
- How to Build a Gold Set for an Agent Before Real Work
- How to Build a Planted Deviation Set for a Clause Library
- How to Catch a Paraphrased Identifier in a Closed Table
- How to Check the Labels Behind a Routing Bar for Noise
- How to Choose Which Ticket Categories Deserve a Model at All
- How to Compare Two Models on Latency Without Fooling Yourself
- How to Compare Two Models on the Same Eval
- How to Compare Two Triage Models on the Same Categories
- How to Decide Which Planted Questions Belong in Regression
- How to Detect a Chunker Change from the Queries It Answers
- How to Eval a Chunked Sheet Against the Whole Sheet
- How to Hold a Reviewer Prompt Stable Across Upgrades
- How to Keep a Score Decision Rule from Leaking the Label
- How to Keep a Triage Set Stable When You Add a Category
- How to Log a Clean Pass So You Can Replay the Bar Later
- How to Log a Rubric Score So a Reviewer Can Audit It Later
- How to Log Model Scores So You Can Replay a Threshold
- How to Measure Inter-Rater Reliability Without a Statistician
- How to Re-Test a Clause Screen After a Model Swap
- How to Re-Test a Coding Agent After a Model Version Change
- How to Sample Rejected Comments for a Second-Model Audit
- How to Seed an Eval Set Before You Have Any Tickets
- How to Set a Citation Rate Bar for an Agent's Report
- How to Set a Coding Baseline Before a Model Touches the Repo
- How to Set a Disagreement Rate Threshold for Model Summaries
- How to Set a Minimum Detectable Effect Before an Eval
- How to Set a Sample Size for a Coding Audit
- How to Set a Working-Set Sampling Rate for Routing Decisions
- How to Set an Agreement Threshold Before You Trust Batch Tags
- How to Size a Go/No-Go Rule for a Coding Agent to Build
- How to Split Tickets into Tune and Holdout Sets
- How to Split Your Eval Set So a Pass Rate Means Something
- How to Tell a Retrieval Miss from a Leak
- How to Test a Categorizer Against a Gold Set of Transactions
- How to Test a Reviewer That Catches Reversals
- How to Test a Schema Card Against a Renamed Column
- How to Test a Table Chunk with a Header-Dependent Question
- How to Test an Embedding Model on Header-Row Distance
- How to Test Whether a Chunk Boundary Bleeds a Neighbor's Fact
- How to Test Whether a Review Model Catches Reversals
- How to Time an Injection Test So It Lands During Peak Load
- How to Turn Codebook Disagreements into a Rubric Revision
- How to Version a Codebook So Model Tags Stay Comparable
- How to Write a Calibration Set That Survives a Rule Change
- How to Write a Codebook a Model Can Apply Without Drift
- How to Write a Null-Field Control into Every Scoring Run
- How to Write a Replay Test That Catches Agent Drift
- How to Write a Retrieval Eval Set from Your Own Questions
- How to Write a Test First So a Coding Agent Can Run It
- How to Write an Eval Set from Your Support Tickets
- When a Per-Category Bar Fails, Is It the Model or the Taxonomy?
- When to Stop Collecting Eval Cases and Ship the Model