Development

Model swaps and quality gates

• A Cost Per Accepted Output: The Metric That Decides a Model Swap
• A Quality Gate for a Cheaper Model: Scoring Before You Switch
• A Test Set for Model Swaps: Label Once, Reuse Across Versions
• Scoring Outputs Before You Switch to a Cheaper Model
• Thresholds That Fire Too Often: Tuning a Gate
• When the Judge Rejects a Cheap Model: Distinguishing Faults
• A Fault-Injection Suite for Model Swaps
• A Golden Set Where the Cheap Model Should Win
• A Rejection Taxonomy: Codes, Owners and Thresholds
• Correlating a Model Swap with a Quality Drop in Traces
• Estimating Model Swap Costs from Your Test Set Runs
• Hidden Costs of a Model Swap: Retries, Re-prompts, Review
• Output Invariants to Check Before Any Model Swap
• Regression Tests for a Prompt Change Without a Human Judge
• Shadow Traffic for a Candidate Model: Running Both and Comparing
• Shadow Traffic for a Cheaper Model: Sampling Rate and Duration
• The Cost of a False Positive: Pricing Each Gate Mistake
• Who Signs Off on a Model Change, and on What Evidence