Choosing and managing AI model providers
Comparing and choosing models
Comparing and choosing models
- A Scorecard for Choosing Which Model to Call
- When a Candidate Model Fails on Long Inputs
- A Scorecard Template for Choosing Which Model to Call
- Comparing Two Models on Cost per Resolved Task
- How to Write a Vendor Scorecard for AI Tools
- One Eval Suite, Two Providers: Making Results Comparable
- What to Log for Every Model Call: Fields That Support a Scorecard
Running a model bake-off
Running a model bake-off
- How to Run a Model Bake-Off: Script, Score and Sign-Off
- When to Accept a Live Call in Your Test Suite
- Building a Baseline Test Set from Production Traces
- How to Keep a Bake-Off Repo a Reviewer Can Rerun
- How to Sample Production Traffic for a Test Set
- How to Set a Cost Ceiling for a Model Evaluation Run
- Logging Two Providers Side by Side: A Cross-Run Comparison Schema
- Mocking an Agent's Tool Calls Without Mocking the Model
- All of Running a model bake-off
Testing a provider
Testing a provider before you sign
- How to Test a Model Provider's Data Retention Through Its API
- Running One Eval on Two Providers Without Rewriting It
- A Canary Checklist for Model Providers
- A Portability Test for a Vendor API Before You Sign
- A Synthetic Record Format That Survives Provider Swaps
- How to Audit an AI Vendor's Service Catalog
- How to Build a Portability Test for an AI Vendor
- How to Test a Model Provider's Data Retention Through the API
- All of Testing a provider
Provider changes and outages
Surviving provider changes and outages
- A Canary Schedule That Matches a Provider's Change Cadence
- How to Read an AI Vendor's Model Change Notice
- When a Provider Changes Retention Terms: Rerun Your Canary
- When a Provider Lowers the Context Limit: A Migration Checklist
- A Decision Log for Provider Swaps
- Budgeting the Cost of an AI Vendor Outage Before It Happens
- How to Track Model Changes Across Multiple Providers
- How to Turn a Vendor's Changelog into a Maintenance Calendar
- All of Provider changes and outages
Contract clauses and remedies
Contract clauses and remedies
- Contract Terms That Make an AI Vendor Exit Cheap
- Estimating the Maintenance Hours Behind an Integration
- How to Vet an AI Vendor's Support and Escalation Terms
- How to Write a Credit and Remedy Table for an AI Service Outage
- How to Write a Data Processing Addendum for AI Tools
- How to Write an Escalation Clause in an AI Vendor Contract
- Setting Credit Caps and Exclusions for a Vendor Remedy Table
- A Remedy Table for Internal AI Services: Credits Between Teams
- All of Contract clauses and remedies