Cost, latency, and budgets for AI agents
Estimating and attributing spend
Estimating and attributing token spend
- How to Control LLM Costs in Production
- Estimating Token Spend Before the First Model Call
- How to Attribute Token Spend to Features and Teams
- Allocating AI Spend to Teams and Customers
- Charging Shared Model Spend Back to Teams and Customers
- Cost per Resolved Ticket for Support Agents
- Cost per Scored Run: Budgeting Repeats, Consistency, Model Spend
- Estimating the Cost of a Model Call Before You Make It
- All of Estimating and attributing spend
Hard caps and refusals
Setting hard caps and refusal handling
- A Downgrade Path When a Model Overruns Its Output Budget
- Budget Caps on Model Spend: Where to Put the Limit
- Structured Output for Cap Exceeded: A Schema for Refusals
- Testing That a Cap Fires: Fixtures for the Refusal Path
- Turning a Token Estimate Into a Hard Cap: Where the Limit Lives
- A Cap on Output Tokens Only: When the Prompt Is Unbounded
- A Cap That Resets Mid-Run: Window Rollover Inside an Agent
- A Log Schema for Recovery Paths: Re-Ask, Escalate, Abandon
- All of Hard caps and refusals
Prompt caching and savings
Prompt caching and cache savings
- Prompt Caching: How It Works and When It Pays
- A Cache Breakpoint Lint Rule for Prompt Templates
- A TTL Matrix for Two Cache Layers
- Cache Hits vs Prompt Freshness: The Trade-off
- Cache Invalidation Across Two Layers: A TTL Matrix
- Comparing Prefix Caching Rules Across Three Providers
- Eviction Policy for a Two-Layer Cache: LRU, LFU or Length
- Stacking Local Cache on Provider Prefix Cache
- All of Prompt caching and savings
Latency budgets and streaming
Latency budgets and streaming behavior
- A Latency Scorecard for Streamed Answers: First Chunk and Tail
- Reconnecting a Stream Without Duplicating Tokens
- A Latency Budget per Agent Hop: Numbers and Triggers
- A Read Latency Budget: p95 Targets for Memory and Model Calls
- A Time Budget for Each Streaming UI State
- Choosing a First-Chunk Budget by Query Class: A Method
- Deadline Budgets That Survive Every Tool Hop
- Time to First Chunk, Chunk Gap and Close: Budgets
- All of Latency budgets and streaming
Retry and concurrency budgets