Development

Inference routing and gateways

• A Routing Rule That Sends Overflow Traffic to a Cheaper Model
• Gateway, Sidecar or In-Process Routing for Model Calls
• Routing by Difficulty: Picking the Smallest Model That Works
• Routing Inference Requests by Cost and Latency
• A Policy Table in Postgres: Schema and Cache
• A Routing Rule That Fires on Latency Before It Fires on Cost
• Draining Connections and Warming Models on Deploy
• Multi-Region Failover for AI Features: What Degrades First
• Routing by Confidence: When to Send a Request to a Larger Model
• Routing by Cost Per Accepted Output: A Production Switch
• Routing by Difficulty: When to Send a Request to a Larger Model
• Routing Queries to Models by Difficulty: A Numeric Rule