Routing Inference Requests by Cost and Latency

A router needs two signals: what kind of task this is, and how hard this particular instance is likely to be. Task type is cheap to derive from the calling code. Difficulty is a prediction, and there are three realistic ways to get it.

Metadata rules. Route on user plan, language, context length and interaction type. These conditions should have a product reason and an observable outcome, and you should log the matched condition, the selected model, cost, latency and product result so you can check the rule later per Inworld. A rule like "long prompt equals expensive model" is often wrong for simple summarization, so measure before you trust it.

A small classifier. Train or prompt a tiny model to score difficulty from the prompt. Semantic routing embeds the request and routes by inferred intent, but the embedding or classifier step itself adds latency to the critical path, so measure it against a plain task tag before adopting it per the same routing overview. Only use it when the caller cannot cheaply label the request itself.

Post-generation checks. Let a cheap model answer first, then grade the output and escalate only if it fails. This is a cascade, and production systems frequently combine pre-generation routing with cascades, using cheap pre-filters to weed out easy cases and escalation to catch the ones that slip through per MLflow.

A practical starting point is a two-field tag: task type from the caller, difficulty from a short heuristic (token count, number of entities to extract, whether the prompt contains code or numbers). Ship that, log it, and only add a classifier once the replay data shows the heuristic is misrouting real volume.

The routing table is a configuration file, not application code. Each row maps a task type and difficulty band to an ordered list of models, with a quality threshold and a latency budget per row.

routes:
  - task: "ticket_classification"
    difficulty: "low"        # score < 0.4
    quality_bar: 0.92        # agreement with gold labels
    latency_budget_ms: 800   # p95 end to end
    chain: ["small-fast", "mid-general", "frontier"]
  - task: "contract_summary"
    difficulty: "high"       # score >= 0.7
    quality_bar: 0.88
    latency_budget_ms: 6000
    chain: ["mid-general", "frontier"]

The decision rule is one sentence: for each request, walk the chain in order and take the first model whose measured quality on that task type meets the row's bar and whose recent p95 latency fits the budget. If the head of the chain is degraded, the chain handles it. If the head fails outright, fall back to the next entry.

Two distinctions matter here. Error fallback and latency fallback solve different problems: error fallback moves after an explicit failure, while a time-to-first-token fallback moves when a model has not begun responding within a threshold. The latter protects responsiveness but can create simultaneous or sequential spend, so define the timeout from your application budget and the primary route's normal p95 per Inworld. Record total time across attempts rather than reporting only the successful model.

Also note that routing is not failover. Routing picks the right model for the task. Failover keeps the same task alive when a provider dies. A provider fallback for the same model reduces behavior change, while a model fallback improves availability at the cost of greater output variance per the same guide. Keep those two policies in separate config sections so an outage does not silently rewrite your quality policy.

The cost trap in cheap-first routing is real. If the cheap model consistently fails the quality bar, you pay for the first attempt and the fallback, adding latency and sometimes increasing total cost per Merge's routing overview. Set an alarm on your fallback rate and treat a sustained rise as a broken routing row, not a busy day.

Retry budgets and circuit breakers

Give every request a retry budget in wall-clock time, not attempts. If the budget is 2.5 seconds and the first model has already burned 1.9, the fallback gets 600 milliseconds and the router should skip any model whose recent p95 exceeds that. This single rule prevents the worst failure mode: a slow primary plus two retries that add up to a timeout three times longer than any user will wait.

Circuit breakers sit per model, not per route. Track a rolling window of errors and timeouts; when the failure rate crosses a threshold, open the breaker and send that route's traffic straight to the next chain entry. Close it after a cooldown and a probe request. Keep the breaker state in shared storage so every instance agrees, otherwise half your fleet keeps hammering a dead provider.

Suppose you process 2 million requests a month across two task types. Your logs, not a vendor page, produced these figures during a one-week replay:

Set the classification bar at 0.92 and the summary bar at 0.90. All three models clear classification, so the cheapest wins and you send 1.8 million classification requests to small-fast. Only frontier and mid-general clear summary, so the 200,000 summary requests start at mid-general and escalate to frontier only when the post-generation check fails.

Run the arithmetic on the failure path. If 5 percent of summary requests fail the check and escalate, you pay for mid-general on all 200,000 and for frontier on 10,000. If instead you had sent all classification to frontier, every one of those 1.8 million requests would cost 30 times the small model, for a quality gain of 0.04 that the bar did not require. That 0.04 is the number to put in front of a product owner: the routing bar exists to make the trade-off explicit.

The cost figures here are illustrative ratios, not prices. Real numbers come from your own token counts multiplied by your contracted rates, and you should recalculate per task type because input and output token mixes differ. The structural insight holds regardless of the actual prices: route to the cheapest model that clears the bar, and let escalation frequency, not intuition, decide when to raise the bar.

Latency deserves the same treatment. Time to first token is the main driver of perceived responsiveness in interactive features, while end-to-end latency captures the full wait, and end-to-end latency is simply time to first token plus total generation time per Anyscale's metrics guide. Budget them separately. A chat endpoint may accept a 500 millisecond first-token budget and a 4 second total, while a batch summarizer only cares about throughput and total time. Route on whichever metric the feature actually feels.

Offline validation is a replay harness. Take a sample of logged requests, run each through the candidate routing table without calling live users, and compute three numbers per task type: quality against your labels, blended cost, and p95 end-to-end latency including retries. Compare the candidate table against the current one on the same sample. A change ships only if quality holds within a small tolerance and cost or latency improves.

Replay has limits worth stating plainly. Logged requests do not capture how users react to a different model's tone, and your quality labels may be sparse. When a model changes behavior, a faster or cheaper model can still reduce retention if quality drops, so export route exposure to product analytics and join it to accepted output, satisfaction and retention per Inworld.

Online validation is a traffic split with sticky assignment, so one user does not alternate between model behaviors mid-session. Predeclare the product metric, the guardrails, the duration and the rollback rule. A feature-flag control plane is the right tool: flags decouple deployment from release, let you target specific cohorts such as regions, device types or customer tiers, and support instant rollback for affected cohorts without redeploying per Datadog's explanation. Progressive rollouts and kill switches shrink the blast radius by exposing a change to a small slice of traffic first, as little as one percent, rather than everyone at once per LaunchDarkly.

The same discipline that keeps development and production close applies here. Keep your router config versioned and reviewed like code, and keep the replay environment stocked with the same model endpoints and tokenizer behavior as production, so a routing decision tested offline behaves the same way live as the twelve-factor parity principle argues. Drift between environments is exactly how a router that looked fine in staging surprises you at peak.

Track these continuously, per route: fallback rate, cache hit rate, cost per request, p95 latency, error rate by model, and the share of requests escalated after a quality check. Continuous monitoring of fallback rate, cache hit rate and cost is what prevents silent drift in the policy per MLflow.

Re-evaluate the routing table on a fixed schedule, monthly or quarterly depending on volume, and whenever a task type's quality drops below its bar for a sustained window. Model providers update their endpoints, your traffic mix shifts, and prompt changes alter difficulty distributions. A routing table nobody revisits is a cost leak with a policy document attached.

Finally, know when a complex selector loses to a static rule. Automatic selection by price or latency is useful for broad, low-risk workloads but dangerous when tool behavior, data policy, model version or quality requirements are strict, and a transparent static rule often outperforms a complex selector when the team cannot explain or audit the decision per Inworld. If your router's decisions cannot be explained to an auditor in one sentence, simplify it.

When the routing logic itself is the work you would rather not build, a gateway that exposes many model types behind one key removes the adapter layer: Quanta Inference offers text, image, video, audio and embedding models through one OpenAI-compatible API, so the routing table points at model names instead of five provider SDKs. That is worth it when your chain spans modalities; it is overkill when all your traffic is one task type.