How to Route on Latency, Not Just Price

You cannot route on latency you do not measure. Start with the grain that matters: every trace should record a route name (the task, not the endpoint), the model, the start time, the duration, and whether the call was a retry. Compute percentiles from raw durations at the grain you report. A p95 of daily p95s is not the weekly p95, because percentiles cannot be recombined after the fact (Metabase).

A minimal schema and query looks like this:

SELECT
  route,
  model,
  COUNT(*) AS calls,
  ROUND(percentile_cont(0.5) WITHIN GROUP (ORDER BY duration_ms)) AS p50_ms,
  ROUND(percentile_cont(0.95) WITHIN GROUP (ORDER BY duration_ms)) AS p95_ms
FROM llm_traces
WHERE environment = 'production'
  AND status = 'success'
GROUP BY route, model
ORDER BY route, p95_ms DESC;

Keep these habits:

• Report p50 and p95 side by side. p50 tells you what the typical request sees; p95 tells you about the tail, which is where most of the annoyance lives. A p95-only spike is usually provider-side congestion: the median barely moves but the slowest users feel it (Metabase).
• Track time to first token separately when you stream. Users feel TTFT, not total duration, so track TTFT for experience and total duration for capacity and cost (Metabase). Streamed chat feels responsive when TTFT stays under about a second even if the full answer takes ten (Metabase).
• Segment by model. A p95 mix that blends models or output lengths can look like a regression when it is really a traffic shift toward a slower model or longer answers (Metabase).
• Never chase the single slowest request. Averages mislead on skewed distributions; report p50 for the typical case and p95 or p99 for the tail (OneUptime's percentile explainer).

If you already have a gateway, the work is mostly configuration. Some gateways track TTFT, p50, p90 and p95 for every model continuously and route against what happened in the last few minutes rather than historical averages (Requesty's routing guide). If you run your own serving, predicted-latency schedulers train a small model on prompt length, cache hit rate, queue depth and KV cache utilization to predict TTFT and time per output token per candidate server, which removes manual weight tuning (the llm-d scheduling post). Either way, the metric grain is the same.

The useful question is not "what is a good p95 for an LLM?" It is "what p95 can this task tolerate?" The ceiling is per surface, not universal: a streamed chat feels fine when time to first token stays under about a second even if the full answer takes much longer, a synchronous autocomplete needs the whole response fast, and background jobs barely care, so set a target per surface and trend against it (Metabase).

The ceiling is a constraint, not a target. Set it just tight enough to protect the experience, then optimize for cost inside it. The most common production pattern is exactly this: "give me the cheapest model that can respond under a set time to first token" (Requesty). A ceiling set too tight forces you onto expensive models for no user benefit; too loose and a slow cheap model quietly eats your users' patience.

Two practical rules:

1. Set the ceiling at p95, not p50. The tail is what your slowest users experience, and the ceiling should be a statement about them.
2. Write the ceiling into the router config, not into a wiki. If the router can see it, the router can enforce it.

This is the step most teams skip, and it is where the decision actually gets made. Queue time and human idle time are real costs, so convert them to dollars and add them to the token bill. Keep the arithmetic obvious.

Define a value for waiting time, say W dollars per hour of blocked human time, and a queue cost Q dollars per hour of added queue delay when requests pile up. For a route, the effective cost per completed task is:

effective_cost = token_cost + (p95_seconds * concurrency * Q_hourly / 3600) + (human_wait_seconds * W_hourly / 3600)

A worked example makes the point. Two candidate models serve the same classification route:

At a queue cost of $12 per hour and an average in-flight concurrency of 5 requests, Model A adds 5 x 4 s = 20 seconds of queue exposure per batch versus 2 seconds for Model B. Across a day of small batches that lands at a few dollars of queue cost, and if any human or agent is blocked waiting, the gap widens fast. The cheap model's rate advantage is real, but it is not the whole bill; it is the token line of the bill.

The same logic explains why self-hosting answers change with load. At low concurrency a GPU that serves five people costs the same as one that serves fifty, so self-hosting often loses to the cheapest API tier; at 100 concurrent users it wins far more often (Navya AI's break-even study). Waiting is cheap when nobody is waiting.

Once you have ceilings and effective costs, the routing policy writes itself.

Tiered routing with a ceiling. Classify each request into a difficulty tier, pick a model per tier, and put a fallback behind each tier, ideally from a different provider so a provider slowdown does not take out both. Routing decisions can be rule-based, cost-based (route to the cheapest model that meets latency requirements), complexity-based, or user-aware (Inworld's routing explainer). The cheapest classifier is one that makes no extra model call: start with a heuristic and reach for a small model call only where the heuristic is wrong often enough to matter.

Cascades: cheap first, escalate on failure. Try the cheap model, and only escalate when its answer fails a verifiable check. This is where the money is: routing only helps if most of your traffic really is simple. Research consistently shows that 60 to 80% of inference cost comes from sending low-complexity tasks through frontier models (Maxim AI's cost-aware routing write-up). Structured output generation (JSON formatting, template filling, data transformation) is well within small-model capability (Maxim AI).

Model routing itself has a research base: it routes a query to the most suitable component, such as a smaller model or a specialist, to improve quality while minimizing cost, and the formal version maximizes a scoring function subject to a cost budget (the routing survey). Latency belongs in that constraint alongside dollars; the survey frames the router as maximizing quality subject to a budget the operator sets (the routing survey).

The escalation decision needs a check that is cheap and honest. A schema validator is the easiest: if the JSON parses and passes the required fields, accept it; if not, escalate. That is the practical reason teams invest in strict structured outputs, which make the model follow a specific format from the start so parsing does not need fragile regex cleanup (Agenta's structured output guide). JSON Schema is the contract that defines required fields, data types and value constraints, so a validator has something concrete to check against (Agenta).

The routing ladder is worth stating plainly, because each rung must earn its complexity with measured benefit (TrueFoundry's routing overview):

1. Static rules by task tag.
2. Weighted splits.
3. Latency-aware routing.
4. Cost-aware routing inside the latency ceiling.
5. Semantic routing (embed the request and route by intent; worth it only when the caller cannot cheaply label the request).
6. Cascades with escalation.

Most teams stop at rung 4 and are right to. One caution from the same source: routing is not failover. Failover is what happens when a provider errors; routing is a policy decision about which model the request should have gone to. Mixing the two hides the fact that your router is quietly betraying you (TrueFoundry).

Use this checklist to move from rate-sheet routing to latency-aware routing without a big-bang rewrite.

The pitfalls that recur:

• Averaging latency hides the tail. Report p50 and p95, and never let a dashboard show only a mean (OneUptime).
• Re-averaging stored percentiles produces a number that is not a percentile at all (Metabase).
• Ignoring streaming makes you route on the wrong metric; track TTFT and total duration separately (Metabase).
• A p95 spike without a deploy is usually the provider: congestion, a routing change, or a degrading region, so keep the latency chart next to the error and retry charts, because rising retries inflate tail latency before they inflate error rates (Metabase).
• Comparing sticker prices only overstates savings. A lower price per million does not necessarily produce a lower total cost, because models can tokenize the same text differently and generate different amounts of output or reasoning; test representative tasks rather than comparing visible response length (OpenAI's token guide).
• Trusting a per-token calculator for self-hosted capacity assumes full utilization and misprices low-traffic deployment badly, which is exactly the case where a slow shared GPU looks cheapest (the concurrency-aware cost paper).

Routing across models means holding keys for several providers and keeping their latency figures comparable. Quanta Inference exposes text, image, video, audio and embedding models through one API key, OpenAI-compatible and prepaid, which removes one source of skew from your measurements: every call lands in the same place, so p50 and p95 per route are computed from one stream of traces (Quanta Inference).

The order of operations does not change: measure p50 and p95 per route, set the ceiling, price the wait, then pick the cheapest model that clears it.