Routing Logic for LLM Tasks: Rules, Thresholds and Fallbacks

Keep the feature set small. Every feature you route on must be computable before the call, cheaply, and stable enough that the same request routes the same way twice. A workable starting set:

1. Estimated input tokens. Count prompt plus context before sending. Tokenization is not word count: English prose averages roughly 0.75 words per token, and JSON adds overhead from brackets and keys (the Worklytics tracking guide). Estimating from bytes with a learned bytes-to-token ratio avoids running a tokenizer on every dispatch (the dual-pool routing paper).
2. Tool surface. The number of tool definitions attached to the request. When an agent is connected to many servers, the tool catalog can represent the majority of input tokens on every request (the Maxim cost tools comparison).
3. Task class. Classification, extraction, summarization, code edit, open-ended reasoning. A request class maps to a fixed tier under static routing (the production playbook).
4. Latency budget. How long the caller can wait. Interactive chat and a nightly labeling job have very different ceilings.
5. Hard constraints. Data residency, a cost ceiling per request, or a required output schema.

Do not route on raw prompt text similarity as a first step. That is what a semantic cache is for, and it should intercept the request before the router sees it, because a cache hit means zero routing decisions and zero model calls (the MLflow routing guide).

The arithmetic behind any policy is simple: cost equals input tokens times input price plus output tokens times output price, for whichever model you called (the production playbook). That means thresholds should be expressed in money and seconds, not vibes.

Take a support assistant. Suppose your frontier tier costs 25 times more per input token than your small tier (the cited comparison shows a 16x gap between one vendor's flagship and its mini model, at $2.50 and $0.15 per million input tokens respectively (the NeuralTrust cost guide)). Your traffic splits like this:

Run 100,000 requests at these prices and read the table right to left. All-frontier costs about $1,090. Routing classification and summarization to the small tier costs about $375, a saving of roughly two thirds. The catch: the small-tier numbers assume those buckets are answered correctly at the cheap tier. If 5 percent of the summarization bucket needs escalation, add a cascade step, and account for its extra scoring call and tail latency (the production playbook).

This is why the published savings ranges differ so much. Routing reports 40 to 70 percent savings, and the figure depends entirely on what share of your traffic is genuinely routine (the Morph cost guide). If 60 to 80 percent of your requests are routine, the weighted average drops sharply; if most of your traffic is hard reasoning, routing does little. Measure your mix before promising a number.

Two threshold design rules follow:

• Set the token threshold from your own distribution, not a round number. The Azure trace analysis found 80 percent of requests fit in 2K tokens and 95 percent in 8K (the dual-pool routing paper). Your long-context bucket may be small enough that a dedicated expensive path is fine.
• Put a hard cost ceiling per request. A prompt with a long document context can cost many times a short question with the same response size (the Worklytics tracking guide). A ceiling turns that into a route change (trim context, or send to a cheaper tier with a shorter answer cap) rather than a surprise invoice.

A routing policy that lives inside application code cannot be reviewed, diffed or rolled back independently. Keep it as a declarative file in version control, keyed by a policy version, with the decision order explicit. The routing matrix published for coding agents is a good shape: classify difficulty and ambiguity, then map each cell to a model tier and reasoning effort, with domain overrides matching first (the Morph cost guide).

policy_version: 7
default_route: standard
rules:
  # Order matters: first match wins.
  - name: pinned_tenant
    when: { tenant_tier: "regulated" }
    route: frontier_pinned
  - name: huge_context
    when: { est_input_tokens: "> 120000" }
    route: long_context
  - name: tool_heavy
    when: { tool_count: "> 20" }
    route: standard
    actions: [collapse_tool_catalog]
  - name: routine
    when: { task_class_in: [classification, extraction, formatting],
            latency_budget_ms: "< 4000" }
    route: small
  - name: open_reasoning
    when: { task_class: open_reasoning }
    route: frontier
routes:
  small:
    model: <small-tier model id>
    max_output_tokens: 500
    timeout_ms: 2500
    fallback: standard
  standard:
    model: <mid-tier model id>
    timeout_ms: 8000
    fallback: frontier
  frontier:
    model: <frontier model id>
    timeout_ms: 20000
    fallback: standard

Four things make this artifact usable. First, an explicit policy version so every logged decision can be traced back to the exact rules in force. Second, first-match ordering, so conflicts resolve predictably. Third, per-route timeout and fallback, so mechanics live with the route rather than in scattered code. Fourth, a changelog in the same repository, so a threshold shift is a reviewable pull request.

Do not test routing with synthetic prompts. Build a replay set from real production traffic: request features, the model actually used, token counts and the outcome. Then score the new policy offline against that set and compare three numbers per route: share of traffic captured, projected cost, and quality on a labeled slice.

The replay is where you find the failure that matters most: requests that the new policy sends to a cheap tier and that the cheap tier gets wrong. Label a few hundred replayed requests by correctness, then bias any threshold that produces a wrong cheap answer toward the stronger tier. This is the same rule that survived the GPT-5 router rollback: a wrong cheap answer costs more than an unnecessary frontier call (the routing economics analysis).

After replay passes, canary. Send a small, stable slice of live traffic through the new policy version while the rest stays on the old one, and compare fallback rate, cache hit rate, cost per resolved task and latency percentiles between the two groups (the MLflow routing guide). Raise the canary share only when the fallback rate and error rate are flat and cost has moved in the expected direction. Roll back by flipping the version pointer, not by rebuilding.

Fallbacks are the part most policies skip. Three triggers, three different responses:

• Timeout. If a model has not produced a useful response by the route's timeout, the cheapest correct action is usually to fail over to an alternate route rather than wait. Note that for streaming responses you may already have sent partial content, so define whether a timeout mid-stream triggers a failover, an apology, or a silent retry (the error handling guide).
• Error. Retry only what is retryable. 4xx client errors are your fault and will not fix themselves; 429 rate limits and 5xx server errors are transient and should use exponential backoff with jitter (the error handling guide). After the retry budget is spent, fall over to another provider or route.
• Quality regression. This is the one teams forget. If an output fails schema validation, a confidence check or a lightweight judge, escalate to the next tier up. Cascades are not free: they add a scoring step and tail latency when escalation fires (the production playbook).

A gateway can carry most of this. Gateway-level routing rules, semantic caching, budget controls on virtual keys and automatic failover all operate below application code (the Maxim cost optimization guide). If you run multiple providers, that single control point is where failover across providers belongs.

If your fallbacks cross providers, keep the request and response shape uniform so a failover is a route change, not a rewrite. Quanta Inference exposes text, image, video, audio and embedding models through one OpenAI-compatible API key, which keeps that failover path short: https://quantaintelligence.ai/inference

Log the routing decision as a first-class event, not just the model's output, so you can audit why a given model was picked (the MLflow routing guide). The minimum viable log per request: input token count, output token count, model, and metadata tags for feature, user and environment (the NeuralTrust token monitoring guide). Add the policy version and the matched rule name.

Watch four numbers per route, not one aggregate:

That last row is easy to miss. On self-hosted fleets, a single long prefill can stall short requests queued behind it; routing short and long requests into separate pools raised GPU-hours savings of 31 to 42 percent in one study (the dual-pool routing paper). Even on hosted APIs, a route that occasionally sends a 100K-token request ahead of interactive chat will show up in p99.

Adjust the policy on evidence, not on instinct. Change a threshold when a route's quality score drops on the labeled slice, when its fallback rate drifts persistently, or when the cost ceiling starts firing often enough to hurt users. Then run the same replay and canary cycle again before it goes wide.

If the policy work itself is the bottleneck, or you want the routing, fallback and observability layer built and run for a line of business rather than maintained in-house, a Quanta expert can take it: https://quantaintelligence.ai/agents