Signals a Router Can Read Before Any Model Call

Token count is the signal most monotonic with cost. Longer inputs cost more, longer outputs cost more, and long contexts are the ones most likely to be truncated. You cannot know the exact token count before tokenizing, but you can get close enough for a routing decision with a character heuristic plus a constant for the response.

def estimate_tokens(messages, chars_per_token=4.0, overhead_per_msg=4, reply_reserve=512):
    chars = sum(len(m.get("content", "")) for m in messages)
    msgs = len(messages) * overhead_per_msg
    return int(chars / chars_per_token) + msgs + reply_reserve

Four characters per token is a rough rule of thumb for English text and should be calibrated against your own traffic: sample a few thousand real requests, tokenize them with the target model's tokenizer, and fit the ratio. Reserve tokens for the reply too. The costliest routing mistake with this signal is ignoring the output side, because a request that looks small on input can generate a large structured payload.

Once you have a number, the decision is a table, not a model:

The upper boundary is not only about cost. Structured output failures cluster around truncation: when max_tokens runs out mid-generation, the model stops producing tokens with a finish reason of length, and most providers guarantee schema conformance only for complete generations the structured output failure taxonomy. A router that sees a large estimate and an output schema should raise the reply reserve rather than let the call fail downstream.

There is a second use for this signal that engineers underrate: protect the input side. If the estimate exceeds the chosen model's context window minus the reserve, route to a path that summarizes or chunks first. That is a code decision, and it should fire before you pay for a call that will be rejected or silently clipped.

Language detection over the first few hundred characters is a small, deterministic computation with a large effect on quality. A model's strengths are uneven across languages, and support queues are uneven too. Two decisions follow:

1. If the detected language is outside the set you have evaluated, route to the model you have actually tested for that language, or to a human queue with a drafted reply. Do not silently send it to your default model and hope.
2. If the language differs from the account's locale, flag the conversation for review, since mixed-language tickets are where translation errors compound.

Run detection on the first 300 to 500 characters rather than the whole message. Early text carries the dominant language for most users, and truncating the input keeps the detector cheap and stable when a ticket contains quoted logs in another language. Treat low-confidence detections as unknown, not as a guess: an unknown language should take the conservative route, which is usually the larger model with a translation step, not the fast lane.

Keep the language decision separate from the quality decision. A common pattern is to pick the model for the language first and then apply the token-count table, so a long French request lands on a large model the same way a long English one does. Two independent lookups are easier to reason about than one combined rule, and much easier to log.

Intent keywords are the least glamorous signal and often the most decisive, because some intents are known to be low-stakes or structurally constrained. A refund status check, a password reset, an order tracking question: these do not need a frontier model, and you probably have a template for the answer. The orchestration patterns literature describes exactly this split, where an orchestrator decides whether the next step is a deterministic function, a tool call, a specialist agent or a human approval the orchestration guide.

Keyword rules also fit naturally into gateway configuration. LiteLLM's auto routing exposes keyword tier rules that run before any classifier, escalating to the highest matched tier, and it logs a cause marker per decision so you can see which rule fired the LiteLLM auto routing docs. That cause marker is the part to copy even if you build your own router: every decision should record why.

Write the rules in a single file with a priority order and an explicit escalation direction.

keyword_rules:
  - match: ["refund status", "where is my order", "track my package"]
    intent: order_status
    route: template_then_small
    priority: 10
  - match: ["cancel my account", "gdpr", "delete my data"]
    intent: account_deletion
    route: human_review
    priority: 100
  - match: ["regex:^\\s*SELECT\\s", "regex:\\bstack trace\\b"]
    intent: technical_payload
    route: large_code_model
    priority: 50

Two rules govern the design. First, escalation beats specialization: if two rules match, take the one with the higher priority, and make destructive or regulated intents the highest priority of all. Second, patterns beat single words. The word "cancel" appears in "how do I cancel a subscription" and in "don't cancel anything", and a regex with context separates them far better than a substring match. The same logic that makes a deterministic intent check cheap also makes it auditable, which is why it belongs in code rather than in a prompt.

The limit of this signal is the long tail. Keyword rules have poor recall on rephrased requests, and Paragon's evaluation work found that prompt wording and description depth moved tool-calling metrics only on multi-tool tasks, with the effect close to flat on single-tool tasks the Paragon tool-calling guide. Read that as a warning: do not spend weeks tuning keyword lists for a lane where the choice barely matters. Spend that effort on the lanes where the signal separates real cost or risk.

The account tier is already in your session context, which makes it the cheapest signal of all. It is also the one with the clearest business logic, because the decision is usually about service level rather than model capability. A free-tier account can be routed to a smaller model with a lower reply reserve; a paid account with a support entitlement can be routed to the larger model and, when the intent rules say so, to a human queue.

There is a good architectural reason to keep tier routing in the gateway rather than scattered through application code. Gateway patterns in the tool-calling literature emphasize centralized controls: unified interfaces across providers, automatic fallbacks, rate limits and budget governance configured in one place the Maxim overview of tool calling. Tier is the signal that most naturally becomes a gateway policy, because it changes on billing events, not on request content.

The engineering discipline here is the same as canary routing on an API: evaluate identity before the routing policy so the user context is populated, and fall back to the default lane when the entitlement lookup fails the canary routing guide. A missing tier should never produce an error page. It should produce the safe default and a logged reason.

Most teams do not have a labeled dataset of requests tagged with the model that should serve them, and waiting for one stalls the work. You can set defensible thresholds without labels if you accept that the thresholds are hypotheses to be validated in production.

The method is to calibrate against observable behavior rather than a ground-truth label:

• Log every request's signal values alongside the model actually used, the finish reason, the output length, the latency and any downstream error. This log is your dataset by proxy, and it costs nothing to collect.
• Set your first thresholds from the distribution of those signals, not from intuition. If 80% of requests fall under a token estimate of 1,200, a cutoff at 1,200 sends the bulk of traffic to the cheap lane, which is exactly the 80/20 split that delivers roughly a 4x cost reduction in the worked example from the routing chapter the model routing chapter.
• Pick a boundary you can defend on cost alone even if quality is unchanged. Token count gives you this: at a known per-token price difference, moving a lane changes the bill by a computable amount. If the threshold is wrong, the symptom is a bill, not a quality regression.
• For signals where quality is the risk (language, intent), set the conservative side of the boundary first and widen it only after you have evidence. An unknown language routes to the tested model until you have evaluated the cheap one.

Here is a worked example. Suppose 100,000 requests a day, 80,000 of them short lookups under 300 estimated tokens and 20,000 of them longer. The short lane runs on a model a tenth the price of the long lane, and both lanes average 400 output tokens at a made-up but consistent unit price of one unit per output token for the large model and one tenth of that for the small one. Sending everything to the large model costs about 40,000,000 units a day on output alone. Routing the 80,000 lookups to the small lane costs about 3,200,000 units for those requests plus 8,000,000 for the long lane, or 11,200,000 units total. The threshold at 300 tokens alone is responsible for roughly a 72% reduction in output cost. That arithmetic is the reason token estimate earns its place as the first signal: the threshold's effect is computable before any quality label exists.

The second discipline is reversibility. Every pre-call rule should be defined in configuration, not buried in a deploy, so you can tighten or loosen it during an incident. LiteLLM's auto router keeps tier decisions in YAML with an LLM classifier as an optional fallback, and its docs describe a semantic auto router being deprecated in favor of the newer configuration-driven approach the LiteLLM auto routing docs. Configuration-first routing means the threshold is a knob, not a release.

Cheapness is measurable. Aim for a pre-call decision budget in the low microseconds: string length, a regex, a lookup table, a language detector over a few hundred characters. Compare that with a classifier call, which the same LiteLLM comparison prices at roughly 100 to 500 milliseconds for an embedding match, versus sub-millisecond for heuristic or keyword rules the LiteLLM auto routing docs. The heuristic layer is two to five orders of magnitude cheaper than the classifier layer, which is why the heuristic layer should capture as many requests as it can.

Auditability is a logging contract. For every request, emit the signal vector, the matched rule, the route chosen, and a reason code. Structured outputs matter here for the same reason they matter anywhere a program consumes model text: valid JSON is not a contract, and validation belongs after generation the TrueFoundry guide to structured outputs. Apply that to your own router's decision record: require the reason code field, validate it against an enum, and log the raw signal values next to it.

Reversibility also means having an escalation path. When a routing decision has no confidence, or when a rule is disabled during an incident, the request should still be served correctly. The orchestration guidance on determinism and fallbacks is written for agents but applies verbatim to routers: prefer known-good routes when uncertainty is high, and bound responses with latency targets that trigger fallbacks if exceeded the Maxim overview of tool calling.

A final piece of discipline: keep the downstream half of routing optional. The pre-call layer should produce a default route for every request, so that a failure in the classifier layer degrades to a reasonable lane instead of an error. That is how cascades are supposed to work, escalating only when a cheap route provably fails, and it keeps the expensive decision from becoming a single point of failure the model routing chapter.

When you want to apply these signals across several providers without rewriting the plumbing, a single API in front of the fleet removes a whole class of variation. Quanta Inference gives every model type one OpenAI-compatible endpoint and prepaid billing in US dollars, which makes the token-estimate threshold the same arithmetic regardless of which vendor answers: https://quantaintelligence.ai/inference.