How to Route Between a Cheap and an Expensive Model

A router can look at many things. Ship with one. The academic survey of routing strategies frames the problem as maximizing answer quality subject to a cost or budget constraint, and notes that the choice of signal is the whole design decision (routing strategies survey). Four signals cover most production cases.

Task label is the best first router for most teams, because the label is free: your application already knows whether it is summarizing a support thread or reasoning about a billing dispute. Input length is the cheapest to compute but the least accurate proxy for difficulty. Confidence scores and cascades (try cheap first, escalate on failure) capture more nuance at the cost of extra calls; the ByteByteGo analysis of production routers describes cascading as trying the cheaper model first and only paying the large model when the small one falls short (ByteByteGo on model routing).

One design rule from the same survey that saves arguments later: keep the router a pure function of the request. Do not let it call a language model to decide which language model to use unless you have measured that the decision call is cheaper than the savings it produces. A frontier model asked to grade difficulty is both slow and billable, a pattern practitioners have flagged as an easy way to reinflate costs (Pondhouse on LLM routing).

The implementation is deliberately small. You need a classify step, a threshold, and a response object that tells your logging layer which model ran and what it cost.

from dataclasses import dataclass

CHEAP = "small-model"
STRONG = "large-model"
MAX_CHEAP_INPUT_TOKENS = 1500
CHEAP_TASKS = {"summarize", "classify", "extract_fields", "rewrite"}

@dataclass
class Route:
    model: str
    reason: str

def choose_route(task: str, input_tokens: int, confidence: float | None) -> Route:
    if task not in CHEAP_TASKS:
        return Route(STRONG, f"task:{task}")
    if input_tokens > MAX_CHEAP_INPUT_TOKENS:
        return Route(STRONG, f"long_input:{input_tokens}")
    if confidence is not None and confidence < 0.6:
        return Route(STRONG, f"low_confidence:{confidence:.2f}")
    return Route(CHEAP, "default")

Three details matter more than the code itself.

First, make the expensive model the default for anything unknown. A new task label that appears in production after a product launch should not silently inherit cheap-model behavior until someone validates it. Fail toward quality, then tighten.

Second, log the routing reason string, not just the model name. When a cost anomaly appears, "long_input:4820" tells you which threshold to revisit, while "large-model" tells you nothing.

Third, get the token count from a real counter rather than a word-split approximation. OpenAI publishes rough English estimates (about four characters per token, three-quarters of a word) but recommends measuring actual usage after the call, because message structure, tools and files all add tokens that a plain-text count misses (OpenAI token documentation). If you want the small model to return a structured routing decision or task label, constrain it with a schema instead of parsing prose; the guide to structured outputs and function calling covers the schema-first approach across providers (Agenta structured outputs guide).

If you are calling several providers from one codebase, a single OpenAI-compatible endpoint for text, image and embedding models removes the per-provider client code that makes routers brittle to maintain. Quanta Inference is one option that exposes one API key, prepaid in US dollars, so the router can switch model strings without touching the billing plumbing.

The only honest comparison is routed traffic versus the same traffic sent entirely to the large model. Run both paths over a representative sample, log input tokens, output tokens, cached tokens and latency, and multiply by your contracted rates.

Here is a worked example using list prices from the published tables, clearly sourced. Suppose you process one million requests a month. After sampling, you find 80 percent are summarization and extraction tasks that the router sends to the small model, and 20 percent need the large one. Each request averages 1,200 input tokens and 300 output tokens. The small model is listed at $0.10 input and $0.50 output per million tokens; the large model at $10.00 and $50.00, figures drawn from the October pricing survey (IntuitionLabs pricing survey).

The routed bill is $5,616 against a $27,000 baseline, a 79 percent reduction. Notice where the money actually moves: the 800,000 cheap requests cost $216, which is under 4 percent of the baseline. The expensive path still dominates the bill at $5,400. That is the insight most routing write-ups skip. Once 80 percent of traffic is cheap, further routing gains are small, and the next dollar is more likely to come from prompt caching on the large path or from shrinking the 20 percent that still needs the frontier model.

Latency tells a second story. Track p50 and p95 separately per path, not a blended average. A blended average hides the tail, and the tail is where users complain. The ByteByteGo analysis notes that routing quality of service depends on the request mix and the price gap, so savings are not guaranteed even when the rule is implemented correctly (ByteByteGo on model routing).

Inworld's analysis puts a scale threshold on the whole exercise: below roughly 100,000 daily active users, a single mid-tier model may be the saner spend, because the engineering complexity of routing does not pay for itself (Inworld on AI model routing). Treat that as a starting hypothesis, then check it against your own request volume and price gap.

Cost savings that quietly degrade answers are not savings. Two cheap controls keep the router honest.

Sample and review. Pull a random fixed-size sample from the cheap path each week, ideally stratified by task label, and have a person or a rubric-based grader score them. Compare against the same sample type run on the large model. If the gap is wider than your tolerance for that task, move the task label to the expensive list.

Track structural failures, not just quality scores. For extraction and summarization, malformed output, missing fields and refusals are countable events. A schema-constrained call makes these visible; the structured outputs guide describes how schema enforcement removes most of the fragile regex cleanup that breaks when a model changes (Agenta structured outputs guide).

Watch confidence calibration. If you route on a small model's self-reported confidence, that number drifts as prompts change. Log the score alongside the outcome so you can recompute the threshold quarterly instead of guessing.

Model routing has a short, well-documented list of ways to go wrong, and almost all of them are operational rather than algorithmic.

Silent threshold rot. Your 1,500-token cutoff was tuned on last quarter's prompts. A template change pushes average inputs past it and the cheap path empties out. Alert on the share of traffic per path, not just on total spend.

Router cost creep. Adding a classification call to every request, or an LLM-based difficulty grader, adds a fixed cost to all traffic before the model runs. The Pondhouse analysis makes the point directly: using a strong model to decide complexity is slow and adds unnecessary cost, while a small local classifier avoids it (Pondhouse on LLM routing).

Token accounting mismatches. Comparing providers by per-token price without accounting for different tokenizers and reasoning-token billing produces forecasts that miss the bill by a wide margin (Amnic cost comparison). Measure the same workload on both models before you commit to a savings number.

Hidden capability gaps in non-English traffic. The character-per-token and word-per-token rules of thumb are English-specific; other languages tokenize differently, so length-based thresholds can misroute entire customer segments (OpenAI token documentation).

For most teams, the sequence below is the shortest path from a single-model bill to a measured routed bill.

1. Export two weeks of requests with task labels if you have them, and token counts if you do not.
2. Compute what the same traffic would have cost on a model roughly two tiers smaller, using published or contracted rates per million input and output tokens (CostGoat comparison).
3. If the projected saving is under the engineering cost of building and maintaining the router, stop. Inworld's threshold is a reasonable sanity check (Inworld on AI model routing).
4. Otherwise, pick the signal with the cleanest lift in your sample: task label first, length second, confidence third.
5. Ship behind a flag, log tokens, latency and routing reason per request, and compare routed output quality against the large-model baseline on a fixed sample.
6. Review the threshold monthly. The router is a living decision, not a one-time patch.

When the analysis says the next real gain is a cleaner multi-model setup rather than more routing logic, that is a conversation about infrastructure. Quanta specialists can take a description of your workload and build the routing layer, evaluation harness and cost dashboard with you, which is what Quanta exists for.