Routing Only the Hard Cases to the Expensive Model

A cascade starts cheap and only pays for the expensive model when the cheap model cannot handle a request, a pattern the survey of dynamic routing and cascading describes as sequential escalation based on a quality estimate. Before you can escalate, you need a number that says "this answer is shaky."

Token probability is the rawest signal: high-entropy distributions spread probability across many candidate tokens, and that spread correlates with uncertainty. The trouble, as a practical write-up on cascades notes, is that LLM self-reported confidence is poorly calibrated. A model can sound authoritative while being wrong, and a correct answer to an ambiguous question can look uncertain. Practical systems therefore combine several signals:

• Sequence likelihood of the cheap model's answer, normalized by length so long answers do not automatically score low.
• Self-verification: a second short cheap call that checks the answer against the retrieved context and returns pass or fail.
• Retrieval quality where the request involves documents. If the retrieved passages are weak, escalate regardless of model confidence.
• Constraint checks for structured output: does the JSON parse, do the required fields exist, does the value fall in range.

Whichever signals you use, collapse them into one score per request and log it. That log is the asset. It lets you choose a threshold from data rather than intuition, and it lets you compare routers on identical traffic later.

One caution on the scoring call itself: it costs tokens. OpenAI's documentation notes that reasoning tokens are billed as output tokens and can dwarf the visible answer, and that a lower price per million tokens does not guarantee a lower total cost because tokenization differs between models (OpenAI token guide). Keep the verifier short and give it a tight output budget.

The escalation rule is a single comparison: if the confidence score falls below a threshold, send the request to the expensive model; otherwise serve the cheap answer. Write it so the decision is logged with the score, the threshold, the model version, and the outcome.

def route(request, threshold=0.62):
    cheap_answer, confidence = cheap_model.answer(request, return_score=True)
    escalate = confidence < threshold
    log(score=confidence, threshold=threshold, escalated=escalate,
        model="cheap-v" , feature=request.feature, agent_run_id=request.run_id)
    if not escalate:
        return cheap_answer
    return expensive_model.answer(request)

Two details in that snippet carry most of the engineering value. First, the threshold is a parameter, not a constant, so you can sweep it offline against logged scores. Second, the log carries feature and run identifiers, which is exactly the request-time metadata that makes per-feature and per-agent-run cost rollups queryable later; Braintrust's cost tracking playbook lists user, feature, deployment, prompt version, agent run and customer as the minimum useful tag set.

There is a subtle trap here. If your confidence check is itself an LLM call, you are paying twice for easy requests: once for the answer and once for the check. A cheaper design is to have the answering call return a structured envelope with both the answer and its self-assessed certainty, so the check costs nothing extra. Where you need an independent verifier, run it only after a cheap heuristic (output too short, schema invalid, retrieval score low) flags the request as suspicious.

The standard lazy baseline is a fixed split: send, say, 20% of traffic to the expensive model by sampling or by hashing request IDs. It is trivially easy to implement and has one endearing property: the escalation rate never drifts. It has one fatal property: the split is blind to difficulty.

Consider a support workload of 10,000 requests. Suppose the cheap model passes 70% of them at a cost of $0.002 per request, and the expensive model passes 95% at $0.02, roughly a 10x price ratio consistent with the frontier-versus-small gap described in ByteByteGo's routing overview. The pass rate target is 90%.

The learned router wins because it escalates the requests the cheap model would have failed and leaves the easy ones alone. In this example it sends 1,400 requests up instead of 2,000 and still clears the 90% pass target, which lands the cost per passing request about 20% below the fixed split and roughly four times below sending everything to the expensive model. The exact numbers depend on your workload; the direction of the result depends on whether your confidence score actually separates passing from failing requests. If the score is noise, the learned router collapses back to the fixed split.

The fixed split has a second weakness that the table hides: it is fragile under traffic shift. On a day when simple lookups dominate, the 20% sample is mostly wasted on requests the cheap model would have passed. On a day when a new hard feature ships, the same 20% starves the hard traffic and the pass rate dips below target. A threshold rule moves with the traffic distribution automatically, provided you recalibrate.

A router can only be judged against labeled outcomes. Build a small evaluation set from production traffic, sample, and grade it with a rubric or a stronger model, then report every routing configuration against the same set. Honeycomb's guide to tracking token cost lays out the minimum fields: capture provider-reported token usage per response, attach business metadata (feature, environment, team, user), compute cost at log time, store it as request-level telemetry, and alert when usage drifts from baseline.

Three rollups carry most of the decision weight:

1. Escalation rate by feature. A single feature creeping from 14% to 30% escalation is usually a prompt regression, not a harder world.
2. Cost per passing request, overall and by feature. This is the number you optimize; everything else is diagnostic.
3. Pass rate of cheap-tier answers that were not escalated. If it sags, the threshold is too loose.

Run the sweep offline. For each candidate threshold, replay the logged confidence scores against their recorded outcomes and compute pass rate and total cost. Pick the loosest threshold that still clears your pass target, then check it against a held-out slice of traffic you did not use to fit the score. This is the recalibration step that separates a router that works in a notebook from one that survives a month of production.

A worked warning: if you tune the threshold on the same 2,000 graded requests you used to choose the scoring prompt, you will overfit. The router will look excellent on paper and escalate too little on new traffic, and the first symptom will be a slow drift in the cheap-tier pass rate that nobody is watching.

The learned-router advantage disappears under several conditions, and knowing them saves you a quarter of engineering time.

Traffic that is uniformly hard or uniformly easy. If 95% of requests need the expensive model, the threshold rule escalates almost everything and you have bought complexity for nothing. If 95% pass cheaply, the fixed split wastes little because the volume of expensive calls is small either way.

Weak separation in the score. Plot the distribution of confidence scores for passing versus failing cheap answers. If the two overlap heavily, no threshold does better than chance, and you should invest in better signals (retrieval quality, schema validation, a verifier) before investing in a router.

Very tight latency budgets. A cascade adds a verification step before the expensive call, and cascading requires good confidence calibration while accepting the latency of potentially running models sequentially (the cascades write-up makes the same trade-off explicit). If your p95 budget cannot absorb a second call on a minority of traffic, a single upfront classifier may fit better than a cascade.

Mixed objectives within one workload. A router optimized for correctness can be the wrong router for security-sensitive queries, since utility and security favor different models and token budgets; the SCRouter paper shows model rankings diverging sharply between the two objectives. If your traffic mixes ordinary requests with adversarial or policy-sensitive ones, route those classes separately rather than tuning one threshold for both.

A useful decision rule: build the learned router when the confidence score separates passing from failing answers with clear margin, when expensive calls are a large share of spend, and when you can afford to grade a few thousand production requests. Otherwise ship the fixed split, instrument it well, and revisit when the data says the score has improved.

The build itself is small if you sequence it correctly.

1. Instrument first. Log model, token counts, feature, run ID, and outcome for every call before you change any routing. You cannot calibrate what you did not record, and request-level telemetry is the prerequisite for every later step.
2. Make the cheap tier emit a score. Structured output with the answer plus a self-assessed certainty, held to a short output budget.
3. Grade a sample. A few thousand requests with pass/fail labels, stratified by feature so rare hard cases are represented.
4. Sweep the threshold offline against the graded sample, then validate on held-out traffic.
5. Ship at the conservative end. Start with a lower threshold than your sweep suggests, watch the cheap-tier pass rate, then loosen.
6. Alert on drift. Escalation rate by feature, cost per passing request, and cheap-tier pass rate, each with a baseline and a threshold.
7. Re-grade monthly. Prompts, models, and traffic all move; the threshold should move with them.

Two cost traps to close while you are in there. Retry storms from transient provider errors bill tokens that never show up in application logs, and context bloat from a longer system prompt raises input tokens across every session, both catalogued in Braintrust's failure-mode list. A router that escalates a request after three failed retries has not saved anything.

If you run more than a handful of models, the routing layer is also the place where vendor sprawl gets expensive in engineering time: separate keys, separate SDKs, separate pricing pages. Quanta Inference exposes text, image, video, audio and embedding models through one OpenAI-compatible API key with prepaid US-dollar pricing, which turns the router's model registry into a single configuration file rather than a set of integrations: Quanta Inference. Pair it with the instrumentation above and the cost-per-passing-request number becomes a line on a dashboard instead of an afternoon of spreadsheet work.