How to Route One Decision Rule Across Cheap and Expensive Models

With a score in hand, the decision rule needs two numbers, not one. Let the score run from 0 to 100 and let the business cut sit at 50, meaning score at or above 50 means approve.

• Low threshold, for example 35: any score below this is a confident reject. Answer locally.
• High threshold, for example 65: any score at or above this is a confident approve. Answer locally.
• The band between them, 35 to 65: escalate to the expensive model.

The band is the whole design. Widen it and you buy accuracy with tokens. Narrow it and you save tokens while pushing near-boundary cases onto a model that is least reliable precisely there.

Two rules keep the band honest. First, the cut at 50 stays where the business wants it, and the thresholds move symmetrically around it, so widening the band does not shift the decision boundary. Second, the band is expressed in score units but tuned in traffic units: you choose the width that escalates a fraction you can afford, then check the errors that fraction produces.

The escalation logic itself is small:

def route(score, low=35, high=65):
    if score < low:
        return "reject", "cheap"
    if score >= high:
        return "approve", "cheap"
    return "escalate", "expensive"

Note what the code does not do. It does not re-ask the cheap model for a second opinion, and it does not let the model decide whether to escalate. The escalation condition is a fixed numeric test, which is the property that makes the whole design auditable. The legal-operations literature draws the same line: an escalation workflow is a governance-controlled conditional branch, and it works only when it enforces delegated authority rather than merely forwarding a task governed escalation workflows. A model that chooses its own escalation is a router without a rule.

Governance frameworks add a useful discipline here: automated triage should catch routine cases so that only unresolved ones reach a human decision-maker Atlan on escalation tiers. Substitute "expensive model" for "human" and the structure is the same three tiers you already see in escalation design: cheap automated tier, capable tier, and human review for the residue.

You cannot set the band from intuition. You set it from a labeled sample, and the label has to come from the business, not from the expensive model. Using the expensive model's answer as ground truth measures agreement with the expensive model, which is a different and much weaker question.

Pull 300 to 500 recent requests, stratified so that maybe half of them fall near the boundary. Have a person label each one with the correct decision. Then run the cheap model on the whole sample, record the score, and build a small table: for every score bucket, how many were truly approve, how many truly reject.

Now the error terms become concrete.

• A false negative is a request the rule rejects that should have been approved. In a refund rule, that is a customer wrongly turned down.
• A false positive is a request the rule approves that should have been rejected. That is money paid out incorrectly.

Both are computed per threshold setting. For a reject threshold at 35, a false negative occurs when the cheap model scores a genuinely approvable request below 35. Count those, divide by the number of genuinely approvable requests, and you have the false negative rate at that setting. Do the same above 65 for false positives. Sweep the two thresholds, or just the width if you keep them symmetric, and plot both rates against the escalated fraction.

Two practical warnings. First, the escalated fraction is measured, not assumed. Novices estimate it from their sense of how hard the traffic is; the sample will correct them. Second, recalibrate when the traffic mix changes, because a band tuned on one distribution of requests will escalate a different share of another.

This is also where most cost-saving claims for cascades quietly fail. The reported cost reductions from routing work, in the range of 45 to 85 percent depending on workload, come from workloads where routing actually separated easy from hard cases well LLM routing and cascades. If your sample shows the cheap model's score barely separates correct from incorrect, no band width will save money without buying errors.

The saving from the design is the escalated fraction times the price gap between the two models. Everything else is overhead. Write it as one line per request:

cost = cheap_price + escalated * (expensive_price + human_review_price * review_rate)

Three terms need care. The cheap price applies to every request, because every request is scored. The expensive price applies only to the escalated slice, and that slice pays both prices, since the cheap call was already made. The human review term applies only if you route a further slice of escalated cases to a person, which you should if the expensive model is still unsure.

A worked example, with prices invented for illustration:

• Cheap model call: 400 input tokens and 20 output tokens, priced at 0.10 and 0.40 per million tokens. That is about 0.000048 dollars per request.
• Expensive model call: the same tokens at 3.00 and 15.00 per million. That is about 0.0015 dollars per request.
• The gap is roughly 0.00145 dollars per escalated request.
• Human review, at 1.20 dollars per reviewed case.

At a million requests a month with no escalation, the expensive model costs about 1,500 dollars and the cheap model about 48 dollars. Now insert the band. If the band escalates 18 percent of traffic and 10 percent of escalated cases go to a human, the blend is:

• Cheap calls on all traffic: about 48 dollars.
• Expensive calls on 180,000 requests: about 270 dollars.
• Human review on 18,000 cases: about 21,600 dollars.
• Total: about 21,918 dollars.

The human review line dominates everything. That is the most common surprise in this design: escalation to a bigger model is cheap relative to escalation to a person, so the review step, not the model choice, decides whether the economics work. If your rule requires human sign-off on escalated cases, the saving versus running the expensive model on everything is the model cost avoided, roughly 1,180 dollars, minus the review bill and minus the cost of the errors.

You also need a price for the errors. Assign each false negative and each false positive a money value from the business side (refund paid in error, customer lost, complaint handled). Multiply the measured error counts at your chosen thresholds by those values and add them to the cost model. The design pays when the escalated fraction times the price gap exceeds the error cost you are adding relative to the single-model baseline.

A comparison helps fix the intuition:

Routing is the one-shot cousin of the cascade: a classifier picks a model per request instead of running stages in sequence ByteByteGo on model routing. It avoids paying twice for escalated requests, but it needs a classifier whose accuracy you trust before the first call. The cascade's advantage is that its decision is made on the actual request content after the cheap model has read it.

One further decision rule, from the cascade research: a two-model cascade captures most of the attainable gain, and adding a third model rarely improved mean accuracy in the comparisons studied the cascade depth study. Build the two-model version first. Add depth only if your measured sample shows the second model's scores also cluster near the boundary.

The design is only as good as its monitoring. Log four things for every request: the score, which route it took, the final decision, and, where you have it, the true outcome. Those four fields let you recompute the escalation rate and error rates on live traffic weekly, without new labeling.

Keep the deterministic layer deterministic. The score comes from a model, but the thresholds belong in configuration and version control, and any change to them should be reviewed like a code change. The rule engine documentation's distinction is useful here: explicit conditions and outcomes are predictable and auditable, while model-backed rules are probabilistic the rule engine documentation. Put as much of the decision as possible in the auditable half.

Watch for three failures, each of which appears in escalation design generally governed escalation workflows:

1. A silent threshold that sits in a document but triggers nothing in the code that routes requests.
2. An escalation target that has moved or been retired, so escalated cases stall.
3. Thresholds set so tight that almost everything escalates, which trains everyone to ignore the queue and returns you to single-model cost with extra latency.

The last one is where the cascade quietly turns back into the thing it replaced. If your measured escalated fraction creeps toward the whole traffic share, the band is no longer doing work; either the score has lost its separation or the traffic has changed shape.

When you outgrow a fixed band, the next step is a query-aware escalation rule that compares the cheap model's estimated error against the cost of running the expensive model on that particular request, rather than a fixed score window the cascade stopping analysis. That is a real improvement, but it is a second project. Ship the band, measure it, and only then decide whether the extra complexity earns its keep.

If you would rather spend your time on the decision rule than on wiring five providers into one billing model, Quanta Inference puts text, image and embedding models behind a single OpenAI-compatible API key with prepaid dollar pricing, which keeps the price-gap arithmetic in your cost model simple (https://quantaintelligence.ai/inference).