What a Routing Table Costs When You Get It Wrong

Take a realistic support ticket. A customer writes about a billing discrepancy across two invoices and wants to know whether they were double-charged. The request needs document lookup, arithmetic, and a careful explanation. It arrives with roughly 3,150 input tokens (system prompt, retrieved help-center passages, ticket history) and produces about 400 output tokens, a shape that pricing guides commonly use to model support workloads (Silicon Data).

The router sends it to the small tier. Assume the small tier is priced near the floor of mainstream APIs and the frontier tier costs several times more on both input and output; the exact rate matters less than the ratio, so I will use a ratio of one to ten on input and one to twelve on output, which is well inside the spread that a single provider's lineup can show between its cheapest and most expensive models (CloudZero).

Here is the ledger for the misroute, with numbers rounded:

Now compare the clean path. The same ticket routed directly to the frontier tier costs one call at roughly $0.07 by the same arithmetic. The difference between the two paths, about $28.95, is the misroute cost. Of that total, $0.02 is tokens and $29 is human time. The token delta is not zero, but it is irrelevant at this scale.

Two more numbers make the shape unmistakable. First, the human share of the misroute cost is over 99 percent. Second, the saving the router was chasing on the first call was a fraction of a cent. No plausible routing policy justifies risking $29 to save a penny unless misroutes are extremely rare, and the paragraphs below show how to price that rarity.

The delay line deserves separate attention because it never appears in a cost report yet is often the largest business cost. Three hours and forty minutes of customer wait on a billing complaint is a churn risk, a possible chargeback, and a trust cost that no token ledger captures. Treat delay as a cost category even when you cannot convert it to dollars, because it changes which misroutes are acceptable: a slow but correct answer is fine for a styling question and unacceptable for a payment dispute.

Every serious accounting of a misroute needs four buckets. Teams that track only the first miss budget badly.

Original call cost. Input tokens sent, output tokens returned, at each tier's actual rate. Cheap and easy to log.

Corrective call cost. Every call made after the failure: retries, escalations, a second pass with more context, tool calls fired in the repair loop. Each is billed separately, and the corrective call is often the largest single token line because it carries the whole thread.

Human repair minutes. The support agent who reads the thread, the engineer who checks the arithmetic, the reviewer who signs off the rewrite. Multiply minutes by loaded hourly cost, not salary, because benefits and overhead are real.

User-absorbed delay. Time from wrong answer to right answer, plus the number of extra messages the customer had to send. This is the bucket most teams skip and the one customers remember.

A useful rule of thumb from the worked example: if human repair minutes exceed fifteen, the misroute cost dominates any plausible per-call price difference between tiers. Measure once on your own traffic and the rule tends to hold, because frontier tiers rarely cost more than pennies per support-sized ticket while an agent minute costs dollars.

The instinct after a misroute is to improve the router. That instinct assumes routing errors come from a classifier that is not smart enough, so a smarter classifier will make fewer errors. The evidence points elsewhere. In a systematic study of routing, learned routers improved relative performance over the strongest fixed-model baseline by about 14.6 percent, and lightweight router designs ranked ahead of heavier ones under tighter cost constraints (LLMRouter, arXiv). That is a real gain and a modest one, and it arrives with a training cost, an evaluation pipeline, and a maintenance burden.

The deeper problem is that routing accuracy is not the same as task success. A router can be right about the category of a request and still be wrong about its difficulty, because difficulty depends on details inside the retrieved documents and the customer's history. That is why cascades exist: send to the small model first, check the answer, escalate when the check fails (TensorOps). The check is the expensive part, and a weak check passes wrong answers through. You have moved the cost from the model to the verification, not eliminated it.

There is a sanitization argument for review too. The same trace-level logging that attributes cost to a workflow step is what lets you find the request types that fail and the retry loops that inflate spend (Braintrust). If you cannot see the misroute in a trace, you cannot price it, and you certainly cannot decide whether to tune.

The decision rule that follows: spend on review before you spend on a classifier, because review catches the errors you did not model, while a classifier only reduces the errors you already knew about. Add review gates to the categories where a misroute costs more than a few dollars: billing, account access, refunds, anything with a deadline. Let the cheap path run unguarded where a wrong answer costs a retry and nothing else.

Tuning feels productive because every round produces a number that looks better. A stopping rule makes it a decision instead of a habit. Use this one.

Let M be the measured cost of one misroute, computed across all four buckets. Let f be the fraction of traffic that misroutes under the current policy, measured from traces, not estimated. Let R be the cost of one more tuning round: engineer-days plus evaluation compute plus the reviewer time to check the new policy on a held-out sample. Let g be the realistic reduction in the misroute fraction that another round would achieve.

Then run one more round only if:

M * f * g * N  >  R

where N is the number of requests the policy will handle before it changes again. If the left side is smaller than the right, stop tuning and spend the money on a review step instead.

Run the numbers from the worked example over a month of 20,000 support tickets. Suppose the current misroute fraction is 2 percent and the average misroute cost is about $29, a mix of cheap recoveries and one expensive billing case. The monthly misroute cost is 0.02 × 20,000 × 29, which is about $11,600. Suppose a tuning round costs $9,000 in engineer time and would cut the misroute fraction by a quarter. The expected saving is 0.25 × 11,600, which is about $2,900. The round does not pay for itself. A review step that costs $2,000 a month and catches most billing misroutes does, because it targets the expensive tail rather than the average.

Swap the assumptions and the rule flips. If the misroute fraction were 10 percent or the average misroute cost $200, the same round pays back within months. That is the whole point of a stopping rule: it tells you which of your two problems you actually have.

Three conditions should override the rule. Raise the priority of review when misroutes cluster in one category, since a single guard is cheap. Raise it when the delay bucket is large, because customer-facing delay has a business cost the formula cannot see. Lower it when the misroute fraction is measured across thousands of traces and is genuinely small. And treat the estimate of g with suspicion: nobody ever expects their next tuning round to fail.

If you are evaluating several models at once and want the routing policy to be a configuration rather than a codebase, Quanta Inference exposes text, image, video, audio and embedding models through one OpenAI-compatible API with prepaid dollar pricing, which makes the small-versus-frontier comparison a matter of two keys rather than two integrations.

Run this before shipping any routing change. It takes an hour and usually saves a week.

Two habits make the checklist stick. First, keep the misroute ledger for the first month after any routing change, because the first month is when the expensive surprises happen. Second, revisit the ledger when traffic shape changes, since a workload that was mostly prose can become mostly structured data, and structured payloads tokenize differently than prose (PE Collective). Routing policies that were safe on the old token mix can flip.

One more measurement helps: compare cost per successful task rather than cost per call. A policy that doubles the number of calls but halves the human repair rate is usually a win. A policy that halves token spend and triples repair minutes is a loss dressed as optimization. Attribution needs request-level detail, including tool calls and retrieval steps, to see which of these you have (Braintrust).

Quanta Experts is the natural place to put a domain owner on this loop: each expert reads your context, does the work and hands back finished deliverables, which is a practical way to run the category-specific review the checklist calls for.

Measure one real misroute, not a synthetic one. Pick a failure from last month that a support agent actually fixed, and fill in the four buckets from logs, timesheets and the ticket thread. The exercise takes two hours and changes the conversation from opinion to arithmetic.

Then compute the stopping rule for your traffic. Write M, f, g and R on one page and decide. If the rule says stop tuning, put the money where the misroutes actually are: a guard, a review step, or a smarter escalation check that catches the billing cases before the customer does. If the rule says tune, tune with a target and a cost ceiling, and keep the ledger open so you can tell whether the next round paid for itself.

The general lesson applies beyond support: any system that picks which engine handles which task is making an indirect cost decision, and the cost of getting it wrong lives in the parts you are not measuring. Fix the measurement first.