Routing Requests by Cost and Quality: Rules You Can Audit

Two tags are enough to start. More than two invites a rule explosion nobody can audit.

The task type names the work: classification, extraction, summarization, drafting, code generation, retrieval-augmented question answering, and so on. Task type matters because the input/output token ratio and the tolerance for error differ enormously between them. A classification call might be 800 input tokens and 5 output tokens; a draft generation call might be 1,000 input and 4,000 output. The same model can be cheap for one and expensive for the other.

The quality tier names the consequence of being wrong: low, medium, high. Define each tier with a testable condition rather than a feeling.

The important discipline is that the tier is assigned by the caller, not inferred by the router. The caller knows whether the output goes to a customer or into a scratch file. Inworld's routing guidance makes the same point: conditions should have a product reason and an observable outcome, and you should avoid rules that encode assumptions you never measured, such as "long prompt means expensive model" Inworld's routing strategies.

A useful convention is to make the tier part of the span attributes from the first day, so you can slice cost by tier later without a migration.

Before writing any rule, calculate what a request of each task type should cost on each candidate model. The arithmetic is simple but easy to get wrong by a factor of ten, usually through unit errors or forgotten retries.

The base formula for an uncached request is:

cost = (input_tokens / 1_000_000) × input_rate
     + (output_tokens / 1_000_000) × output_rate

Then multiply by billed attempts, not user-visible jobs. A document pipeline that makes three model calls per document has three charges before any retry, and a retry is another billable attempt unless the provider says otherwise BenchLM's token pricing guide.

Token counts also vary by provider. The same text can tokenize differently across vendors, so a prompt measured on one model is an estimate, not a measurement, on another Silicon Data's cost per token guide. Measure with the target model's own tokenizer before you commit to a ceiling.

Here is a worked example using the synthetic workloads and dated list rates from BenchLM's calculator, which is a useful template even if your own rates differ:

Plug your own published rates into the formula for each row and you get a per-workload budget. BenchLM's own comparison, computed from rates it read on a stated date, put the same retrieval workload at $168 to $2,560 per month depending on the model and, for one provider, the time of day BenchLM's dated rate table. That spread is your routing headroom. If your ceiling for a low-tier summarization request is set below the cheapest option's computed cost, your rule is broken before it ships.

LiteLLM ships a pricing calculator that does this arithmetic interactively, showing per-request, daily and monthly cost from input tokens, output tokens and request volume LiteLLM's pricing calculator docs. Whatever tool you use, write the resulting ceiling into the rule, not into a spreadsheet nobody opens.

A routing rule should be readable by someone who is not the author. The cleanest form is a table with a default and a small number of overrides.

routes:
  - match: {task: classification, tier: low}
    model: cheap-fast
    max_cost_usd_per_request: 0.00012
    validate: {schema: label_enum, confidence_min: 0.6}
    fallback: mid-tier

  - match: {task: summarization, tier: medium}
    model: mid-tier
    max_cost_usd_per_request: 0.0040
    validate: {schema: summary_length, sampling: 0.05}
    fallback: strong-tier

  - match: {task: code_generation, tier: high}
    model: strong-tier
    max_cost_usd_per_request: null   # no ceiling, quality wins
    validate: {tests: must_pass}
    fallback: none

Three design choices in that block matter more than the syntax.

First, every cheap route has a validation step. Validation is what turns a cost decision into a quality decision. For classification, it is a schema check and a confidence floor. For summarization, it is a length bound plus a sample audit. For code, it is the test suite.

Second, every route with a fallback has a bound on how many attempts it can make. Inworld separates error fallback (move after an explicit failure) from latency fallback (move when the model has not started responding within a threshold), and warns that the latter can create simultaneous or sequential spend Inworld's failover guidance. Record total time and total cost across attempts, not just the winning attempt.

Third, the high tier has no cost ceiling. Writing max_cost_usd_per_request: null is a deliberate act of documentation. It tells an auditor that the absence of a cap is a decision, not an oversight.

A comparison of the three common routing patterns helps when you are choosing:

For a first deployment, the static rule wins on auditability. Inworld's own conclusion is that a transparent static rule often beats a complex selector the team cannot explain.

Routing without logs is unprovable. The OpenTelemetry GenAI semantic conventions give you a vendor-neutral shape for this, and the SRE guidance is to track attributes such as model, token counts, latency and cost per call, because LLM billing is per token rather than per request OpenTelemetry for LLMs from OpenObserve. PostHog's documentation on calculating LLM costs shows the same mechanics at the product-analytics layer PostHog's cost calculation docs.

At minimum, emit one event per request with these fields:

• task_type and quality_tier (the tags the caller set)
• route_matched and model_selected
• input_tokens, output_tokens, cached_input_tokens
• computed_cost_usd, calculated in your code from the rate table you stored
• validation_result and fallback_used (true or false, plus the reason)
• accepted (the outcome that actually matters)

Storing the rate table alongside the event is the trick most teams miss. If you only store the computed cost, a rate change months later makes your history unauditable. If you store the rate you applied, anyone can recompute the number.

Mavvrik's three-layer model is a good mental checklist for what comes next: a usage layer (what happened inside the system), an ownership layer (team, product, feature, customer) and a finance layer (allocation rule, budget, cost-to-serve) Mavvrik's cost tracking guide. Routing gives you the usage layer almost for free. The ownership layer is what lets you say "the support classifier saved this much and the drafting tool did not."

One warning from the same source is worth repeating: AI cost tracking starts with usage events and expands outward into infrastructure, so model costs are not the whole bill. A route that is cheap on tokens but triggers a long vector retrieval is not cheap.

The review meeting should compare three numbers per route, and refuse to discuss any of them alone.

Cost per accepted outcome. Total route cost divided by the count of outputs that passed validation or were accepted downstream. This is the number that matters, and it is the one DoiT argues is missing from most ROI conversations DoiT.

Acceptance rate, and rework rate. If the cheap route's acceptance rate drops enough, savings evaporate. The concrete trap is documented in AI ROI guidance: if AI produces a draft in 20 minutes instead of 60 but you spend 25 minutes fixing errors, the saving is 15 minutes, not 40 the AI ROI metrics rundown from Naitive. Apply the identical logic to routing. Compare cost per accepted output, not cost per call.

Fallback rate. The share of requests that fell back to the stronger model. A high fallback rate means the cheap route is a toll you pay on the way to the expensive one, and you should either fix its validation or delete it.

Waydev's coding-tool ROI framework offers a useful unit metric you can borrow: cost per merged pull request. Their worked example divides total AI spend by merged PRs to get a single number that finance and engineering can both read Waydev's AI coding ROI guide. The routing analog is cost per accepted task, and the same caution applies: if cost per accepted task falls while cycle time rises, you have moved the work, not removed it.

A simple review table, filled monthly, keeps the conversation honest:

That last row is the decision rule this whole article is built around. A route with a high fallback rate and a weak acceptance rate is not a saving. It is a second bill with extra latency. Delete it, or move the task to a higher tier and stop pretending the cheap route works.

If the numbers are new to your team and the exercise needs a second pair of eyes before you commit engineering time, a free 30 minute AI discovery call is a low-commitment way to pressure-test the plan against your own workloads Quanta Consulting.

The rules above assume you can call several models from one place and see one bill. That is the operational prerequisite, and it is where a lot of teams stall: three vendors, three SDKs, three rate tables to keep in sync. A single OpenAI-compatible endpoint that covers text, image, audio and embedding models, with prepaid pricing in US dollars, removes the integration work and leaves you with the routing logic itself Quanta Inference. Keep the rate table in your own code regardless of the gateway, because the audit depends on you storing the rate you applied.

For teams that want the routing policy reviewed against a specific line of business rather than as a general exercise, Quanta's domain experts read your context and return finished deliverables, which is a reasonable way to get a second opinion on thresholds without a long consulting engagement Quanta Experts.