When a Misroute Is Worth Fixing: A Cost Ledger

The core discipline is simple: one row per request, emitted by the gateway or router, containing enough to reconstruct what happened. Providers report token usage in each response, and using those reported counts beats estimating, because a billed token is a real token (Honeycomb's guide to token cost tracking recommends capturing the provider response layer: model, input tokens, output tokens, cached tokens, finish reason). Tokenizers are not standard across providers, so the same text can produce different counts on different platforms, which means only the provider's own numbers are safe for a ledger (Silicon Data on cost per token).

A minimal log schema:

{
  "request_id": "req_8f21c",
  "ts": "2025-11-04T14:02:11Z",
  "intent": "refund_request",
  "expected_tier": "mid",
  "chosen_tier": "cheap",
  "model": "recorded-at-call-time",
  "input_tokens": 3120,
  "output_tokens": 410,
  "cached_input_tokens": 2400,
  "latency_ms": 1840,
  "fallback_used": false,
  "retry_count": 0,
  "outcome": "escalated_to_human",
  "user_id_hash": "u_55a1"
}

The fields that earn their keep are expected_tier, chosen_tier, and outcome. Most teams already log model, tokens, and latency, because observability tooling captures them automatically: instrumenting a model call can record token counts and cost per call with two lines of setup (Logfire's cost-tracking cookbook). What they usually lack is the intent and the ground truth about what the user actually needed. Add those two and the logs become an accounting system.

Watch for hidden multipliers. A single user question does not always mean a single model call: agents checking tools and retries firing silently can turn one request into several, each billed separately (DigitalOcean's cost calculation guide). If your agent loop makes five calls, log five spans under one request_id and sum them. Otherwise your cost per error will be understated by the retry factor, which is exactly the factor that makes some routes expensive.

Routing logs tell you where a request went. They do not tell you what it cost the business. That needs a second and third table.

The second table is billing or invoice data: what you were actually charged, per model, per period. This is your price source. Rate cards are a starting point, but you want the invoice line that includes cached input pricing, batch discounts, and any negotiated rate. Cached input is typically priced around a tenth of the standard input rate, and batch jobs at about half, so a ledger built on list prices alone will misprice workloads that lean on either (CloudZero's pricing comparison). Input and output are separate meters, and output usually costs several times input per token, so a verbose route is not cheap even when it reaches a cheap tier (PE Collective on token math).

The third table is the human cost of escalation: your ticketing or CRM system, with timestamps for handoff and resolution. Join on request_id or on the ticket ID your router writes into the escalation record. From this you get two numbers: minutes of human time per escalated request, and your loaded hourly cost for that queue. Multiply them and you have the staff-time component.

-- expected cost of one routing error class, per 1000 requests
SELECT
  intent,
  chosen_tier,
  expected_tier,
  COUNT(*)                                            AS errors,
  AVG(model_cost_usd)                                 AS avg_model_cost,
  AVG(human_minutes) * :loaded_hourly_cost / 60.0     AS avg_human_cost
FROM routing_ledger
WHERE chosen_tier <> expected_tier
  AND ts BETWEEN :window_start AND :window_end
GROUP BY 1, 2, 3
ORDER BY errors * (avg_model_cost + avg_human_cost) DESC;

Notice the ordering. Ranking error classes by frequency times cost per error surfaces the one worth fixing, not the one that sounds most alarming in a meeting. A rare, expensive error and a frequent, cheap one can land at the same total, and only one of them is usually fixable.

One caution on the join: incomplete joins bias the answer downward. If only 60 percent of escalations carry a ticket ID, your human-cost average covers the well-instrumented half, and those may be structured differently from the rest. Measure the join rate explicitly and treat it as a data quality metric in the ledger.

Suppose a support assistant handles 1,000,000 requests a month. The policy table says refund requests go to the mid tier. The router, using a semantic classifier, sends 6 percent of refund requests to the cheap tier instead, because the classifier misreads short, vague complaints.

That is 60,000 misrouted requests a month. Here is what each costs, using invented but internally consistent figures you would replace with your own bills and staff records:

The misroute saves about half a cent of model spend and costs about $2.07 in extra human time. Over 60,000 misroutes a month, that is roughly $124,000 in avoidable cost, minus the model savings of about $340. Call it $123,600 a month, or $124 per 1,000 refund requests.

Now the decision rule. The fix here is a classifier retrain plus a hard rule that refund intents cannot land below the mid tier. Estimate the fix at two engineer-weeks plus a month of evaluation runs, call it $18,000 all in. Payback period is $18,000 divided by $123,600, or about 0.15 months, roughly four and a half days. Fix it.

Compare that against a different error class in the same system: three requests a month where a contract-clause question reached the cheap tier, each costing $40 in rework. That is $120 a month. A fix costing two weeks of engineer time has a payback of 150 months. Do not fix it. Log it, set a review trigger if volume triples, and move on.

The decision rule in one sentence: a sizeable error class is worth fixing only when monthly cost of errors exceeds monthly cost of ownership of the fix within a payback window your finance team already accepts for other projects. Use whichever window your organization uses for software work; the point is that the comparison is explicit, not vibes.

Two refinements matter in practice. First, include the cost of the fix's ongoing maintenance, not just the build, because routing rules rot as models and intents change. Second, track the error rate after the fix. If the classifier retrain cuts misroutes from 6 percent to 2 percent but a new model release pushes it back to 4 percent, you need to know that the fix decayed rather than assuming it held.

The ledger will sometimes tell you that the biggest number on the page is not a routing error at all. That is a useful finding. Three common cases:

The right tier, wrong answer. If the model reached the expected tier and still produced a bad result, the cost shows up in your escalation join, but routing changes will not help. Look at prompts, retrieval quality, or acceptance testing instead.

Cheap-first cascades that escalate. A cascade that sends everything cheap and retries on failure can cut blended cost when most traffic resolves on the first try, but the escalation rate is the variable that decides it (TrueFoundry on cost- and quality-aware routing). If your ledger shows a 35 percent escalation rate like the example above, the cascade is not misrouting, it is mispriced. The fix is the classifier or the tier assignment.

Over-escalation. A router that sends refunds to humans too readily is "safe" and expensive. The same ledger formula applies in reverse: compare the human cost per request against the model cost of handling it at the frontier tier with a review step. Sometimes the correct route is cheaper than the cautious one.

If your model menu is broad and the routing policy keeps drifting, a single gateway that prices and routes every model from one API key makes the ledger simpler to build, because every call lands in the same place with the same token fields (Quanta Inference).

The ledger fails quietly more often than loudly. Use this checklist before you trust a number in a planning meeting.

Two habits keep this alive. First, run the join reconciliation on a fixed cadence, the same day your invoice arrives, so discrepancies are caught while the cause is fresh. Second, keep the log schema boring. The temptation is to add fields for every question; resist it, because a schema that grows faster than the queries against it becomes expensive to maintain and hard to compare across months.

The insight that separates a useful ledger from a dashboard: label errors by the decision they would change. An error class whose total cost sits well below your fix cost should be tagged "monitor" and removed from the working queue. An error class that clears the threshold gets owner, deadline, and a post-fix measurement. Everything else is noise that competes for attention with the one route that actually deserves engineering time.