Costing a Wrong Tool Pick in Tokens and Retries

Start with the inputs. A typical agent request has four parts that get billed as input on every turn:

1. The system prompt, which holds instructions and policy.
2. The tool schema list, serialized as JSON. This is usually the largest fixed block and it is resent on every turn.
3. Conversation history, including prior tool results.
4. The current user message.

The MCP resources spec is useful here for understanding why context is so heavy: resources can carry file contents, database schemas and live system data, and a server may return multiple contents in one read the MCP resources specification. A server that returns a whole directory on one read can inflate the input block in a single step.

Output is smaller but not negligible. Tool-call arguments are JSON, and JSON has overhead: braces, quotes, field names repeated per call. A wrong pick may also include a short justification, which the model was never asked for.

Here is a working estimator you can adapt. It takes token counts and per-million rates and returns cost per attempt:

def attempt_cost(input_tokens, output_tokens, in_rate, out_rate):
    # rates are dollars per million tokens
    return (input_tokens * in_rate + output_tokens * out_rate) / 1_000_000

def misroute_cost(input_tokens, wrong_output_tokens,
                  retry_output_tokens, in_rate, out_rate,
                  retries=1):
    failed = attempt_cost(input_tokens, wrong_output_tokens, in_rate, out_rate)
    # the retry resends the same input plus the wrong tool output
    retry_input = input_tokens + wrong_output_tokens
    ok = attempt_cost(retry_input, retry_output_tokens, in_rate, out_rate)
    return failed + retries * ok

Run it with the numbers from the summary. A prompt of 4,000 input tokens and 300 wrong output tokens at $0.50 per million input and $1.50 per million output costs $0.00245 for the failed attempt. If you exclude the routing overhead and assume the retry resends the same input, the successful attempt costs about $0.00245 as well, so the pair lands near $0.0049. If you add the wrong output to the retry input, the total rises slightly. That seems trivial per call, and it is: the interesting number is the product of that figure, your request volume and your error rate.

The number that should drive your architecture is expected cost per request, not the cost of a bad call in isolation. Write it as a mix of the clean path and the dirty path:

expected cost = (1 - p) * clean_cost + p * failed_cost + p * q * second_retry_cost

where p is the probability that the model picks the wrong tool and q is the probability that the retry also fails. If your logs do not record tool selection outcomes, you are estimating p blind. The Scalekit analysis of production tool-call failures reports that individual tool calls fail 3 to 15 percent of the time in well-engineered production systems, and that at 99 percent per-step reliability across a 10-step workflow, overall reliability falls to 90.4 percent the Scalekit guide to tool call failures. Those are failure rates, not specifically misroute rates, but they set the order of magnitude for p in a real system.

Use that to build a small table for your own traffic. The figures below are illustrative arithmetic, not published prices; substitute your own token counts and rates.

The lesson of the table is that token cost alone rarely justifies a routing redesign. Even the heavy row is cents per misroute. The reason to care is the second and third columns: latency, and the fraction of misroutes that take a wrong action rather than a wrong answer.

Latency per round trip is where the money argument turns into a product argument. A misroute adds at least one full model call before the user sees anything, and that call has to regenerate the output from scratch. A request that would have taken one round trip now takes two, plus a tool execution, plus the network hop for each.

Two published figures bound the delay. The same Scalekit analysis notes that as per-step reliability falls, workflow reliability collapses multiplicatively, which is the mechanism behind long agent runs that feel slow: each step carries its own latency and its own chance of restart. And a study of semantic routing in vLLM serving found that routing queries before inference reduced response latency by 47.1 percent and token consumption by 48.5 percent on the benchmark it tested, while improving accuracy by 10.2 percentage points the When to Reason paper. That comparison is between routed and unrouted inference, not between a misroute and a clean call, but it establishes the shape of the trade: routing decisions change both latency and spend in the same direction.

A useful practical rule: count round trips, not milliseconds, when you are deciding whether a misroute is worth fixing. A single retry inside an interactive chat is usually tolerable. Two retries is a visible stall. A retry inside a background batch job is invisible and cheap, which is why batch pipelines can afford a weaker router than a chat interface can.

If a small model were a strict substitute for a larger one at a fraction of the rate, everyone would route with it. The problem is that tool selection and parameter identification are exactly the tasks where small models degrade most. A study presented at ACL 2026 found that small language models hallucinate plausible but nonexistent tool names that reflect naming conventions they internalized during pretraining, and that the failure is driven by schema misalignment rather than by capability the ACL paper on tool schemas. The same work reports that adapting the tool schema to the model's pretraining patterns, rather than training the model to the schema, improved results by up to 17 percent and cut schema misalignment errors by 80 percent the preprint version.

That is a sharp and underused insight: your tool names are part of the routing problem, and a cheap fix is to rename your tools. If a small model consistently invents get_user_profile when you expose fetch_account_by_id, the naming is doing the damage. Renaming costs nothing, a schema rewrite costs nothing, and both shrink the misroute rate before you spend a cent on a bigger model.

The complementary point is about the choice itself. If the routing decision is "which of these 12 tools applies to this message," that is a classification task, and classification is cheap. If the work is "draft the refund response using the customer's order history," that is generation, and generation is where the larger model earns its rate. Conflating the two is the most common costing error in agent design.

The rule is a comparison of two differences. Route with the small model when the extra cost of routing with the large model exceeds the expected cost of the misroutes the small model will cause. In symbols:

use_small_for_routing if (rate_large - rate_small) * routing_tokens > p_small_error * (cost_of_one_misroute + latency_penalty)

Read the right side carefully. It is not the misroute token cost alone, which we saw is tiny. It is the misroute cost plus whatever you charge yourself for a slower, wronger request. For most chat products, the latency penalty dominates. For most batch jobs, it does not.

A few practical corollaries:

• Route with the small model when the tool set is small, stable and well named. The ACL result suggests naming alignment does most of the work.
• Do the work with the larger model when the output is user-facing, when the tool has side effects, or when a wrong argument costs more than a wrong token.
• Add a confidence gate. If the small router's output is low confidence or names a tool that does not exist, escalate to the large model for the routing step rather than letting the large model do the work. This costs one extra small call and avoids the misroute.
• Defer rarely used tools. Sprawling schemas are a leading cause of misroutes; loading them only when needed keeps the input block small and the selection cleaner, a pattern the OpenAI docs recommend for large tool sets the OpenAI tool search guidance.

Here is a checklist to run before you pick a router:

That last row is the one most teams skip. Without selection telemetry you cannot estimate p, and without p the decision rule has no right-hand side. The MCP specification's attention to capability negotiation and versioning is a reminder that the protocol layer will happily carry whatever you send; it will not tell you whether you sent the right thing the MCP specification.

The architecture that follows from all of this is a two-tier router. A small model, or even a classifier, examines the user message plus a compact tool index and returns a tool choice or an escalation flag. A larger model receives the chosen tool schema, the full context, and the instruction to produce the arguments and the final answer. You avoid paying large-model rates for the selection step while keeping the large model where its quality shows.

Structured output support is what makes this practical. The schemas feature in the LLM command line tool, for example, lets you constrain a model to a named JSON shape and works across several providers the LLM schemas documentation. Constrain the small router to a schema with one field (the tool name) and a confidence score, and you have a clean, cheap decision point. The same discipline applies to arguments: a schema with strict types and no extra properties eliminates a whole class of wrong-argument failures before the call leaves your app the Agenta guide to structured outputs.

If you want to run the router and the worker against the same endpoint, Quanta Inference exposes text, image, video, audio and embedding models through one OpenAI-compatible API with prepaid US dollar pricing, which makes the per-million arithmetic above map directly to a single invoice. For teams that would rather not build the routing tier themselves, Quanta Experts takes your context, does the work and hands back finished deliverables.

The final step is measurement. Log three things per request: the tool the router chose, the tool the worker would have chosen, and whether the outcome was correct. After a week you will have a real p, a real cost per misroute, and a real number for the right-hand side of the rule. Until then you are guessing at the cheapest line item and ignoring the expensive one.