When to Let the Model Choose the Tool vs. When to Route in Code

Four dimensions matter: latency, cost, reliability, and debuggability. Each moves in a predictable direction.

Latency. A model-chosen tool call requires at least one full generation round trip before any tool executes, plus a second generation to synthesize the result. A code route can dispatch in microseconds. If you need sub-100ms overhead, the model cannot be in the selection path. Constrained decoding itself adds only a small per-step overhead, typically a few percent for regex and JSON-schema constraints, per ZeroEntropy, so the selection turn, not the constraint machinery, is where your latency goes.

Cost. Every model-chosen call spends input tokens on tool schemas and output tokens on the name and arguments. With a large tool set, schema tokens grow with every call. A code route spends nothing on selection.

Reliability. Here the naive comparison misleads. Function calling guarantees syntactic validity of the argument JSON but not semantic correctness: the model might generate valid JSON with nonsensical parameter values, Blaxel's function calling overview notes. A code route has no selection error at all, but it can route the wrong way if your rules are incomplete. The honest comparison is between model selection error and rule coverage gaps.

Debuggability. This is the least discussed and most decisive factor. A code route leaves a stack trace. A model route leaves a prompt, a sampling temperature, and a tool name. When a call goes wrong, you need to know whether the failure was tool selection, argument extraction, tool design, missing permissions, or excessive context, and those should be tracked as separate metrics, Paragon's tool-calling benchmark writeup argues.

Suppose a support assistant handles 200,000 requests a day and must pick among 4 tools: look up order, start refund, update address, escalate to human. Assume one model-chosen call adds 600 input tokens of tool schemas and 40 output tokens of arguments, at a blended rate of $0.50 per million tokens (a rate chosen for arithmetic, not a quoted price). Selection cost per request is 640 tokens times $0.50 divided by 1,000,000, which is $0.00032. Across 200,000 requests that is about $64 a day on selection alone, before the synthesis turn.

Now add the reliability math. If the model picks the wrong tool on 2% of requests, that is 4,000 wrong calls a day. If 10% of those wrong calls touch money (a refund started instead of an order lookup), you have 400 financially consequential errors daily. A four-branch router written as validation-then-dispatch can drive that class of error to zero for the cases it covers.

The decision flips when the tool set grows and the intents are open-ended. If a research assistant has 40 tools registered and users ask in free text, a router cannot enumerate the branches. At that point the model's semantic matching is worth its cost, and the right move is to shrink the tool set it sees per request rather than to abandon model selection.

Apply these four tests in order. The first one that fires decides.

1. Is the tool set small (roughly a handful) and fixed? Route in code. You can enumerate the branches and test them.
2. Is a wrong tool call expensive? Route in code, or add a confirmation gate before the model's choice executes. A typed tool name and argument object describe an intended action, and runtime authorization is a separate decision, TrueFoundry's structured outputs guide stresses.
3. Must arguments be validated before execution? Validate either way, but validation before dispatch is easier when your code produced the arguments. Tool calling is structured output with consequence, as the same guide puts it.
4. Do you need sub-100ms selection overhead? Route in code. If none of these fire, let the model choose.

The VC-backed default of "always let the model pick" fails test 1 and test 2 simultaneously for most internal tools.

Whichever path you choose, the argument object needs a schema and a validator. Pydantic is the common choice: you define a model, generate a JSON schema for the tool definition, then validate the returned arguments against the same model. Jason Liu's Pydantic and LLM tutorial shows both halves, including Packages.model_validate_json on the tool-call arguments and Packages.model_json_schema() inside the tool definition. The separation of data from its description makes prompts easier to organize.

from pydantic import BaseModel, field_validator

class RefundRequest(BaseModel):
    order_id: str
    amount_cents: int

    @field_validator("amount_cents")
    @classmethod
    def cap_refund(cls, v: int) -> int:
        if v <= 0 or v > 50_000:
            raise ValueError("refund outside allowed range")
        return v

# same class feeds the tool schema and validates the model's arguments
args = RefundRequest.model_validate_json(tool_call.function.arguments)

That validator is your router's second safety net. It runs identically whether the arguments came from a model or from your own parser, and it enforces the domain rule (a refund cap) that no JSON schema keyword can express on its own.

Do not rely on the schema alone for correctness. Strict schema output narrows the interface but only within its supported contract; shape is not truth, and a schema-valid customer ID, amount, or date can still violate domain rules, per TrueFoundry. Keep schemas minimal with required fields, strict types, and enums to prevent ambiguous outputs, and log raw and parsed outputs with validation failure metrics so violations surface early, MLflow's structured outputs guide recommends.

If the model must choose, improve the choice rather than the model. Tool descriptions carry more weight than most teams expect: models rely on the description to match user intent to the right function, and clear, specific descriptions reduce routing errors, Blaxel notes. Ambiguous queries sometimes trigger the wrong function or no function at all.

Three levers matter, in this order:

• Model choice. Swapping the underlying LLM produced larger and more consistent swings than any prompt, schema, or routing change tested, per Paragon's evals. It is the lever most teams cannot pull freely, which is why the next two matter.
• Description depth and prompt wording. These moved the numbers mostly on multi-tool tasks and were close to flat on single-tool tasks, same source. If you have one tool, stop rewriting the prompt.
• Dynamic filtering. Send only the tools plausible for the current request. Fewer choices means fewer wrong choices and fewer schema tokens.

Then instrument. Distributed tracing that captures sessions, traces, and spans for each tool call, including inputs, outputs, latency, and errors, is the minimum for finding out why a selection went wrong, Maxim's tool-calling deep dive describes this pattern. Evaluate on the full trajectory, not just the final answer.

One more trap: silent truncation. If generation hits the token limit mid-payload, the JSON is cut off and most providers guarantee schema conformance only for complete generations. Always check the finish reason before parsing, per Tianpan's failure taxonomy, which also notes that complex schemas break constrained decoding in framework-specific ways.

The rule is to measure first and promote second. Log four fields for every request: the branch taken, the tool name the model chose, the validated arguments, and the wall-clock cost of selection plus execution. After two weeks you will see a distribution, not a guess.

Promote a path to code when all three hold: the same tool is chosen for more than 90% of requests in a recognizable intent class, the arguments follow a stable template that your code can construct from parsed input, and the path has produced at least one misroute you can explain. Keep it model-chosen while any of these fail.

A hybrid is often best. Route the top three intents in code, pass everything else to a model picker with a narrowed tool set, and require a confirmation gate on any tool that moves money or touches external systems. This keeps the deterministic path fast and the open-ended path possible.

When your tools span text, image, audio, and embedding models, a single API key for all of them removes a layer of provider-specific routing from the orchestration problem, which is what Quanta Inference provides. For teams that want the orchestration and the deliverables handled rather than assembled, Quanta Experts is the natural next step.