Routing Errors Versus Prompt Errors: Telling Them Apart

Before you open a single prompt, extract these fields for the failed request ID. If any are missing, adding them is the first fix.

A gateway is the natural place to capture these, because all model traffic passes through one chokepoint. Gateway telemetry can record tenant and application identity, the user making the call, the model and provider serving it, the request route, status, latency, token usage and estimated cost for every request the Splunk write-up on LLM gateways. If you do not run a gateway, the same fields can be attached as attributes on the span that wraps the model call; the OpenTelemetry GenAI conventions give you standard names for the model and provider at minimum the Logfire cost tracking guide.

A minimal structured log line for each request looks like this:

{
  "request_id": "req_8f21",
  "route_selected": "support.billing.refund",
  "route_runner_up": "support.billing.invoice",
  "route_score": 0.52,
  "route_threshold": 0.50,
  "policy_version": "router-14",
  "prompt_template_id": "refund_v3",
  "prompt_hash": "sha256:...",
  "retrieved_doc_ids": ["kb-1042", "kb-887"],
  "tool_calls": [{"name": "lookup_order", "args": {"order_id": "A-19"}}],
  "finish_reason": "stop"
}

The prompt hash matters more than it looks. Teams use correlation IDs and prompt hashes precisely so they can identify duplicate and near duplicate requests and spot which prompt version actually ran the Latitude guide to debugging LLM failures. When the renderer is the suspect, the hash lets you group failures by exact prompt content without storing every string twice.

Once the fields are in front of you, the routing check is a comparison against intent, not against the answer. You already know what the user wanted, because the ticket or the transcript says so. The question is whether the selected route describes that intent.

Three patterns account for most routing failures.

Wrong route, confident score. The selected route is clearly for another intent, and the score is high, say 0.89. This is a classifier or keyword rule problem. The fix belongs in the routing policy or its training examples. Looking at the prompt will waste your time.

Right route, borderline score. The correct route wins with 0.52 against a threshold of 0.50, with the runner up at 0.49. Nothing is broken today, but a small input change flips the decision. This is a latent routing error. The user phrasing "my invoice is wrong, I want my money back" can reasonably score as either an invoice question or a refund request, and the two routes answer differently.

Right route, no route at all. The request fell through to a default or catch-all route because no score cleared the threshold. Default routes are usually configured with a generic prompt, which explains the vague answer. The fix is either more training data for the uncovered intent or a rewrite of the fallback behavior.

A decision rule that works well in practice: treat the request as a routing error if any of the following holds, and as a prompt error only if none do.

1. The selected route's intent label differs from the intent you would assign by hand.
2. The correct route appears among the runner ups with a score within 0.05 of the selected route.
3. The selected route is the fallback route.
4. The policy version changed after the last routing evaluation passed.

That fourth condition is the quiet one. A policy deployed without re-running the evaluation suite can regress on intents nobody re-tested. Versioning the policy and tying it to the evaluation record turns "the route got worse" into "policy version X failed intent Y," which is a fixable statement.

It helps to remember that routing itself is a formal discipline. LLM routing is best described as a sequential decision process built from a context encoder, a scoring function, a decision rule and a learning signal, and the choice of decision rule is one of the design variables that determines behavior at the boundary the LLMRouter paper. When you debug a borderline score, you are usually debugging a threshold, not a model.

If the route checks out, the next question is whether the model saw what you think it saw. The rendered prompt is the ground truth here, not the template. Compare the two directly and look for four failure modes: an unfilled variable, a context block that arrived empty, a truncated history, and instructions that contradict each other after the route specific preamble was concatenated.

The retrieval span deserves its own check. In sequence, confirm that retrieval returned relevant material, that prompt assembly carried it intact to the model call, and that the tool arguments were well formed; the first span that diverges from expected behavior is where the fault lives the NHIMG trace analysis. A retrieval step that returns nothing is an easy catch. A retrieval step that returns a plausible but stale document is not, which is why logging the document IDs and versions matters.

Prompt length is a frequent silent culprit, and it has a cost dimension. Output tokens are typically priced at several times the input rate, and a request's bill is driven by how much text the workload sends and receives rather than by the headline rate alone the Silicon Data cost per token guide. A prompt that grew because a retrieved document expanded from 2,000 to 20,000 tokens can degrade quality and multiply spend at the same time. Checking the input token count against the baseline for that route tells you whether prompt bloat is part of the incident.

Here the distinction between routing and prompt design becomes practical. Routing exists to send each query class to the least expensive model that meets its quality bar, and a route should only be released to a cheaper model after that model clears the acceptance criteria for the class on representative queries the Braintrust routing guide. That framing gives you two clean repair paths: change the route's model assignment, or change the prompt that the route sends. Mixing the two in one change means you cannot tell which one fixed the failure, or which one caused the next one.

Take the message "I was charged twice for order A-19, please fix it." The support agent answers that it cannot find the order and suggests contacting the bank. Three different incidents can produce that identical output.

Case one: routing error. The request went to the support.account.access route because the word "charged" matched a login-issues rule set. The route's prompt is written for account access and has no order lookup tool attached. The model responded as well as it could given the tools it had. The evidence is in the route fields: selected route is account access, the correct refund route appears in the runner ups, and the tool call trace contains no lookup_order call at all. Fix the routing rule, add the example to the routing evaluation set, and the prompt needs no change.

Case two: prompt error. The correct refund route ran, the tool call trace shows lookup_order returning the right order, but the rendered prompt's context block is empty because the template referenced a variable name that changed in a recent deploy. The model was told to look up the order and was handed nothing. The evidence is the mismatch between the retrieval and tool output on one side and the rendered prompt on the other.

Case three: tool error. The route is right and the prompt is intact, but lookup_order was called with order_id: "A19" instead of A-19, and the tool returned not found. The model reported the truth it was given. The evidence is the tool call arguments in the trace.

The same user message, three different fixes. This is why the ledger alone cannot get you to a cause, and why teams that rely on logs and uptime charts without tracing end up guessing at the layer and fixing the wrong one the NHIMG analysis of LLM trace analysis.

Classification is only useful if it changes something. Route each incident to a permanent control based on its class.

The evaluation seat is the important one. Promoting each failed trace into a regression dataset with a defined expected outcome, then running that case before release, turns a one-off incident into part of the control set the NHIMG trace analysis. For routing specifically, the expected outcome is the route label, not the answer, so a routing regression suite stays fast and cheap to run.

Alerting on borderline scores is worth building early. The score margin is a leading indicator: requests with a thin margin will fail after the next prompt tweak, model swap or traffic shift, and they will fail in a way that looks random. A daily count of low-margin requests per intent gives you a ranked list of routes that are one small change away from misrouting.

Token and cost telemetry belongs in the same place. If you already compute request level cost from provider reported usage and tag it by feature, model and environment, you can correlate the failure with spend and catch the requests where a retry loop or an oversized context multiplied the bill the Honeycomb token cost guide. Runaway spend is often a routing or prompt symptom wearing a finance costume.

Some routing failures are not yours to fix in a threshold. If your application routes across several providers and models, the selection policy may live in a gateway or router you do not control directly. Managed routers classify prompts into task types and rank candidate models using aggregate signals, and they let you constrain the eligible model set the Braintrust routing guide. Knowing whether your route came from your own rule, a classifier you trained, or a vendor's default policy changes the fix entirely, so log the policy source alongside the version.

The same reasoning applies to the model pool. Model choice is often the largest single lever on an AI budget, and the spread inside one provider's lineup can run to a factor of many times between the cheapest and most capable tier the CloudZero pricing comparison. A route that points at the most capable model for a task a cheap model handles well is a routing error with a quality alibi: the answers look fine, so nobody investigates. Reviewing route to model assignments on a schedule, with real token counts per request rather than vendor examples, keeps that error from becoming a permanent line item the NHI cost estimation guide.

If you are consolidating several providers behind one key while you build this instrumentation, Quanta Inference offers text, image, video, audio and embedding models through one OpenAI-compatible API, which keeps the gateway fields consistent across model families. That is a convenience decision, not a substitute for the logging work above.