Routing to Nothing: When the Right Answer Is No Skill

Two errors are possible. A miss is when a relevant skill exists and the system routes to none or to a person. A false load is when no skill fits and the system loads one anyway, or loads skill B when the request needed skill A.

A misrouted input does not stay contained. Routing errors trickle down through later agents and compound hallucination and cost, and each wrong step spends tokens it did not need to spend (Patronus on agent routing). Tool-learning surveys describe the same shape: planning, tool selection, execution, response generation, with selection as a discrete stage whose mistakes propagate (tool learning survey).

The asymmetry is not automatic, and that is the decision rule worth internalizing. If a wrong skill fails loudly and cheaply (it errors in two seconds, the user retries, nothing is written anywhere), a false load is cheap and an aggressive threshold is fine. If a wrong skill produces a plausible report, a deployment, a customer email or a migration, a false load is expensive and the none route is the cheapest guardrail you can build. Write down which regime you are in before you tune anything.

The none route is an output, not an error state. It should be a named value that flows through the same logging, metrics and UI path as every real skill.

{
  "outcome": "none",
  "request_id": "req_8f21",
  "top_candidates": [
    {"skill": "incident-triage", "score": 0.61},
    {"skill": "kubernetes-diagnostics", "score": 0.58}
  ],
  "threshold": 0.74,
  "reason": "no candidate above threshold",
  "handoff": {"queue": "platform-oncall", "sla_minutes": 30}
}

Three properties do most of the work. The outcome field is explicit, so downstream code cannot treat a weak match as a hit. The candidates, scores and threshold travel with the request, so whoever picks it up sees how close the call was. The reason separates "nothing was close" from "two skills were close," which are different problems: the first may need a new skill, the second two rewritten descriptions.

Design the pipeline the same way. Real systems separate capability identification from implementation selection, so an agent first decides what kind of capability is needed and only then which implementation serves it (two-step tool selection). If capability identification returns nothing confident, you never reach implementation selection.

You also need a fallback that behaves predictably under load, since routing without a fallback is a known production gap (Patronus). Decide what the none route does when the human queue is saturated: wait with an honest estimate, degrade to a general-purpose answer clearly marked as unassisted, or ask for more detail. Silently falling back to the top-ranked skill is not on that list, because it recreates the false-load problem and hides it.

You cannot calibrate a threshold against a router you have not measured on your own traffic. Borrow the practice skill authors already use: build realistic queries labeled with whether they should trigger, about 20 to start with an even split, and make the negatives near-misses rather than obviously irrelevant prompts (optimizing skill descriptions). "Write a fibonacci function" tests nothing; "update the formulas in my Excel budget spreadsheet" is a real negative for a CSV analysis skill because it shares vocabulary and needs a different tool.

For a none route you want three labels rather than two:

The ambiguous bucket is the honest one. Many real requests straddle skills, and forcing them into one of the other buckets during labeling bakes in a fiction. Keep them separate, decide a policy per overlapping pair (often by making one description defer explicitly to the other), and measure resolution separately.

Realism matters more than volume. Real prompts carry file paths, column names, typos and backstory, and the trigger tests that discriminate are the ones where the skill would help but the connection is not obvious from the wording (optimizing skill descriptions). Sample from production logs where you can, redact what you must, and include the odd requests: one-word messages, pasted stack traces, requests that arrived in the wrong channel.

On sample size, the guardrail calibration literature is directly usable even though it is written about prompt-injection screening: 100 examples of each class is the minimum for numbers you can loosely trust, 300 or more per class is noticeably more reliable, and 1,000 or more per class lets you distinguish a 2 percent rate from a 5 percent rate (guardrail calibration). Test with every model you route through, because the same instructions behave differently on a fast economical model and a stronger reasoning model (skill authoring best practices).

A threshold converts a confidence score into a binary decision, and placement trades two error types against each other: lower it and you catch more true matches while admitting more false loads; raise it and you block legitimate loads while letting requests fall to none (guardrail calibration). The right cutoff depends on your traffic and your costs, so compute it rather than copying one.

Assign costs. With the workflow owner, suppose a false load costs 40 units (rework, review, occasional cleanup) and a miss costs 6 units (a handoff and a short wait). The ratio is what matters, not the units.

Sweep the threshold on your labeled sample. For each cutoff, count errors per 1,000 requests.

The minimum sits near 0.74. Note what the table shows: pushing false loads to almost zero (0.90) more than doubles total cost, because misses are cheap but not free and there are many of them. Note also that the answer is a statement about your cost ratio. Set the costs to 10 and 20 and the optimum moves sharply upward, because misses now dominate.

Turn the costs into a floor. The minimum acceptable confidence is false_load_cost / (false_load_cost + miss_cost). With 40 and 6 that is about 0.87, meaning you should auto-load only when the router is at least 87 percent likely to be right. Use the floor as a sanity check on the swept threshold. If the two disagree, your scores are not calibrated probabilities and the sweep is the safer number.

Add a tie margin. When the top two candidates are within a few points, route to none or a reviewer regardless of absolute score. A 0.80 top score with a 0.78 runner-up is a coin flip with good presentation.

Re-check after every catalog change. Adding a skill moves decision boundaries without touching other files (catalog reliability), so re-run the labeled set on each catalog or model change and treat a shift in false loads as a regression.

A none route without a review loop becomes a place requests go to die. The loop that pays for itself is near-miss review.

Log every request with the outcome, a redacted copy of the request text, the top five candidates with scores, the threshold in force, the router and model configuration, and the eventual disposition. Explicit routing traces let you answer the harder question later: was a bad answer an agent fault or a router misclassification (Patronus).

Review weekly with three questions per case. Would a skill have helped? If yes three or more times in a row, you have a coverage gap, not a threshold problem. Would the top candidate have been right anyway? If yes and the score was close, the threshold is too conservative or the score is miscalibrated. Did two skills compete? If yes, fix the descriptions with explicit deferral language ("use this only for X; for Y use the Z skill") and re-test with those cases as negatives.

Track four numbers: the none rate, the false-load rate from labeled spot checks, the ambiguous resolution rate, and the median time from none to human answer. The none rate alone is a poor metric, since a rise can mean a tighter threshold, a bigger catalog or a traffic shift, and only the other three numbers tell you which.

Borrow one structural lesson from contact center routing. Over-segmentation, cutting the pool into slices too thin to absorb normal variation, leaves some queues empty while others miss service levels, and the fix is structural: collapse skills that do not change outcomes, widen tolerances, and use bounded overflow before demanding a perfect match (skills-based routing design). Skill catalogs fail the same way when every niche gets its own narrow skill.

The handoff is the product surface of the none route, and most handoffs fail for a boring reason: the person receives a ticket that says "unable to route" and rebuilds everything from scratch.

Attach the whole decision. The request in the user's own words, unparaphrased. The top three candidates with scores and the threshold. A one-line note on why none fired (nothing close, or a tie). What the system already tried. Who the requester is and which channel they came from. The contact center pattern is instructive: when an automated agent escalates, it persists the detected intent fields onto the work item before routing, so the downstream queue inherits context and does not re-ask (intent-driven routing). Copy that idea even when the "person" is an engineer on rotation.

Set the human's job precisely. They are not answering from scratch; they are resolving the routing question and then either doing the work or creating the artifact that makes the next occurrence routable. Give them two buttons in the review tool: "should have loaded skill X" or "genuinely out of scope." Both write to the same log the thresholds were tuned on, which is how the loop closes and how the next sweep gets fresh labeled data.

Then close the path. A "should have loaded skill X" event is a description bug and a candidate for the near-miss test set. An "out of scope" event that repeats often is a signal to write a new skill or decline that class of request explicitly at intake. An honest none today is a correctly routed request next month.

If you are standing up the routing layer, the smallest useful version is a skill catalog plus a gateway that can serve embeddings for retrieval and a capable model for the rerank step. Quanta Inference puts text, image, video, audio and embedding models behind one OpenAI-compatible API key, so you can prototype the pipeline and the none threshold on the same traffic before committing (https://quantaintelligence.ai/inference).