Writing a Skill Router: How Models Choose Between Skills

Most routers are assembled from the same four stages, even when the marketing names differ. The Patronus routing guide separates routing approaches into rule-based (keyword spotting and pattern matching), supervised machine learning on routing data, and LLM-based selection, with the last described as the state of the art for agent routing (Patronus AI agent routing guide). Production systems typically blend the first and third.

Stage 1, embedding. The request and every skill description are converted into vectors, and similarity is computed. This is cheap, runs against the whole catalog, and is good at paraphrase but blind to specifics that embeddings treat as noise, such as a product name or a file format.

Stage 2, keyword match. A lexical pass rewards exact terms. It catches "use the parquet loader" when the description says "parquet," and it is the reason a description with no shared vocabulary with the request almost never surfaces. Rule-based keyword routing is simple and inflexible, which is exactly what you want for a must-fire trigger (Patronus guide).

Stage 3, scoring and ranking. Scores from the passes are combined and sorted. In a two-stage retrieve-and-rerank design, a bi-encoder retrieves a shortlist (SkillRouter retrieves 20 candidates) and a cross-encoder reads the query alongside each candidate's name, description, and body to rerank them (SkillRouter paper).

Stage 4, threshold. The top candidate is loaded only if it clears a cut. Threshold is the parameter most likely to be set once and never revisited, and it is the one that most often causes silent, total failure.

For a selector that needs a hard trigger to fire regardless of scores, treat it as a routing rule rather than a retrieval problem. The classic mitigation is to separate capability identification from implementation selection: first decide which capability is needed, then choose among implementations registered under it (two-step tool selection disclosure). Applied to skills, that means one explicit capability mapping per request class, with rival skills demoted to implementations.

There are four distinct failure modes, and mixing them up is why people rewrite a description that was never the problem.

The vocabulary gap. The request says "sales spreadsheet," the description says "tabular data ingestion." The embedding pass may bridge some of that, but the keyword pass will not, and the description guide is explicit that agents match against what the user asked for, so you should list contexts where the skill applies "even if they don't explicitly mention" the domain terms (Agent Skills description guide).

The stronger rival. Catalog reliability breaks at scale in a specific way: a new skill does not corrupt an old one, it "changed the decision boundary around it." The documented example is an operations agent where a new kubernetes-diagnostics skill wins a request about deployment errors over the existing incident-triage skill, because the newer description carries more lexical matches for "deployment" and "errors." Both skills may be defensible choices, but the wrong one was chosen, and the individual skill tests still pass (catalog reliability article).

The task that does not need a skill. Agents typically consult skills only for tasks that need knowledge or capabilities beyond what they can handle alone, so a simple one-step request may not trigger a skill even when the description matches perfectly (Agent Skills description guide). If you are testing activation with trivial requests, you are measuring nothing.

The threshold and the body gap. A good score that does not clear your cut is a configuration failure, not a wording failure. Separately, if you route on metadata only, you lose information that lives in the body. On a benchmark of about 80,000 candidate skills, hiding the skill body lowered Hit@1 from 68.0% to 24.0% for the strongest base pipeline, and metadata-only encoding trailed all-field encoding by 14.0 points even when trained on the same 37,979 pairs (SkillRouter analysis). Those numbers come from one heavy-overlap benchmark, so read them as "bodies matter more than you think in crowded catalogs," not as a universal law.

The description guide recommends building an eval set of about 20 queries, 8 to 10 that should trigger and 8 to 10 that should not, mixing formal and casual phrasing, explicit and implicit references, terse and context-heavy prompts, and single-step versus multi-step tasks (Agent Skills description guide). The most valuable negatives are near-misses: a CSV analysis skill should not fire on "update the formulas in my Excel budget" or on a database ETL request, because those share keywords but need a different capability.

Once you have the set, run four experiments. Log, do not guess.

Experiment 1, vocabulary audit. For each should-trigger query, record which words in the request appear in any skill description. If the domain noun in the request ("parquet," "postmortem," "invoice") never appears in any description, you have found a vocabulary gap without touching the model.

Experiment 2, score trace. Log the candidate list with scores and the threshold value. This single artifact separates "not retrieved" from "retrieved but ranked second" from "ranked first but under threshold," and each has a different fix.

Experiment 3, A/B the description only. Change nothing but the description, include both what the skill does and specific triggers for when to use it, then rerun the full set (Claude skill authoring best practices). Keeping everything else fixed is what makes the result attributable.

Experiment 4, remove a rival. Pull each competing skill out of the catalog one at a time and rerun. If activation recovers when a rival is removed, your problem is the decision boundary, not the skill.

A minimal harness looks like this: load every description from frontmatter, score the fixed query set, and print the top five with scores so regressions are visible in a diff.

import json, math, re

def tokens(text):
    return set(re.findall(r"[a-z0-9]+", text.lower()))

def jaccard(a, b):
    return len(a & b) / max(1, len(a | b))

skills = json.load(open("skill_descriptions.json"))
queries = json.load(open("eval_queries.json"))

for q in queries:
    qt = tokens(q["query"])
    ranked = sorted(
        ((round(jaccard(qt, tokens(s["description"])), 3), s["name"]) for s in skills),
        reverse=True,
    )[:5]
    print(q["should_trigger"], q["query"][:60], "->", ranked)

Run the same loop against your production router's actual scores where you can, since a hand-rolled metric only shows what lexical overlap sees.

The description guide lists four principles, and each maps to a pipeline stage. Use imperative phrasing ("Use this skill when...") because the agent is deciding whether to act. Describe user intent, not internal mechanics, because the agent matches against what the user asked. Err on the side of pushy by listing contexts explicitly, including ones where the user does not name the domain. Keep it concise, since long enough to cover scope and short enough not to bloat context across many skills is the whole target, and the spec enforces a 1,024-character ceiling (Agent Skills description guide).

The authoring guidance adds a naming convention worth stealing: gerund-style names such as processing-pdfs or analyzing-spreadsheets, and never vague names like helper or utils, which carry no signal for either a human or a selector. It also stresses that the description should include both what the skill does and when to use it (Claude skill authoring best practices).

A comparison makes the difference concrete:

That third column is the decision rule: a description earns its tokens by containing words the request will use, not by describing the implementation well. One more boundary to respect: keep one skill focused on one outcome. Red Hat's applied AI team reports that broad, comprehensive skills are harder to route and maintain, and that splitting work between deterministic scripts and model reasoning cut cost by 26% in their experiments (Red Hat skill-building pitfalls). A skill that does five things has five sets of trigger words competing in one description, and loses all five races to five focused rivals.

Skill selection is a small instance of a larger job. LangChain's framing is that agents usually fail for one of two reasons: the model is not capable enough, or "the 'right' context was not passed to the LLM," and that the second is more often the cause, which makes context engineering "the number one job of AI Engineers" (LangChain context engineering docs). A router that surfaces the wrong skill has passed the wrong context before the model writes a token.

That framing gives you a checklist that generalizes. For every request class, ask what context the model needs (a skill, a tool, a document), how that context is represented to the selector (name and description, or full text), what budget it consumes, and what happens when selection misses. The LangChain guide separates transient model context from persistent state and store, which is the right mental split for skills too: metadata is transient per call, the loaded body is a per-task decision, and bundled scripts and references are on-demand resources.

Progressive disclosure is a context optimization first, so it needs a spec: the body should be loaded on demand, and the parts a selector needs must reach it through summary. That is exactly why skill text is treated as a resource and not as prompt text. The practical implication is that when you fix a routing bug, you are usually improving a description or a spec, and that improvement helps every agent that reads it rather than one prompt.

Task routing itself is a generalized version of this. If you also route between agents or specialists, the routing patterns and evaluation metrics described for multi-agent systems (single-agent routing, multi-agent fan-out, hierarchical routing, and accuracy or recall checks against a labeled set) apply directly to skill selection with a change of vocabulary (Patronus AI agent routing guide). The same guide recommends logging, testing, and fallback mechanisms, and those three practices are what turn a router from a black box into something you can operate.

That is where a skills library becomes a shared asset rather than a personal folder. Installable skills with explicit metadata and routing-friendly descriptions are what make a catalog readable to an agent at scale, and the same discipline you apply to one description is what keeps a library honest. If your next step is comparing several models on routing or generation quality, doing it through one endpoint with consistent request shapes removes a variable from your experiments; Quanta Inference puts those models behind a single OpenAI-compatible key.