A Description File Is Code: Review and Ownership

The single most effective control is also the cheapest: every entry in the registry has one person's name on it. Not a team alias, not a channel, one person who is expected to answer when the skill misroutes.

Ownership works because the failure mode is almost never a broken file. It is an unchanged file whose meaning shifted when a neighbor arrived. In the worked example from the catalog writeup, an incident-triage skill and a later kubernetes-diagnostics skill are both defensible choices for the request "Checkout errors increased after this morning's deployment. Determine the probable cause and prepare a remediation plan." The second skill wins on lexical matches for deployment, errors and Kubernetes, even though the first holds the correct end-to-end workflow the catalog reliability writeup. No skill-level test catches that, because neither skill changed and both still pass their own checks.

Governance checklists for agent systems already converge on this point. Microsoft's guidance for managing agents asks teams to assign owners responsible for long-term maintenance and to define a review process before anything moves to production the Copilot Studio manage checklist. Legal and risk reviews for agent deployments likewise require naming both a business owner and a technical operator before approval the agent governance checklist. Apply the same discipline to a skill entry and record it in the registry metadata, not in a wiki page that drifts.

A minimal owner record looks like this:

skill: incident-triage
owner: [email protected]
backup_owner: oncall-sre-lead
approved_version: 7
last_reviewed_by: j.okafor
neighbor_skills:
  - kubernetes-diagnostics
  - log-analysis
  - postmortem-writing

Two fields do the heavy lifting. approved_version is the rollback target. neighbor_skills is the list the reviewer must actually read before approving a wording change, because those are the skills a request could plausibly land on instead.

A description change deserves the same treatment as a code change to a router. The review should be short enough that people actually do it, and specific enough that a rubber stamp is visible.

The positive and negative example requirement is not bureaucratic. Prompt testing as a discipline works by running a fixed set of representative inputs against explicit expectations and scoring every run the same way, so a change becomes a visible regression rather than a vibe Braintrust's guide to prompt testing. That guide also makes the point that behavior depends on the prompt text, the model, the parameters, the tools and the retrieved context, so a change in any of those can alter results even when the prompt looks similar the same guide. For a skill registry that means a description review is also a moment to check whether the underlying model or router changed since the last approval.

A golden dataset, meaning a fixed set of test cases run before every meaningful change, is the standard way to keep those examples honest over time Langfuse on golden datasets. Keep the example prompts in the registry next to the description, versioned with it, so the negatives accumulate instead of being reinvented each review.

The overlap check is where most reviews should slow down. Two skills that both claim "investigate failures" and "production" will trade traffic unpredictably, and the loser is whichever request happens to be phrased with the neighbor's vocabulary. The fix is usually to make one description narrower: name the artifact, name the tool, name the outcome, and drop generic words like investigate, analyze or help. Nate Jones's breakdown of prompting as four separate disciplines makes the same underlying argument for specifications: precise, structured intent produces reliable autonomous output, and vague intent produces cleanup work Nate's Substack on prompting skills.

Suppose your registry holds 50 skills and you add one. Before the addition, a sample of 200 logged requests routed with 170 correct top-1 selections, an 85% rate. After the addition, the same 200 requests route with 161 correct, an 80.5% rate. The new skill itself is not obviously at fault. The drop is 9 requests, and the honest question is which of them changed destination.

Pull the 9 and look at where they went. Suppose 6 of them now land on the new skill, which shares the word "deployment" with a skill the requests used to reach. That is a description overlap problem, not a model problem, and no amount of skill-level testing finds it. The catalog writeup describes exactly this: the new skill did not corrupt the old one, it changed the decision boundary around it the catalog reliability writeup.

Three practical consequences follow. First, treat 200 as a floor rather than a target for the logged sample, because a 9-request movement is small enough that sampling noise can mimic it. Second, record the routing destination per request, not just pass or fail, or you cannot compute the movement at all. Third, add the 6 misfires as negative examples on the old skill and as positive examples on the new one, so the next review cannot regress them silently.

The SkillRouter results suggest one more lever. If your router can read the full body during selection, use it, because in the large-overlap setting that signal is what separates close candidates, and the compact 1.2B retrieve-and-rerank pipeline reached 74.0% Hit@1 while running 5.8 times faster than the strongest base pipeline the SkillRouter paper summary. If your stack only exposes name and description, the review checklist above is the compensating control, and it is not optional.

Over-triggering usually arrives as a complaint: a routine request got answered by a specialist skill, or a specialist skill started appearing in unrelated conversations. The flow below keeps the response short and leaves a record.

1. Freeze the entry. Set the skill to frozen in the registry so no further wording change merges while the incident is open. Freezing is not disabling. The skill stays live on its last approved wording.
2. Capture the failing examples verbatim. Copy the requests that routed wrongly, with the destination each one actually reached. These are the raw material for the next review, and paraphrase loses the vocabulary that caused the misfire.
3. Restore the last approved description. Roll back to approved_version from registry history rather than hand-editing the current text. Hand edits reintroduce the bug and break the lineage.
4. Confirm the rollback. Replay the captured examples and confirm they route as they did before the problem change. If they do not, the problem is upstream of the description and the incident escalates.
5. Write the post-incident note. Record which change caused the misfire, which examples exposed it, and which neighboring skill should have been checked in review.

Rollback to a preserved, approved artifact is the standard practice for prompt and configuration changes in production. A prompt registry that stores versioned payloads with metadata and lineage makes rollback a pointer update rather than a reconstruction exercise, and canary or blue-green deployment patterns limit blast radius while a new version is validated versioning and rollback for agent prompts. The same source stresses immutable artifacts, because rollback only works if the exact old payload still exists and can be verified.

Fallback logic belongs in the plan too. Treating a fallback as a first-class part of the control plane, versioned and validated against golden sets before it reaches a user, is the discipline MLflow describes for prompt fallbacks MLflow on fallback prompts. In a skill registry the equivalent is a defined behavior for ambiguous requests, such as asking the user which of two skills they meant, instead of letting the router guess between near neighbors.

If the incident involved an agent taking an action it should not have taken, widen the review. Guidance for agent governance suggests testing the workaround as well as the initial refusal, because a blocked task is exactly when an agent may improvise the agent governance checklist. A misrouted request that lands on a skill with write access is a permissions question on top of a routing question.

Processes decay when the person who wrote them leaves. Three conventions keep a shared registry workable across team changes.

Descriptions live only in the registry. No copies in wikis, slide decks or chat threads. A duplicated description is a second source of truth, and the copy is always the one someone edits.

Every change points at a version. A review comment should name the approved version it supersedes and the new version it approves. This is the same auditability principle Microsoft's checklist captures when it asks whether versioning requirements are established for agents and components and whether a rollback plan exists for failed deployments the Copilot Studio manage checklist.

Ambiguity is a bug, not a user error. When a request could reasonably go to two skills, that is a description defect. The fix is to narrow one description or add a disambiguation step. Research on tool selection in production separates capability identification from implementation selection precisely because multiple tools often serve the same capability, and conflating the two produces brittle routing the two-step tool selection disclosure. A skill registry has the same shape: the task needs a capability, and several skills may claim it.

A broader survey of tool-use pipelines places tool selection as one of four stages alongside task planning, execution and response generation, which is a useful reminder that routing accuracy is a precondition for everything downstream rather than a metric to optimize in isolation the tool learning survey. A production-oriented survey of tool and agent selection makes the same point about the backend layer, where retrieval, context engineering and evaluation determine whether the right capability is reachable at all the production tool selection survey.

When the registry grows past a few dozen entries and several teams publish overlapping capabilities, the review work starts to compete with delivery. Teams that standardize on a shared skill library, with an expert per line of business reading their own context before handing back work, can keep the review load distributed; Quanta Skills is the natural next step for teams wiring reviewed skills into coding agents Quanta Skills.