A Trigger Regression Set for Skills: Sampling and Pass Criteria

A trigger regression set is a fixed, versioned list of realistic user requests, each labeled with one bit: should this fire the skill, yes or no. You run the same list against every new draft of the description and compare outcomes. This is the same idea as a golden dataset in LLM evaluation practice: a fixed set of cases run before every meaningful change, so that an improvement in one place cannot silently break another.

The published guidance on trigger evals suggests roughly 20 queries split into 8 to 10 that should trigger and 8 to 10 that should not, per the skill description guide. For a practitioner's set that you actually maintain by hand, a slightly smaller range works: 8 to 12 positives and 4 to 8 negatives, for 12 to 20 total. Fewer cases mean each one gets written carefully and stays realistic.

The format is deliberately boring. A JSON file next to the skill:

[
  { "id": "csv-01", "query": "I've got ~/data/q4_results.xlsx with revenue in col C and expenses in col D. Add a profit margin column and flag anything under 10%.", "should_trigger": true },
  { "id": "csv-02", "query": "boss wants a chart out of this data file, can you just make it", "should_trigger": true },
  { "id": "csv-07", "query": "I need to update the formulas in my Excel budget spreadsheet", "should_trigger": false },
  { "id": "csv-08", "query": "write a python script that reads a csv and uploads each row to our postgres database", "should_trigger": false }
]

Each case carries a stable id so that a failure in CI names a case rather than "the fifth one".

Vary the positives along axes that actually change the routing decision. The description guide lists four worth covering:

Your best positives are the cases where the skill would help but the connection is not obvious from the words alone. Those are the ones where description wording decides the outcome. If a query already asks for exactly what the skill does, any reasonable description will trigger it, so the case tests nothing, a point the description guide makes directly.

Keep the requests realistic. Real users paste file paths such as ~/Downloads/report_final_v2.xlsx, give backstory, and use inconsistent capitalization. Synthetic-sounding queries leak vocabulary from your description and inflate your pass rate.

One practice worth borrowing from the wider evaluation literature: test on more than one model if you ship to more than one, because a description tuned for a strong reasoning model may under-trigger on a faster, cheaper one, as the authoring guide warns when it recommends testing with every model you plan to use. If your regression run is model-specific, record which model each score came from.

There is a measured reason to take the routing decision seriously. A study of routing across a large skill registry, reported in the SkillRouter paper, found that hiding skill bodies from the router dropped Hit@1 sharply across representative retrieval baselines. In the companion summary of the same work, the strongest base retrieve-and-rerank pipeline fell from 68.0% to 24.0% Hit@1 when the body text was withheld, with drops of 37.3, 38.7 and 44.0 percentage points across three model configurations. Those numbers describe one benchmark with around 80K heavily overlapping skills, not every deployment the summary. But the direction is clear: when the selector works from a short field, small wording changes carry a lot of weight.

This is the half of the set that earns its keep. The most valuable negatives are near-misses: requests that share keywords or concepts with your skill but need something different, per the description guide. Weak negatives ("what's the weather today?") test nothing because there is no overlap to confuse a router. Strong negatives sit close to the boundary.

Using a CSV analysis skill as the running example, the guide's own strong negatives are:

• "I need to update the formulas in my Excel budget spreadsheet", which shares spreadsheet and data vocabulary but needs spreadsheet editing, not analysis.
• "can you write a python script that reads a csv and uploads each row to our postgres database", which names CSV but is a database ETL task.

Three ways to generate near-misses systematically:

1. Same noun, different verb. Take the object your skill operates on and pair it with an adjacent job. Lists the skill does not process, files it does not edit, events it does not triage.
2. Same verb, different object. Your skill reviews something; a near-miss asks to review a different class of artifact that a sibling skill owns.
3. Adjacent skill lookup. For each skill in the catalog, ask "which other skill could a user plausibly mean?" Each answer is a near-miss. The catalog-level article on agent skill routing at scale shows why this matters: a new, more specialized skill can take over requests that used to go to a general one, and each individual skill can still pass its own tests while the catalog gets less reliable. The decision boundary moved, not the skill.

That catalog effect is why negatives should include cases that belong to a neighboring skill rather than to no skill at all. A near-miss that should fall through to database-etl is a better test than a near-miss that belongs to nothing.

If you keep a variant of a task on both sides of the line, document the distinction in one sentence in the case file. Two reviewers looking at "clean up my sales data" should agree whether it fires. Disagreement between humans means the case is ambiguous, not that the skill is wrong. Fix the case or move it.

A pass bar has to be simple enough that nobody argues about it at review time. Two rules cover most needs:

1. Every must-trigger case fires. Zero exceptions. Each positive you added represents a user whose task the skill should have handled. A miss is a silent failure: the user gets a generic answer and never knows a skill existed.
2. No more than one near-miss in ten fires. With 4 to 8 near-miss cases, that means at most one false positive per run for a set of the recommended size. If you keep ten near-misses, one is your ceiling.

Why not zero on the negatives? Because triggering is probabilistic across models and phrasings, and a strict zero bar on a small negative set invites pointless description churn; you end up over-narrowing the description to kill one stubborn case and losing positives elsewhere. Why so tight at one in ten? Because a false trigger is not a small cost. A misfire loads a whole body of instructions into context and pushes the agent down the wrong workflow, and the routing article on catalog reliability shows how the wrong skill can execute perfectly while answering the wrong question.

A worked example makes the math concrete. Suppose the skill has 10 positives and 5 near-misses, and you run the set three ways across two models:

Draft A is the classic over-broad edit: the positives all pass, so a lazy check says ship it, but two-thirds of the near-misses now fire. Draft B overcorrected and lost a positive. Draft C is the one you ship. Note that you cannot get from A to C without the negatives, because A looks perfect on the positive side alone.

Keep the bar fixed across the whole catalog. If one skill is allowed a 20% positive miss rate and another is held to zero, nobody can compare results across skills, and reviewers start negotiating with the test instead of the description. Record the bar in the repository next to the case files.

For a team running many skills through a single endpoint, Quanta Inference gives you one OpenAI-compatible API key for the model calls in the harness, with prepaid pricing in US dollars, so a regression run across text and routing calls is billed from one account.

The pass bar will fail descriptions that are too vague or too broad, so it helps to know what the passing shape looks like. The description guide recommends:

• Imperative phrasing. "Use this skill when..." rather than "This skill does...". The agent is deciding whether to act, so tell it when to act.
• User intent, not implementation. Describe what the user is trying to achieve.
• Pushing toward coverage. List contexts where the skill applies, including when the user does not name the domain, for example "even if they don't explicitly mention CSV or analysis."
• Concision. A few sentences to a short paragraph, within the specification's character limit.

Those four can pull against each other, and the near-miss set is how you find the tension. Adding a coverage clause raises positives and false positives together. Adding an exclusion clause lowers both. The regression set tells you which side of the trade you landed on.

A practical editing loop:

1. Change one clause at a time. Two changes at once and a failing case tells you nothing about which edit caused it.
2. Run the full set, positive and negative.
3. If a positive regressed, add coverage wording aimed at that case's intent.
4. If a near-miss fired, add a scope clause that excludes the adjacent job, phrased in terms of what the user wants rather than negative instructions (an agent reads "use this when the task is extraction and not ETL" more reliably than "do not do ETL").
5. Re-run until both rules pass, then commit the description and the case file together.

Keep the case file in version control next to SKILL.md. The skill authoring guide emphasizes concision because the description is pre-loaded for every skill in the catalog, so a few lines saved in each description also reduce the routing confusion that the SkillRouter work measures.

Borrow one more idea from production tool-selection design: separate "which capability is needed" from "which implementation handles it," as the two-step tool selection disclosure describes. In practice, that means each must-trigger case should be labeled with the capability it needs, so when it fails you know whether the problem is your description or a genuinely missing capability. If the case needs something no skill in the catalog offers, it belongs in a backlog, not in another skill's negative set.

If you would rather start from a working reference than from a blank SKILL.md, the trigger description and body patterns behind Quanta Skills can be installed into a coding agent directly, which gives you a description shape to copy and then a regression set to hold it to.

The set only pays off if it runs at the moment the risk appears: on the description diff. Three ways to wire that up, from lightest to heaviest.

Manual, before you ship. Copy the full description into a fresh session, run each case one at a time, and record fire or no fire. Fifteen cases take minutes. This is the fallback when nothing else exists, and it is far better than shipping untested wording.

Scripted, on a pre-commit hook or CI job. Call your model with the skill catalog loaded and parse whether the skill was selected. The harness should fail the build on the two rules: any positive miss, or more than the allowed number of near-miss fires. Braintrust's guide to prompt testing describes this shape of check: run the same representative cases against the same expectations whenever anything in the configuration changes, and keep a stored baseline so score differences can be traced to the change that caused them. For skills, the configuration includes the description text, the model, the other skills in the catalog, and the harness that assembles them.

Scheduled or catalog-wide. Once you have more than a handful of skills, a new skill's description can change routing for old skills that nobody touched, which is the catalog regression effect. Running every skill's set whenever any description in the catalog changes catches that class of break. Run each skill's set alone against a fixed catalog snapshot, plus one pass with the full catalog.

Three operational rules keep the set honest over time:

• Version the cases with the skill. A description edit and its case-file update go in the same change, or the review cannot tell whether the bar still holds.
• Never delete a failing near-miss to make the bar pass. If a near-miss is genuinely wrong, say why in the change: the case is mislabeled, or the neighboring skill changed. Deleting cases to go green is the one failure mode that makes the whole set useless.
• Add a case whenever production surprises you. When a user reports that a skill fired for a task it should not have handled, add that request as a near-miss before fixing the description. That is how the set stays aligned with real failures, the same feed-production-failures-back approach prompt testing practice recommends.

A short run log keeps the history readable:

skill: csv-analysis
description-sha: 9f2c1a
catalog-sha: 3b7d0e
model: <the one your harness uses>
positives: 10/10
near-miss fires: 1/5
bar: PASS (all positives, <=1 near-miss fire)

If your team runs several skills across different business functions, the trigger descriptions, bodies and regression cases for each are the kind of asset Quanta Experts package with each expert, so the work of testing a description is done once and reused rather than rebuilt per project.