A Golden Set for Intent Classification: Building and Keeping It

A label is only useful if it reproduces. The test is simple: hand the same 50 queries to two annotators working from the rulebook alone and compare. Write rules as a priority-ordered decision list, not prose descriptions of each intent, so that when two rules could fire the earlier one wins.

Add explicit tie-breakers for known ambiguities. A query asking for a price and a demo is transactional, because the user is ready to act the Siteimprove intent guide. A query saying "best X" or "X vs Y" is commercial, because the user is comparing before deciding the Semrush keyword types guide. A query naming your product plus a support action is support even when phrased as a question. A query with no recognizable topic and no recognizable action is out_of_scope, and you keep the row.

This matters more than it sounds because heuristics misclassify. A "how to" query can land on product pages, and a query an SEO tool tagged commercial may be ranking educational content the Grow and Convert intent guide. Your human labels encode what you actually want, which is why they must be rules and not vibes.

When agreement drops below the threshold, do not relabel quarreling rows by fiat. Fix the rule, bump the version, re-annotate the affected rows. Each resolved disagreement becomes an edge case you would otherwise have met in production.

Split the labeled pool once, by row, into a development split (fair game while tuning) and a locked test split you never tune against. Record the split's row count, label distribution, rulebook version and a hash of the sorted row IDs in a manifest, and treat any change to those as a new dataset version.

{
  "golden_set": "intent-golden",
  "version": "v3",
  "rows": 812,
  "split": {"dev": 650, "test": 162},
  "label_distribution": {"support": 344, "commercial": 201, "transactional": 158, "informational": 79, "out_of_scope": 30},
  "rows_sha256": "<hash of sorted row ids>",
  "labeling_rule_version": "rules-7"
}

Three rules keep the split honest. If a row moves between dev and test, that is a new version with a new hash, and the old numbers are not comparable. Never add a rule derived from studying test rows; if you relabel, relabel both splits. Report dev and test separately and put the test number in the release note. A team that tunes a prompt until the test score rises and then reuses the same test set next cycle has turned the test set into a development set with extra steps. Teams that layer few-shot examples into prompts and then evaluate on those same examples are reading the answer key the practical intent classification walkthrough. Prompt examples come from dev, or from generically correct paraphrasings, never from test.

Name the events that force a full rerun, in writing, in the same checklist as your deploy steps: any change to the taxonomy, the prompt, the model or version serving it, the labeling rules that alter labels, or preprocessing that touches query text such as lowercasing or Unicode normalization. Add a fixed calendar cadence so slow drift is caught even when nothing intentional changed.

Cost is not the obstacle. Batch inference is commonly half the price of synchronous calls for workloads that do not need an immediate answer, and scoring a frozen set is exactly that the batching guide. If your prompt carries a long stable prefix, prompt caching cuts the repetition the Claude prompt caching documentation. Neither trick makes the evaluation cheap in the way that matters: the expensive part is the label.

Then read results off a matrix of predicted rows by true columns, and read off-diagonal cells as pairs:

Commercial leaking into transactional (5 of 48, about 10%) may be tolerable, since both route to a sales surface. Transactional leaking into commercial (6 of 38, about 16%) is worse, because buyers are sent to comparison content. Support leaking into out_of_scope (4 of 75) means customers are told their request is not understood.

Turn that into a decision rule. Compute per-intent recall and precision for the previous release and the candidate. Block the deploy if any intent's recall drops more than your tolerance (5 percentage points is a reasonable default) or if any intent loses more than a tenth of its correct rows. Permit tiny dips only with a written note from that intent's owner plus a fix ticket. Never accept a trade that gains overall accuracy while collapsing a small intent, because small intents are usually high-value or high-risk.

For wiring it in, the Langfuse cookbook walks an intent classification pipeline end to end and is a useful reference for attaching scores to traces so each result ties back to a prompt and model the Langfuse pipeline cookbook. Store raw model output for every row, not just the parsed label, because prompt changes often shift output format and you will want to re-parse old runs after a bug. And when you compare models through one API, keep the model identifier and prompt hash in every run record, which is exactly what a single OpenAI-compatible, prepaid endpoint lets you do (https://quantaintelligence.ai/inference). If you would rather have the harness built inside the surrounding app, describing it to Quanta hands it to a specialist who builds it with you (https://quantaintelligence.ai/).

Your classifier serves 200,000 queries a month and your set has 1,000 rows: 350 support, 250 commercial, 200 transactional, 150 informational, 50 out of scope. Release A gets 60 support rows right. Release B gets 52 right, even though overall accuracy rose from 812 to 830 rows. Support recall fell from roughly 86% to roughly 83.6%, about 2.4 points, under a 5-point tolerance. Commercial, however, fell from 47 correct to 41, also 2.4 points but a 12.8% relative loss, so the relative test blocks the deploy until there is a named fix.

Weigh the traffic. Support is 35% of volume, so a 2.4-point recall loss routes roughly 1,680 support queries a month wrongly (200,000 × 0.35 × 0.024). That number, not the accuracy delta, belongs in front of a product owner.

Now the traps. Synthetic examples written in a prompt or by a model are a second training set; check your sampler for raw, ugly log text. Few-shot examples copied from test inflate scores; diff the prompt against test rows. Quietly adding three edge cases and comparing to last month's score breaks comparability; check the manifest hash. One aggregate accuracy number hides the pair of intents that flipped; read the matrix every time. Relabeling without a version makes two runs incomparable; rules-8 against rules-7. And a rare intent with 12 test rows cannot be measured with confidence, so grow it deliberately or mark it unmeasured rather than declaring it fine.