Writing an Intent Labeling Guide Two Raters Can Apply

Take a support inbox with four intents: Order Status, Returns, Billing Dispute, and Product Question. Here is the rules table as a single portable block. Note that every row carries a precedence number, so a message that matches two rows has a fixed answer.

Now the pilot batch. Twelve messages, labeled independently by Rater A and Rater B using only the table above, with no discussion between them.

Eight of twelve match. That is 66.7 percent agreement, and the four misses are all in the Returns and Billing Dispute boundary, which is exactly where the tie-breaker was written too loosely.

Percent agreement is the count of matching labels divided by the total. Here it is 8 / 12 = 0.667. That number alone is misleading, because some categories get used far more often than others. Cohen's kappa corrects for agreement that would happen by chance given each rater's distribution of labels as the Wikipedia entry on Cohen's kappa sets out.

Build the confusion table first. Rows are Rater A, columns are Rater B.

Observed agreement is the diagonal sum over N: (4 + 2 + 2 + 2) / 12 = 0.833. That is higher than the crude match count suggested, because both raters spread their labels the same way overall.

Expected agreement is the sum of each category's row total times its column total, divided by N squared:

pe = (5*5 + 3*3 + 2*2 + 2*2) / (12*12)
   = (25 + 9 + 4 + 4) / 144
   = 42 / 144
   = 0.2917

kappa = (0.833 - 0.2917) / (1 - 0.2917)
      = 0.5417 / 0.7083
      = 0.765

A kappa of 0.765 sits in the "substantial" band of the Landis and Koch interpretation table, which labels 0.61 to 0.80 as substantial and 0.81 to 1.00 as almost perfect per the kappa interpretation summary. Clinical fields often require 0.80 or above, and that same source warns that the labels are conventions, not laws. If your target was 0.80, this guide fails. If it was 0.70, it passes with room.

The distinction between percent agreement and kappa matters more than the arithmetic. A worked example from the same reference shows 90 percent agreement collapsing to a kappa near 0.33 once you account for a lopsided distribution. When one intent dominates your inbox, raw agreement will look flattering and kappa will expose it.

Pick the threshold and write it into the guide document before the first message is labeled. Three inputs set it:

• Downstream cost of a wrong label. If labels feed a routing rule that sends a message to the wrong queue, the cost is a delay. If labels train a classifier you will ship, errors propagate.
• Category count. Kappa gets harder to move as categories increase, because chance agreement falls and near-miss confusions multiply.
• How close the intents genuinely are. Returns and Billing Dispute share vocabulary by nature. Two intents that share words will never reach the ceiling that two unrelated intents reach.

A practical working bar is 0.80, with a documented fallback: if kappa lands between 0.70 and 0.80 and every remaining disagreement traces to one identifiable message pattern, accept the guide, log the pattern, and monitor. If kappa is below 0.70, stop and rewrite.

The logic is the same one used for confidence thresholds in AI systems, where the cutoff is a business decision layered on top of a model score, not a property of the model per LlamaIndex's explanation of confidence thresholds. Their guidance applies directly to human labeling: measure your baseline error before you tune, because a false positive in one domain is not the same cost as in another.

Record the decision in a one-line table so nobody relitigates it later.

That last row is the insight most guides miss. Very low kappa usually means your taxonomy is broken, not your wording. If raters cannot separate Returns from Billing Dispute even after a clear tie-breaker, the two intents may not be separate concepts in the data.

Do not average away the disagreement. Read the off-diagonal cells one at a time and ask which rule would have decided the message.

Here, all four misses are between Billing Dispute and Returns, in both directions. Messages 6 and 9 are the reverse of each other, which is a strong signal: the two raters are not biased, they are reading the same tie-breaker differently. Message 6 reads "Refund for order 44120 still not showing." Rater A applied priority 1 because money is contested. Rater B applied priority 2 because the word "refund" appears. The tie-breaker as written says Billing wins when money is contested, which is vague: a refund is money.

The fix is to make the trigger lexical and observable rather than conceptual:

• Old: "Billing wins over Returns when money is contested."
• New: "Billing Dispute requires an assertion that a charge is wrong, duplicate or unrecognized, stated with a specific amount or the word 'charged'. A refund that is merely delayed belongs to Returns."

Under the new rule, message 6 is Returns and message 9 is Returns, because neither asserts a wrong charge. Message 1 stays Billing because "charged twice" is an assertion. Message 12 stays Billing for the same reason.

This is the discipline that separates a guide from a vibe. Every rewrite should change one rule, name the specific messages it reclassifies, and leave the rest alone so you can attribute the next kappa to that change.

Keep a revision log next to the guide. It is the audit trail that makes the next pilot comparable.

Rerun on fresh messages, not the pilot batch. Raters who have just discussed messages 6 and 9 will label them correctly from memory, and your second kappa will be inflated by recall rather than by the guide. Draw a new sample from the same distribution, same size, same independence rule. For real reliability work, put 50 to 100 messages in each pilot so a handful of confusions does not swing the number; that range also gives you enough off-diagonal cells to see a pattern rather than a coincidence. Inter-rater reliability work in annotation projects makes the same point: low agreement signals that instructions or training need refinement before the dataset moves forward per Ultralytics on inter-rater reliability.

If you are computing this repeatedly, a short script beats a spreadsheet and removes arithmetic from the list of suspects:

from collections import Counter

def agreement(a, b):
    n = len(a)
    pairs = list(zip(a, b))
    po = sum(x == y for x, y in pairs) / n
    ca, cb = Counter(a), Counter(b)
    pe = sum(ca[k] * cb[k] for k in set(ca) | set(cb)) / (n * n)
    kappa = (po - pe) / (1 - pe) if pe < 1 else 1.0
    return po, pe, kappa

rater_a = ["billing","status","returns","product","billing","billing",
           "product","returns","returns","product","status","billing"]
rater_b = ["billing","status","returns","product","billing","returns",
           "product","returns","billing","product","status","billing"]
print(agreement(rater_a, rater_b))  # po 0.833, pe 0.292, kappa 0.765

One caveat worth knowing before you quote kappa in a review: the statistic is unreliable when one category is rare, because expected agreement gets dominated by the majority class. If an intent appears in fewer than a handful of pilot messages, report the per-category confusion counts alongside the overall number and say so explicitly. Fleiss' kappa extends the same correction to more than two raters, which matters once you add a third labeler or a model as a tie-breaker as the Imerit guide to annotator agreement notes.

Once two humans clear the bar, the guide becomes the specification for an automated classifier, and the pilot set becomes a regression test. The practical sequence is unchanged: run the model on messages the humans labeled, build the same confusion table with the model in the column position, and compare. A model scoring well below the human kappa is not automatically wrong; it may be picking up signals your raters discounted, or your raters may be applying an unstated rule. The gap is the conversation.

Two operational notes save money here. First, the guide itself is a stable prefix you send with every classification request, so it is worth structuring it as a fixed block ahead of the variable message; providers price repeated prefixes at a discount, and Anthropic's documentation describes a 5-minute default cache lifetime with refreshes at no extra cost on reuse per the Claude prompt caching docs. Second, classification jobs that nobody is waiting on can run asynchronously for a large discount; OpenAI's and Anthropic's batch endpoints both advertise a 50 percent discount for async processing with results within 24 hours per NeuralTrust's batching guide. Labeling 40,000 historical tickets to enrich a training set is exactly that workload. Live inbox routing is not.

If you want one key across text, image and embedding models for this pipeline, Quanta Inference exposes them through an OpenAI-compatible endpoint.