How to Tag and Route Inbound Questions by Topic Without an LLM

A rule-based classifier costs nothing per prediction and resolves in under a millisecond, while an LLM classification call typically costs between one cent and ten cents per prediction with 200 to 2000 milliseconds of latency per the Institute of AI PM's classification guide. At a thousand questions a day, that gap is the difference between a rounding error and a line item.

Cost is only half the argument. Rules are deterministic and explainable: when a question routes to the billing queue, you can point to the exact phrase that sent it there. A model gives you a label and a confidence score, and neither is stable across identical inputs unless you pin the temperature. OpenAI's own agent guidance recommends deterministic solutions before reaching for a model, reserving LLM control of a workflow for cases where rule-based approaches genuinely fall short in its practical guide to building agents.

There is a second cost argument. Research comparing fine-tuned encoders against prompt-based LLM classification found encoder models matched or beat LLM accuracy on fixed-label classification while operating at one to two orders of magnitude lower cost and latency in this arXiv study on cost-aware model selection. That is for trained models. A hand-written rule set is cheaper still, with its only real cost being the maintenance of the rules themselves.

The decision rule: use rules for topics you can enumerate and that change slowly, and reserve model calls for the tail. The Institute of AI PM frames this as a PM signal: if your team can list the classification logic in a meeting, start with rules in the same guide.

A rule is a list of patterns plus a weight. Start with three pattern types and keep them strict:

• Exact phrases carry the most weight. Multi-word phrases like "cancel my subscription" or "reset my password" are far less ambiguous than single words.
• Single keywords carry less weight and are riskier. "Refund" in "when will my refund arrive" is billing, but "refund" in "your refund policy is terrible" is a complaint.
• Negative patterns subtract weight. If a phrase sends something to the wrong queue, add it to the negative list for the topic it falsely matches rather than hunting for new positive rules.

Give every rule an explicit priority number. Priority resolves ties and handles overlaps: a question containing both "log in" and "billing" should probably route to account access first, because no billing conversation can proceed if the person cannot sign in. Encode that decision in the priority field, not in the rule text, so it is visible and editable in one place.

Here is a minimal implementation in Python that scores phrases against a set of weighted rules:

RULES = [
    {"topic": "account_access", "priority": 1, "weight": 0.5,
     "phrases": ["can't log in", "cant login", "reset my password", "locked out", "sign in failed"],
     "negative": ["api key"]},
    {"topic": "billing", "priority": 2, "weight": 0.4,
     "phrases": ["refund", "invoice", "charged twice", "update payment", "billing address"],
     "negative": ["refund policy"]},
    {"topic": "shipping", "priority": 2, "weight": 0.5,
     "phrases": ["where is my order", "tracking number", "not arrived", "delivery status"],
     "negative": []},
]

def score(text):
    text = text.lower()
    scores = {}
    for rule in RULES:
        s = 0.0
        for phrase in rule["phrases"]:
            if phrase in text:
                s += rule["weight"]
        for phrase in rule["negative"]:
            if phrase in text:
                s -= rule["weight"]
        scores[rule["topic"]] = max(s, 0.0)
    return scores

Keep the scoring simple and additive. You want a number a human can recompute by hand during a review, because that is what makes the system auditable.

A threshold converts scores into decisions. A common starting point is a score of one full phrase match, meaning any single positive phrase from a topic routes the question to that topic. Test the threshold against a labeled sample of a few hundred questions before you trust it.

Three outcomes matter:

1. One topic clears the threshold and is strictly highest. Route it. This is your happy path and it should handle the bulk of volume once tuned.
2. Two topics clear the threshold and tie, or the margin is tiny. Do not guess. Send it to the model with the scores attached. A near-tie usually means a genuinely ambiguous question, and the model has a real chance of adding information.
3. No topic clears the threshold. Send it to the model. These are your misses, and they are the raw material for next week's rules.

Track the share of questions hitting each outcome. If the rule path is not catching most traffic, your phrases are too narrow or your threshold is too high. If the model path never fires, your rules may be too loose and quietly misrouting.

The model call should not start from scratch. Pass it the original question, the top topics with their scores, and the list of queues. Ask it for structured output: a topic, a queue, and a short reason. This turns the escalation into a decision about the specific ambiguity rather than a fresh classification that might disagree with obvious signals.

Structured output also makes logging trivial and keeps the prompt short. Limiting the defined options helps accuracy: guidance on intent prompts suggests keeping the list of candidate intents under about twenty per prompt per IrisAgent. The same applies to queues: if you have more than twenty, split the routing into two stages rather than one long prompt.

When you are choosing which model handles the fallback, you can use a cascade. Run a cheap model on everything that misses the rules and escalate to a stronger model only when the cheap one is not confident; a well-tuned cascade can match a larger model's answers most of the time while cutting the large majority of cost as described in this walkthrough of model cascades. Model routing that assigns each question class to the cheapest model that meets its quality bar follows the same principle per Braintrust's routing guide. If you would rather not manage several providers and keys yourself, a single OpenAI-compatible endpoint for every kind of model, prepaid and priced in US dollars, is what Quanta Inference offers (https://quantaintelligence.ai/inference).

The model is not the end of the pipeline; it is a data collection point. Log every fallback with the original text, the rule scores, the model's answer, the queue it chose, and later, whether that queue was right. Without a ground-truth check, your log tells you what the model said, not whether it was correct.

Review the log weekly with three questions in hand:

• Which questions fired no rule? Look for repeated phrasings. Promote the top recurring phrase into the matching topic's rule list.
• Which questions fired a rule but landed in the wrong queue? Add the offending phrase to the negative list for the topic that wrongly caught it.
• Which ties keep recurring? A persistent tie often means two topics should merge, or a priority order needs to change.

This loop is the whole point of the architecture. Every model call that becomes a rule is a cost that disappears and an outcome that becomes explainable. The same promotion cycle applies to your help content: issues that repeatedly reach an agent are candidates for a new article, and support teams that measure content health against actual contact reasons catch gaps earlier as Zendesk notes in its knowledge base guidance.

Routing sends a question to a queue, and the queue answers it. If your answers come from a knowledge base with retrieval, the quality of the chunks feeding that retrieval matters as much as the classifier. Chunk size affects accuracy measurably: an NVIDIA experiment across several datasets found page-level chunking achieved the highest average end-to-end retrieval accuracy at 0.648, while extreme chunk sizes at both ends underperformed in NVIDIA's chunking study. Medium-sized chunks tended to win, with optimal size varying by query type: factoid questions favored smaller chunks and analytical questions favored larger ones.

Two practical consequences for routing. First, keep each knowledge base topic self-contained, so that a correctly routed question has a decent chance of finding an answer. Second, write content that avoids vague pronouns and covers one topic per article, because retrieval compares question meaning against chunk meaning and mixed-topic chunks dilute both according to Zendesk's content optimization guide. A clean rule set plus clean chunks is what makes a route actually resolve, not just land.

Suppose you receive 1,000 inbound questions a day. Your rules cover 78 percent and route them to the right queue with a two percent error rate, verified against a labeled sample. The remaining 220 questions go to the model fallback.

At a conservative one cent per classification call, fallback costs 2.20 dollars a day, or roughly 66 dollars a month at 30 days. Running every question through the model instead would cost 10 dollars a day, or about 300 dollars a month, before any retries. The savings are modest at this volume, which is exactly the point: the real justification is not the money at small scale, it is the audit trail and the ability to explain any route without opening a model console.

Now scale it. At 100,000 questions a day, rules handle 78,000 at no marginal cost. Fallback handles 22,000 at one cent each, or 220 dollars a day. The all-model approach costs 1,000 dollars a day. The gap is 780 dollars a day, and it grows with volume. If you spend one hour a week promoting misses into rules and push rule coverage from 78 to 85 percent, fallback volume drops from 22,000 to 15,000 questions a day and you save another 70 dollars a day.

The numbers move with model prices, which change constantly, so treat the cost per call as a variable you read from your own invoice rather than a fixed number. What does not change is the shape: rules scale at zero marginal cost, model calls scale linearly, and every rule you add is permanent savings.