Routing Inbox Messages to the Right Queue with Rules and a Model

Rules fail in a predictable pattern. They handle the cases you thought of and nothing else. A keyword filter on "refund" catches "I want a refund" and misses "you charged me twice and I'd like my money back." A rule on the word "invoice" catches invoices and also catches a customer complaining that an invoice was wrong, which is a different queue. The customer support literature is blunt about this: traditional keyword systems break the moment a customer uses slightly different wording or does not use the exact term you configured (How to use AI to classify or tag support tickets).

That is the residual. After your rules run, whatever is left is the set of messages whose destination depends on meaning rather than on a field. That is exactly what a classifier is for. Intent classification, the task of mapping a piece of text to one of a predefined set of purposes, is the standard framing here, and it can be done with rules, with machine learning, or with both (Intent classification: understanding text with the power of AI). You already have the rules layer; the model covers the rest.

There is a second reason to keep the model's job narrow. Classifiers are described as learning the intent behind a message, reading "I can't get into my account" and "my login isn't working" as the same problem (A practical guide: How to use AI to classify or tag support tickets). That is a real gain, and it is also a source of failure: the model will confidently produce a category for a message that belongs to no category at all. AWS's reference architecture for automated email response handles this by sending the email to a support team for a manual reply when the predicted intent does not match an existing knowledge base entry (Automate email responses using Amazon Comprehend). Copy that pattern. A model with no escape hatch is a model that will silently misroute.

The architecture that holds up is a cascade. Rules first, model second, confidence bands last, and a human path always at the end. Here is the shape of it in code:

def route(message, account, rules, classifier):
    # Tier 1: policy and compliance. These rules cannot be overridden.
    for rule in rules.tier(1):
        if rule.matches(message, account):
            return Route(queue=rule.queue, reason=f"rule:{rule.id}")

    # Tier 2-5: identity, account, language, SLA.
    for rule in rules.tier(2, 3, 4, 5):
        if rule.matches(message, account):
            return Route(queue=rule.queue, reason=f"rule:{rule.id}")

    # Residual: ask the model.
    prediction = classifier.predict(message.text)
    if prediction.confidence >= 0.90:
        return Route(queue=prediction.queue, reason="model:high")
    if prediction.confidence >= 0.60:
        return Route(queue="review", reason="model:mid",
                     suggested=prediction.queue)
    return Route(queue="triage", reason="model:low")

Three numbers do most of the work: the high threshold, the low threshold, and the review queue's staffing. Everything else is bookkeeping. The high threshold is the point above which you are willing to let the model act alone. The low threshold is the point below which you stop pretending the model knows anything. Between them sits the review queue, where a person sees the model's suggestion and either accepts it or corrects it.

Two design choices make this work in practice. First, log the confidence score, the model version, the rules that ran, and the final destination on every message. You will need that log to tune the thresholds, and you will need it when someone asks why a message went to the wrong team three weeks ago. Second, treat the review queue as a product, not a dumping ground. It needs an owner, a target turnaround, and a visible count. An unowned review queue becomes a second catch-all within a month.

The same architecture applies to structured extraction, not just classification. If your router pulls fields out of a message (an order number, a date, a currency), structured outputs can look correct while being wrong. Cleanlab's benchmark found that trust scores detected LLM output errors with about 25% greater precision and recall than alternatives including LLM-as-a-judge and token log probabilities, and it frames the goal as focusing human review on roughly 1 to 5% of cases while automating the rest (Real-Time Error Detection for LLM Structured Outputs). That framing is the right one for routing: automate the bulk, and spend human attention where it changes the outcome.

Thresholds are not a matter of taste. They follow from the cost of each kind of mistake. Take a support inbox with 10,000 messages a day. Suppose your rules resolve 55 percent, leaving 4,500 for the classifier. Among those, assume 60 percent genuinely belong in one of five queues and 40 percent are ambiguous or belong elsewhere.

Now set numbers. Suppose the classifier returns confidence at or above 0.90 on 3,000 messages, and on those it is right 96 percent of the time. That is 120 misroutes a day. Suppose between 0.60 and 0.90 it covers 1,200 messages at 85 percent accuracy, which is 180 errors if you route them automatically, or 180 corrections if you send them to review instead. Below 0.60 sit 300 messages where accuracy is close to a coin flip.

The decision rule falls out of the arithmetic. If a misroute costs a support agent 10 minutes to detect and re-route, and the review queue costs 2 minutes per message, then a message at 85 percent accuracy costs 1.5 minutes of expected rework (0.15 × 10) against 2 minutes of guaranteed review. That is close enough that other factors decide it: how fast the review queue turns around, and whether the message is time sensitive. If a misroute instead risks an SLA breach or a wrong refund, the expected cost of the error jumps and automatic routing stops being defensible, because the damage is not reversible.

This is the same reasoning used to place human review gates in AI workflows, which sorts steps by reversibility and blast radius: gate anything irreversible with wide reach, let reversible low-impact work run on its own, and monitor the middle (Human in the loop: when to keep people in AI workflows). Routing a message to the wrong internal queue is usually reversible and low reach, so a confidence band is enough. Routing a message that triggers a refund, a permission change, or an outbound customer email is not, so those actions need a person in front of them regardless of confidence.

Write the thresholds down with the reasoning, then revisit them on a schedule. A threshold without a stated error cost is just a number someone liked.

Every routing system misroutes. The question is what happens next. Four mechanisms cover most of it.

First, a catch-all queue with a named owner. Messages below the low threshold go there, and the owner's job is to route them, not to resolve them. Track two metrics on that queue: volume as a share of total traffic, and median time to re-route. A catch-all that grows is telling you a rule is missing or a threshold is wrong.

Second, an escalation path from every queue. An agent who receives a misrouted message needs a one-click way to send it to the right team, and that action must be logged with the original message, the original destination, the new destination, and the actor. That log is your labeled data.

Third, a feedback loop that turns corrections into training data and into rule candidates. Each correction is a row with the message text, the model's prediction, its confidence, and the human's answer. Once a pattern appears repeatedly, promote it to a rule. This is the mechanism that keeps the review queue from growing forever: the system gets better at the named cases, so the residual shrinks.

Fourth, a distinction between the three oversight postures, because most teams blur them. In a human in the loop design the model proposes and a person approves. In a human on the loop design the model acts and a person monitors and can roll back. In a fully autonomous design the model acts and humans audit after the fact (Human in the loop: when to keep people in AI workflows). Use all three at once, routed by the stakes of each message.

One warning about measurement. If you evaluate routing quality using email open rates, you will be reading noise. Apple's Mail Privacy Protection preloads tracking pixels, so emails sitting in an Apple Mail inbox are reported as opened whether or not anyone read them, which inflates open rates and makes open-based metrics unreliable (Apple Mail Privacy Protection FAQs). Mailchimp's guidance is to measure clicks, bounces, unsubscribes and conversions instead, and to expand automation triggers beyond opens (Apple Mail Privacy Protection FAQs). The same applies to routing evaluations that use engagement as a proxy for correctness: clicks survive MPP, opens do not. Benchmark data on open and click rates varies widely by industry, with one benchmark report showing click rates in the low single digits for most sectors (Email Marketing Benchmarks), which is a reminder that an open-based metric will look healthy long after routing quality has decayed.

The parts you need are a rules engine, a classifier, a confidence gate, a review queue, and a log. All of them can be assembled from scratch, and none of them is where the interesting work is. The interesting work is the rule order, the threshold arithmetic, and the correction loop.

Two shortcuts are worth knowing. The first is to keep the model layer boring by calling every model through one interface. If your classifier is one endpoint, your embedding-based similarity search is another, and your extraction model is a third, you have three sets of credentials to rotate and three per-message cost models to explain. Quanta Inference puts text, image, video, audio and embedding models behind one API key, OpenAI-compatible and prepaid in US dollars, which removes that overhead (https://quantaintelligence.ai/inference). The second is to get the workflow itself built rather than hand-rolling the queue and the review UI. If you would rather describe the routing system you need and have someone build it with you, Quanta works that way (https://quantaintelligence.ai/).

Whichever route you take, budget time for the unglamorous part: seed the rule table with the cases your support team already knows, and set up the correction log on day one. A routing system without a correction log is a system that cannot improve, and it will look exactly as good on day 400 as it did on day one.