Classifying Requests Before They Reach the Router

The schema is where most classification projects fail, and it is the part you cannot fix later with more data. Keep it small enough that two careful people agree on a label in under ten seconds.

Intent. Pick five to ten values that map to distinct downstream behaviour. If two intents would route to the same model, same tools, and same prompt template, merge them. A working set for a customer-support surface looks like this: account_lookup, billing_question, refund_request, technical_troubleshooting, policy_question, data_export, human_handoff. Seven values, each with a different handling path. The eight-value lists people start with usually contain two pairs that are the same intent wearing different names.

Risk. Three tiers, defined by the action rather than the topic.

The example column matters more than the definition column. Labelers disagree about abstractions and agree about examples.

Confidence and the abstain band. Your classifier should emit a score per label and a single decision: assign, or abstain. An abstention sends the request to the router's default (usually the strong model) plus a log entry for review. Set the abstain threshold by cost, not by accuracy: if a wrong cheap route costs more than an unnecessary frontier call, bias the threshold toward the strong model. That asymmetry is the one rule that survives every case study in the routing literature (the routing analysis from Adnan Masood).

Do not add a topic label. Teams consistently want one, for dashboards. Build it from the intent and risk pair instead. Every extra output head is another thing to label, calibrate, and monitor.

You need roughly a few thousand examples to train a small encoder well. The cheap path is a blend: half sampled from logs, half written by hand.

Start from logs that already carry an outcome. Requests you have already served include the tool call the agent made, the model that answered, and whether a human escalated. Those outcomes are weak labels. If the agent called issue_refund, the intent was refund_request and the risk was high. Weak supervision is a genuinely useful bootstrap, though a controlled comparison found that weakly supervised classifiers lose on precision relative to LLM-assisted labelling even when they win on recall (the SIGIR study on weak supervision versus LLMs). Use the weak labels to fill the head of the distribution, then correct a sample by hand.

Sample by stratum, not at random. Pull 150 examples per intent and 100 per risk tier. A uniform sample of logs gives you thousands of policy_question rows and perhaps fifteen data_export rows, and the classifier will learn to predict the majority class and nothing else.

Write the tail by hand, in pairs. For each intent, write ten examples with the obvious phrasing and ten with the phrasing a frustrated user actually types. Pairs where only one word differs (a negation, a number, a name) teach the model what does not matter. Few-shot work on intent classification consistently finds that data augmentation and contrastive objectives help most when examples per class are scarce, which is exactly the tail (Microsoft's few-shot intent study).

Label in a fixed order, with a rubric. Intent first, then risk, then a short note if the two labelers disagreed. Disagreements are the most valuable rows in the set: they tell you which boundary in your schema is under-specified. Fix the schema, not the labelers.

Hold out a challenge set. Twenty percent of the hand-written tail goes into a set you never train on. Report accuracy on it separately from accuracy on the log sample. The gap between the two numbers is your real quality estimate, and it is the number to quote when someone asks whether the classifier is ready.

A small fine-tuned encoder is usually the right shape. It is cheap per call, fast enough to sit in the request path, and easy to version. The work is in calibration, not architecture.

Split the job into two heads rather than one joint label space. Intent is a multi-class problem over five to ten values; risk is a three-class problem. Two heads means you can tune the risk threshold without disturbing intent predictions, and a request can legitimately be account_lookup at high risk when it carries an unusual identifier.

Calibrate each head separately. Raw softmax scores from a fine-tuned encoder are usually overconfident, which is fatal for abstention. Fit a temperature on the held-out challenge set, then check the reliability of the abstain band directly: of the requests scored between 0.55 and 0.75, what share were actually correct? If the answer is 0.6 when the score says 0.7, widen the abstain band.

A minimal training and scoring loop looks like this:

# two heads over one shared encoder; calibrate on the challenge split
encoder = load_encoder("small-encoder")
intent_head = Linear(encoder.dim, len(INTENTS))
risk_head = Linear(encoder.dim, 3)

for batch in loader(train):
    h = encoder(batch.text)
    loss = ce(intent_head(h), batch.intent) + ce(risk_head(h), batch.risk)
    loss.backward(); step()

calibrator_intent = fit_temperature(intent_head, loader(challenge))
calibrator_risk = fit_temperature(risk_head, loader(challenge))

def classify(text, tau=0.70):
    h = encoder(text)
    p_intent = calibrator_intent(intent_head(h))
    p_risk = calibrator_risk(risk_head(h))
    i, c = p_intent.max(dim=-1), p_risk.max(dim=-1)
    if c_i < tau or c_r < tau:
        return {"abstain": True, "reason": "low confidence"}
    return {"intent": INTENTS[i], "risk": RISK[c]}

Measure the classifier the way you will use it. Overall accuracy hides the errors that matter. Report intent accuracy per class, risk recall at the high tier (you care about high-risk misses far more than high-risk false alarms), and the abstain rate. If high-risk recall is below your bar, lower the risk threshold and accept more abstentions; the abstention path is the strong model, which is where you wanted high-risk traffic anyway.

Keep a durable record per call: the text hash, the model version, both labels, both calibrated scores, the abstain flag, the routing decision, and the model that served the request. That row is the raw material for everything in the next section.

Labels arrive late, which is why drift detection for this classifier has to lean on proxies. Data drift is a change in the distribution of inputs; prediction drift is a change in the distribution of outputs; both are useful signals when ground truth is delayed (the Evidently guide to data drift). For a pre-router classifier, three monitors cover almost everything.

Label mix drift. Track the share of each intent and risk tier per day, per tenant, per entry point. A sudden jump in billing_question is usually a product change, a marketing campaign, or a new client integration. Alert on a shift of more than a few percentage points relative to a trailing baseline.

Abstain rate drift. A rising abstain rate is the earliest signal that input text has changed in ways the encoder does not recognise. It is cheap to compute and it fires before accuracy numbers move, because accuracy needs labels. Watch it per intent, not just in aggregate: an abstain spike confined to technical_troubleshooting points at a vocabulary change in one product area.

Score-to-outcome divergence. This is the monitor most teams skip. For every routed request, log what actually happened: did the agent call the tool the intent predicted, did a human escalate, did the user rephrase and ask again within a minute, did the request fail validation. Then compare. If the classifier said risk=low and the downstream tool call wrote a record, the risk head was wrong. If requests labelled policy_question escalate to a human at twice the rate they did last month, the intent definition has drifted from reality.

The comparison is what makes the classifier auditable, and it catches the failure mode that labels alone never will: a schema that no longer matches the business. It also gives you a free label pipeline. The downstream outcomes are labels you did not pay for, arriving hours or days after the request, ready to fold into the next training set.

Restrict what the comparison covers. A tool call is a reliable weak label for intent. Escalation is a reliable weak label for "this needed a stronger model or a human", but it is noisy about which. Do not treat user rephrasing as ground truth for anything except "the first answer failed", which is itself useful. Pair the tooling with per-request token and cost logging, since attribution is the prerequisite for knowing whether the classifier is saving what you think it is (Worklytics on token tracking).

Take a support assistant serving 1,000,000 requests a month. The router has two tiers: a small model and a frontier model. Per million tokens, the small model costs a fraction of the frontier model, and the gap is on the order of an order of magnitude or more (NeuralTrust on model right-sizing). Set the exact per-token figures from your provider's current price list before you run the arithmetic; the shape of the calculation is what transfers.

Assume the average request is 800 input tokens and 300 output tokens, so 1.1 billion tokens a month in and out combined. Suppose sending all of it to the frontier model costs 1,000 cost units. The classifier sends 70 percent of traffic to the small tier at roughly a tenth of the unit price, and 30 percent to the frontier tier: the blended bill lands near 0.37 of the original on model spend. The classifier itself, running on a small encoder over a short request, adds a small fraction of a unit per call, and you should measure it rather than assume it.

Now the part the spreadsheet misses. The classifier will be wrong on some share of requests, and wrong cheap routes can cost more than unnecessary expensive ones. Model that explicitly: if 2 percent of the 70 percent low-tier traffic gets a wrong cheap answer that a user has to retry, you pay for the retry plus the trust. The rule that survives: bias the abstain threshold toward the frontier tier until production evidence says otherwise, then move it down one notch at a time and watch the divergence monitor.

One more number worth computing before you build: the share of traffic that is genuinely high risk. In most support surfaces it is small, single digits. If your sampled logs say 40 percent, your risk definition is too broad and the tier has stopped discriminating. Narrow it until high risk means "a human should probably see this".

Once the tags exist, the routing policy is configuration, and the classifier is doing the job a gateway is meant to do: applying one set of decisions across every provider without touching application code. If you would rather not run the model layer yourself, Quanta Inference gives you text, image, video, audio and embedding models plus a hosted way to run classification through one OpenAI-compatible API key, which keeps the classifier and the downstream models on the same bill.

Treat the classifier as a product with a version, an owner, and a rollback.

Version everything together. Encoder, calibration temperatures, thresholds, intent list, risk rubric, and routing policy travel as one artifact. When someone asks why a request went to the frontier model in March, you should be able to reload the exact policy that was live.

Shadow before you enforce. Run the classifier on live traffic for a week, log its tags, and route everything to the default. Compare the tags against the downstream outcomes before you let them change a single routing decision. This step catches rubric problems that no offline evaluation will.

Review abstentions on a schedule. A weekly sample of 50 abstained rows, read by a person, will surface schema gaps faster than any metric. When the same kind of abstention shows up three weeks running, it is a new intent, not noise.

Retrain on a cadence, not on a hunch. Monthly is usually enough for a small classifier on a stable domain. Each retrain takes the previous month's outcome-labelled traffic, re-runs the challenge set, and ships only if the challenge-set numbers hold. If they drop, the schema changed and no amount of retraining will fix it.

Give the policy a circuit breaker. If the abstain rate crosses a hard ceiling in any hour, bypass the classifier and route everything to the default tier. A classifier outage should degrade cost, never safety.

Keep the human in the loop for high risk. The classifier's job on a risk=high request is to make the escalation obvious, not to decide it. Route it, log it, and let the confirmation step live in the application layer where it can be tested independently.

Teams that run this well end up with three assets they did not have before: a routing policy that can be reasoned about, a labelled dataset that grows for free from outcomes, and an audit trail that answers the question every cost review eventually asks. If you want the plumbing and the classifier hosted together, the Quanta platform takes a described system and builds it with you; if your team would rather have someone review the schema and the monitoring plan before you commit, a consulting call covers that.