Ticket Signals That Predict a First-Pass Failure

Lift is the difference between the failure rate in a signal bucket and the base rate, expressed either as an absolute percentage point gap or as a ratio. Both are useful; absolute points tell you how much a routed cohort improves, the ratio tells you how much more likely a failure is inside the bucket.

Here is the decision rule in one sentence: keep the signal only if the lower bound of the confidence interval around the bucket's failure rate is still above the base rate, and the field is populated before routing happens.

The confidence interval for a proportion uses the binomial standard error. Jason Brownlee's walkthrough of reporting classifier performance gives the formula and the constants: 1.64 for 90 percent, 1.96 for 95 percent, 2.58 for 99 percent, applied to the error, or in our case the failure rate, rather than the accuracy. The same page notes the assumptions: independent observations and at least 30 observations in the evaluated group.

Worked example. A queue takes 4,000 tickets per quarter. Base failure rate is 18 percent, so 720 failures. Now split by a candidate signal, body length under 120 characters:

The short-body group fails at 28.0 percent against a base of 18.0 percent, a 10 point lift. The standard error for that group is the square root of (0.28 times 0.72) divided by 400, which is 0.0225. Multiplying by 1.96 gives a half-width of 0.044, so the 95 percent interval runs from 23.6 to 32.4 percent. The lower bound is well above 18 percent. The signal survives.

Now the same test on a signal that looks similar but is not: body length under 60 characters.

Lift looks like 3.1 points, but the standard error is 0.043, and the 95 percent interval runs from 12.7 to 29.5 percent. The base rate sits inside the interval. You cannot tell this bucket apart from the queue. Do not ship the rule; collect more tickets in that bucket or merge it into the wider short-body rule until the sample is big enough.

A quick way to run the same calculation in a script:

from math import sqrt

def lift(n, failures, base_rate, z=1.96):
    p = failures / n
    se = sqrt(p * (1 - p) / n)
    lo, hi = p - z * se, p + z * se
    return {
        "rate": round(p, 4),
        "lift_pts": round((p - base_rate) * 100, 2),
        "ci_low": round(lo, 4),
        "ci_high": round(hi, 4),
        "clears_base": lo > base_rate,
    }

print(lift(400, 112, 0.18))  # short body: clears_base True
print(lift(90, 19, 0.18))    # very short: clears_base False

Three practical cautions. First, run the base rate for the same window as the bucket split, otherwise you are comparing cohorts from different traffic mixes. Second, if you test twenty signals at 95 percent you should expect roughly one false positive by chance alone; either lower the threshold or validate surviving signals on a fresh holdout window. Third, treat the interval as a sanity check, not a proof. It assumes a stable queue, and queues shift with product launches and outages.

The table below ranks the signal families by how often they tend to survive the interval test, based on what practitioners report about escalation and routing signals. The "populated before routing" column is the one that disqualifies more candidates than any statistical test.

A few notes on the table. On prior history, Supportbench's summary of escalation prediction reports that accounts with four or more escalations in 90 days are flagged as higher risk, and that a response gap of more than two days or unresolved threads raise risk. Those thresholds are useful starting buckets, but the article does not say whether they were validated on independent data, so treat them as hypotheses to test on your own queue, not as fixed rules.

On attachments and structured features, SupportLogic's account of feature engineering for escalation prediction describes building features from ticket state over time and from the customer's prior history, then combining and selecting them with automated tools. The useful lesson is that static fields rarely carry the whole signal; ratios and histories (this ticket's age versus this customer's median resolution time, for example) do better. They are also populated before routing, which is why they clear the second gate in the table.

On intent phrasing, the TickIt paper from ByteDance analyzes over 20,000 tickets on Volcano Engine and argues that topic-aware, dynamic ticket state matters because routing to the right team depends on what the ticket is about, and that state changes over a ticket's life. If you build a phrasing classifier, remember you are measuring the classifier, not the underlying customer behavior, so report both its own accuracy and the lift of its output as a routing signal.

The most common mistake in pre-router signal work is training on a field that only exists after a human has acted. Escalation flags, agent notes, resolution codes and final priority are all post-routing or post-resolution. A classifier using them will look excellent in offline evaluation and be useless in production, because at decision time the field is empty.

The practical test: take 200 tickets from your queue and, for each candidate field, answer a single question. Would this field have had the same value at the moment the ticket entered the queue? If no, drop it or replace it with a pre-routing proxy. Prior escalation count is fine because it references older tickets. The current ticket's escalation flag is not.

The same discipline applies to the label side. A first-pass failure is defined by outcome, which is fine, but you need the outcome recorded consistently. If reopened tickets are only logged when a customer replies, and some reopenings happen silently in the tool, your base rate is understated and every signal's lift is measured against a wrong yardstick.

There is a related question of cost. Every pre-router signal adds computation before routing, and routing decisions often stack a cheap model ahead of an expensive one. Bouchard's analysis of LLM cascades finds that while two-model cascades improve mean accuracy over single-model selection by 2.1 to 8.2 percentage points, adding more than two models did not improve accuracy in 118 of 135 comparisons and added at most 0.43 points. The same logic applies to signals: every added signal costs an evaluation, so a signal with a 2 point lift and an interval that barely clears the base is rarely worth the operational complexity of maintaining it.

If your signal set grows large and you want to test combinations rather than single signals, the AI Engineer topic page on model routing and gateways frames evaluation as a first-class responsibility: evaluate the complete policy, record why a route ran, and release policy changes with an audit trail. That framing applies directly. Your router's decisions should include a record of which signal fired and the observed failure rate for the bucket.

The signal set is not static. Queues change when products ship, when channels get added, and when the customer mix shifts. Build a monthly review with a fixed structure:

1. Recompute the base failure rate for the queue over the review window and compare it to the prior window.
2. For each live signal, recompute the bucket failure rate and interval, and flag any signal whose interval now includes the base.
3. For each flagged signal, either retune the bucket boundary, retire the rule, or mark it as needing a holdout.
4. Run one or two candidate signals pulled from the last month's backlog of ideas through the same two-gate test.

Two guards keep this honest. Track both the false positive rate and the false negative rate of the router against the full queue, because a signal that catches failures but routes too much traffic to the specialist path costs you specialist hours. And review contained and escalated conversations together, as the Userflow breakdown of escalation criteria recommends, so that fixing a threshold on one side does not quietly damage the other.

It also helps to distinguish triggers by type when you review. Conduit's escalation framework separates confidence thresholds, sentiment detection, topic rules and VIP flagging, and assigns each a different role: sentiment is about relationship repair rather than correctness, and topic rules are hard-coded because the stakes are too high for a probability. If you mix sentiment, which is expensive to compute and noisy, with structural fields like attachment presence, the interval test will wash out both. Test sentiment as a signal family in its own right, on a holdout window, before it enters the router.

If you want an expert to build this signal review into your own queue, Quanta Intelligence experts read your context and return finished deliverables, including the signal test harness and the monthly review checklist. One free consulting call is also available if you would rather scope the definitions and the base rate first.

The most useful mental shift is to stop looking for signals that "feel predictive" and start treating each candidate signal as a hypothesis with a verdict. The verdict has two parts: the lift clears the confidence interval, and the field exists at decision time. A signal that passes one gate and fails the other is not a weak signal; it is not a signal.

The failure cost side matters too. An escalation path that triples the cost of resolution, as Supportbench reports, sets the bar for the router higher than a path that merely reassigns within a team. High-cost paths deserve a wider confidence interval, not a narrower one.

Finally, keep the evaluation loop closed. IBM's overview of human-in-the-loop is a useful reminder that the router is a system with a human on the other end, and the metric that matters is whether the human ends up doing the work the router intended to send them. Instrument the handoff: did the receiving agent confirm the routing decision was right, and did the customer reply within the first response? Those downstream signals are the best evidence available for whether a candidate rule deserves to stay.