Combining Two Signals Without Double-Counting: A Lift Test

Suppose you run a support assistant and you have two candidate signals for "this request needs the stronger model":

• Signal A: length. The prompt exceeds a token cutoff.
• Signal B: sentiment. The ticket text scores below a negativity threshold.

You have 10,000 logged requests with a known outcome (resolved first pass or not). Here is the table, with success rates:

The base success rate is 44.0%. Now compute lift for each signal as the ratio of success rate in the present group to the base rate, and the same for the absent group, so you can see both directions:

• Lift of A present: 16.7% / 44.0% = 0.38. Lift of A absent: 55.7% / 44.0% = 1.27.
• Lift of B present: 34.0% / 44.0% = 0.77. Lift of B absent: 54.0% / 44.0% = 1.23.

Both signals point the same way: presence is bad for first-pass success, which is exactly what you want from a routing signal. Now the interaction test. Multiply the two "present" lifts:

0.38 x 0.77 = 0.293

The observed lift in the both-present cell is 10.0% / 44.0% = 0.227. The observed lift is lower than the product, which means the signals overlap: each one is partly telling you the same thing, and stacking them double-counts. This is multiplicative interaction on the risk scale, the same quantity the epidemiology literature calls interaction on a multiplicative scale, which is distinct from additive interaction and can disagree with it (the interaction tutorial).

What the overlap costs you

If you kept both signals and scored them as two independent votes, you would predict a much lower success rate for the both-present group than the 10.0% you actually saw. You would route some of those requests to the expensive model unnecessarily, or you would over-weight the pair in a learned scorer. The practical size of the error is the gap between 0.227 and 0.293, about 22% relative overcounting of the pair's effect.

Here is a compact way to run the whole calculation:

base = 0.440
lift_a = 0.167 / base
lift_b = 0.340 / base
lift_ab = 0.100 / base
product = lift_a * lift_b
print(f"lift A={lift_a:.3f} lift B={lift_b:.3f} product={product:.3f} observed={lift_ab:.3f}")
print(f"ratio observed/product = {lift_ab / product:.3f}")
# ratio near 1.0: additive in the multiplicative sense, keep both
# ratio below 1.0: overlap, drop the weaker signal or add an interaction term

A ratio near 1.0 says the signals are close to additive on this scale. A ratio meaningfully below 1.0 says overlap. A ratio above 1.0 says synergy, and that is the one case where an explicit interaction feature earns its keep.

The short answer to whether to keep one signal or both is: compare the combined lift to the product, not to the sum. Multiplication is the right operation because each lift is a ratio to the same base rate, and ratios compound when independent signals are applied together.

Three outcomes, three actions:

The trap is treating a below-1.0 ratio as a reason to keep both anyway "because both have lift." A signal can have lift and still be redundant with another. Redundancy is not free: every extra feature in a router is a chance to overfit the training slice, and the overfitting shows up as a deeper cascade that costs more without improving quality. Work on cascades found that two-model cascades improved mean test-set accuracy over single-model selection by 2.1 to 8.2 percentage points, but adding more than two models failed to improve mean test-set accuracy in 118 of 135 comparisons and added at most 0.43 points (the cascade paper). Extra depth is a real cost with a thin payoff.

That finding has a direct reading for signal selection: the second and third signals behave like extra cascade depth. The work also found that a lightweight pre-generation router beat the best cascade policy on four of five datasets, suggesting cascade performance is limited more by paying the cheap model before escalation than by a shortage of intermediate stages (the ICML abstract). A router that decides before generation avoids that overhead. Your two signals are the router's inputs, so keeping the minimal set that carries the information is the whole game.

The multiplicative test is the right default, but it has two known failure modes.

Scale dependence. Interaction on the multiplicative scale and interaction on the additive scale are different questions, and one can be present while the other is absent. The standard tutorial is explicit that the two scales must be evaluated separately and that a claim of "no interaction" is only true for the scale you tested (the interaction tutorial). If your router adds the two signal scores into a linear score, you are on the additive scale and should check the difference of differences instead of the ratio of ratios.

The interaction term is often a bad default. A review of including interactions in linear regression found that interaction terms are easy to add, inflate variance and raise the risk of overfitting, and are therefore best reserved for cases with a clear hypothesis and enough data (the pros and cons review). That is why the decision rule above puts the interaction feature last. You add it only when the combined lift is actually higher than the product, not as a reflex whenever the ratio is off.

There is a third, quieter failure: measurement noise in the signal. The 2x2 table tutorial notes that measurement error distorts the observed relationship between the two binary quantities, and that sample size drives how wide the interval on each cell gets (the tutorial on 2x2 tables). A tiny both-present cell can produce a ratio far from 1.0 by chance alone.

Before you act on a ratio, put bounds around the both-present cell. Treat the success count as a binomial proportion and use the simple normal approximation: standard error equals sqrt(p times (1 minus p) divided by n), and a 95% interval runs about 1.96 standard errors either side when the cell has enough observations (the interaction tutorial covers power and sample size for additive and multiplicative interaction, which is the same arithmetic).

For our worked example, the both-present cell has 200 successes out of 2,000, so p = 0.10 and the standard error is sqrt(0.10 x 0.90 / 2,000) = 0.0067. A 95% interval runs from 0.087 to 0.113. Translate that to lift: 0.087/0.44 = 0.20 up to 0.113/0.44 = 0.26. The product was 0.293, which sits above the top of the interval, so the overlap here is not noise.

When it is noise, the intervals overlap and the honest conclusion is "no detectable interaction, fall back to the simplest model." Report the two lifts and the ratio with intervals, not a single ratio, so the next person can see how firm the call was.

One more guard: if the outcome is rare, the normal approximation gets shaky and you need more data per cell. The escalation literature is a useful warning here, because escalations are a minority of tickets and models are tuned for recall rather than precision on that small slice (AI escalation prediction). A 5% positive rate inside a both-present cell of a few hundred rows will not settle an interaction question.

The output of the test is a decision about feature set, and the decision has to survive contact with production. Three habits make that happen.

First, keep the test offline and the policy frozen until the test is done. Moving thresholds or reweighting between the test and the deployment means the interaction result no longer describes the router you shipped. The routing overview is explicit that release should not restore old authority, so a bad threshold change should not be able to sneak back in behind a clean interaction result (the routing overview).

Second, evaluate the complete policy, not the individual signals. The same overview distinguishes mediation (a gateway choosing a destination) from quality decisions and from failure fallback, and it insists on evaluating the whole policy rather than isolated components (the routing overview). Your lift test is a feature-selection step inside that larger evaluation.

Third, watch the cost side, not just accuracy. Token pricing varies enormously across providers and within a provider across model tiers, and the same workload can differ by more than an order of magnitude in spend depending on which model handles it (the cost per token guide, the pricing comparison). A router that keeps a redundant signal and over-escalates pays that gap on every affected request. The lift interaction test is, in the end, a way of not paying twice for the same information.

For teams that want to run routing experiments across many models without wiring each provider separately, Quanta Inference gives text, image, video, audio and embedding models through one OpenAI-compatible API key with prepaid pricing in US dollars (https://quantaintelligence.ai/inference).