Routing by Difficulty: Picking the Smallest Model That Works

The difficulty score does not need to be a trained model. It can be. But before you train anything, start with features you already have, because the biggest savings usually come from simpler levers first. One practitioner analysis of enterprise routing argues that price spreads of 10x to 30x inside model families make workload-to-model matching worth doing, yet the largest and safest savings come from static tiering and capping reasoning effort before any learned router fires per this routing guide.

A practical first score is a weighted blend of observable input features:

The last row matters most and is the one most teams skip. If you already log model calls with OpenTelemetry, you have gen_ai.request.model, gen_ai.usage.input_tokens and gen_ai.usage.output_tokens on every span, plus finish reasons, and these are captured even when content capture is disabled per the OpenTelemetry GenAI walkthrough. That gives you the raw material to compute, for each cluster of requests, how often the small model actually produced an accepted answer. That empirical rate is a better difficulty feature than any hand-tuned weight.

You can also record your own routing decision as an attribute on the span, which is the cheapest place to put it. Adding data to the current span is usually cheaper than emitting a separate log or span, and it lets you correlate the decision with outcomes on the same event per Honeycomb's instrumentation guidance. A minimal version:

span.set_attribute("routing.difficulty", d)
span.set_attribute("routing.confidence", conf)
span.set_attribute("routing.decision", "small")  # person | small | large
span.set_attribute("routing.t1", t1)
span.set_attribute("routing.t2", t2)

If you need a custom attribute name outside the standard set, follow the general conventions for stable, low-cardinality keys rather than inventing dotted names that collide with the GenAI namespace see the general attributes spec. Note that the GenAI attributes themselves have moved to a dedicated repository, so check there before assuming a key is current per the registry page.

A difficulty score alone is not enough, because two requests with the same predicted difficulty can differ in how sure you are about that prediction. That is where the confidence signal comes in. The combined score is what you compare to each cutoff.

The simplest combination is an adjustment:

def adjusted_difficulty(d, conf):
    # conf in [0, 1]; lower confidence makes a request look harder
    return min(1.0, d + (1.0 - conf) * ALPHA)

ALPHA controls how much uncertainty inflates difficulty. With ALPHA = 0.3, a request scored d = 0.5 with conf = 0.9 becomes 0.53, while the same request with conf = 0.4 becomes 0.68. The second one is more likely to clear t2 and go to the large model. That is the behavior you want: a request you are unsure about should be treated as harder than its point estimate suggests.

This matters because models are often overconfident. Neural networks can stay confident while accuracy drops on new inputs, which is exactly the failure mode that confidence-based routing is meant to catch as Appen notes in its HITL review. If your confidence signal is raw model output, calibrate it first; the confidence-versus-accuracy relationship is the thing you should be measuring, not the raw number per Mindee's treatment of confidence scores.

One caution on where the confidence comes from. If you generate it with a small model call, you have added a cost to every request, including the ones that end up on the small model anyway. Keep that scoring call tiny, batch it, or compute confidence from features rather than a model. The routing decision should cost far less than the routing savings.

Now the part most write-ups skip: how to pick t1 and t2. Do not guess. Write down a cost model and minimize it on a held-out split.

The cost model has five numbers per request:

• c_person: the loaded cost of a human handling the request (reviewer time times loaded rate).
• c_small: the cost of running the small model.
• c_large: the cost of running the large model.
• c_error: the cost of a wrong answer that reaches the user.
• p_small(d), p_large(d): the probability each model succeeds at difficulty d, estimated from your labeled sample.

The expected cost of routing a request of difficulty d to an option is the option's cost plus the error cost times the probability of failure:

def expected_cost(option, d, probs, costs):
    if option == "person":
        # assume a person succeeds, but costs more
        return costs["person"] + 0.0
    p = probs[option](d)
    return costs[option] + (1 - p) * costs["error"]

Then sweep t1 and t2 over a grid, assign every held-out request to a bucket, and sum the expected costs. Pick the pair with the lowest total.

A worked example makes the shape clear. Suppose, per 1,000 requests:

• Human: c_person = 2.50, error cost on a person-routed task is zero (person fixes it).
• Small model: c_small = 0.002, and it succeeds on 70 percent of requests it is given.
• Large model: c_large = 0.020, and it succeeds on 95 percent of requests it is given.
• Error cost c_error = 5.00.

At t2 set so that the large model handles 10 percent of traffic and the small model handles everything above t1:

Total expected cost: about 1,428.70. Now move t1 down so the person handles 10 percent instead of 5 percent and the small model takes the difference:

Total: about 1,478.60. Sending more to the person raised cost, because human time is expensive relative to the error cost in this scenario. Flip the error cost to 50.00 and the answer reverses: the small bucket's error cost becomes 12,000, and moving 5 percent of traffic to a person saves far more than the person costs. That reversal is the whole point of writing the cost model down. The cutoff is not a quality opinion; it is a consequence of your numbers.

This is also why the ubiquitous advice to bias thresholds toward quality is right on average but wrong as a rule. A wrong cheap answer costs more than an unnecessary frontier call in many settings per the routing guide above, but "more" is a number you should compute, not assume.

Fit on one slice, measure on another. Otherwise you will tune the cutoffs to noise.

Three checks on the held-out split:

1. Cost versus the all-large baseline. Compute expected cost for your router and for sending everything to the large model. If the router is not cheaper, you have a calibration problem or your error cost is mis-set.
2. Cost versus the all-small baseline. Same comparison with everything on the small model. This is the baseline most teams forget, and it is often the strongest competitor when the error cost is low.
3. Per-cluster error rates. Slice by domain tag. A single global t2 can hide a domain where the small model fails 40 percent of the time. If you find one, add a domain-specific offset rather than moving the global cutoff.

Keep the validation honest by using traces you already capture. A trace links the steps of one request into a span tree, with parent and child spans that mirror how the code executed, which is what lets you attribute a failure to the model call rather than to retrieval or a tool per the Braintrust comparison of tracing and logging. Logs alone give you a stream of standalone events; when several requests run concurrently, entries from different runs interleave and the failing path becomes hard to reconstruct as described in the same piece. Tags such as gen_ai.conversation.id, gen_ai.agent.name and gen_ai.operation.name are what bind every span in a run into one view per Honeycomb's agent instrumentation guide.

A useful detail for evaluation: the spec defines an evaluation event plus score attributes such as gen_ai.evaluation.name, gen_ai.evaluation.score.value and gen_ai.evaluation.score.label, and these can ride on the same trace as the model call per the GenAI attribute registry. That means your held-out labels and your routing decisions live in the same place. You do not need a separate labeling pipeline to re-fit the cutoffs.

Not every workload needs three buckets. The second cutoff earns its complexity only under specific conditions. Use this checklist before you build it.

The fifth row is the one that quietly breaks designs. A person cutoff that routes 20 percent of traffic to reviewers is a staffing plan, not just a threshold. Reviewers introduce moderate-to-high latency on the items they touch and scale with reviewer capacity, which is the main cost of putting a human in the loop per Appen's comparison of in-the-loop and on-the-loop designs. If you cannot staff it, route those requests to the large model and revisit later.

There is also a sequencing question. If you are unsure whether the small model will hold up, a cascade can be safer than a router: try the small model first, check the answer, and escalate only when it fails a quality test. The survey above treats cascading and routing as complementary approaches, with production systems often combining them per the routing survey. A cascade costs you a second call on the escalated share; a router costs you a misroute when the score is wrong. Early on, misroutes are the more expensive failure.

If your model pool spans several providers and you want to test small versus large without re-plumbing each one, Quanta Inference gives you text, image, video, audio and embedding models behind one OpenAI-compatible API key, which keeps the swap a configuration change rather than an integration project: https://quantaintelligence.ai/inference

Once traffic is flowing, watch four things, and re-fit on the same schedule you use for evals.

• The share in each bucket. It should track the held-out prediction. A shift means your difficulty score has drifted.
• The small model's success rate. If it climbs, lower t2 and capture savings. If it falls, raise it.
• The escalation rate out of the small bucket. This is where a cascade shows up even if you did not design one.
• Cost per successful request. Not cost per request. A cheap route that fails and retries is not cheap.

Two subtleties in the telemetry: token counts differ by provider, with specific guidance for how some providers report input tokens, so normalize before comparing costs across models per the semantic conventions evolution write-up. And span payloads grow with every attribute you add, so keep high-cardinality data out of the hot path; the same write-up notes that these conventions are still in Development status, so expect attribute names to move per the same source. A dedicated collector is often worth running locally while you tune, and a free OTLP-compatible viewer such as the Aspire Dashboard accepts the data directly with no cloud account per the OpenTelemetry walkthrough.

If you would rather have someone set the cost model and cutoffs with you on your own traffic, the Quanta team does that as consulting work and can build the surrounding app once the routing is settled: https://quantaintelligence.ai/