A Routing Rule That Sends Overflow Traffic to a Cheaper Model

The rule has four parts, and each is a place a naive build fails.

1. A reservation budget. Track consumed input and output tokens per minute against the committed rate, not a running request count. Bedrock reserved capacity is expressed as a tokens-per-minute rate for one- or three-month terms, and traffic above it spills to on-demand rather than being throttled per Inventive HQ.
2. A difficulty classifier. Score the request on cheap, deterministic features before any model call.
3. A routing decision. Hard requests go to the reserved model, easy requests to the overflow model, and unplaceable requests stay reserved.
4. An evaluation gate. Sample overflow outputs, score them against the golden set, and let the pass rate decide whether overflow stays enabled.

Start the classifier as rules, not a model: prompt length, whether the request asks for code, whether it names several constraints, whether an earlier turn in the session hit a retry, and an allowlist of forgiving endpoints. A learned router beats this once you have labels, but the gate is identical either way, so you can swap the classifier without touching it.

The payoff is real. The RouteLLM study found that sending easy queries to a cheaper model cut costs by over 85 percent on MT Bench while keeping 95 percent of the frontier model's quality per Maxim AI. That also names the risk: the quality you give up is not spread evenly across prompts, and the gate exists to catch the case where it clusters.

A gate is only as good as the set it scores against. The golden set is a frozen, versioned collection of prompts with reference outputs and a scorer for each, built from production traffic rather than public benchmarks. Sample a few hundred real requests over a normal week, stratify by endpoint and by the classifier's own score bands, and hand-label what the reserved model produced. Those become the reference; then score the overflow model on the same inputs with the same scorer.

Keep scorers mechanical where you can: exact match on extracted fields, schema validation on structured output, regex for required content. Rubric scoring with a judge model is expensive and drifts, so pin the judge and re-validate the judge itself whenever the set changes. Freeze the set during a gate cycle, because editing prompts mid-cycle makes a pass-rate drop meaningless. Version the set and record which version each gate result used.

Evaluation tooling exists to do exactly this: hold the set, run scorers, keep results versioned per Braintrust. If you already run evals in CI, reuse the harness and scorers. If not, a small script and a JSON file are enough to start.

The gate has two numbers to tune, both from measurement.

The threshold. Use the overflow model's pass rate on the golden set before you switched any production traffic, minus a safety margin. If the cheap model passed 91 percent and the reserved model passed 97 percent on the same set, 88 percent tolerates small sampling noise while catching a real drop.

The sample rate. You need enough scored outputs per window for a pass rate that is not noise, and 20 scored outputs is the practical floor. At a 5 percent sample, that means roughly 400 overflow requests in the window. Raise the rate during the first week of an overflow event and lower it once the pass rate stabilizes. Scoring every output recreates the expense the cheap model was meant to save.

scored = score_against_golden_set(sample(overflow_window, rate=sample_rate))

if len(scored) < min_scored:
    decision = "hold"            # not enough evidence, keep the last decision
elif pass_rate(scored) >= threshold:
    decision = "overflow_on"     # cheap model may keep serving the easy band
else:
    decision = "overflow_off"    # flip back to reserved, queue the excess

record(decision, pass_rate(scored), len(scored), threshold, sample_rate)

The "hold" branch matters most. A gate firing on 6 scored outputs oscillates every few seconds and produces an outage nobody can reproduce from logs, so make "keep the last decision" the default when evidence is thin. The gate must also flip in both directions: sustained recovery above the threshold brings overflow back, and every flip is logged with the pass rate that caused it.

Now the failure modes. Set the threshold too high and the gate fires on noise, so every overflow event bounces back and queues: a service-level failure dressed up as caution. Set it too low and the gate sits quietly while a cheaper model degrades quality, and you hear about it from users. Start slightly high, count unnecessary flips for a few days, and lower the threshold only when that count is measured and unacceptable.

A flip-back is not free. The excess returns to a reserved model that is full, so the router needs a queue that does not break the agreement worse than the cheap model would have.

Every request carries a deadline, and requests that would miss it get an honest degraded response rather than a silent drop. Cache helps more than most teams expect: repeated prompts within a short window can be served at a reduced rate on the same provider per the AWS Builders guide, and a cache hit never touches the reservation's token rate. Reach for on-demand spill last, and only at the margin, because it keeps the expensive model anyway per Inventive HQ. If spill is more than a few percent of a flip-back window, the threshold is wrong, not the infrastructure.

The capacity literature reaches the same conclusion in different words: reservations pay off only above a break-even, and burst workloads belong on on-demand or spot capacity per OneUptime. The router decides which bursts get the reserved capacity and which go elsewhere.

Take a reservation of 2,000,000 input and 800,000 output tokens per minute. Baseline traffic runs at 55 percent, averaging about 1,200 input and 400 output tokens per request, roughly 900 requests per minute. At 2.5x baseline the reservation is 25 percent over, and about 450 requests per minute overflow.

At the rates cited for a legacy mid-tier model ($6.00 input, $30.00 output per million tokens) per Inventive HQ, each overflow request costs about $0.0072 in and $0.012 out, so $0.0192, or about $8.65 per minute. The small-model tier at $1.00 and $5.00 costs about $0.0012 in and $0.0020 out, so $0.0032 each, or about $1.44 per minute. A 40-minute spike costs about $346 on the reserved model and about $58 on the cheap one.

Now the gate. The overflow model's golden-set pass rate was 91 percent against a threshold of 88 percent. At a 5 percent sample, the spike yields about 22 scored outputs, just above the floor. If they pass at 86 percent, the gate flips and the excess queues: the 40 minutes cost $346 instead of $58, and no user gets a poor answer. If they pass at 92 percent, the saving stands.

The decision rule is that the threshold sits so the expected cost of a false negative (quality degrades, you learn from users) exceeds the expected cost of a false positive (the cheap model was fine, you pay the reserved rate briefly). At a 4 to 5 times price gap, a quality regression costs far more than four minutes of peak traffic, so the threshold belongs confidently above the noise floor.

One asymmetry to plan for: if traffic shifts harder, the easy band shrinks, overflow volume falls and the gate sees fewer scored outputs per window. Set the sample rate as a target count per window, not a fixed percentage, so the gate keeps getting evidence.

Three rules keep the classifier useful.

Score the request, not the answer. The classifier never calls a model. A classifier on the critical path that calls a model adds latency everywhere and doubles the cost of requests it routes cheaply.

Calibrate against outcomes. Label a few thousand historical requests by whether the reserved model's answer was actually better, then fit the classifier against those labels. Classifiers tuned on gut feel overestimate difficulty, which is safe but expensive.

Re-fit on every flip. Each gate flip is a labeled example of a request the classifier called easy and the scorer rejected. Feed them back. Learned routers need training data and drift as the underlying models change per Maxim AI, and the flip log is the cheapest training data you will get. If you would rather not build the routing plumbing across providers, Quanta Inference puts text, image, video, audio and embedding models behind one API key at https://quantaintelligence.ai/inference, so the overflow policy lives in one place.