Small Model First, Larger on Retry: Routing by Tier

The token profile comes first, because the retry carries more of it. A first pass typically reads the ticket and the retrieved knowledge article and writes a short reply. A retry reads all of that plus the failed attempt and its critique, so it is materially larger on the input side.

Take a profile in line with the support-ticket workload used in Silicon Data's token cost guide, which frames a typical request at roughly 3,150 input tokens and 400 output tokens:

The retry is 44 percent larger on input because it carries the first attempt, the critique and the retrieved passages again. That is the single most under-modelled line in most cascade budgets: teams price the retry at the same token volume as the first pass and understate the bill by a third or more.

Now apply a price ratio. Suppose the small model is priced at 1 unit of input cost and the larger one at 20 units for the same input, with a similar output multiple. Scale by whatever absolute rate you actually pay: BenchLM's pricing table shows the frontier tier sitting one to two orders of magnitude above the cheap tier across vendors, which is exactly the range that makes this architecture tempting. Treat the numbers below as ratios, not as a quote.

Read the third row against the second. The price ratio did not change. The failure rate went from 10 to 30 percent and the bill went from 3 to 7 units. This is the insight the price list hides: the break-even moves with the failure rate, not with the price gap. A 30 percent failure rate on a retry that costs 20 times a first pass still multiplies your bill by roughly seven relative to a clean first pass.

The same arithmetic says something less comfortable. If your all-large baseline costs, say, 12 units per ticket for one pass, then the small-first design at 30 percent failure (7 units) is worth doing but saves under half. At 50 percent failure (11 units) it saves almost nothing while adding a second round trip and a second chance to say something wrong. Run that comparison before you build the router.

A single blended failure rate is a planning number. It is a terrible routing number. Split it by failure type and the design changes.

The useful split is between failures the first pass could not have avoided (hard tickets: multi-turn reasoning, conflicting policy, account-specific exceptions) and failures it should have avoided (easy tickets the small model fumbled). The first category justifies the larger model. The second is a prompt, retrieval or tool-calling defect, and paying a frontier price to paper over it is how cascades get expensive without getting better.

Concentrate's writeup on confidence-threshold escalation makes the structural point well: cascading beats upfront routing because nobody has to predict how hard a query is before seeing an answer. But it also flags the tradeoff that matters here, that cascade performance is limited by paying the cheap model before escalation. If the cheap model is failing on tickets it should handle, you are paying twice for the same mistake.

So instrument the failure reasons before you tune the gate. Tag every failed first pass with one of a short list: retrieval miss, policy gap, tool error, misunderstood intent, out-of-scope request. VocalOps frames these as the failure modes worth testing, and the list transfers directly to text support: misunderstanding, insufficient knowledge, impossible action, higher-risk request. If retrieval misses dominate, fix retrieval. If out-of-scope requests dominate, those tickets should never have gone to the small model at all.

That last point is where a decision rule earns its keep. Route by type first, then by score:

• Never first-pass: refunds, billing disputes, security or account-access issues, legal or compliance language, safety concerns. These escalate on topic alone, before any model runs. Conduit's escalation framework treats these as hard-coded rules that no confidence score can override, and SearchUnify's escalation guide lists severity as the first trigger for the same reason.
• Small first, escalate on score: the long tail of routine how-to, status and configuration questions.
• Large first: anything the small model has failed on before, clustered by intent, once you have the data to see the cluster.

This type-first split lowers the effective failure rate you are paying for. Moving 8 percent of tickets out of the first-pass pool and straight to the larger model costs more on those tickets and less overall, because the retry attempt is skipped entirely.

The threshold is one number and it is easy to obsess over. The signal feeding it is where the money is.

Verbalized confidence, where the model says it is 80 percent sure, is the weakest option available. The userflow escalation guide cites work testing model-stated confidence and reports an average AUROC of 62.7 percent for separating right from wrong answers, which is better than chance and nowhere near good enough to run a budget on. The same guide recommends pairing the score with a signal the model cannot inflate: no passage clearing your retrieval threshold, contradictory top passages, an intent mapping to no configured topic, or the user restating the question.

Here is a gate that encodes that, written as plain policy rather than framework code:

escalate_to_large(ticket, first_pass):
  if topic in HARD_ESCALATE:            return "large, skip first pass"
  if first_pass.tool_error:             return "large, with error attached"
  if first_pass.retrieved == 0:         return "large, retrieval was empty"
  if first_pass.top_score < T:          return "large"
  if first_pass.score_gap < G:          return "large"
  if user_restated_question:            return "large"
  if sentiment_negative_for(2 turns):   return "large, flagged for human"
  return "accept"

# on the retry, one attempt only
if retry_failed: route to human queue with full transcript

Two details carry real money. First, the score gap G: a high top score with a close second candidate is a weak answer, and comparing the top two intents catches cases a single number misses, an approach explained in this walkthrough of confidence scoring and escalation logic. Second, the retry cap. One retry, then a human. Retries beyond the first almost never converge and each one costs full freight.

There is also a structural argument for stopping at two tiers. Bouchard's analysis of cascade depth optimized sequences from a pool of eight models and found that allowing more than two models did not improve mean accuracy in 118 of 135 comparisons across scorers and datasets, adding at most 0.43 percentage points. The workshop version of the same work goes further, finding that a lightweight router that chooses before generation beats the best cascade policy on four of five datasets. If you are about to build a three-tier waterfall to chase the last few points, measure a pre-generation router against it first.

The formula is only useful if the numbers going into it are observed. Build the ledger before the router.

Two failure modes hide in this table. A rising escalation rate with flat resolution means the gate is firing more often without buying quality, the classic over-escalation signature. A falling escalation rate with falling resolution means the gate got looser and someone stopped reading the transcripts. Both look like normal operation in a dashboard, which is why Userflow's guidance recommends reviewing contained and escalated conversations together: tuning one signal alone fixes one problem and creates another.

Worth adding: HubSpot's documentation of its customer agent handoff is a useful reference for what a handoff record should carry, because the retry and the human escalation should both inherit the first pass's context rather than starting cold. A retry that re-reads the ticket from scratch wastes input tokens and produces a second independent guess.

The saving is real but bounded, and it concentrates in one place: the tickets the small model resolves correctly on the first attempt. Everything else is overhead.

Four rules follow from the arithmetic.

Rule one: measure before you route. If your first-pass resolved rate is below 60 percent on the routine bucket, the cascade is not yet a cost story, it is a defect story. Fix retrieval and prompts first. The formula's denominator punishes a mid-range failure rate hard.

Rule two: choose the retry model for a different job, not a bigger one. The retry sees a failed attempt plus a critique. That is a different task from answering cold, and it rewards a model that is good at critique-and-revise, not simply the most expensive option available. A mid tier at five times the first-pass cost with a 30 percent failure rate lands at 2.5 units per resolved ticket, against 7 for the 20-times tier at the same failure rate. Test both on your own failed transcripts before assuming the frontier tier earns its price.

Rule three: skip the small model where the topic is hard. The pre-generation router result above points the same way: paying the cheap model on tickets you know will fail is pure waste.

Rule four: cap retries at one and price the human fallback. The human queue is not free, and a cascade that pushes 15 percent of volume into it has traded a token bill for a labor bill. Model both.

Where this gets practical is the plumbing. Routing between two models means two providers, two sets of credentials, two billing relationships and two rate limits, and the retry lands on the model you call least often, so it is also the one whose key is most likely to have rotated. If you are standing up this pipeline, Quanta Inference exposes text, image, video, audio and embedding models through one OpenAI-compatible key with prepaid pricing in US dollars, which collapses the two-provider problem into one bill you can reconcile against the ledger above. Nothing about the routing logic changes; you just stop maintaining two integrations to run a two-tier cascade.