How to Write a Fallback Ladder When the Strong Model Is Down

Start with a table. For each band, write the ordered sequence of models the router may try, ending in a rung that always works.

Three rules make this table useful rather than decorative.

First, the order is per band, not global. A top-band row should never fall to the cheapest model because the router's global list ran short. The top band's ladder exists to protect that band's quality contract, or to hand the row to a person.

Second, a cheaper same-vendor model is the default step down from a strong model because it usually shares API shape, auth and output schema, which lowers integration risk. The next rung is cross-vendor, so that an outage affecting one provider does not remove two rungs at once. Multi-provider failover guidance treats a single provider as a single point of failure and recommends ordered chains across providers, because the provider's incident becomes your incident TrueFoundry's failover guide. The same guide notes that a fallback chain trades availability for some output variance, which is the cost you accept when you cross vendors.

Third, some rungs should not exist. A safety refusal, a malformed request or a 401 should not be retried down the chain. Only availability-shaped failures belong in the ladder: server errors, timeouts and rate limits, plus schema failures where you can tolerate a variant response. That distinction is the difference between a resilient router and one that burns its whole chain on a bad request the fallback routing guide. The same guide flags the case most routers get wrong: a content-policy denial is not a fallback trigger, and sending a refused request to another vendor is policy circumvention, so it should be terminal.

One more constraint is often skipped: the authorized-endpoint set. If some rows contain data that may only reach certain providers under contract, the ladder must check membership before each attempt and skip unauthorized endpoints. Falling back to a non-covered provider is a compliance failure that looks like a success in your metrics. The routing guide makes this point directly: constrain the chain to endpoints the request may reach, and log an explicit "no authorized fallback" entry when the authorized set is exhausted.

Retries and fallbacks add latency, and latency is what users feel. If each attempt gets its own generous timeout, a four-rung ladder can leave a row waiting a minute or more before it lands anywhere useful.

Invert the budget instead: one total wait per row, divided across attempts.

The total cap is 20 seconds. Attempt 1 gets the largest slice because it carries the quality contract. Later rungs get shorter slices because a slow fallback is barely better than none: by rung three the user has waited long enough that a fast, decent answer beats a slow, marginal one.

Two rules keep the cap honest. Never retry the same model past its slice; the timeout is the timeout and the ladder moves on. And log a per-attempt timeout differently from a failure, because a timeout says nothing about whether the model would have answered correctly, only that it was slow. Timeouts are not one problem: a text model that is slow because of a large input is a different diagnosis from a provider queueing requests at peak load EvoLink's timeout guide. That guide's practical recommendation matches this design: a shorter timeout plus a fallback model is often better than a long timeout and no fallback.

CAP_SECONDS = 20.0
def run_row(row):
    started = time.monotonic()
    trace = []
    for rung in ladder_for(row.band, row.data_class):
        left = CAP_SECONDS - (time.monotonic() - started)
        if left <= rung.reserve_for_handoff:
            trace.append(("skipped_no_budget", rung.name))
            break
        outcome = call_model(rung, row, timeout=min(rung.timeout, left))
        trace.append(outcome)
        if outcome.ok:
            return outcome, trace
        if outcome.terminal:      # refusal, 401, malformed request
            break
    return handoff_to_human(row, trace)

The reserve_for_handoff field matters. The last slice exists to enqueue the row and write its context, and the router must not spend it on another speculative model call.

The human queue is the rung that makes the ladder safe to run, and it fails most often for a boring reason: whoever picks up the row has to reconstruct what happened from scratch.

The handoff payload should be a first-class artifact, not a chat transcript. It carries:

• The original input, byte for byte, before trimming or prompt wrapping.
• The band decision and the signal that produced it.
• The full attempt trace: each rung tried, its timeout slice, the failure class, and any partial output discarded.
• The data classification and the authorized-endpoint set the router applied.
• A stable request ID plus one ID per attempt.

That last point is about audit, not operations. A fallback that succeeds against an unauthorized endpoint is invisible in success metrics and surfaces only when someone asks which model saw a given prompt the same fallback routing guide. Per-attempt IDs written into the audit record make the ladder defensible.

For tool-calling workloads, add a replay token per tool call. If a model produced a create_order call, the caller executed it, and a later check failed, a fallback can ask for create_order again, which leaves two orders. A caller-supplied replay token lets the tool server reject the duplicate, and the routing guide recommends it for any tool call that may execute twice.

Finally, treat the queue as a product surface. A queue that rows enter but never leave is a slow error. Track time-to-first-touch and time-to-resolution per band. If the high band's queue keeps backing up, your ladder needs another model rung or a better banding threshold, not a looser handoff.

An untested fallback path is a hypothesis. Most teams discover their fallback logic is broken during an actual incident, because simulating provider outages in staging is hard TrueFoundry's failover guide. You can make it much less hard.

Run this suite on every router change and on a schedule, because the failures that matter appear when nobody is watching. Evaluation practice makes the same point: a mature strategy turns every production incident into a permanent regression test so a known failure pattern cannot reach users again Splunk's evaluation framework.

Two metrics belong in the output beyond pass and fail. Band degradation rate is the share of attempts ending below attempt 1, which tells you how often the quality contract is stretched. Cap hit rate is the share of rows reaching a person because the budget ran out. A cap hit rate that spikes during a provider incident is good news, because the ladder is doing its job instead of hanging. A high cap hit rate on a normal day means your slices are too tight or the top band is routinely overloaded.

It helps to treat the ladder as having an error budget: an amount of degradation you accept inside a period, tracked against your service objective. Error budgets are a standard reliability tool for deciding when to spend on stability versus features, and the arithmetic applies here Sedai's error budget guide. If your human queue absorbs 4 percent of high-band rows because one provider is flaky, you are likely over budget and should add a rung or renegotiate capacity rather than keep absorbing.

The instinct is to keep adding rungs. The evidence says stop early.

A study of LLM cascades optimized sequences from a pool of eight models across five benchmarks and nine deferral scores, and found two-model cascades improved test-set accuracy over the best single model by 2.1 to 8.2 percentage points. Allowing more than two models did not improve mean test-set accuracy in 118 of 135 comparisons and added at most 0.43 percentage points the cascade depth study. Gains from depth depended on how well the confidence score separated correct from incorrect answers for the current model compared with later models.

That is a usable decision rule. Each rung past the second buys little on average, and every rung costs latency, test surface and audit complexity. A strong model, one cross-vendor equivalent, one mid model and a human queue is usually enough. The exception is a well-calibrated confidence signal, which is exactly where the study found depth can pay.

The same research stream reframes escalation itself. A cascade is an optimal stopping problem, and its optimal policy compares current estimated error against the cost of continuing rather than a fixed confidence threshold; a fixed threshold matches the optimum only in a degenerate case and otherwise leaves money on the table the cascade regret analysis. In practice, decide whether to try another rung from the remaining budget and the expected value of the next rung, not from a static "always try three models" rule. Under heavy load, a five-second slice on a third model may be worth less than getting the row to a person sooner.

Model-agnostic plumbing earns its keep here. If model IDs are hardcoded in route handlers, changing the ladder is a code change with a deploy cycle. Keep the model ID as runtime configuration behind one boundary and an availability failure becomes a config edit the model-agnostic fallback layer guide. That source documents a case where a provider disabled models for every customer, and teams whose IDs were configuration recovered by changing one value.

Deterministic rungs carry a real advantage too. Rule engines and decision tables are fully deterministic with a row-level trace, while an AI agent rule is probabilistic with caching available, which makes auditing harder the DecisionRules AI agent docs. A deterministic rung is cheap, explainable and never times out, which is why it belongs between the last model and the human queue wherever a rules answer is acceptable.

When a fallback rung gets a rewritten prompt, give it tools to find context rather than stuffing more context in. Production agent teams report that minimal instruction plus context loading beats overprompting the Linear Agent discussion. A tight, narrowly scoped prompt on a smaller model often outperforms the full top-band prompt, which was tuned for another model's strengths.

If you would rather not maintain per-provider SDKs, one API key across text, image, video, audio and embedding models removes a whole class of failover code: Quanta Inference is OpenAI-compatible and prepaid, so each rung becomes a configuration entry.