Ordering a Fallback Ladder by Cost and Latency

The core insight is that a slow primary and a slow fallback are not the same kind of problem. If the primary is slow, you have already paid part of the latency. The fallback starts a new clock, and its full latency is added on top of what the user has already waited. That is why a fast-looking fallback can be the more expensive choice.

So each candidate model needs three numbers, measured on your own traffic rather than read off a vendor page:

1. Price per call for your typical input and output shape, in dollars.
2. Median and 95th percentile latency for that same shape, in seconds. Use the high percentile, because fallbacks are triggered by slow conditions.
3. A waiting penalty in dollars per second, derived from your latency budget. If a second of overrun costs you $0.002 in lost completion or support, that is the coefficient.

The expected cost per candidate is:

expected_cost = price_per_call + waiting_penalty * added_latency_seconds

where added_latency_seconds is the candidate's latency, not the primary's. You are comparing what it costs to serve the request from this candidate right now.

A worked example with three models shows why raw ordering fails. Assume a waiting penalty of $0.002 per second, and a primary that has already spent 4 seconds and missed its 3 second share of the budget.

Sorted by expected cost, the ladder is C, B, A. Sorted by price, it is C, B, A too, but only because latency happens to fall in the same direction. Change the numbers and the answers diverge. Suppose Model C is a tiny model with a cold start problem on some infrastructure, pushing its p95 to 5 seconds. Its expected cost becomes $0.0015 + $0.0100 = $0.0115, which now sits between B and A. A cheaper model with a long tail belongs lower on the ladder than its price suggests.

The same logic explains why serverless cold starts matter here. A function that has not been called recently pays a startup cost that Lambda cold starts can put anywhere from half a second to a few seconds, and that cost lands squarely in your waiting penalty. If your fallback lives on a cold edge function, its measured p95 will reflect the cold path, and it will rank accordingly. Keeping the fallback warm (provisioned concurrency, periodic warm-up requests) can move it up the ladder without changing its price at all.

The penalty coefficient is the lever that decides everything. If you plot expected cost against the penalty rate for two candidates, they cross at a break-even point. Below that point, price dominates. Above it, latency dominates.

Take Model A at $0.0120 and 6.0 seconds against Model B at $0.0060 and 2.5 seconds. The price gap is $0.0060 and the latency gap is 3.5 seconds, so they break even at a penalty of $0.0060 / 3.5 = $0.0017 per second. If your users tolerate the wait well enough that a second is worth less than 0.17 cents, put A first on price grounds. If a second is worth more than that, B wins and A falls behind it.

This break-even number is the decision rule. Write it next to the ladder. It tells the next engineer why the order is what it is, and it tells you when to revisit the order: whenever the penalty coefficient moves, or whenever measured p95 latency shifts. It also handles a subtle case. A provider can be technically up while returning time to first token of several seconds instead of the usual fraction of a second, a failure mode that fallback routing addresses with explicit latency thresholds rather than error rates. Your ladder is ordered by the same quantity those thresholds measure.

Two practical rules fall out of the arithmetic.

Do not retry the same model before moving down the ladder. A retry re-pays the primary's latency in full and leaves you with the same expected result. Bounded retries are useful for transient network errors, but for a slow call the correct next action is the next rung, not the same one again.

Never let a fallback's latency exceed the budget that remains. If 1 second is left and the next candidate's p95 is 2.5 seconds, it cannot serve this request at all. Skip it. This is why the ladder must be sorted by latency as well as cost: a candidate that cannot finish within the remaining budget has infinite expected cost, whatever its price.

A ladder that only ever escalates to better models is a ladder that always gets slower under load. Sometimes the correct move is downward: return a shorter answer, skip an optional tool call, or serve a cached result. Deciding when to do that is a policy question, and the latency budget already contains the answer.

The rule: escalate only while the next rung's expected latency fits inside the remaining budget. When no rung fits, degrade. Degradation options, roughly in order of user impact:

• Return the streamed partial answer with a note that it was truncated.
• Serve a cached or precomputed answer for the same input, accepting that it may be stale.
• Fall back to a deterministic, non-model path (template, search result, rules engine).
• Return an honest error with a retry affordance.

Caching deserves a caveat here. A cache is not a fallback rung in the same sense, because its hit rate is unpredictable and a cold cache behaves exactly like a missing model. The distinction between a cache and a precomputed read model matters: a read model is built for the query shape and has predictable performance, while a cache warms on first request and leaves some requests slow. If you need the degraded path to be dependable, build it as a read model, not as a cache.

The same reasoning applies to replication lag. If your degraded answer reads from a replica, it may be serving data that lags the primary by seconds or minutes, and applications reading from replicas must accept that the data is somewhat out of date. Decide up front whether a stale answer is better than a slow one for this endpoint. For a product listing, yes. For a balance check, no, and the ladder should end in an error rather than a wrong number.

The ladder is a small amount of code. The important part is that the ordering comes from data, not from configuration order.

from dataclasses import dataclass

@dataclass
class Candidate:
    name: str
    price_usd: float
    p95_seconds: float

def expected_cost(c: Candidate, penalty_per_sec: float) -> float:
    return c.price_usd + penalty_per_sec * c.p95_seconds

def order_ladder(candidates, penalty_per_sec):
    return sorted(candidates, key=lambda c: expected_cost(c, penalty_per_sec))

def rest_of_ladder(remaining_budget_s, candidates, penalty_per_sec):
    ordered = order_ladder(candidates, penalty_per_sec)
    return [c for c in ordered if c.p95_seconds <= remaining_budget_s]

Two details make this production-ready. First, p95_seconds should be recomputed from real telemetry on a schedule, not typed in once. Second, rest_of_ladder is evaluated at the moment of the failure, with remaining_budget_s calculated from the request start, not from a fixed constant.

Test the ordering with synthetic traffic before trusting it. Send a share of requests through each rung and record which model actually served each one, the latency, and the cost. The difference between the planned ladder and the observed serving distribution tells you whether your penalty coefficient is realistic.

One caution on governance: a fallback chain that selects models outside an approved pool during an incident creates a compliance failure exactly when the system is already stressed. Fallback models should be pre-selected from the approved pool, not chosen at failure time. Fix the candidate set per workload class, then order within it.

A fallback ladder is a narrow tool. It handles one request whose primary call is slow or failing. It does not handle choosing between models for quality, splitting traffic by tenant, or negotiating cost across a fleet of workloads. Keep the layers separate so each one stays testable.

Cost discipline across the whole system starts earlier than the ladder. The largest drivers are model complexity, workload demand and infrastructure efficiency, and optimization should begin with baselines for cost, quality, latency and throughput before any routing change. The ladder is a latency tool with a cost tiebreak, not a general cost-optimization strategy. If your unit economics are wrong at the baseline, reordering fallbacks will not fix them.

The other structural point: a ladder that fans out to many providers needs a single place to hold credentials and see per-model spend. If you are building that routing layer yourself, one option is to put the provider surface behind a single key so the ladder only changes model identifiers. Quanta Inference exposes text, image, video, audio and embedding models through one OpenAI-compatible API, which keeps the ladder definition in your code rather than in five SDK setups.

Run this checklist when a new model is added to the candidate set, and again whenever the penalty coefficient changes. The ladder is a small model of your own economics. Like any model, it is only as good as the numbers you feed it, and those numbers come from your telemetry, not from a vendor's landing page.