When the Second Retry Climbs: A Ladder for Three Model Tiers

A failure condition is observable if it can be evaluated from the response without human judgment. "The answer looked weak" is not a condition. These are:

• Schema validation failed against a named validator.
• Time to first token exceeded a budget derived from your p95 for that tier.
• A required tool call is absent from the response.
• Output hit the token ceiling and truncated mid-structure.
• The provider returned a retryable error class after a single attempt.

The trap is stacking several conditions behind an OR and calling that a policy. Two schema failures and a latency overrun are different problems, and they should not promote a request to the same expensive tier. Pick the primary condition for your workload and make the rest recoverable on the same rung. For a JSON extraction pipeline, that primary condition is almost always schema validation; for a chat product it is time to first token.

Braintrust makes the related point for routing generally: prompt complexity and price can estimate cost, but they cannot show whether a model will return a correctly formatted response (Braintrust on routing without hurting quality). The same applies one step later. The only trustworthy escalation trigger is a failure you measured on your own traffic.

Watch the output budget when the condition is truncation. Reasoning tokens are billed as output and share the completion budget, so a response can run out of room with the visible text still short (BenchLM on token pricing). In that case the fix is a bigger budget on the same tier, not a promotion to a frontier model that will also truncate.

Here is the ladder in full. Each rung gets one attempt. The only thing that advances the request is the condition from the previous section.

Two attempts at rung 1 sound generous and are usually a mistake. If the same model with the same prompt produced invalid JSON twice, the third attempt has the same probability of working as the second, and you have paid for it. Promote instead.

The tier price spread is the reason the ladder is worth building at all. Published list rates collected by Silicon Data show output prices that differ by an order of magnitude between a small model and a large one, while input prices differ by roughly the same factor (Silicon Data on cost per token). PostHog's observability documentation makes the operational point that follows: model cost tracking has to attribute spend per request and per attempt, or the ladder is invisible on the bill (PostHog on calculating LLM costs).

Take a support ticket classifier. Input is 3,150 tokens, output is 400 tokens per request, a workload shape Silicon Data uses in its own examples (Silicon Data). Suppose your measured rung 1 schema failure rate is 12 percent, rung 2 reduces that to 4 percent, and rung 3 to under 1 percent.

Per 10,000 requests, roughly 1,200 climb from rung 1 to rung 2, and about 48 of those climb again to rung 3. The blended cost is dominated by rung 1, which serves 88 percent of traffic at its own price. The 48 frontier calls cost almost nothing in aggregate even at a high per-token rate, because the volume is tiny. This is the worked example that surprises people: the ladder is cheap not because the top rung is cheap but because almost nothing reaches it.

The bad outcome is a ladder with a loose trigger. Raise the rung 1 failure rate to 40 percent (a common result when a schema is new and the prompt is untested) and 4,000 requests climb to rung 2, with 160 reaching rung 3. The blended bill more than doubles on output tokens alone. Braintrust's advice applies directly here: a lower-cost model should receive a query class only after its results meet the acceptance criteria for that traffic (Braintrust). The same gate should sit between rungs.

LLM proxy pricing calculators exist for exactly this forecast. LiteLLM's calculator takes input tokens, output tokens and request volume and returns per-request, daily and monthly cost, exportable as CSV (LiteLLM pricing calculator docs). Run it three times, once per rung, weight by your measured escalation rates, and you have a defensible number before anyone commits to the ladder.

The top rung is where most ladders quietly become infinite loops, because a frontier model failing feels like a reason to try harder. It is not. The stop rule has three parts.

First, a fixed attempt count across the whole ladder, not per rung. One cheap attempt, one mid attempt, one frontier attempt, then stop. Second, a degraded but valid response: the best parsed partial result from any rung, with an explicit error code naming the rung that failed. A user is better served by a complete answer to half their ticket than by a spinner. Third, a circuit breaker on the failing dependency so the next fifty requests do not repeat the experiment.

Circuit breaker mechanics are well documented for LLM apps: closed passes traffic, open fails fast, half-open probes with a small number of requests after a cooldown, and trips are driven by evidence such as error rate, timeouts and p95 latency rather than one noisy request (KryptoMindz on circuit breakers for LLM apps). The breaker is what keeps a provider outage from turning into a ladder that promotes every request to the most expensive tier simultaneously.

The escalation-matrix discipline from support operations transfers cleanly. A matrix works because it answers "who handles this, by when, and what happens if they cannot" in under a minute, with owners written as roles and criteria in plain language (IssueLinker escalation matrix). Your ladder is an escalation matrix for models: severity is the failure condition, owner is the tier, and response time is the attempt budget. Incident response guidance makes the same point about thresholds and ownership being explicit rather than inferred (incident.io on escalation policy).

A ladder you cannot measure is a budget leak. Log these fields on every attempt, tagged with the rung:

The latency field needs a warning attached. Total time across attempts is the number users feel, not the latency of the model that finally answered. Inworld lists both TTFT and completion in its recommended routing telemetry for exactly this reason, alongside input, cached input, output and cost (Inworld). Reporting only the successful model's latency makes the ladder look faster than it is.

Shadow mode is the cheapest way to calibrate the trigger before it spends real money. Run the candidate rung 2 model on real production requests without returning its output, capture the results, and compare pass rates against the rung 1 responses on the same inputs (Christopher Samiullah on shadow mode). Advanced deployment pattern guides describe the same technique as a pre-release check for load and input handling (APXML on canary and shadow testing). Once shadow data shows rung 2 clearing the schema at a materially higher rate, the promotion condition has evidence behind it.

The ladder is policy, so it belongs in one place that sees every model call. If your application calls three providers through three SDKs, you will implement the promotion rule three times and it will drift. A single inference endpoint with one API key per environment makes the ladder a config change rather than a refactor, which is the practical argument for routing all tiers through one gateway. Quanta Inference puts text, image, video, audio and embedding models behind one OpenAI-compatible, prepaid API priced in US dollars, so the three rungs can be swapped or re-priced without touching application code (Quanta Inference).

Two habits keep the ladder healthy after launch. First, review the promotion rate weekly; a rung that suddenly serves 30 percent of traffic is either a prompt regression or a provider problem, and both cost money. Second, test the stop rule on purpose. Switch the frontier tier to a failing model in staging and confirm that the degraded response, the error code and the breaker all behave as written. An escalation policy nobody has fired is a policy nobody can trust, which is the lesson support escalation design has been teaching for years (TechTarget on service tiers and escalation paths).