When Escalation Is Cheaper Than Repair: A Cost Ledger

Take one task: extract a structured record from a messy invoice email, then check it against a schema. The receiver needs 8 output fields, one of which is a currency code drawn from a fixed list of 12.

One strong call. The strong model receives the instruction plus the email and returns the record. For the ledger I use round numbers and label them as illustrative, since your own rates and token counts will differ: 2,000 input tokens and 400 output tokens, billed at 10 times the cheap model's rate. The per-call price is therefore 10 times the cheap baseline, call it 10 units. It succeeds on the first attempt. A reviewer checks one record.

Three cheap attempts on the same task.

• Rung one, tightened prompt: same cheap model, instructions rewritten to name the 12 currency codes explicitly. Adds about 150 input tokens. Cheap enough that the token cost is noise, but it consumes one retry slot.
• Rung two, retrieved context: the same cheap model receives two previously accepted examples from the same vendor plus the schema. The input bill roughly doubles, and retrieval itself adds latency.
• Rung three, a different cheap model: a second provider with different tokenizer behavior and a different failure profile. You pay both models, and you pay a format-normalization step so downstream code sees one shape.

The important asymmetry is not the token arithmetic. It is the human. If a reviewer must read each cheap attempt to know whether it failed, the ladder's true cost is three readings plus one strong reading, not one. That is the wrapped cost the break-even rule is built on.

Escalation research points the same way. In a study that optimized model sequences and deferral thresholds from a pool of eight models, two-model cascades improved mean test-set accuracy over single-model selection by 2.1 to 8.2 percentage points, but allowing more than two models did not improve mean test-set accuracy in 118 of 135 comparisons and added at most 0.43 points the cascade depth paper. Depth beyond two rarely paid. That is an accuracy finding, not a cost finding, but it is a strong prior: the third cheap rung is usually the weakest rung.

Let S be the fully loaded cost of one strong accepted result: model cost plus the reviewer minute to check it. Let C be the fully loaded cost of one cheap attempt, including its reviewer or validator pass. Let p be the probability that the cheap attempt produces an accepted result and let q be the probability the strong call does.

Repair wins when:

p * S  >  C  +  (1 - p) * failure_penalty

Read it plainly: the expected saving from a cheap success has to clear what you spent on the attempt plus what a cheap failure drags in (a downstream retry, a re-run, a support message). If p is 0.5 and C is a fifth of S, repair wins comfortably. If p is 0.5 and C is three quarters of S because a person has to sit with each attempt, repair loses and you should have gone up a tier.

Worked example. Suppose S is 10 units for the strong model plus 6 units of reviewer time, so 16. A cheap attempt costs 1 unit of tokens plus 5 units of reviewer time, so 6. With p at 0.5, expected cheap cost per accepted result is 6 divided by 0.5, or 12, which beats 16. Now run the same numbers with a validator instead of a reviewer. The cheap attempt costs 1 unit and the validator is nearly free, so C is 1. Expected cost is 2 per accepted result. The strong call is 16. Repair wins by a wide margin.

The same task flips if the reviewer cannot check cheap output faster than strong output. Cheap models tend to fail in ways that read plausibly, so the reviewer spends the same minutes and catches fewer errors. When reviewer time is equal on both sides and tokens are a rounding error against salary, the ladder has almost nothing left to save. One survey of practitioners puts the fixed tension sharply: routing short, straightforward queries to a fast small model and reserving the large model for complex work is how teams hit latency targets no single model can hold the model routing chapter, and it notes that frontier models cost roughly 10 to 50 times more per token than a well-tuned small open model.

The decision rule I use in production has four gates. If any gate fails, go straight to the strong model.

Gate one: can a deterministic validator reject bad output without a human? Schema validation, regex checks against a fixed code list, length limits, tool-call argument checks, unit tests. If yes, cheap attempts are almost free to evaluate and the ladder has a real chance. If no, every attempt costs reviewer minutes and the math above collapses. Guidance on output parsing and validation treats format failures as a distinct engineering problem with its own tooling, which is why they should be handled by a parser and a schema rather than by re-prompting the parsing and validation guide.

Gate two: is the failure structural or stylistic? Structural failures include missing fields, wrong enum values, truncated output, and agent loops that repeat a tool call. Retrying the same model with a slightly different prompt does not change the failure mode. Those are exactly the cases where a stronger model is the cheap option in practice. Stylistic failures (tone, phrasing, verbosity) are the ones a tightened prompt can fix.

Gate three: does the added latency land inside the user's patience? Each rung adds a round trip, and rung two may add a retrieval step on top. Observability guidance for LLM and agent workflows exists precisely because cost and delay hide inside specific prompts, retrieval steps and repeated agent loops rather than showing up in aggregate totals the observability tooling survey. Trace each rung so you know where the seconds go.

Gate four: is the traffic volume meaningful? At ten requests a day, the token saving across the entire ladder is smaller than the engineering hour spent tuning it. Run the ladder where volume makes the arithmetic worth the complexity.

A checklist for the go-or-skip call:

Note how the routing literature frames the same tradeoff. Cascading is described as sequential escalation in which a cheap model's output is checked against a threshold and the query moves up only when confidence is low, while routing is a one-shot decision made before any generation the routing and cascades guide. The same source warns that LLM self-reported confidence is poorly calibrated, and that confidence thresholds must be evaluated on your own workload rather than set by intuition. If you cannot validate the cheap output, you cannot run a reliable cascade, and you are back to reading every attempt.

Retries exist for transient faults: rate limits, network timeouts, temporary API outages, context length errors. Libraries such as Pydantic's evaluation tooling expose retry configuration with bounded attempt counts and exponential backoff for exactly this class of failure the retry strategy docs. That is a different job from escalating a task that is too hard for the model. Mixing the two in one retry loop is how teams end up with twelve billable attempts on a task that needed a stronger model three months ago.

Keep them separate in code and separate in the ledger:

def solve(task, strong, cheap_a, cheap_b, validator, ledger):
    # transient faults only; 1 immediate try plus 2 backoff retries
    out = strong.call_with_backoff(task, attempts=3)
    if validator.ok(out):
        ledger.record("strong", attempts=1, review=1)
        return out

    # one rung only if a machine can reject bad output
    for rung, model in (("tight_prompt", cheap_a), ("new_model", cheap_b)):
        candidate = model.call(task, rung=rung)
        if validator.ok(candidate):
            ledger.record(rung, attempts=1, review=1)
            return candidate
    ledger.record("fallback", attempts=3, review=1)
    return strong.call(task)

Two details in that sketch carry the money. The validator runs before a human sees anything, and the loop is bounded at two rungs. A third cheap attempt is the one that rarely pays, per the cascade depth finding above, and it is the one that most often turns a five second wait into a fifteen second wait.

The cap also protects the reviewer. If a cheap attempt fails and you escalate, the reviewer sees two artifacts, the failed one and the accepted one, and may be tempted to compare them. Give the reviewer the accepted record only. Comparison is a debugging activity, not a production one, and it doubles reviewer minutes without improving the output.

There is a habit worth stealing from content management systems: drafts are private and working copies are explicit. A draft is saved automatically and is not visible to others, while a working copy is a deliberate checkout that locks the asset and can be committed, broken or reassigned the Cascade drafts documentation. Apply the same separation to model attempts. Cheap attempts are drafts, visible in logs and to engineers, never to the reviewer or the downstream system, until a validator promotes one to a committed result. A second help page makes the same point about resuming stale drafts and recommends submitting work rather than leaving edits parked in a draft the Northwestern Cascade help page, which is the review-queue version of "do not merge a stale attempt."

Record five columns per task class, over at least a few hundred real requests: model or rung, tokens in and out per attempt, attempts per accepted result, latency at the 50th and 95th percentile, and reviewer minutes per accepted result. Multiply tokens by your own rates, multiply reviewer minutes by a loaded hourly cost, and you have a per-task number you can compare across rungs.

Then run the comparison honestly, with the same acceptance criteria on every rung. The trap is grading cheap output against a looser standard, which inflates the ladder's win rate and hides its cost. One study of AI web search makes the same point in a different domain: access to a tool alone produced only a modest accuracy change in the cited results, while retrieval design moved accuracy from 0.40 with no retrieval to 0.66 in another benchmark the web search evaluation framework. The lesson generalizes: the pipeline around the model, not the model alone, determines whether the cheap path works.

Once you have the ledger, most task classes sort themselves into three buckets. Validator-friendly extraction tasks run a two-rung ladder. Reasoning-heavy tasks with judgment calls go straight to the strong model. High-volume chat with short outputs gets static tiering by intent, which the practitioner guides treat as the practical starting point before confidence-based escalation the routing and cascades guide. When you need to run several tiers side by side and price them on one key, Quanta Inference puts text, image, video, audio and embedding models behind a single OpenAI-compatible API so the ledger has one bill instead of five.