When the Pass Threshold Fails: A Fallback Ladder

Write the rule down before you write the code, because the rule is the part that survives model changes.

Two details make this work in practice.

First, escalate with the same prompt. If you change the prompt and the tier at the same time, you learn nothing from the outcome. If the stronger draft passes, you now know the prompt was adequate and the tier was not. If it fails too, the prompt or the task definition is the problem, which is a different ticket.

Second, retry once, not three times. A schema retry that carries the validator's actual error message ("field deadline must be an ISO date, got 'next Friday'") is a different request from the original. A second identical retry is not. Pydantic's retry documentation shows bounded attempts with exponential backoff as the standard pattern for transient failures, and the same discipline applies here: a stop condition you can name (Pydantic retry strategies).

Here is the routing function in a shape you can paste into a pipeline and adapt:

def route_after_draft(draft, validation, score, threshold):
    if draft.policy_flags:
        return "human_review", "policy flag"

    if not validation.ok:
        if not draft.retried:
            return "retry_same_tier", validation.error_message
        return "human_review", "schema failed twice"

    if score < threshold:
        if draft.tier < MAX_TIER:
            return "escalate_one_tier", None
        return "human_review", "strongest tier still under threshold"

    return "accept", None

The function returns a named action plus a reason string. Log both. After a few weeks you will have a distribution of reasons, and that distribution tells you where to spend engineering time: a pile of schema failures means your output contract or context length is wrong, not that your model is weak.

The case for escalation rests on one claim: when a draft passes structurally but scores low, the model has hit its ceiling for that task. Retrying the cheap tier is buying lottery tickets with the same numbers.

There is research support for a two-tier cascade. A study of LLM cascades, where a cheap model defers to an expensive one on low-confidence queries, found that two-model cascades improved mean test-set accuracy over the best single model by 2.1 to 8.2 percentage points across five benchmarks (Is Escalation Worth It?). That is a meaningful lift, and it is the lift you are buying when you escalate.

The same study carries the more useful warning. Allowing more than two models did not improve mean accuracy in 118 of 135 comparisons, and added at most 0.43 percentage points when it did. So the ladder should have exactly one escalation rung. Cheap tier, one step up, then a person. Deep chains of three and four tiers look thorough and mostly buy latency.

Now the cost side. Price spreads between tiers are wide enough that escalation is not free and not catastrophic. A pricing tracker that compares LLM APIs lists, for example, a mid-tier model at $2 per million input tokens and $10 per million output tokens against a top-tier model at $4 and $20 for the same volumes (LLM API pricing comparison). A 2x to 5x spread is typical within a family.

Work the numbers on a batch of 10,000 drafts:

• Average draft: 2,000 input tokens, 500 output tokens.
• Cheap tier at $2/$10 per million: 4M input tokens plus 1M output tokens, so $8 plus $10, which is $18 for the batch.
• Strong tier at $4/$20 per million: $16 plus $20, which is $36 for the batch.
• Escalate 20 percent of drafts: $18 for all of them at the cheap tier, plus $36 times 0.2 which is $7.20, so $25.20.

Escalating a fifth of traffic costs about 40 percent more than the all-cheap baseline and buys a stronger model on exactly the drafts that needed it. Compare that with the alternative of retrying the cheap tier twice on those 2,000 drafts: two more cheap passes cost about $7.20, roughly the same as escalating them once, with a smaller chance of clearing the bar. That is the decision rule in one sentence: if a retry and an escalation cost about the same, buy the one that changes the model.

For a broader treatment of the routing ladder from static rules up to cascades, the TrueFoundry write-up is a good structural reference (TrueFoundry on LLM routing). Their framing of routing as distinct from failover matters here: failover is for outages, routing is for fit, and your ladder is a routing decision made after the first response exists.

If you run drafts against several providers, keeping the escalation comparison honest is easier when every tier is behind one API surface. That is the role Quanta Inference plays: text, image and embedding models through one OpenAI-compatible key, prepaid in US dollars (https://quantaintelligence.ai/inference).

A schema failure is not a quality signal. It is an output contract violation, and the most common causes are boring: the model ran out of output tokens mid-JSON, it wrapped the object in prose, it used a key name from an earlier prompt version, or it returned a string where you asked for a number.

The fix is to retry the same tier with the validator's complaint appended. Guidance on handling LLM parsing errors recommends exactly this loop: validate the output, catch the parse failure, and send a corrective message back to the model rather than failing the request outright (Handling LLM output parsing errors).

A concrete repair message that works:

Your previous response failed validation.

Error: field "start_date" must match YYYY-MM-DD; received "March 3".
Also: the response was truncated before the closing brace.

Return only the corrected JSON object, with every field present,
and keep the total under 800 tokens.

Three things make this effective. It names the field and the expected format. It names the structural problem (truncation), which the model cannot see from the outside. And it restates the output-only constraint, which prevents a chatty apology in front of the JSON.

Two guardrails:

• Cap the retry at one. If the repair attempt also fails validation, hand off or fail cleanly with the raw output attached. A loop of four repairs on a model that keeps truncating is a token furnace.
• Fix the cause when the reason repeats. If your logs show truncation as the dominant schema reason, raise the output token ceiling or shorten the requested object. If you see missing optional fields, your schema may be stricter than your prompt.

One more distinction worth making: schema retries are not the same as transport retries. Rate limits, timeouts and temporary outages should be retried with backoff and no prompt change, and they should not count against the schema retry budget. Keeping those two retry counters separate keeps your metrics readable.

Policy flags are the rung people get wrong most often, because a policy failure feels like a quality failure. It is not. When a draft includes a customer's email address in a public summary field, or makes a claim your compliance team has not approved, or reproduces text from a source you do not have rights to, the model did what it was capable of doing. A stronger model will do it more fluently.

The routing decision is therefore: stop, flag, and send it to a person with context. That person needs the draft, the specific rule that fired, the source material, and one or two clicks to accept, edit or reject. Do not send them a bare queue entry that says "flagged."

Policy tiers deserve their own thresholds. A hard block (credentials, personal data in an outward-facing field) should block the pipeline entirely and page someone. A soft flag (an unsupported superlative, a competitor mention) can route to review without stopping the batch. Write the list down, keep it versioned, and log which rule fired, because the frequency of each rule is your roadmap for prompt and retrieval fixes.

This is also where an evaluation setup pays off. When the same policy rule fires repeatedly on the same content type, the durable fix is usually upstream: remove that content from retrieval, or add an explicit instruction with a counterexample. The human review rung is a safety net, not a training loop, and if it is catching 30 percent of drafts your upstream context is the problem.

A fallback ladder that nobody measures is a cost center with good intentions. Five counters cover most of it:

1. First-pass pass rate, by task and by tier. This is your baseline and the number that should move when you fix prompts.
2. Failure reason distribution. Scored, schema, policy, transport. Each has a different owner.
3. Escalation lift. For escalated drafts, the difference between the cheap score and the strong score. A healthy ladder shows a consistent positive lift; a lift near zero means your threshold is mis-set or the task is prompt-bound.
4. Schema retry success rate. What share of schema retries pass on the second attempt. Below roughly half, fix the contract or the output length instead of adding retries.
5. Human review volume and agreement. What share of handed-off drafts the reviewer accepts, edits or rejects. A high reject rate on escalated drafts means your escalation target is too weak for the task.

Set the score threshold from data, not instinct. Pull a few hundred past drafts that a human rated, sort them by evaluator score, and find the score below which humans consistently reject. That number is your threshold, and it will differ by task. The routing literature is blunt about this: thresholds set by intuition will be miscalibrated, and the right approach is to measure what fraction of queries the cheaper model gets right at each score on your own workload (LLM routing and model cascades).

A worked example of why per-task thresholds matter. Suppose your support-summary task has a first-pass pass rate of 85 percent with the cheap tier, and your marketing-copy task sits at 55 percent. If both share a 0.7 threshold and a single escalation policy, the copy task will escalate nearly half its traffic and blow the budget you planned around summaries. Either raise the cheap-tier quality on copy through better prompts and examples, or accept a higher escalation rate there and budget for it explicitly. One threshold for all tasks hides this in the average.

The same logic applies to choosing which model sits at each tier. The right-sizing guide to LLM routing makes the point that a large share of savings comes from static tiering, caching and effort caps before any learned router is involved, and that a wrong cheap answer costs more than an unnecessary strong call (Right-Sizing the Frontier). Your ladder is the static part. Get it right before you automate it.

Retrying the cheap tier on scored failures. The most expensive mistake, because it looks thrifty. Three cheap attempts at $0.0018 each is not cheaper than one escalation at $0.0036 if the cheap attempts have a low chance of clearing the bar. Measure the lift before you defend the retry.

Escalating on schema errors. You pay strong-tier prices to fix a missing comma. Worse, the strong model may return a different but still invalid shape, and now your logs mix two failure classes.

Auto-retrying policy flags with a "be careful" instruction appended. This sometimes works and always hides the underlying problem. The flag exists because you cannot rely on the model's judgment for that rule; adding more instructions does not change that.

Unbounded ladders. Cheap, mid, strong, stronger, human is three escalation hops. The cascade research found a third tier rarely helps. Spend the latency budget on a better first prompt instead.

No reason logging. Without the reason string you cannot tell whether the ladder is working, and you cannot show anyone the cost per failure class. The reason string is the cheapest observability you will ever add.

Thresholds copied from a benchmark. Another team's 0.75 is not your 0.75. Your task, your rubric and your reviewers define it.