A Fallback Ladder That Ends at a Human When Extraction Fails

Here is the sequence, written so an engineer and a reviewer can both read it.

Two details make this ladder work rather than merely exist.

First, the cheap retry must carry a reason. Passing the validator's error message back to the model turns an unproductive re-roll into a repair attempt. Research on schema-guided extraction found that combining schema optimization with a validation-checked retry reduced extraction errors by 92 percent within the first retry (PARSE paper). The retry is cheap precisely because it is narrow: one more call against the same priced model, no human involved.

Second, rung 2 is not "the same call with more emphasis." It is a different model, chosen for capability rather than price, receiving more context: the original document, the previous attempts, and an explicit statement of what was missing. The routing policy should also allow narrowing the task before escalating. An invoice schema that extracts supplier, line items, totals, tax, and payment terms in one call fails as a unit when one field is ambiguous; splitting totals from line items isolates the failure (AI Tool Stack).

A cap without a definition is arbitrary. Define failure as a predicate that returns a boolean and a reason code, evaluated the same way on every rung.

def ladder(item, max_attempts=3):
    log = []
    out = extract(model="cheap", text=item.text, schema=SCHEMA)
    log.append(("rung0", out))

    verdict = validate(out, SCHEMA)
    if verdict.ok and verdict.confidence >= 0.8:
        return finish(out, log, path="auto")

    # rung 1: cheap repair, one shot
    out = extract(model="cheap", text=item.text, schema=SCHEMA,
                  prior=out, errors=verdict.errors, temperature=0.0)
    log.append(("rung1", out))
    verdict = validate(out, SCHEMA)
    if verdict.ok and verdict.confidence >= 0.8:
        return finish(out, log, path="retry")

    # rung 2: stronger model with full context
    out = extract(model="strong", text=item.text, schema=SCHEMA,
                  prior=[o for _, o in log], gap=verdict.errors)
    log.append(("rung2", out))
    verdict = validate(out, SCHEMA)
    if verdict.ok and verdict.confidence >= 0.8:
        return finish(out, log, path="escalated")

    # rung 3: human, always with context
    return enqueue_human(item, log, reason=verdict.errors, priority=item.risk)

Three automated attempts, counting the first, is a defensible default: the standard retry example in the tenacity documentation wraps an async call with a maximum of three attempts, exponential backoff starting at two seconds capped at thirty, and jitter of up to two seconds (tenacity documentation excerpt). Agent pipelines commonly settle on the same bound. The cap matters because every additional rung multiplies spend on items that are failing for structural reasons, and structural failures do not improve with repetition.

Treat rate limits differently from quality failures. Rate limits and timeouts deserve exponential backoff with jitter and do not consume your quality-attempt budget (Clarion.ai). A schema violation should never be retried with backoff, because waiting does not fix a missing field.

Circuit breakers belong at the same level as the cap. After a run of consecutive failures on the same schema or the same source system, stop dispatching automated attempts and route straight to review, which prevents a retry storm from hammering an upstream API while every item fails the same way (Clarion.ai).

This is the decision most teams skip. The question is not "how many retries can we afford" but "which rungs cost less than the human read they might prevent."

Do the arithmetic in the same unit for both sides. A reviewer reading one extraction, checking it against the source, and correcting it takes some number of minutes. Multiply by your loaded hourly cost to get a cost per human read. Then convert each automated rung into the same unit.

Worked example. Suppose a reviewer spends 4 minutes per item at a loaded cost of $45 per hour, so a human read is $3.00. Suppose the cheap model's extraction call for a two-page document costs a fraction of a cent, and the stronger model costs roughly ten times that but still well under $0.10. Now compare:

The decision rule falls out of the table. If the stronger model's call, including its longer input, costs less than a human minute, it is worth trying on every item that fails the cheap retry. In this example a human minute is $0.75, and rung 2 costs about $0.06, so escalation is clearly worth it: you would need it to succeed less than one time in fifty for the math to flip. That break-even calculation, not intuition, is what sizes the ladder.

Two corrections to apply before you trust the number. First, use your own figures: substitute the real per-call price you pay rather than a rounded estimate. Quanta Inference publishes prepaid per-model pricing in US dollars through one OpenAI-compatible API, so you can read the exact number for each rung from the same source (Quanta Inference). Second, include the cost of a wrong automatic answer, not just the cost of the call. On an invoice total, a wrong value caught later costs far more than $3.00, which pushes the break-even far in favor of escalation. On a low-stakes tagging task, the reverse is true, and a single retry with a human tail may be all the ladder you need.

A ladder you cannot measure is a ladder you cannot tune. Emit one event per item with the rung that produced the final answer, and aggregate weekly.

{"item_id": "inv_88213", "schema_version": "3", "final_rung": "rung2",
 "rung0_ok": false, "rung1_ok": false, "rung2_ok": true,
 "failure_codes": ["missing_field:tax_total"],
 "cost_usd": 0.061, "review_minutes": 0.0}

Four numbers tell you almost everything:

• Share resolved at rung 0. If this is high, the ladder is mostly insurance, which is fine.
• Share resolved at rung 1. A healthy repair rate means your validator messages are specific enough for the model to act on.
• Share reaching rung 3, the human rate. This is the headline metric. It is also the one that usually reveals a schema problem rather than a model problem.
• Cost per resolved item, rung by rung. If rung 2's cost per resolution exceeds a human read, delete rung 2.

Human review itself must produce structured data, or the ladder never improves. A defensible pattern, drawn from annotation tooling, is to agree on one sentence defining the criterion ("the extracted total matches the final revised total on the document"), record pass, neutral, or fail against it, attach a comment naming the evidence, and tag the failure type (Logfire human review docs). Those annotations then become test cases and calibration data for your validators, which is how a rung 3 rate that starts at 8 percent gets to 3 percent without buying a bigger model.

The same source notes that a saved verdict, comment, and tags are stored in your project and stay visible to your team, and that comments can contain user data (Pydantic docs). Treat review comments like production data and keep sensitive values out of them.

One more instrument worth having: persist retry state outside the process. If the orchestrator restarts mid-item, an externally stored attempt count prevents the ladder from silently starting over and exceeding its cap (fallback orchestration lesson).

Many teams jump from a failed extraction straight to a stronger model. Often the better rung 1.5 is a rule engine, not a larger model. Rules are explicit, auditable, and do not hallucinate, and they handle exactly the cases you have already seen: a total that must equal the sum of line items, a date that must precede today, a vendor code that must appear in an approved list (particula.tech).

The reason to insert rules before a stronger model is economics. A rule check costs approximately nothing and resolves a predictable class of failures with certainty. The three-tier pattern of model inference, then deterministic rules, then human judgment exists because each layer handles what it is best at: models handle ambiguity and pattern matching, rules handle compliance and known exceptions, humans handle novel situations and judgment calls (particula.tech).

Checklist for the rule rung:

• Does the rule have a single, testable assertion (sums match, dates in range, enum in allowed set)?
• Does a rule failure produce a human-readable reason that can feed the retry prompt?
• Is the rule versioned alongside the schema so a change is traceable?
• Does every rule failure still count toward the attempt cap?

If any answer is no, the rule belongs in the validator (rung 0), not as a separate rung. A rule that cannot explain itself to a reviewer is a rule that will confuse the review queue.

The middle tier also changes the economics of the top tier. Every item the rule engine resolves is a human read you never pay for, which means the rule rung is often the cheapest way to move the human rate, cheaper than buying a stronger model that may also be wrong (particula.tech).

"Sending it to a human" is not a fallback unless the reviewer receives the input, the output, the evidence, and the trigger reason together with the actions available (AI Tool Stack). A reviewer who has to re-open the PDF and re-derive what the model did is not reviewing, they are redoing the task, and the ladder has saved nothing.

Build the review card so the decision takes minutes, not tens of minutes:

1. The source passage or document page, with bounding boxes where the extractor supports them. Document extraction tools commonly offer bounding boxes in their reconciliation views for exactly this reason (Appian extraction docs).
2. The failing output, the validator message, and both prior attempts side by side.
3. One-click accept, edit, or reject, with reject requiring a failure tag.
4. The reason the item escalated, written in the vocabulary of the failure taxonomy, not the model's error string.

Track reviewer minutes per item as a first-class metric. If the review card takes 12 minutes when your cost model assumed 4, every escalation-cost comparison in this article shifts, and usually in favor of more automation rather than less.

There is also a quality argument for a human rung that a stronger model cannot replace. In a study comparing two AI tools against human reviewers for data extraction in systematic reviews, the human reviewers remained the reference standard against which the AI tools were measured, which is why the escalation target must be a competent reader rather than a passing check (Cochrane Evidence Synthesis and Methods). Where the consequence of a wrong value is high, the rung exists to catch what the ladder could not, and it should be staffed accordingly, not treated as a leftover bucket.

Run the ladder for two weeks, then read the four metrics together.

• Rung 3 above 10 percent. This is a schema problem, not a model problem. Tighten field descriptions, split the schema into field families, and re-measure before spending money on a stronger model. Ambiguous or incomplete schemas are a known driver of hallucination and non-adherence in extraction, and the original paper's framing is that schemas were designed for human developers rather than for models reading them (PARSE).
• Rung 1 near zero. Your retry prompt is too vague. Feed the validator's actual error message back, not a generic "try again."
• Rung 2 with a low success rate and a high cost. Delete it. A rung that rarely converts is a tax on the rung below it.
• Human rate falling over weeks. The ladder is working. Resist the urge to add a fourth automated rung; past three attempts, marginal returns collapse and the cost of latency and complexity rises.

Stop adding rungs when the next candidate's expected cost per resolution exceeds the human read it would prevent. That is the same rule you used to size the ladder, applied once more.

If the bottleneck turns out to be the schema design or the review card rather than the models, that is a build problem an expert can take on directly rather than a tuning problem (Quanta Experts).