When to Repair a Failed Response and When to Start Over

Read down the failure column, then across to the action.

The last two rows are the ones teams skip. A single repair attempt is cheap and worth trying. The second attempt is where the economics flip: if you sent a precise instruction naming the exact field and the model still got it wrong, the instruction is probably wrong, the schema is unsuited to the task, or the input genuinely does not contain the information. A third repair pays for the same failure.

A retry that resends the identical prompt is the reflex when a validation error is raised, and it is the weakest move available. Same prompt, same temperature, roughly the same odds: you pay for another round and hope the dice land differently (n hirschfeld on feeding errors back into the retry).

Compare three options with rough numbers. Suppose each call costs a cent or two and takes a couple of seconds end to end. The exact figure does not change the shape of the argument:

• Plain retry: you re-send the same input and instructions. If the failure was randomness, this works. If the failure was a structural mismatch between schema and data, it fails identically.
• Targeted repair: you re-send the original prompt plus the bad output plus the error message. It costs more per attempt than a retry because the payload is larger, and it usually succeeds on the first try because the model is editing rather than generating.
• Fresh call with a changed prompt: you spend one call on a better prompt, which is the only option that fixes a deterministic failure.

Repair is justified because it is more expensive per attempt than a retry and cheaper than a redesigned prompt plus the engineering time to replan it. The moment repair stops converging, you pay the higher per-attempt price for nothing.

Pushing structural constraints out of prose prompts and into a native schema also cuts the constant input cost of every call, since you stop resending formatting instructions. The schema itself is still billed as input tokens, but a compact schema plus a short semantic prompt usually costs less than a paragraph of rules and few-shot examples (agenta on structured outputs and function calling). A repair pass then adds only the delta of the bad output and the error line on top of that leaner base.

A repair pass built the same way as a plain retry defeats the purpose. The message has to be written for the model, not for a log line. There is a widely used pattern for this: capture the validation error, format it as plain feedback, add it to the conversation, and ask the model to try again with that information (Instructor retry mechanisms).

Here is a small example. The validator catches an enum miss on sentiment and an out-of-range confidence, and the repair message names both.

repair_message = f"""
The last response failed validation:
<errors>
- sentiment: value was "somewhat positive"; allowed values are positive, negative, neutral
- confidence: value was 85; must be a number between 0 and 1
</errors>

Return the corrected object. Change only the fields named in the errors.
Preserve every other field exactly as it was.

<previous>
{previous_output}
</previous>
"""

Two details make this work. The error is described in the model's language, not as a stack trace or exception type. And the model receives its own prior output as the artifact to edit, which keeps the parts that were already correct instead of gambling on full regeneration. Instructor's error formatting follows the same shape: a short bulleted list of failing fields with the plain-English correction, then the request to try again.

The example also hints at a design constraint. For the model to edit rather than regenerate, the output must be parseable enough to serialize back into the prompt. Truly empty or truncated output has nothing to hand back, so you need a normal fresh call under the repair path as the fallback (n hirschfeld).

Sometimes the failure is not model flakiness but an over-constrained contract. The simpler the schema, the more reliable the output; the more complex and restrictive the schema, the more likely the model fails the restrictions (nuric on structuring LLM responses with JSON schemas). If repair passes keep failing on the same field, check whether the constraint is one the model can satisfy at all.

The most common trap is portability. A schema that works with one provider can fail with another, because strict modes support only a subset of JSON Schema. Optional fields expressed as a union of a type and null, numeric constraints expressed as minimum and maximum keywords, recursive models using references, and union types can all be rejected or behave differently in strict mode, which is why teams write sanitizers that reshape a rich schema into a provider-compatible subset for the call (LangChain forum on JSON Schema sanitization). If you move between providers, this mismatch shows up as a validation failure no repair prompt can fix, because the failure sits at the schema boundary, not inside the response.

The other trap is depth. Deeply nested schemas, such as rich text documents with indefinite list nesting, are hard for a model to emit in one pass even with schema enforcement. The fix is dividing the task: break the schema into smaller, simpler pieces and prompt for each part separately rather than generating the whole tree in one response (Stack Exchange discussion on complex JSON generation). If repair keeps failing on a deeply nested object, the repair is a symptom and the schema needs splitting.

A useful check before blaming the model: generate the JSON Schema from your application type and read it. Pydantic emits schemas that follow the JSON Schema 2020-12 dialect and the OpenAPI 3.1 specification, with constructs like $defs, anyOf and exclusiveMinimum appearing where you might not expect them (Pydantic JSON Schema docs). Each of those is a potential provider compatibility issue.

Escalation needs an owner, a payload and a threshold, because "log the error and move on" is not escalation, it is data loss with extra steps.

A workable threshold has two triggers. The first is a failed second repair attempt on the same input. The second is the same failure class appearing on the same input twice, even across separate requests. The second trigger matters because deterministic failures quietly burn retry budget in every request: if a prompt consistently produces an enum miss on category, every request that hits it fails once and repairs once, and your dashboard shows a steady drip that looks like flakiness but is a schema-design problem.

The payload should carry enough to reproduce and fix the problem without asking the reporter anything: the original input, the exact prompt sent, the bad output, the validator error, the repair instruction, the repair output, the attempt count and the token usage per attempt. The Instructor exception object is a good model, since it exposes every failed attempt with its exception, raw completion, attempt count and token usage (Instructor retry mechanisms).

For tool calls, escalation has a second meaning: authorization. A structurally valid tool call is not the same as a permitted action. The runtime has to decide whether a schema-valid object may cause an external side effect, and that decision belongs on a separate branch from the repair loop (TrueFoundry). A validation failure on a refund request should never route to a repair pass that quietly produces a different, still-valid refund.

You cannot apply the rule without data, and the data has to be recorded at the moment of failure rather than reconstructed later. The guidance is blunt: log raw and parsed outputs, set validation failure metrics, and build in retries or fallbacks so schema violations surface early instead of silently regressing (MLflow on enforcing, validating and observing structured outputs).

The fields worth logging, per attempt:

Two fields do more work than they look like. Defect path is the classifier: a populated path means the failure was local and repair was a legitimate option, while an empty path means the validator could not localize it and repair was a gamble. Output characters changed tells you whether repairs are genuinely targeted. If repair outputs routinely differ from the original in most of their characters, your repair is a full regeneration, and you should charge it to the fresh-call budget and ask why fresh calls were not the first choice.

Aggregate defect path by value and you get a leaderboard of your most frequent local failures. The top entry is almost always cheaper to fix at the schema or prompt level than to keep repairing. Aggregate outcome and you get a convergence rate: successful first repairs divided by repairs attempted. When that rate drops, something upstream changed.

If you run this across more than one model provider, the same metrics tell you whether failures are provider-specific. Routing through one gateway with an OpenAI-compatible interface means the same schema and the same repair loop go to every model, which is the only way a side-by-side failure comparison means anything. Quanta Inference (https://quantaintelligence.ai/inference) offers that shape: text, image, video, audio and embedding models through one API key, prepaid and priced in US dollars.

Suppose a document extraction pipeline processes 10,000 requests a day, and the initial call fails validation on 8 percent, so 800 failures. The logs split them: 70 percent local and nameable, 20 percent global, 10 percent truncated.

Applying the rule: 560 requests go to targeted repair, 160 return to the caller for more context, and 80 go to a fresh call with a larger output budget. The repair pass succeeds on 80 percent of its 560, so 448 are fixed in one extra call and 112 escalate. That is 560 repair calls, 240 fresh calls and 112 escalations a day.

Compare that to plain-retry-on-failure for all 800. If a blind retry resolves a local failure half the time, you spend 800 retry calls and still escalate 400. That is 240 extra calls and 288 extra escalations compared with the rule-based policy, and escalation is the expensive part because each one is a human or engineering minute.

The leverage sits in the middle column. Cutting the repair failure rate from 20 percent to 10 percent by writing better repair prompts and trimming the schema saves 56 escalations a day for the same number of calls. Cutting the local failure rate itself by fixing the top defect path saves more. Both are cheaper than adding a third repair attempt.

If many of your escalations are really "an expert needs to look at this input and decide," that is not a retry problem. Quanta Experts (https://quantaintelligence.ai/agents) is where that handoff lands: an expert reads the context, does the work and returns a finished deliverable, which is roughly what an escalation queue becomes once it has a stable input format.