Repairing Invalid JSON from LLM Responses

Before writing any recovery code, decide how much structure you can enforce at generation time. Three tiers exist, and they differ in guarantees, not just convenience.

OpenAI's Structured Outputs takes a supplied JSON Schema and ensures the model cannot omit a required key or invent an invalid enum value, and refusals become programmatically detectable (OpenAI structured outputs docs). On self-hosted models, vLLM exposes guided_choice, guided_regex, guided_json and guided_grammar for the same effect (HPE developer portal). The benchmark paper behind JSONSchemaBench evaluates six such frameworks on efficiency, coverage and output quality, so "it compiles" is not the same as "it preserves reasoning quality" (arXiv:2501.10868).

A practical decision rule: if you own the serving stack, constrain decoding at the engine. If you call a hosted API, use native structured outputs. Only fall back to prompt-only JSON when a provider supports neither, and budget for the repair layer you are about to read about. The trade-offs between native structured outputs, JSON mode and function calling, provider by provider, are laid out in Vellum's comparison.

One caveat on constrained decoding: if the tokenizer cannot cleanly express your grammar, some engines reject the schema at compile time or drop to character-level generation, which is slow (ZeroEntropy). Test your schema against your tokenizer before you ship it.

Recovery is only as good as diagnosis. Run these three checks in sequence and record which one failed.

1. Parse with a strict parser. json.loads or JSON.parse throws on the first syntax error and, crucially, reports a character position. Capture the position; it tells you whether the failure is at the start (fence or prose) or at the end (truncation). Python's error gives you a line and column you can map back to the offset.

2. Count brackets outside strings. A single left-to-right scan tracking string state and escapes tells you the imbalance. If depth ends at 1, you are almost certainly truncated mid-object. If depth returns to zero but extra content follows, you have prose or a second object.

def bracket_state(text: str) -> tuple[int, bool]:
    depth, in_str, esc = 0, False, False
    for ch in text:
        if esc:
            esc = False
        elif ch == "\\" and in_str:
            esc = True
        elif ch == '"':
            in_str = not in_str
        elif not in_str:
            if ch in "{[":
                depth += 1
            elif ch in "}]":
                depth -= 1
    return depth, in_str

A positive depth means missing closers. in_str being true means an unterminated string, usually a truncation or an unescaped quote.

3. Validate against the schema. Syntax-valid JSON can still be wrong: field renamed, type changed, enum value hallucinated. Validate with Pydantic, Zod or a JSON Schema validator and keep the error list. Model outputs are untrusted input at a trust boundary, and validation belongs exactly there (dev.to discussion on truncated responses).

Log all three results together: raw response, finish reason, parse error position, bracket depth, schema errors, model identifier, prompt hash. Without that tuple, you cannot tell a systematic problem from noise.

A blind retry with the same prompt tends to reproduce the same failure on the same deterministic settings. A corrective retry works far better because the model can see what went wrong. Send back the original input, the invalid output (clipped to the failure region if it is long) and the precise validator message.

Your previous response was not valid JSON.
Parse error: Expecting ',' delimiter at position 143
Invalid fragment: {"items": [{"sku": "A-12" "qty": 3}]}
Return the same data as valid JSON only. No prose, no code fences.

Two rules make this reliable. First, one corrective retry, not a loop; after two failures the cause is usually not formatting. Second, use a repair library's built-in reask pattern instead of hand-rolling it. Instructor wraps any of 15+ providers with automatic validation and retries on failure, and returns a typed object rather than a string you have to guess about (Instructor docs, GitHub repository).

Numbers matter here. If your raw failure rate is around 5 percent, which is the order of magnitude several practitioners report for prompt-only structured output (ZeroEntropy), a single corrective retry clears roughly 90 percent of those failures in typical extraction workloads, leaving around 0.5 percent needing repair or escalation. Each retry costs one more call, so at 20,000 requests a day you spend about 1,000 extra calls and rescue about 950 records. That trade is almost always worth it; a retry loop of five attempts is not.

Retries are for format errors. Do not retry on rate limits, timeouts or 5xx responses with the same logic, because those need exponential backoff with jitter, not a corrective prompt (error handling and retries).

When the retry fails, you have two recovery options: repair the syntax, or parse around it.

Repair. Libraries such as json_repair accept malformed input and reconstruct valid output, handling the common cases: missing quotes, trailing commas, unquoted keys, a missing opening brace, and text wrapped around the object (json_repair repository). Use it, but treat its output as untrusted, because repair guesses. Always run the schema validation step after repair and never skip it.

Partial parsing. For truncated output, close the structure yourself and parse what is complete, then accept only the fields that validate:

import json

def salvage(text: str) -> dict:
    depth, in_str = bracket_state(text)
    patched = text + ('"' if in_str else "") + ("}" * max(depth, 0))
    try:
        return json.loads(patched)
    except json.JSONDecodeError:
        return {}

A concrete example. Suppose a truncated response reads:

{"invoice": "INV-8891", "total": 1280.00, "vendor": {"name": "Acme", "tax_id": "GB

Bracket depth is 2 and a string is open. The repair gives you invoice, total and the nested vendor.name. You lose tax_id. Whether that is acceptable is a policy question, not a technical one: if tax_id appears on an expense claim, the record is not usable, and you should not let a half-parsed vendor object through.

The same logic applies to array truncation, where any complete objects already emitted are valid elements and the partial one is simply dropped. That is genuinely useful for lists of extracted line items.

The rule that keeps salvage safe: tag every recovered record with a provenance flag, for example recovered: true plus the specific fields that failed validation, and keep the raw response alongside it. Downstream consumers then decide whether a record with missing tax_id is admissible. Without the flag you have silent data corruption, which one practitioner describes as the worst outcome because the half-parsed object flows on and breaks something three steps later (dev.to comment).

Repair has a hard limit: it can fix syntax, not meaning. Decide your stop conditions in advance and encode them.

On the last point, the economics are unforgiving. Retry costs are real: if 10 percent of calls fail and each failure triggers 2 to 3 retries, you pay roughly 20 to 30 percent more for the same work and multiply latency accordingly, as Brightlume's analysis of agent failures notes (Brightlume). A prompt version that starts failing after a model update is a defect to be fixed, not a repair problem to absorb.

Logging is what makes escalation actionable. Store the raw response, the three detection results, the repair action taken and the human outcome. Over a few weeks you will see whether failures cluster on one schema field, one model, one input length range or one streaming path, and that cluster is usually fixable at the source. Background on how parsing errors surface and how to structure that telemetry is in the ApXML chapter on parsing errors.

One organisational note. If you run extraction pipelines across several providers, the repair layer, retry policy and validation schema should be shared rather than reimplemented per integration. A single gateway in front of every model keeps the failure taxonomy consistent; that is the problem Quanta Inference (https://quantaintelligence.ai/inference) is built for, with text, image, video, audio and embedding models behind one OpenAI-compatible API key, so your validation and retry code has one shape to reason about.

The whole thing is a short decision sequence, and each step should be one function with one log line:

1. Request structured output at the provider if supported.
2. On response, check finish_reason. If truncated, re-request with a smaller scope.
3. Parse strictly. On failure, run bracket analysis, then one corrective retry.
4. If the retry fails, attempt repair or partial parse, then validate.
5. Tag recovered records and keep the raw response.
6. Escalate per your stop table, and alert when failure rate crosses threshold.

The insight worth carrying away is that these steps fail in a specific order and each one tells you something different. Truncation is a budget problem. Fences and prose are a prompting or provider-feature problem. Schema violations are a prompt clarity problem. Repair is only ever a bridge to a human or a fixed pipeline, never a permanent substitute. The formal framing of why generation fails without constraints, and how grammar-constrained decoding eliminates the class, is covered in the constrained decoding overview and the survey of constrained decoding libraries.