What to Do When a Model Ignores the Schema: Diagnostic Playbook

Look at the raw text and sort it into one of four buckets before touching anything else.

The first bucket is no structure. The model replied with prose, an apology, or a clarifying question. That is a prompt or task-understanding problem, not a format problem.

The second is structure, but not parseable. The JSON is there, but wrapped in a code fence or preceded by a sentence. The Pydantic guide shows both cases producing the same JSONDecodeError: Expecting value: line 1 column 1 because json.loads rejects anything that is not the whole document.

The third is truncated. The JSON starts correctly and stops mid-token. This comment on JSON parsing failures makes the right point: a model response is untrusted input that can be cut off by a token limit, a timeout or a dropped stream, and treating it as guaranteed-valid is the error.

The fourth is valid JSON that violates the schema. It parses cleanly but has a wrong type, a missing required field or an invented one. The constrained decoding walkthrough lists exactly these failure modes: syntax errors, schema violations, encoding issues and format drift.

If the response is prose, your schema and retry logic are innocent. Changing the temperature will not fix a model that did not understand what you asked for.

A model that returns a fenced code block is not misbehaving: it is reproducing a pattern from its training data, where JSON often appears inside markdown. The Vellum article gives this as the canonical example, noting that the assistant emits backticks because those are likely next tokens in that context.

You have two honest options. The first is to prevent the wrappers at the source by using your provider's native structured output mode rather than a plain text instruction. When you do that, the constraint is applied during decoding, so the output cannot leave the grammar. This overview of constrained decoding describes the mechanism: at each step, the state machine derived from your schema masks all tokens that would make the partial output invalid, so the model samples only from legal continuations.

The second is to strip the wrapper defensively before parsing. Do it narrowly: detect a leading triple-backtick fence, read the optional language tag, and find the matching closing fence. Do not write a regex that grabs the first { and the last } in the string. That pattern corrupts any response where braces appear inside a string value, and you will only discover it after it has bitten you.

The decision rule is simple. If the endpoint supports schema-constrained generation, use it and treat wrappers as impossible. If it does not, or you are on a model or gateway where you cannot rely on it, strip explicitly and log every strip. That log count is a useful health metric: it should fall toward zero as you move work onto constrained endpoints.

A pass-or-fail validator wastes your time. You want the first failing path and the reason, in the same form the model uses.

Pydantic is the easiest way to get there in Python because the schema and the parser share a single definition. The Pydantic guide shows the pattern: define a BaseModel, call model_validate_json on the raw string, and read the ValidationError if it fires. The error carries the location, for example packages.0.author, and a message. That path tells you immediately whether the failure is structural (top level wrong), a missing field or a type mismatch deep in an array.

The practical value of naming the path is that it sorts the failure into a repair action. If the path is empty and the error says "invalid JSON", you go back to the wrapper or truncation checks. If the path is a required field, the prompt needs to describe that field better, or the schema needs to make it optional. If the path is a type mismatch, the field description probably does not make the expected type obvious.

The MLflow guide to structured outputs adds the operational half: log both raw and parsed outputs, set a metric for validation failures and watch it over time. A schema-constrained endpoint guarantees syntactic validity but not value correctness, so you still need runtime validation. That distinction is the one teams most often miss: valid JSON is not the same as correct JSON.

Keep your validator independent of the API's own claim of compliance. Even when the provider says the output adhered to your schema, validate on receipt. A community report on schema validation changing without notice is a useful reminder that the validator on the other side of the wire is not something you control.

When the failure is not a wrapper and not truncation, the next question is which of the two other suspects is guilty: the prompt, or the schema. The fastest way to split them is a bisection.

Take your real prompt and pair it with a minimal schema: one required string field, one required integer. Run it ten times. Then take a minimal prompt ("extract the name and age from this text") and pair it with your real schema. Run that ten times.

The results tell you where to look:

• Minimal schema works, real schema fails: the schema is the problem. It is probably too strict, uses a construct the endpoint does not support, or asks for values the model cannot derive from the input.
• Real schema works with a trivial prompt, real prompt fails: the prompt is the problem. It is probably too long, ambiguous or missing the data the schema demands.
• Both fail: the model or the endpoint is the problem. Check whether you are actually in structured output mode.

This is why the schema matters as a contract. A production guide to schema validation recommends keeping schemas small and purposeful, with enums for fields that drive branching and constraints that match downstream logic. A schema is not a wish list. Every required field is a promise the model can find that value in the input.

The JSONSchemaBench paper is worth reading if you suspect the schema. It benchmarks constrained decoding across thousands of real-world schemas and reports that coverage of JSON Schema features varies a lot between frameworks. A schema feature your framework does not support can be silently dropped or cause the constraint to fail, which looks from your side like the model ignoring you.

One more detail that trips people up: optional versus required. If a field is the only thing in a nested object and it is optional, some models will omit the whole object rather than emit null. Making the object required with a nullable field inside is usually more predictable than leaving both optional.

After the diagnostics, you should have a one-line diagnosis. Map it to one of three repairs, and do not split the difference.

You fix the schema when the model returns valid JSON that fails validation for reasons the model cannot control. Examples: an enum that excludes a value the input genuinely contains, a minLength that rejects a legitimate empty answer, or a nested structure the endpoint's constrained decoder does not fully support. The repair is to relax the constraint or restructure the object, then re-run the minimal-schema test.

You fix the prompt when the model fails on content, not form. Missing required fields, wrong types when the value is ambiguous, or a prose answer instead of an object. Concrete fixes that work: move the schema out of the prose and into the API parameter so it is not competing for attention, describe each field in one sentence, and give one short example of a filled object. The Agenta guide to structured outputs makes the separation-of-concerns point: when the schema lives in the tools or response_format parameter, the prompt holds only the data.

You fix the retry loop only when the failure is genuinely transient or stochastic. A rate limit, a timeout or a dropped stream deserves a retry with backoff. A schema violation caused by a systematic prompt gap does not. Retrying a deterministic failure just multiplies cost. This review of LLM error handling is blunt about it: 4xx errors mean the request is wrong and retrying the same request will not help, while 429s and 5xxs are transient and should be retried with exponential backoff and jitter.

A worked example with numbers makes the cost real. Suppose a pipeline runs 10,000 extraction calls a day and 4% come back failing validation. That is 400 failures. If each failure triggers three retries and half of those retries succeed, you are paying for 1,000 extra calls a day to recover 400 records, and you still have 200 records that never validate. Fix the prompt or schema once and you remove the 4% at the source, which is cheaper than absorbing a 10% call overhead forever.

When the fix is done, feed every captured raw failure into a regression set, so the next prompt edit is tested against the exact cases that broke before.

If the underlying problem is that you are juggling several providers with different structured output support, one option is to put text, image and other model calls behind a single OpenAI-compatible endpoint so your validation code stays in one place; Quanta Inference is built for that.

Truncation deserves its own section because it is the failure that most often gets fixed in the wrong layer. The tell is a response that ends mid-string or mid-object and a finish_reason of length. A JSON repair library will happily "close" the missing braces and hand you a valid object with half the records in it, which is worse than a crash because it fails silently three steps downstream.

The right sequence is: detect truncation from the finish reason, treat it as a distinct error class, and decide explicitly whether to raise the limit, retry with a smaller input, or fall back. The json_repair library is useful for recovering malformed output, but it should sit behind a truncation check, not in front of one. Repair is a safety net for the cases that slip through, not a substitute for knowing why the bytes stopped.

The same logic applies to function calling versus structured outputs. If you only need data reshaped from context the model already has, use structured outputs on the final response. If the model must decide to act and then receive a result, use tool calling. Mixing them up produces exactly the kind of half-structured response that is hard to diagnose.