When a Model Ignores the Schema: Diagnostic Playbook

Before theorizing, replay. Take the failing request from your logs, including the full prompt, the schema you actually passed, and the model's raw output. Send it again with temperature near zero so you get a reproducible result, and keep the raw text of every response, since most "not valid JSON" reports come from stray commentary wrapped around the object the MLflow guide to structured outputs.

Then do one thing that surprises most teams: parse the raw output yourself with the same validator, in a scratch shell. You want to know whether the model produced something the validator rejected, or something your pipeline mangled before the validator saw it. Markdown fences, leading prose, and trailing explanations are all classic parse failures, and they are not model failures at all.

A minimal replay loop looks like this:

raw = call_model(prompt=prompt, schema=schema, temperature=0.0)

with open("trace_raw.txt", "w") as f:
    f.write(raw)                 # keep the bytes the model actually sent

parsed = json.loads(strip_fences(raw))   # your extraction layer, not the model
errors = validator.iter_errors(parsed)
for e in errors:
    print(e.json_path, e.message)        # fails at the exact field

If strip_fences is doing heavy lifting, that is already your answer. Fix the extraction layer first, because a validator that never sees valid JSON will keep generating false alarms.

This is the most common and least discussed cause. The model was given a schema, but the validator enforcing it applies extra rules: additionalProperties: false where the prompt allowed extra keys, stricter formats, a required array the prompt never mentioned, or a newer schema dialect whose keywords the validator interprets differently the JSON Schema validation guide.

Two concrete traps. First, defining a property under properties does not make it mandatory; only the explicit required array does, and a model reading a schema can miss that distinction just as a human reviewer can same guide. Second, dialect drift: JSON Schema is a family of drafts, and keywords can differ by dialect, so the schema's $schema declaration and the validator's configuration must agree same guide.

The decision rule: if the failing field appears in the schema the model saw but in a weaker form than the validator enforces, you have a validator problem. Fix the validator or regenerate the schema the prompt embeds from the authoritative validator config. Do not "tell the model to try harder" for a rule it was never given.

Test the schema itself, not just the responses

Schemas are production code and deserve their own tests. Feed each schema a known-good fixture and a known-bad fixture, and confirm the validator accepts and rejects them respectively. Teams that do this catch the validator-drift bug before it becomes an incident, and it is the same discipline as validating third-party responses in place rather than only in mocks validating third-party API responses with JSON Schema.

If the schema and validator agree, the model was told what to do and did not do it. The usual cause is salience. A schema embedded as a dense JSON blob, with the critical field buried three levels deep, competes with everything else in the context window. Prompted JSON gives no guarantee, which is exactly why schema-constrained generation and function calling became the default for production pipelines the MLflow guide.

Cheap prompt fixes, in order:

1. Restate the required array as a plain sentence above the schema, for example "Every response must include id, status, currency, total, and items."
2. Use the schema's exact property names everywhere, including in examples. Renamed fields are almost always a case where the prose used one word and the schema used another.
3. Add one filled example of the exact output shape. One in-context example does more work than a paragraph of instructions, and practitioners report valid-JSON rates moving from roughly 60 percent to above 95 percent after adding schema instructions and strict field rules, on their own measurement the structured output prompting guide.
4. Tell the model what not to do: no markdown fences, no commentary before or after the object. Parsers handle fewer surprises when the prompt names the failure mode the MLflow guide.

If none of this moves the failure rate, stop editing prose. You are at the capability boundary.

When a model follows simple schemas and misses complex ones, the fix is schema design, not a longer prompt. Three levers, in order of payoff.

Flatten nesting. Every level of nesting is another place to lose the thread. Splitting a deeply nested extraction into two calls, one per object, is often cheaper than fighting the model.

Replace open strings with explicit enums. A status field that allows any string invites paraphrasing; an enum of pending, paid, cancelled does not. Keep schemas minimal with required fields, strict types, and enums to prevent ambiguous outputs the MLflow guide.

Remove optional fields you do not consume. Every optional property is a chance for the model to include it when it should not, or format it in a way your code does not expect.

There is a ceiling here, and it is worth naming. A template-driven control that nearly eliminates syntax errors still leaves substantial structural and semantic errors, which shows the bottleneck sits beyond formatting the empirical study of structured output control. Simplifying the schema buys reliability, but it does not buy correctness of values.

If you are still on prompted JSON, move to schema-constrained generation or function calling. Constrained decoding guarantees syntactically valid output before a token is spent on something a parser will reject, and it reduces parsing errors better than prompted JSON, though it does not ensure value correctness the MLflow guide. This is the single highest-leverage change for teams still hand-parsing.

Before you swap models, stratify. Group failures by schema complexity and input length. A model that fails only on the most complex schema in your fleet is a schema problem wearing a model costume. A model that fails across simple and complex schemas alike is a capability problem. When you do need to compare candidates, compare models under the same extraction frame: API-native structured outputs and function calling are built-in provider features that enforce strict formats, while library approaches work across models but stay fragile the structured outputs and function calling guide. Comparing a constrained-output model against a prompted-output model tells you nothing useful.

If your failure rate is split across several providers, routing all of them through one endpoint removes a variable from the comparison. Quanta Inference puts text, image, video, audio and embedding models behind one OpenAI-compatible API key, which makes an A/B run against the same schema a config change rather than an integration project: https://quantaintelligence.ai/inference

A schema failure is a validation failure, and validation failures are the category where the orchestrator cannot repair the payload itself. The model must read the error, adjust, and produce a corrected request the LLM tool calling error handling guide. That distinction drives your retry policy: do not retry a schema rejection blindly, and do not route it to the same circuit-breaker path as a 503.

Two rules do most of the work.

First, feed the error back in structured form. An LLM given "invalid date format, expected ISO 8601 (YYYY-MM-DD)" can self-correct on the next attempt; an LLM given a raw stack trace usually cannot error handling and recovery patterns for agents. Include the failing JSON path, the expected type, and the observed value.

Second, classify the error before you retry it. Transient infrastructure errors deserve exponential backoff with jitter; permanent errors such as invalid input or a malformed request body will fail on every attempt and should fail fast AWS retry behavior, the Temporal non-retryable errors pattern. Cap retries at three to five attempts; if it has not worked in three, it will not work in thirty AI agent error handling that works.

One caution: retrying a run that already committed a side effect is a different problem. If the schema failure happens after a tool call fired, a retry can duplicate the write. Idempotency keys generated by the orchestration layer, not by the model, make the retry safe agent tool call retries and idempotency, 7 patterns that make agent retries idempotent.

Log both the raw and parsed output for every call, and track a validation failure rate as a first-class metric the MLflow guide. A method that works one day and fails after a model change is a regression you can only see if you measured it.

Keep a small regression suite of the inputs that once failed, each with its exact prompt and schema. A passing fixture suite of easy cases hides real-world misses precisely because the fixtures were generated from the same schema the model finds easy. The fixtures that matter are the ones that broke.

Watch latency and timeouts alongside correctness. A call that returns malformed JSON after a 30-second wait costs you twice, and recommended timeouts vary by operation type rather than being one global number commercetools error handling.

If the failures cluster in one line of business rather than one endpoint, the schema is a symptom of a workflow nobody has modeled end to end. That is when it helps to have AI experts read your context and hand back finished deliverables rather than another prompt to tune: https://quantaintelligence.ai/agents