JSON Mode, a Schema or a Tool Call for Structured Output

Failure modes are not shared evenly. Each approach has a signature break, and if you know it you can instrument for it.

Two failure modes deserve a closer look because they survive even the strongest constraint.

Truncation is a budget problem, not a schema problem. If the model hits max_tokens mid-object, the JSON is incomplete no matter how strict the decoding constraint was. GMI Cloud calls truncation the most common production failure for schemas with array fields, because array length is variable and a schema that usually produces three items occasionally produces thirty (GMI Cloud). The fix is a finish_reason check on every response plus a retry with a larger budget, not a stricter schema.

Silent coercion is the other one. A response schema that permits "42" where you wanted 42 will happily emit the string, and your downstream code may call int() on it and move on until 4,200 becomes 42. The schema is not wrong; it was under-specified. Put "type": "integer" in the schema, mark the field required, and let validation reject the rest.

One more category matters for anything that feeds an agent rather than a database. If you validate a tool response, remember that schema validation is a security control, not just a quality control. OWASP's MCP guidance recommends constraining tool response formats, requiring structured output with a fixed schema and rejecting responses that do not match, because free-text tool responses carry injected instructions into the model context (OWASP). A schema will not detect a polite instruction hidden inside a valid string field, so it sits alongside access controls, not in place of them.

Here is a rule you can apply in one pass, then the arithmetic behind it.

Use a tool call when the model must choose whether and how to act. Use a response schema when you need a fixed typed object back and nothing else. Use JSON mode only for simple shapes where you already validate strictly and cannot use a schema for some reason.

The arithmetic turns on retries, because each one is a full inference call at full cost and latency. Take a pipeline doing 50,000 extractions a day. Prompt-only JSON at a 32 percent post-processing error rate has been documented dropping to 0.4 percent once constrained decoding replaced the prompt-and-parse loop (tianpan). At 50,000 requests a day, 32 percent means 16,000 failures before you add any retry logic. At 0.4 percent it means 200. Even if every failure costs only one retry, that is 15,800 extra inference calls a day removed, before counting the ones your validator caught but your retry did not fix.

The benchmark literature says the same thing from the other direction. The JSONSchemaBench paper evaluates six constrained-decoding frameworks against 10,000 real-world JSON schemas and finds Wide differences in coverage, speed and output quality across implementations (arXiv). The takeaway for a practitioner is that "supports structured outputs" is not a single yes/no; test your actual schema against your actual provider before you trust the guarantee.

Gardening analogy aside, the rule survives contact with numbers. Retry cost, not token cost, is the variable that separates these approaches.

Yes, in one specific way: tool calling schemas are not portable.

Every major provider supports function calling, but each wraps the definition differently. In Anthropic's API the tool carries an input_schema; in OpenAI's the same fields sit under parameters inside a function object. If you want the tool to work across providers, you maintain two definitions or write a translation layer (Prefect). A response schema has the same problem at a lower level, since providers differ in which JSON Schema keywords they support.

This portability gap is why the Model Context Protocol exists: it moves the schema and the execution into a server that any MCP client can discover at runtime, so you define a tool once rather than once per provider (MCP specification). The protocol specifies a tools/list call that returns each tool's name, description and inputSchema, and a tools/call that passes arguments and returns content (MCP tools). Note the design detail: servers should return tools in deterministic order, which lets clients cache the list and improves prompt cache hit rates when tools ride along in the model context (MCP tools).

For a single-provider app, a direct tool schema costs less to build and debug. For an app that might switch providers, or that wants one tool exposed to several client surfaces, MCP pays for itself in the first migration. If you would rather not maintain per-provider wrappers at all, one API key across models removes the translation layer entirely; Quanta Inference is the route we publish for that at quantaintelligence.ai/inference.

The failure mode that does not show up in benchmarks is the silent regression after a model upgrade.

A prompt-tuned JSON request has no contract. The model can start renaming status to current_state or start emitting a string where it used to emit a number, and nothing in your code knows until a downstream query returns zero rows (Agenta). A response schema freezes the contract, so a model swap that still honors the schema cannot break the shape. A tool call freezes the contract and the decision surface, so you also notice if the new model starts picking a different tool for the same input.

The other upgrade hazard is over-constraint. James M. documents the common mistake of tightening enums until the model has no legal token sequence for a legitimate input, then falling back to a refusal or a hallucinated nearest match (jamesm.blog). A schema that is too strict converts a data-extraction problem into a decoding dead end. Leave a nullable field or an "other" enum value where real inputs are genuinely open-ended.

Provider support is uneven enough that you should check before committing. Vellum's comparison shows JSON mode and function calling available across OpenAI, Anthropic, Gemini and Mistral, while standalone structured outputs were available on OpenAI and Gemini but not the other two at the time of writing (Vellum). Treat that as a pattern rather than a permanent fact, and verify against current docs before you design around it.

Whatever you pick, the validation layer after generation should be the same, because shape is not truth. A schema-valid customer ID can still be wrong (TrueFoundry).

from pydantic import BaseModel, ValidationError, field_validator

class LineItem(BaseModel):
    sku: str
    qty: int
    unit_price_cents: int

    @field_validator("qty")
    @classmethod
    def qty_positive(cls, v: int) -> int:
        if v <= 0:
            raise ValueError("qty must be positive")
        return v

def parse_or_retry(raw: str, finish_reason: str, max_retries: int = 1):
    if finish_reason == "length":
        raise TruncatedOutput  # retry with a larger token budget
    try:
        return LineItem.model_validate_json(raw)
    except ValidationError as e:
        if max_retries == 0:
            raise
        return retry_with_error_context(e)

Three checks belong in every pipeline regardless of approach:

1. Parse check. Did the output parse at all? This only fails on JSON mode and on truncation elsewhere.
2. Shape check. Validate against the same schema you sent. For a response schema this should be a no-op, which makes it an excellent alerting signal: any failure here means a provider regression or a truncation.
3. Domain check. Enforce the rules the schema cannot express: date ranges, referential integrity, business invariants. This is where wrong-but-valid values die.

Add a fourth for tool calls specifically: authorization. A structurally valid argument object does not mean the caller was permitted to request it. MCP's specification is blunt that tools represent arbitrary code execution and that tool behavior descriptions should be treated as untrusted unless the server is trusted (MCP specification). Enforce permissions at the execution layer, not by asking the model nicely in the system prompt (OWASP).

If you are building the tool surface itself, that checklist is exactly what a Quanta skill encodes, so a coding agent scaffolds the schema, validator and permission check together rather than one at a time; the library lives at quantaintelligence.ai/skills.

The short answer from the top holds up. Tool call for decisions, response schema for objects, JSON mode only as a last resort with strict validation behind it.