Structured Output from LLMs: Schemas and Validation

There are three common ways to get structured data from a model, and they offer different guarantees:

OpenAI's documentation draws the line between the last two clearly: JSON mode and Structured Outputs both produce valid JSON, but "only Structured Outputs ensure schema adherence," and it recommends using Structured Outputs instead of JSON mode wherever possible (OpenAI, Structured outputs). It lists three benefits: no need to validate or retry incorrectly formatted responses, safety refusals that can be detected programmatically, and simpler prompts that no longer need strongly worded formatting instructions.

Other providers and open source servers offer equivalent features. The HPE developer guide to vLLM, for example, describes guided output for a fixed list of choices, a regular expression, a JSON schema or a full grammar (HPE, Structured outputs in vLLM). If your provider supports schema enforcement for the model you use, turn it on. Prompt-only formatting should be a fallback for models that do not.

Schema enforcement is usually implemented with constrained decoding. ZeroEntropy's explanation is concise: at every step, the model produces scores for every token in its vocabulary, and the decoder sets the score of any token that would make the partial output invalid to negative infinity before sampling, so only legal continuations remain (ZeroEntropy, Constrained decoding).

Schema:   {"name": string, "age": integer}
So far:   {"name": "Ada", "age":
Allowed:  digits, minus sign, whitespace
Masked:   letters, quotes, "true", "null", closing brace

A state machine built from the schema tracks where the output is and which tokens can come next. ZeroEntropy notes two properties worth knowing. The model's relative preferences among the allowed tokens are preserved, so constraints remove invalid options without otherwise changing what the model wanted to say. And because the rule is enforced during generation rather than afterward, a parse error cannot happen. Its summary of where the field has landed: for "any output that downstream code parses," enforcement at inference time has won over prompt-only formatting.

The same mechanism can enforce other formats. Regular expressions suit dates, phone numbers and fixed labels. Full grammars suit SQL or a domain-specific language. Function calling is a specialization of the same idea, where the schema describes a tool's arguments.

Constrained decoding has a cost in some tasks. Research summarized by Brijesh Nambiar, including the paper "Let Me Speak Freely?", found that rigid formats can hurt reasoning-heavy tasks such as math and multi-step questions while helping classification tasks. The recommended pattern is a free-form reasoning field followed by a constrained final answer (Nambiar, Beyond free-form text).

A schema is a prompt as well as a contract. The model reads field names and descriptions to decide what to put in each field, so design them for clarity.

Define the schema in code, not as a hand-written JSON string. OpenAI's documentation notes that its Python and JavaScript libraries accept Pydantic and Zod models and convert them to JSON Schema (OpenAI). That keeps one source of truth for both the request and the validation of the response.

from enum import Enum
from typing import Optional
from pydantic import BaseModel, Field

class Category(str, Enum):
    billing = "billing"
    shipping = "shipping"
    technical = "technical"
    other = "other"

class TicketTriage(BaseModel):
    reasoning: str = Field(description="Two or three sentences on how you classified the ticket")
    category: Category
    order_id: Optional[str] = Field(description="Order number if the customer gave one, otherwise null")
    refund_requested: bool
    summary: str = Field(description="One sentence summary for the agent queue, under 25 words")

Design rules that hold across providers:

• Use enums for any field with a fixed set of values. The model cannot invent a category that does not exist.
• Make missing data explicit. A nullable field with a description such as "null if not stated" gives the model a correct answer when the information is absent, instead of pressure to invent one.
• Describe every field. Include units, formats and limits: "price in cents," "ISO 8601 date," "under 25 words."
• Forbid extra fields. OpenAI's schema examples set additionalProperties to false so the model cannot add keys you did not define.
• Put reasoning before the answer. Fields are generated in order, so a reasoning field placed before category lets the model think before it commits, which addresses the reasoning cost described above.
• Keep nesting shallow. Deeply nested and highly conditional schemas are harder for models and for constrained decoding engines.

A schema guarantees structure. It does not guarantee the content is right. An order ID can be well-formed and belong to a different customer. A date can be valid and in the past when it must be in the future. These checks belong in code.

The Instructor library, built on Pydantic, shows a common pattern: define validators on the model, such as a minimum title length or a rule that estimated hours come in half-hour steps, and let the library retry automatically when validation fails (Instructor). On a retry, the validation error is sent back to the model so it can correct the specific problem rather than guess again.

from pydantic import field_validator

class Refund(BaseModel):
    order_id: str
    amount_cents: int

    @field_validator("amount_cents")
    @classmethod
    def positive_and_capped(cls, v):
        if v <= 0 or v > 50_000:
            raise ValueError("amount_cents must be between 1 and 50000")
        return v

A complete handling path covers four outcomes:

1. Valid and correct: use it.
2. Valid shape, failed business rule: retry once or twice with the validation error in the prompt, then route to a person or return a clear error.
3. Refusal: OpenAI's documentation notes that safety refusals are detectable programmatically when using Structured Outputs (OpenAI). Check for them before reading fields, and handle them as their own case.
4. Truncated output: if generation stopped at the token limit, the object may be incomplete. Treat it as a failure, not as data.

Cap retries. A loop that retries forever on a request the model cannot satisfy burns money and hides the problem. Log every validation failure with the input, so you can see which fields fail and fix the schema or prompt.

Structured output and function calling use similar machinery and solve different problems. OpenAI's guidance: if you are connecting the model to tools, functions or data in your system, use function calling. If you want to structure the model's response itself, use structured output (OpenAI).

Machine Learning Mastery frames the choice by what the model needs (Machine Learning Mastery, Structured outputs vs function calling):

It also points out the cost difference. Function calling usually needs several round trips, each adding tokens and latency, while structured output typically completes in a single turn. Its hybrid pattern is worth copying: let an orchestrating agent use tools to gather what it needs, then pass the results to a final step that uses only structured output, so the response always matches what your UI or API expects.

Constrained decoding engines do not all support the same schema features. The JSONSchemaBench study notes that constrained decoding frameworks have standardized on JSON Schema, yet "there is poor understanding of the effectiveness of the methods in practice," and it evaluates engines on how efficiently and how completely they handle real-world schemas (JSONSchemaBench, arXiv). Nambiar's summary adds that not every engine handles advanced features such as oneOf or patternProperties (Nambiar).

In practice:

• Test your exact schema on your exact model and provider before relying on it. A feature that works in one engine may be ignored or rejected in another.
• Build an evaluation set of real inputs with expected outputs, including inputs where fields should be null and inputs designed to tempt the model into inventing data.
• Score fields separately. Category accuracy, extraction accuracy and null handling fail for different reasons.
• Version your schemas alongside your prompts, and run the evaluation set when either changes.
• Monitor in production: track validation failure rate, retry rate, refusal rate and truncation rate per schema.

If you build extraction or triage agents on Quanta Intelligence, the same rule applies: define the schema once in code, enforce it at generation time and validate the meaning before anything downstream trusts it.