When to Use Structured Output vs Function Calling

Structured output generally works by constrained decoding. At each step the decoder masks tokens that would break the schema, so the model can only sample from legal continuations. The output is valid by construction rather than by retry ZeroEntropy on constrained decoding.

That has three consequences you should plan for.

First, schema-compilation overhead. Constrained decoding frameworks preprocess the grammar against the tokenizer to work out which tokens are valid prefixes. Regex and JSON Schema constraints are cheap to evaluate, while deeply nested grammars cost more per step ZeroEntropy on constrained decoding. In practice the latency added by constrained decoding is rarely the bottleneck, but the compile step shows up in tail latency when schemas are large or built dynamically per request.

Second, coverage limits. A benchmark of 10,000 real-world JSON schemas found very different schema coverage between constrained decoding frameworks: one widely used library covered around 3 percent of difficult GitHub schemas because compilation timed out, while another reached 96 percent with compile times around a hundredth of a second the JSONSchemaBench paper. The lesson is not "pick the better library" so much as "your schema complexity is a production variable, and it gets tested in staging, not in the demo."

Third, strict mode has shape limits. OpenAI's strict mode caps schema nesting depth and does not support recursive schemas, and providers reject schemas that cross internal complexity thresholds the structured output failure taxonomy. If your domain object is a tree of arbitrary depth, flatten it before you constrain it.

Function calling pays a different bill. Every tool round trip is an extra model call, so a three-step chain is three completions plus two executions. Tool definitions also sit in the prompt, so a large tool catalog adds input tokens to every turn. And reliability splits into four separate things worth measuring: whether the right tool was chosen, whether arguments were correct, whether the task finished, and how many tokens and retries it took Paragon's tool-calling optimization guide. A system that eventually gets the right answer after retries can still be too slow or expensive for production.

A rough cost comparison looks like this:

Single-pass extraction with a fixed schema. Invoices, contracts, support transcripts, resumes. The model has everything it needs in the prompt; it just reshapes it. Structured output is the right tool, and the schema library route is mature: Pydantic models produce JSON Schema and validate the response, and libraries like Instructor wrap the retry loop when validation fails the Pydantic guide to LLMs and the Instructor documentation. A worked example from Instructors documentation extracts a person from the sentence "John is a 30-year-old software engineer" into a typed model with name, age and occupation fields the Instructor documentation.

Classification into a closed set. Sentiment, intent, routing labels, severity levels. Enums in the schema do the work that prompt engineering used to do badly, and an invalid enum value becomes impossible rather than rare the OpenAI structured outputs guide.

Model-chosen tools with dependent arguments. "Book the cheapest flight that lands before my meeting" requires the model to decide which search tool to call, read the result, then decide whether to call the booking tool with arguments derived from the search output. No fixed schema can express that, because the second step's arguments are not known until the first step returns. This is function calling territory the Agenta guide to structured outputs and function calling.

Chat where the user supplies parameters over several turns. "Text to SQL" is a good example: the first user message often lacks enough context to write the query, so the assistant needs extra messages before the call. Structured outputs combined with function calling, where the function fetches schema information, works better than either alone the Vellum comparison.

Structured intermediate reasoning. An agent's "thoughts" can be a schema with a reasoning field, an assumptions field and a decision field. That forces a parseable chain of thought and keeps the audit trail typed. This is structured output used inside an agent, not instead of tool calls the Machine Learning Mastery decision framework.

Take an invoice-processing pipeline. Two designs, same model, same documents.

Design A: one structured output call per invoice. Input is the document text, output is a schema with invoice number, vendor, currency, total, due date and a line items array capped at twenty entries. One completion per document.

Design B: a function-calling agent with four tools: lookup_vendor, extract_line_items, convert_currency, post_to_ledger. The model decides which to call and in what order.

Now put plausible numbers on it, clearly labeled as an illustrative model rather than a measurement. Suppose the average invoice takes 900 output tokens in Design A and the average structured call costs 1.4 seconds at the 50th percentile. Suppose Design B averages 2.6 model calls per invoice (one decision, one extraction, one posting confirmation), plus two tool executions at 120 milliseconds each.

Design B's median is roughly 2.6 times the per-call generation time plus the tool time, which puts it in the several seconds range, and its prompt carries four tool definitions on every turn. Design A's cost is linear in documents. Design B's cost is linear in documents multiplied by an average number of round trips that can spike on ambiguous invoices.

Design B buys you something Design A cannot: currency conversion and vendor lookup happen against live systems, and the model can skip extraction when the invoice is a duplicate. Whether that is worth the multiple depends on how often the extra step fires. If 5 percent of invoices need a vendor lookup, Design A plus a rule-based lookup is almost certainly cheaper. If 60 percent do, the agent earns its cost.

That is the decision rule in numeric form: estimate the fraction of requests that need an out-of-band action or a decision. Below roughly a fifth, a single structured call plus deterministic code around it wins on cost and latency. Above that, function calling stops being overhead and starts being the product.

Structured output does not fail by returning invalid JSON when strict mode is on. It fails in quieter ways.

Silent truncation. When the token budget runs out mid-generation, the payload is incomplete and the finish reason says length, not stop. Providers guarantee schema conformance only for complete generations. Always inspect the finish reason before parsing, and retry with a larger budget or split the task the structured output failure taxonomy.

Schema-valid nonsense. Constrained decoding guarantees syntax and shape, not truth. A customer ID, amount or date can satisfy every constraint and still be wrong. Validation belongs after generation too, in the form of domain rules the schema cannot express the TrueFoundry guide.

Schema drift. A prompt that worked in testing breaks after a model update, and a field named status becomes current_state. Without a contract test, the first signal is a crash in production the Agenta guide.

Partial JSON from prompted output. If you skip strict mode and just ask for JSON in the prompt, you reintroduce the classic failures: markdown fences around the object, a chatty preamble, an omitted required field, a string where a number belongs. At scale, parse-error rates of one to five percent are common for prompt-only approaches ZeroEntropy on constrained decoding.

Function calling has its own set:

• Wrong tool chosen for an unambiguous request.
• Right tool, wrong arguments (a customer ID from a different record, a date range that misses).
• Unnecessary loops that re-fetch data the model already had.
• Confident action with no authorization check.

That last one matters most. A structurally valid tool call is an intent, not a permission. Authorization decides whether a well-formed object may cause an external action the TrueFoundry guide.

Three tests catch most of the damage.

Contract tests. Pin the schema as a fixture, generate a response, and assert that the parsed object matches the expected type. Run this on every prompt or model change. The point is to fail in CI rather than in a downstream parser the MLflow guide to structured output pipelines.

Latency percentiles, with and without strict mode. Measure the 95th percentile, not the mean, because schema compilation and long outputs both land in the tail. Compare a strict-schema run against an unconstrained run on the same inputs to get your real overhead, rather than assuming it is negligible ZeroEntropy on constrained decoding.

Validation-failure logging. Keep raw and parsed outputs, count validation failures per schema field, and alert on the rate. A schema field that fails 4 percent of the time is telling you the field is underspecified, not that the model is bad the MLflow guide.

Here is a minimal harness that makes the contract explicit:

from pydantic import BaseModel, ValidationError

class LineItem(BaseModel):
    description: str
    amount: float

class Invoice(BaseModel):
    invoice_number: str
    vendor: str
    total: float
    line_items: list[LineItem]

def parse_invoice(raw: str, finish_reason: str) -> Invoice:
    if finish_reason == "length":
        raise ValueError("truncated payload, raise the token budget")
    try:
        return Invoice.model_validate_json(raw)
    except ValidationError as exc:
        log_validation_failure(raw, exc.errors())
        raise

For tool-calling paths, add a fourth test: a fixed set of twenty requests with known correct tool and arguments, scored per call. Track tool correctness and argument accuracy separately, because they fail independently and need different fixes Paragon's tool-calling optimization guide.

Provider support is uneven, and that constrains the design. In a comparison of four major providers, JSON mode and tool calling were broadly available, while standalone structured outputs were not offered by every one, with one major provider supporting structured output only through function calling the Vellum comparison.

Two practical responses. First, treat the schema as yours and the enforcement as the provider's, so switching providers changes one adapter rather than every prompt. Second, prefer libraries that normalize the difference. Instructor exposes a unified interface across many providers so the same typed extraction call works whether the backend is a hosted API or a local model served by vLLM the Instructor documentation. Self-hosted serving engines expose the same idea through guided JSON, regex and grammar parameters the HPE guide to structured outputs in vLLM, and the ecosystem of constrained decoding libraries keeps growing the Awesome LLM Constrained Decoding list.

If you are routing across several model families for cost or capability reasons, the adapter layer is the place to keep this uniform. Quanta Inference exposes text, image, video, audio and embedding models behind one OpenAI-compatible API key, which lets you keep the schema constant while the backend model changes (https://quantaintelligence.ai/inference).

Before you ship, confirm each of these:

Work through the checklist top to bottom and the choice usually resolves itself. Most systems need function calling at exactly one layer, where the model picks an action, and strict structured output everywhere else. If you would rather have someone build that split with you against your own workload, a specialist through Quanta can take the spec and ship it alongside your team (https://quantaintelligence.ai/).