Streaming Structured Output: Handling Partial JSON

A partial JSON parser is not a relaxed parser. It is a state machine that explicitly models the open positions in the document: which strings are mid-character, which arrays are mid-element, which objects have an unbalanced brace (tianpan.co). JSON's grammar is small enough that you can enumerate the states by hand.

Two details trip people up. First, the escape state is not optional: "path": "C:\\temp" and "note": "line\nbreak" both contain a backslash that must not be treated as content, and a \u escape spans six characters. Second, numbers are greedy. You cannot know 12 has ended until you see a character that cannot continue it, which is why number values should only be committed when the delimiter arrives.

Here is a minimal character-level implementation. It is deliberately small; wire it into your own transport.

CONT = set("0123456789+-.eE")
LITERALS = {"t": "true", "f": "false", "n": "null"}

class StreamParser:
    def __init__(self):
        self.stack = []          # container frames
        self.state = "VALUE_START"
        self.buf = ""            # current token text
        self.pending_key = None
        self.root = None

    def push(self, chunk):
        for ch in chunk:
            self._step(ch)
        return self.snapshot()

    def _step(self, ch):
        s = self.state
        if s == "IN_STRING":
            self.buf += ch
            if ch == "\\":
                self.state = "ESCAPE"
            elif ch == '"':
                self._close_string()
            return
        if s == "ESCAPE":
            self.buf += ch
            self.state = "IN_STRING"
            return
        if s == "IN_NUMBER":
            if ch in CONT:
                self.buf += ch
                return
            self._commit_number()
            self._step(ch)
            return
        if s == "IN_LITERAL":
            self.buf += ch
            if self.buf in LITERALS.values():
                self._commit_literal()
            return
        # VALUE_START and container handling
        if ch.isspace() or ch == ",":
            return
        if ch == '"':
            self.state = "IN_STRING"
            self.buf = ""
        elif ch == "{":
            self._open("object")
        elif ch == "[":
            self._open("array")
        elif ch in LITERALS:
            self.state = "IN_LITERAL"
            self.buf = ch
        elif ch in CONT or ch.isdigit():
            self.state = "IN_NUMBER"
            self.buf = ch
        elif ch in "}]":
            self._close_container(ch)

    def snapshot(self):
        return self.root

The _open, _close_container, _close_string, _commit_number and _commit_literal methods attach to self.stack[-1] and, on the last close, set self.root. Keep them separate from _step so you can add validation hooks there without touching the state transitions.

The important invariant: snapshot() must never throw and must never return a value that is inconsistent with what has already been committed. That is the property that makes incremental rendering safe.

The single most useful decision rule is this: values that a user can read while growing are safe to render continuously; values that a machine consumes are not.

The last row is the one that bites hardest. A chat UI and a function-call argument want opposite behavior from the same stream: a chat UI probably wants partial strings, a function-call argument probably does not, because executing a tool with a half-typed argument is worse than waiting one more chunk (tianpan.co). Make that a per-field policy in your schema, not a global setting.

The practical way to implement it is to have the parser emit events rather than only a snapshot:

• key_start(path) when a key's opening quote is consumed.
• string_grow(path, delta) for each character inside a streamed string.
• value_complete(path, value) when a string closes, a number is delimited, or a container closes.
• container_close(path) when a brace or bracket balances the outermost frame.

Your renderer subscribes to string_grow for prose fields and to value_complete for everything else. This keeps the buffering policy in the view layer where it belongs.

If you would rather not maintain the machine, best-effort-json-parser follows exactly this contract: it returns partial data for incomplete input, so parse('[1, 2, {"a": "apple') yields [1, 2, { a: 'apple' }] (best-effort-json-parser docs). jsonchunk advertises the same shape with a typed object on every chunk (jsonchunk), and jsonriver is an incremental parser built specifically to feed UI refresh from LLM JSON, with an API that returns a series of partial values (JSON River on Hacker News). Note the author's own caveat there: jsonriver mutates existing values where possible rather than allocating new ones, so React or Lit memoization needs short-circuiting to notice updates.

Per-token validation is a tempting default and it is almost always wrong. Most checks are undecidable mid-token: you cannot reject "score": "hi as a type error before the closing quote, and you cannot reject a number as out of range until the delimiter arrives. Running a schema validator on every chunk also gives you the O(n²) problem in a different costume.

Validate when a value is decided: the moment its type is fixed and either the value is complete or growing in a known direction (tianpan.co). Concretely, put checks on these events:

• value_complete for scalars. Type check, enum membership, range, regex, required-field presence.
• container_close for objects. Cross-field invariants such as "if status is refunded then refund_id must be present."
• key_start for unknown keys. If your schema is strict, an unexpected key is a failure you can detect immediately, before the value is even generated.

There is a real cost argument for early detection. If the model emits {"score": "high" and your schema says score is a number, catching it at the closing quote lets you re-prompt or retry cheaply, because the model has not yet generated the rest of the document. Catching it after the closing brace is a wasted full generation (tianpan.co).

The tools you already use fit here. Pydantic models can be validated from JSON text and can generate the JSON schema that constrains generation in the first place (Pydantic for LLMs). The pattern is to parse incrementally, then hand each completed object to the same validator you use on the final result. That way there is exactly one definition of correct, and the streaming path cannot drift from the batch path.

One habit worth building: treat every completion as untrusted input, because it is. Truncation from a token limit, a timeout or a dropped stream are all ordinary events, and the worst outcome is not a crash but a half-parsed object that flows downstream and corrupts something three steps later (discussion of truncated LLM responses). The second link is a comment thread; the point holds regardless of who says it.

A stream that dies mid-document is not a failure of your parser. It is a normal transport event, and the question is what state you are left holding. Straightforward retry-on-error advice covers the transport layer: exponential backoff with jitter for transient errors, and extra care for streaming because you may already have sent partial content to the user (error handling and retries for LLM calls). The application layer needs its own answer.

The design that works is a commit log, not a live mutation:

1. Maintain committed, a deep copy of the object as of the last validated boundary.
2. Maintain working, the object the parser mutates as tokens arrive.
3. On each validated value_complete or container_close, promote working to committed.
4. On failure or disconnect, discard working and re-render from committed immediately.
5. Retry the request. When the new stream arrives, either replay it into a fresh working or, if your API supports it, resume and merge by key.

Step 4 is the one people skip, and it is why a mid-stream retry corrupts state: the UI has already rendered half of a value that will never be completed, and the retry writes a different half on top. Restoring the snapshot makes the regression visible and recoverable instead of silently wrong.

A worked example makes the cost concrete. Suppose the model is generating a 40-field extraction with an average of 12 characters per value, so roughly 480 characters of payload, not counting keys and punctuation. At a streaming rate that takes 4 seconds, a disconnect at second 3 leaves you with about 75 percent of the field values committed and one field half-written. If you promoted that partial field, your retry must either overwrite it (fine if it is a streamed prose field, dangerous if it is an ID) or carry a flag through every consumer. If you did not promote it, nothing downstream ever saw it, and the retry is a clean overwrite. The snapshot costs one deep copy per validated boundary, which for a 40-field object is trivial next to the retry it saves.

Two rules keep the snapshots honest. First, snapshot only at validated boundaries, never on a timer. Second, make the committed object immutable from the renderer's point of view, so a stale reference cannot be mutated by the parser after you promoted it.

It is worth being precise about what constrained decoding buys you, because it does not remove the need for a partial parser, and expecting it to is a common misread.

Constrained decoding masks the next-token distribution so only tokens that keep the partial output valid against a grammar or schema remain, which means the final document is valid by construction (ZeroEntropy). A 2025 benchmark assembled 10,000 real-world JSON schemas and evaluated six constrained-decoding frameworks on efficiency, coverage and quality, treating JSON Schema as the standardized constraint format (structured outputs benchmark). The curated list of papers, libraries and benchmarks in this space is also a good map of the terrain (awesome list).

What none of that does is tell your UI what to draw at token 40. Validity by construction is a property of the finished document, not of an arbitrary prefix. You still need the state machine to know which strings are open. You still need buffering policy to know which fields are safe to render. What you gain is that the failure mode shifts from "the JSON is broken" to "the JSON is valid but the content is wrong," which is a validation problem rather than a parsing problem, and a much better place to spend your effort.

There is also an architectural point worth remembering when you choose an interface. Structured outputs and function calling both pass schemas under the hood but serve different purposes: structured outputs are a single-turn constraint on form, while function calling is a multi-turn flow where the model pauses for your code to execute a tool (structured outputs vs function calling). Streaming a tool call means streaming arguments that must be buffered whole before execution, so if you find yourself streaming and executing arguments field by field, the design is fighting the tool.

If you are comparing the mechanism to JSON mode, structured outputs are the stricter of the two and JSON mode guarantees validity without schema adherence (Vellum on JSON mode, structured outputs and function calling). Pair a strict schema with the incremental parser and you get both guarantees: valid shape from generation, and a usable rendering at every step.

A reference pipeline, in order:

1. Request schema-constrained output if the provider supports it, so shape errors are impossible rather than repaired.
2. Feeding chunk text into the state machine, not into a JSON parser. Track string, escape, number and container context.
3. Emit events, not just snapshots, so the view layer can choose per-field render policy.
4. Buffer atomic values, render streamed ones, and never let a buffered field emit a partial value.
5. Validate at value_complete and container_close, using the same schema object that constrained generation.
6. Promote to a committed snapshot only at validated boundaries, and keep the working copy separate.
7. On failure, re-render from the committed snapshot, then retry with backoff.

The complexity is genuinely in steps 2 and 6, which is why the ecosystem exists rather than every team writing their own. The ZeroEntropy overview notes that building a robust constrained-decoding system for arbitrary tokenizers is non-trivial, and most production teams use established libraries rather than rolling their own; the same advice applies to the parsing side. Use a library, wrap it in the event and snapshot layers, and spend your own effort on the validation rules that are specific to your domain.

If you are routing several model families through one endpoint, keeping the parser and schema in one place gets easier. Quanta Inference serves text, image, video, audio and embedding models through a single OpenAI-compatible API key with prepaid US dollar pricing, so the streaming client and its partial parser stay the same across providers (https://quantaintelligence.ai/inference). For teams that would rather hand the whole extraction pipeline to someone who does it weekly, Quanta Experts takes your context and returns finished deliverables (https://quantaintelligence.ai/agents).