Naming the First Failing Field: A Validator Wrapper

Before writing any code, know that you do not have to parse error strings. jsonschema's ValidationError carries the information in roughly three buckets: what happened (message), why it happened (validator, validator_value, schema_path), and what was being validated (instance, path, json_path) the official error-handling documentation.

The distinction between path and absolute_path matters when $ref is involved: absolute_schema_path is always relative to the original schema, not to whatever subschema was followed the jsonschema error reference. Use absolute_path for the data side and you will not care.

One trap: iterating errors and picking the first is not the same as picking the shallowest. Errors from iter_errors arrive in validation order, which is not sorted by how useful the message is. Sorting by path length, then by insertion order, gives a stable "first failing field" that reviewers can predict.

Take a small product catalog. The schema requires an array of items, each with an integer id, a non-empty name, and a non-negative price; the draft 2020-12 shape below follows the pattern used in a worked API validation tutorial the learn-api JSON Schema walkthrough.

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "type": "object",
  "required": ["items"],
  "properties": {
    "items": {
      "type": "array",
      "minItems": 1,
      "items": {
        "type": "object",
        "required": ["id", "name", "price"],
        "properties": {
          "id": {"type": "integer", "minimum": 1},
          "name": {"type": "string", "minLength": 1},
          "price": {"type": "number", "minimum": 0}
        },
        "additionalProperties": false
      }
    }
  }
}

A plausible model response:

{
  "items": [
    {"id": 1, "name": "Kettle", "price": 29.0},
    {"id": 2, "name": "Mug", "price": 8.5},
    {"id": 3, "name": "Grinder", "price": "nine ninety-nine"}
  ]
}

Strictly, more than one error exists: "nine ninety-nine" fails the type check, and minimum also fails on a string for some validators. The wrapper's job is to report the one that a fix should target, which is the type failure at items[2].price, not a cascade of downstream complaints.

The wrapper is a thin function. It returns a discriminated result instead of raising, so the retry loop decides what to do. Everything below uses only jsonschema public attributes.

from dataclasses import dataclass, field, asdict
from typing import Any
from jsonschema import Draft202012Validator

@dataclass
class FieldError:
    path: str            # "$.items[2].price"
    segments: list       # ["items", 2, "price"]
    keyword: str         # "type"
    expected: Any        # "number"
    actual: Any          # "nine ninety-nine"
    message: str         # validator's own text, as a fallback

@dataclass
class Result:
    ok: bool
    data: Any = None
    error: FieldError | None = None

def to_json_path(segments) -> str:
    out = "$"
    for seg in segments:
        out += f"[{seg}]" if isinstance(seg, int) else f".{seg}"
    return out

def validate_first(schema: dict, instance: Any) -> Result:
    validator = Draft202012Validator(schema)
    errors = list(validator.iter_errors(instance))
    if not errors:
        return Result(ok=True, data=instance)
    errors.sort(key=lambda e: (len(e.absolute_path), e.absolute_path))
    e = errors[0]
    segments = list(e.absolute_path)
    return Result(ok=False, error=FieldError(
        path=to_json_path(segments),
        segments=segments,
        keyword=e.validator,
        expected=e.validator_value,
        actual=e.instance,
        message=e.message,
    ))

Two details earn their keep. The sort by path length means a missing top-level key is reported before a type mismatch three levels deep, which is usually the right repair order. And segments travels alongside the JSONPath string because the prompt text wants items[2].price while your logging and dashboards want the list form, so keep both.

Draft202012Validator is reused here because constructing a validator once and iterating many instances is the pattern the library documents; jsonschema.validate() is a convenience call that raises on the first failure and returns None on success, which is fine for one-off checks but not for a loop the learn-api tutorial.

For the sample above the wrapper produces:

{
  "path": "$.items[2].price",
  "segments": ["items", 2, "price"],
  "keyword": "type",
  "expected": "number",
  "actual": "nine ninety-nine",
  "message": "'nine ninety-nine' is not of type 'number'"
}

Not every failure deserves the same response, and this is the decision rule most write-ups skip. Classify by where the failure occurs and what the payload looks like.

The bottom row is the one that burns teams. Structured-output providers validate the schema itself, and their interpretation can tighten without warning; a schema that worked for weeks began failing validation with no client-side change, forcing a rewrite to satisfy undocumented validator behavior an OpenAI community report on a schema validator change. Treat schema rejection as a deployment bug, not a model bug, and do not spend retry budget on it.

There is also a prior question: should you be asking the model for structured output at all, versus tool calling? Structured outputs suit single-turn data transformation where the model has everything it needs and just needs to reshape it, while function calling suits multi-turn flows where the model picks a tool and your code executes it the Machine Learning Mastery comparison. If the task is extraction, the wrapper belongs on a structured-output call. If the task is routing to a tool, validate the arguments with the same wrapper but expect a different retry shape.

The loop has one job: on failure, send back the model's own output plus a description of the wrong field, and ask it to fix that field. Describing the error for the model, not for a log, is the detail that does the work; the follow-up prompt should read like "field X must be a number, you sent a string", and the prior payload should ride along so the model edits rather than regenerates the dev.to writeup on error feedback retries.

def feedback_message(err: FieldError, prior_json: str) -> str:
    return (
        "Your previous response failed validation at one field.\n"
        f"Path: {err.path}\n"
        f"Rule: {err.keyword}\n"
        f"Expected: {err.expected!r}\n"
        f"Received: {err.actual!r}\n"
        "Return the full corrected JSON object. Change only that field.\n"
        f"Previous response:\n{prior_json}"
    )

def generate_validated(generate, schema, prompt, max_attempts=3):
    messages = [{"role": "user", "content": prompt}]
    last_error = None
    for attempt in range(1, max_attempts + 1):
        raw = generate(messages)              # your model call
        try:
            parsed = json.loads(raw)
        except json.JSONDecodeError:
            last_error = "unparseable"
            messages = messages[:1]           # plain retry
            continue
        result = validate_first(schema, parsed)
        if result.ok:
            return parsed, attempt
        last_error = result.error
        messages = [{"role": "user", "content": prompt},
                    {"role": "assistant", "content": raw},
                    {"role": "user", "content": feedback_message(result.error, raw)}]
    raise RuntimeError(f"validation failed after {max_attempts}: {last_error}")

Two operational notes. The follow-up prompt is longer, since it carries the error plus the prior payload, so cap attempts and track the per-path failure counter; a path that fails twice is a schema or prompt design problem, not a sampling problem. And if you fail over between providers mid-loop, do not count the provider swap as a real attempt, because prompt semantics do not always transfer the same writeup.

Suppose 1,000 calls, a 12 percent first-pass validation failure rate, and a per-request cost of one unit. Under blind retry with an optimistic 60 percent success per retry: 120 failures produce 48 residual failures after one retry, and roughly 19 after two. You also pay 120 extra calls, a 12 percent cost overhead.

Under feedback retry, the same 120 failures split by the table above. Truncated or unparseable bodies (say a third, 40 calls) go to plain retry; the rest (80 calls) get a named field. If feedback converts 85 percent of those to a pass on the next call, residual failures drop to 12 plus whatever the truncated group leaves, and the following call count is the same 120, but aimed. The saving is not calls, it is double retries and escalations. If you also shrink the schema to fewer required fields, you shrink the base failure rate directly; keeping schemas minimal with required fields, strict types and enums is the cheapest reliability lever available the MLflow guide.

Worth remembering that constrained decoding changes the mix. Grammar-based approaches filter invalid tokens during generation, so syntax errors shrink and the wrapper's findings concentrate on semantic problems like wrong values and wrong enum members; constrained decoding guarantees the output is syntactically valid but does not ensure the values are correct the MLflow guide. That means your feedback prompts should name values, not just types, as context counts shift.

When you generate the same schema across several model families, running one code path against every vendor keeps the retry loop honest. Quanta Inference exposes text, image, video, audio and embedding models behind one OpenAI-compatible API key, so generate(messages) above stays a single function while you compare validation failure rates per model. If the failure is really a schema design problem, a Quanta expert can take the schema, the failing payloads and the retry logs and hand back a tightened contract: Quanta Experts.

Malformed JSON is not always a model mistake, and you should not silently repair your way past a broken pipeline. Repair libraries exist and are useful: json_repair parses otherwise invalid JSON and tries to recover the missing values the json_repair repository. The key discipline is ordering. A three-stage pipeline of syntactic repair, fallback extraction, then schema validation is a reasonable shape, but the comment that prompted it is the real lesson: an LLM response is untrusted input, and the worst version of the failure is not the crash, it is the half-parsed object that flows downstream and corrupts something three steps later the dev.to comment thread on truncated responses.

Concretely: never let a repaired object skip the wrapper. Repair gets you something to validate; validation decides whether it enters your system. And make truncation a loud, handled case with its own counter, because a truncated response and a wrong field need different retries.

Two more edges are worth naming. If your schema uses additionalProperties: false and the model invents a field, the error path points at the parent object, not the invented key; the unexpected names live inside the message text, and extracting them means either parsing or a manual set difference between instance keys and schema properties, which is the workaround the jsonschema issue thread settled on the jsonschema issue on removing failing fields. Decide once whether invented fields are fatal or stripped, and encode that in the classifier rather than in the wrapper.

If you want human-friendly messages, the supported route is adding your own annotation to the schema and looking it up on the error, for example reading error.schema.get("error_msg", error.message), rather than regexing the default text the Stack Overflow thread on custom jsonschema messages. That keeps the wrapper dumb and the wording declarative.

Finally, observability. Log the raw output, the parsed object and the structured error for every failure so you can count failures per path and per model over time; the recommended pattern is exactly that, raw and parsed output logging with validation failure metrics, so regressions show up before users find them the MLflow guide. A wrapper that reports items[2].price is useful in a log; a dashboard of the top failing paths is what stops the next incident.