A Compliance Scorecard for Two Candidate Models

The scoring loop has one rule: no retries and no repairs count as a pass. Send every fixture to both candidate models once, capture the raw response, and validate exactly what came back. If the model returned JSON wrapped in markdown fences or with commentary around it, that is still what your pipeline would receive, so record it as a failure unless your real pipeline strips fences before parsing.

Three things belong in the request so the comparison is fair. First, the same system instruction and the same schema description for both models, since providers differ in how much schema detail they need: the provider comparison at collinwilkins.com notes that some vendors silently ignore JSON Schema keywords such as external $ref, so a schema that works on one provider may not be enforced on another. Second, a temperature setting low enough that a single run is representative. Third, a hard output token limit that matches production, because truncation is a compliance failure in its own right: a truncated response has nothing to repair, and the comment thread on truncated LLM responses makes the point that a half-parsed object flowing downstream is worse than a loud crash.

Handle transport errors separately from schema failures. A 429 rate limit or a 500 is not the model failing to comply with the schema, it is the call failing, and the retry guidance in this error handling writeup is clear that transient errors should be retried with backoff and client errors should not be retried at all. Log transport failures in their own column so they do not contaminate the compliance rate.

Validation is the heart of the method. Python's jsonschema library raises a ValidationError carrying a validator keyword, a message, and an absolute_path or json_path that locates the failing value, as documented in the jsonschema error handling reference. That path is the thing you want on the scorecard: it tells you which field broke, not just that something broke.

A compact scoring function:

import json
from jsonschema import Draft202012Validator

validator_cache = {}

def score_output(raw_text, schema, schema_id):
    validator = validator_cache.setdefault(
        schema_id, Draft202012Validator(schema)
    )
    try:
        payload = json.loads(raw_text)
    except json.JSONDecodeError as err:
        return "FAIL", f"$ (parse error at char {err.pos})"

    errors = sorted(validator.iter_errors(payload), key=lambda e: list(e.path))
    if not errors:
        return "PASS", None

    first = errors[0]
    return "FAIL", first.json_path

Two details are worth keeping. First, sort the errors before taking the first one. iter_errors does not guarantee ordering, and the Stack Overflow thread on reading every error from jsonschema shows the same sorting pattern used to produce readable output. Without sorting, "first failing path" is not reproducible across runs, and a scorecard that changes when you rerun it is worthless. Sorting by path is deterministic; sorting by message is not, because message text changes when the schema changes.

Second, treat a JSON parse error as its own category with the path set to the document root. Parse errors are the most severe class: no field-level diagnosis is possible, so the frequency of parse errors per model is the first number you should look at. The APXML chapter on parsing errors and the Brightlume analysis both separate syntax failures from structural ones for exactly this reason: the fixes differ. Syntax failures sometimes respond to repair, as libraries like json_repair exist to recover malformed JSON, whereas structural failures need schema or prompt changes, and repair hides the real problem.

For additional detail, you can also record first.validator (for example type, required or enum) alongside the path. The validator keyword tells you what kind of mistake it was; the path tells you where. Together they usually name the fix.

Here is a hypothetical result set, constructed to show how to read the scorecard rather than drawn from a live benchmark. Two models, A and B, are scored on 120 fixtures in a single pass.

Aggregate the rows into a summary:

Read the summary in order, not just by the top number. A compliance rate of 85.0% against 95.0% looks like a clear win for B, and the failure profile agrees: B had no parse errors and no type mismatches, which are the two categories hardest to fix downstream. A's failures cluster in one place, type errors, which is a prompt or schema-description problem you can plausibly fix, while B's five required-field failures suggest it sometimes drops a key when the instruction is ambiguous. That is the decision rule: prefer the model with fewer failures in the categories your pipeline cannot absorb, and check whether the leading model's failures are fixable before committing.

The arithmetic also tells you what the rate costs you. At 85.0% compliance on 10,000 calls, you are handling 1,500 failures. At 95.0%, you handle 500. The Brightlume writeup notes that if a 10% failure rate triggers two to three retries per failure, you pay roughly 20 to 30 percent more for the same work and your end-to-end latency multiplies by two or three. Apply that to your own numbers: a 10-point gap in compliance is not cosmetic, it is a budget line and a p95 latency figure.

If you plan to run this comparison across more than two models, or rerun it whenever a model changes, the number of SDKs and request formats you have to maintain becomes the bottleneck rather than the scoring script. Quanta Inference (https://quantaintelligence.ai/inference) puts text, image, video, audio and embedding models behind one OpenAI-compatible API key with prepaid US dollar pricing, which means the fixture runner can hit every candidate without a rewrite per provider.

The scorecard is a diagnostic, not a verdict. After the first pass, group the failures by path and validator keyword and fix the upstream cause before you rescore. Recurring required-field failures usually mean the schema or the instruction needs to spell out the field more explicitly. Enum failures usually mean the model needs an example of the allowed values. Type failures usually mean the model is converting values (a price as a string) and needs a stricter instruction or a schema-constrained generation mode where the provider supports it.

The retry strategy should follow the evidence. The pattern that consistently beats a blind retry is feeding the validation error and the model's own bad output back into the next prompt and asking for a targeted correction, which is described in detail in this writeup on error-feedback retries. For a budget calculation, decide how many retries you will allow per failure and multiply by the failure rate from the scorecard; the model with the higher first-pass rate wins on both cost and latency unless its residual failures are in a category you cannot repair.

Keep a delta table so rescoring is comparable:

Two practical warnings. First, schema behavior can change without your code changing. The OpenAI community report on structured output schema validation changes describes a schema that was accepted in production and then rejected after a server-side validator change, so keep the scorecard in version control and rerun it when your provider changes anything. Second, re-score on the full fixture set after every prompt or schema edit; fixing one failure often moves another. The agenta guide to structured outputs and function calling makes the same point from the opposite direction: a prompt that works in testing can break after a model update, and only a fixed fixture set with recorded results tells you when that happened.

When you have settled on a model and a schema, the next step is wiring the validated output into the system that consumes it, and that is the point where the fixture set becomes an ongoing regression test rather than a one-off comparison. If the consumer is a web application or an internal tool, a specialist can take the validated pipeline and build it out with you at https://quantaintelligence.ai/.