Catching a Small Model's Fill-in-the-Blanks Before a Retry

Not every semantic failure deserves the same treatment. Group them into three buckets and your lint code stays small enough to reason about.

Invented references. The model produces an ID that is well-formed but absent from the source set you supplied. This covers customer IDs, product SKUs, ticket numbers, order IDs and account codes. It is the most common and the most damaging, because it usually passes every type check and fails only at the database write.

Skipped or hollow fields. The key exists because the schema forces it, but the value is "", " ", "N/A", "unknown" or null. Schemas rarely forbid the empty string, and small models use it as an escape hatch when they cannot infer a value. A field that is present but hollow is more dangerous than a missing field, because nothing downstream flags it.

Out-of-range values and illegal combinations. A total that does not equal the sum of line items, a percentage of 140, a status of "pending" on an object whose enum only permits open, closed and escalated, a start date after an end date. These are business rules, and no JSON Schema you would want to maintain will encode them all.

The column worth staring at is the third one. Every row in that table is something a JSON Schema can be taught, but the last four only stay correct if the rules live near the business logic, not inside a generated schema you treat as configuration. Practical guidance from the structured-output literature points the same way: keep schemas minimal with required fields, strict types and enums, then add application validation for the invariants the schema cannot know the MLflow structured-output guide.

The lint pass is a function that takes the parsed object plus the same context you handed the model (the allowed IDs, the known accounts, the catalogue) and returns either a clean object or a list of rule violations. Keep it deterministic, keep it fast, and never call a model from inside it.

ALLOWED_PRIORITY = {"low", "normal", "high"}
PLACEHOLDERS = {"", "n/a", "na", "unknown", "none", "tbd"}

def lint_ticket(obj, ctx):
    errors = []

    # 1. Invented references: every ID must exist in the source set
    if obj.get("customer_id") not in ctx["customer_ids"]:
        errors.append(("unknown_reference", "customer_id"))
    for sku in obj.get("line_items", []):
        if sku.get("sku") not in ctx["catalog_skus"]:
            errors.append(("unknown_reference", f"sku:{sku.get('sku')}"))

    # 2. Hollow fields: trim, then reject placeholders
    for field in ctx["must_be_filled"]:
        val = obj.get(field)
        if not isinstance(val, str) or val.strip().lower() in PLACEHOLDERS:
            errors.append(("hollow_field", field))

    # 3. Ranges and enumerations
    if obj.get("priority") not in ALLOWED_PRIORITY:
        errors.append(("bad_enum", "priority"))
    if not (0 <= obj.get("discount_pct", 0) <= 100):
        errors.append(("out_of_range", "discount_pct"))

    # 4. Aggregates must reconcile with their parts
    parts = sum(li["qty"] * li["unit_price"] for li in obj.get("line_items", []))
    if abs(parts - obj.get("total", 0)) > 0.01:
        errors.append(("aggregate_mismatch", "total"))

    return obj, errors

Three design notes that matter more than the code.

Pass the source set down with the request, not just into the lint. If the model is never shown the legal customer IDs, you are testing its ability to guess. If it is shown a list of 40 IDs from a retrieval step, invented references drop sharply and the lint catches what remains.

Normalize before you validate. Trim whitespace, collapse internal spaces, lowercase enum candidates, and parse dates into a single format. Half of all "hollow field" and "bad enum" rejections are formatting noise, not model error, and burning a large-model retry on a value that needed .strip() is the most common way this pattern gets abandoned.

Record the rule name, not just pass or fail. The tuple format above exists so that every rejection writes ("unknown_reference", "customer_id") into your logs. Without that, you can measure your failure rate but not fix it. With it, you can see that 70 percent of your small model's rejections are invented SKUs and solve that with better retrieval, not a bigger model.

The wider pattern is the same one evaluators describe for LLM systems generally: you need a check that produces a pass or fail signal you can count, and you should evaluate against a representative test set rather than a handful of hand-picked calls, because probabilistic regressions do not show up in single examples Sebastian Raschka's overview of LLM evaluation methods.

You cannot budget a retry policy without two numbers per model: the share of calls that pass lint, and the tokens each attempt consumes. Get them the same way you would get any quality metric, by running both models over the same frozen sample of real requests and diffing.

A suggested protocol:

1. Freeze a sample of 200 to 500 production requests with their context (the source sets, the catalogue, the customer records) into a fixture file.
2. Run each candidate model over the whole fixture, capture the raw output, and store the parsed object plus the lint verdict and rule names.
3. Run it twice more. Determinism is not something you get to assume, and the spread between runs tells you how much of your measured failure rate is noise.
4. Report pass rate per model, rules broken per 100 calls, and mean output tokens per attempt.

Do this before you pick a large model for retries, and re-run it whenever you swap either tier out. Model names and prices change faster than your business rules, which is exactly why routing decisions should be made against your own workload rather than a leaderboard the PrompTessor LLM routing guide. The same guide's checklist is worth copying almost verbatim: classify what the request needs, filter out models that cannot meet hard constraints, then optimize cost among the survivors.

Two practical cautions. First, the lint pass makes the small model look worse than it feels in a demo, because a demo never checks whether the ID exists. That is the point. Second, run the lint on the large model's output too, as an acceptance gate rather than a comparison. If the large model also fails a rule, the problem is your context or your prompt, and retrying it will not help.

The economics are simple in structure and easy to get wrong in practice. Let c_s be the cost of a small-model attempt, c_l the cost of a large-model attempt, and f the share of small-model calls that fail lint and get retried. The blended cost per successful response is:

blended = (c_s + f * c_l) / (1 - f * e)

where e is the share of large-model retries that also fail. If the large model is reliable enough that e is near zero, the second denominator term disappears and the decision rule is just c_s + f * c_l < c_l, which simplifies to f < 1 - c_s/c_l. In plain terms: retrying is cheaper than calling the large model every time as long as the failure rate stays below the fraction of large-model cost you save by starting small.

Work an example with round numbers so you can redo it with your own. Suppose a small-model attempt costs about a tenth of a large-model attempt, and you can measure tokens per attempt from your fixture run. Then c_s/c_l = 0.1, and the break-even failure rate is 0.9. That looks generous, and it is: almost any measured failure rate will still beat calling the large model every time. The real constraint is not break-even, it is latency and operational sanity, plus the token budget you set for retries.

A retry policy that survives contact with production has four parts:

• A hard retry share. Cap retries at 20 percent of calls in a rolling window. Above that, stop retrying and alert, because a jump in failure rate usually means a broken source set or an upstream context bug, not a model that got worse overnight.
• A per-request token cap. Count prompt and completion tokens for every attempt and store them with the request so you can compute blended cost per successful response directly from logs. This is the same cost-aware instrumentation that gateway teams argue for when routing traffic across a menu of models the TrueFoundry routing and cost articles.
• One retry, not a loop. Send the failing object back to the large model exactly once, with the lint errors included in the retry prompt. If the second attempt fails lint, return a typed failure to the caller instead of retrying again. Cascades that escalate on failure work because most traffic resolves cheaply, not because they retry indefinitely the same routing guide.
• A kill switch. If the retry rate crosses the cap for a full window, route everything to the large model until a human looks. That trades cost for service, which is the right trade when the alternative is a queue of invented IDs.

The retry prompt is where a lot of implementations lose the savings. Do not resend the whole conversation. Send the source object, the lint errors, and an instruction to fix exactly those fields, in the same schema. If your retry prompt is as large as the original, you have doubled the token cost of the failing subset for a small quality gain and made your blended-cost math meaningless.

For teams running more than one model tier, a single endpoint that takes a model name and returns token usage makes this accounting much simpler than juggling SDKs, which is what Quanta Inference provides through one OpenAI-compatible API key priced in US dollars.

The lint log answers a question that prompt engineers usually answer by intuition: which lever actually moves the number. Read the rule histogram before you change anything.

The last row is the cheapest win in the whole article. Every number you can compute deterministically is a number the model cannot get wrong, and every field you remove from the output is a field your lint no longer has to police. Do that pass before you touch the prompt.

On the prompt side, the durable lesson from production prompt engineering is that bigger instructions do not fix structural failures. A voice AI practitioner with hundreds of production prompts behind him describes the model as a new developer who needs context, examples and positively framed guidance rather than a long list of prohibitions, and notes that not every prompt follows the same template the voice prompt framework talk. Vendor documentation makes the same point from the other direction: a prompt is re-executed on every turn and every token in it costs latency, so structure and brevity beat volume the Vapi prompting guide. If you want positive examples to shape enums better, a few correct output samples for the failing rule belong in the prompt before a bigger model does.

The academic literature supports the same bias toward verification over instruction. Work on prompt engineering for large language models emphasizes that prompts shape behavior but do not guarantee it, which is why evaluation and iteration remain necessary the PMC review of prompt engineering. The structured-output study reaches a matching conclusion: enforcing structure is necessary but not sufficient, and the remaining gap is semantic the structured-output control study.

Three operational choices keep the lint and retry logic from becoming a subsystem of its own.

Version the rules with the prompt. A lint rule is part of the contract in the same way the schema is. When you change the enum whitelist or the tolerance on totals, bump the version and record it alongside the parsed output, so a spike in rejections can be traced to a rule change rather than blamed on the model. Treating prompts and their constraints as immutable, versioned artifacts with staged promotion is the established discipline here the LangWatch prompt management guide, and the same reasoning applies to the validators sitting next to them the MLflow prompt versioning comparison.

Log three things per request: the model that produced the accepted answer, the lint verdict and rule names for every attempt, and the total tokens across attempts. That triad is enough to compute blended cost per successful response, which is the only number your finance partner will ask about.

Test the failure path deliberately. Feed your lint pass objects with a valid schema and a fake ID, a whitespace-only field, and a total that is off by a cent. If any of those passes, your gate has a hole, and the hole will be found by a customer rather than by your test suite.

If you would rather not assemble the routing, retry and token accounting yourself, Quanta can take a written spec for the pipeline (source sets, schema, rules, retry cap) and build it with you, including the logging needed to prove the blended cost claim.