When a Tool Description and Its Schema Disagree

An agent calls a transfer_funds tool. The prose says the currency is required. The schema omits currency from required and gives it no default. The model reads the prose, so it usually sends the currency. Occasionally it does not, because the conversation never mentioned one. The call validates, reaches your handler, and currency is undefined. A database insert with a null currency violates a constraint. Nothing was a schema violation, so no structure-level retry catches it; it surfaces as a wrong balance or a crash two services downstream.

Published write-ups catalogue exactly this family: malformed JSON, missing required fields, wrong types, out-of-range values, and injection through parameters a tool call validation guide. The same source notes that models omit required fields when context is ambiguous, which is the soft-constraint case where prose is the only source of truth. One widely read practitioner post calls the pattern "tool argument rot": valid JSON, wrong payload, tool either crashes or silently does the wrong thing the argument rot write-up.

The compounding matters more than any single failure. A model that produces a valid tool call 95 percent of the time looks fine until you compute the chance of a clean 20-step session: 1 minus 0.95 to the 20th power, about 64 percent failure, and even 99 percent per step leaves roughly an 18 percent chance of at least one bad output the structured outputs chapter. Each silent mismatch lowers the per-step number, and session-level reliability falls faster than intuition.

A controlled study that varied only the interface, holding tool semantics constant, found that schema-based contracts reduced interface misuse (malformed calls, missing fields, invalid enums) but did not reduce semantic misuse, meaning schema-valid calls that were wrong for the task the schema-first tool APIs study. The study used one open local model under tight step budgets, so treat the direction as informative rather than universal. The separation is the useful part: schema quality buys contract adherence, and prose quality remains responsible for choosing the right action. A conflict makes the model excellent at satisfying a contract that does not describe the task.

Sort conflicts by what the model does with each artifact, because the resolution differs.

The decision rule is short: the schema is normative for structure, the prose is normative for intent, and any constraint that matters at runtime belongs in both. If a constraint lives only in prose, the model may or may not honor it and your validator cannot catch a violation. If it lives only in the schema, the model sees a structural rule without knowing why, and may satisfy the letter while defeating the purpose.

Tool design guidance reaches the same conclusion from the other direction: tight schemas that encode constraints, enums for small value sets, and descriptions that define scope rather than just purpose AI agent tool design: what works and what doesn't. That source shows a field carrying description, pattern and bounds together, which is the ideal: the constraint lives in the schema and is explained in the adjacent description, so the two cannot disagree.

The mechanism determines what you can trust. The schema reaches the model as a typed contract that the serving stack may enforce during generation. With structured outputs, responses cannot omit required keys or invent invalid enum values the structured outputs guide. Function calling carries no hard guarantee in the same way: the model produces a structured call, and the application still validates before running it, as the tool calling overview describes LLM tool calling explained. Either way, whatever the schema permits the model may produce, and whatever it forbids the model usually cannot produce even when the prose says it should.

That asymmetry creates the nastiest class of conflict. If the prose says "the client ID must be a UUID" and the schema says {"type": "string"} with a loose pattern, the model may send anything the schema accepts. If the prose says "use at most 1000 rows" and the schema has no maximum, a schema-valid call can request a million. If the prose says "call this only for refunds" and the schema is generic, nothing stops a call for a charge. In each case the schema wins on the letter, the prose loses on the spirit, and no validator complains.

The reverse conflict is the one that gets caught. Prose says optional, schema requires it, the model omits it, strict enforcement rejects the call, and you see a failure and fix something. Conflict types differ in visibility, and the invisible ones matter more. That is why the table above records which side wins for each row: design for the failures you cannot see.

One empirical detail reinforces the split. In the schema-first study, interface misuse fell under schema conditions while semantic misuse persisted and dominated outcomes under constrained budgets the schema-first tool APIs study. Structural rigor makes calls well formed; it does not make them the right calls. The description's statement of purpose is the only lever on semantic misuse.

The audit has three steps: extract candidate constraints from prose, map each to a schema keyword, and report the ones without a home.

Step one, extract. Descriptions use a small vocabulary for hard constraints. Scan for modal and quantitative language: "must", "always", "required", "never", "at most", "between", "one of", "ISO", "positive", "non-empty", "read-only". Each hit is a candidate.

Step two, map. For each candidate, decide which JSON Schema keyword should encode it.

Step three, diff. Any prose constraint whose keyword is absent from the schema gets flagged. Then run the check in reverse, because schemas accumulate constraints the prose never explains: a maximum of 500000 on an amount field with no unit in the description is a trap for the model and the next engineer.

Extraction can start with regex plus a rules table, or use a structured-output classifier that reads the description and returns implied constraints in a fixed JSON shape. The structured-output route is more robust, and strict structured outputs are designed to guarantee the response matches your schema, so the classifier's output needs no defensive parsing the structured outputs guide. Keep the perspective that every layer has its own reliability profile: JSON mode guarantees valid JSON but not a specific schema, function calling gives high but imperfect compliance, and strict structured outputs approach near-perfect compliance the structured outputs chapter.

Here is the deterministic half in Python. It reads a tool definition, pulls constraint-bearing phrases from the description, and reports which schema keywords appear on each property.

import re

PATTERNS = [
    (r"\bmust\b|\brequired\b|\balways\b", "required"),
    (r"\bone of\b|\beither\b", "enum"),
    (r"\bpositive\b|\bat least\b|\bminimum\b", "minimum"),
    (r"\bat most\b|\bmaximum\b|\bno more than\b", "maximum"),
    (r"\bcharacters\b|\blength\b", "maxLength"),
    (r"\bISO\b|\bYYYY\b|\bformat\b", "pattern/format"),
    (r"\bdefaults? to\b", "default"),
]
KEYWORDS = {"enum", "minimum", "maximum", "maxLength", "pattern", "format", "default"}

def audit(tool):
    desc = tool.get("description", "")
    props = tool.get("inputSchema", {}).get("properties", {})
    required = set(tool.get("inputSchema", {}).get("required", []))
    findings = [(k, "implied by description") for p, k in PATTERNS if re.search(p, desc, re.I)]
    for name, spec in props.items():
        if name not in required:
            findings.append((name, "not in required"))
        findings.append((name, sorted(KEYWORDS & spec.keys())))
    return findings

Run it on an invoice tool whose prose promises a positive bounded amount, a currency enum and an ISO date, and the output shows the prose constraints the schema does not encode. The classifier pass, which handles paraphrase, orbits this rule table and never replaces it.

Resolution follows a fixed order. First, decide whether the constraint is real. If a field truly is optional, delete "always" from the prose and describe when to include it. If it truly is required, add it to required. Never leave a constraint in prose alone if a violation would break something.

Second, move checkable format guidance into the schema. Dates become format: "date" or a pattern. Currencies become an enum. Row limits become maximum. Where the format is genuinely fuzzy, say so in the description and do not pretend the schema enforces it.

Third, make errors actionable, because a mismatch that reaches runtime should teach the agent how to recover. Agent-facing error design has a clear standard: state what failed, why, and what to do next, since an agent cannot look up a 400 Bad Request and a vague body leaves it retrying the same broken payload error handling for AI agents. The API design guide makes the same point for agent-callable endpoints: a stable error code, a human explanation, recovery suggestions, and contextual metadata how to design APIs that AI agents can actually use. A field-level diagnostic ("currency missing; allowed values USD, EUR, GBP") is a reconciliation message delivered when the model can act on it.

Fourth, add idempotency to any mutating tool so a retry after a validation failure cannot double-apply. The monday.com documentation lays out the pattern: send an Idempotency-Key, the first request executes and caches its response for 30 minutes, a retry with the same key returns the cached response with Idempotency-Replayed: true, and a concurrent duplicate returns 409 with Retry-After the monday.com idempotency documentation. That reference also draws a useful line: retrying after a timeout or a 5xx warrants a key, retrying after a 429 does not, because the original request was never processed.

Fifth, keep a gate in continuous integration. Whether you write tools against one vendor's API or route several providers through one endpoint, the reconciliation belongs in the build. Quanta Inference puts text, image, video, audio and embedding models behind one OpenAI-compatible API key, which keeps tool definitions identical across providers and limits the number of places a schema can drift Quanta Inference.

The harness has three layers, each cheap to build.

Layer one is schema validity. For every tool, assert the definition is a valid JSON Schema and that every name in required exists in properties. This catches hand-editing mistakes in seconds and needs no model.

Layer two is golden prompts. Write five to ten user utterances per tool, each with an expected call shape. Run them against your model, validate each returned call, and compare the argument set to expectations. This catches the optional-versus-required conflict in practice, because the golden prompts include the ambiguous cases where prose is the only guidance. If a prompt that should produce a currency omits it, the model followed the schema and your prose needs to become a constraint.

Layer three is a prose-to-schema diff test. Run the extractor in CI, snapshot its findings, and fail the build when a new one appears. Drive the list toward zero, and any later edit that reintroduces a prose-only constraint shows up as a diff.

from jsonschema import Draft202012Validator

def check_tool(tool, golden):
    schema = tool["inputSchema"]
    Draft202012Validator.check_schema(schema)      # layer one
    validator = Draft202012Validator(schema)
    for prompt, expected in golden:               # layer two
        args = run_model_for_call(prompt, tool)["arguments"]
        assert not list(validator.iter_errors(args)), prompt
        assert set(expected) <= set(args), (prompt, args)

Run layer two against more than one model if you can. Families differ in how closely they honor a description when it conflicts with a schema, and a mismatch invisible on one family can be a steady failure on another. A multi-provider gateway keeps the switch cheap; Quanta Inference exposes the same tool-calling interface across its catalog, so a harness written once runs against each model without a rewrite Quanta Inference.

Two notes on the harness itself. Keep the golden prompts adversarial with respect to the conflict: include the prompt where the user never mentions the disputed field, because that is where optionality or a default decides the outcome. And log the raw call beside the validation result, so a failure report contains both what the model produced and the rule it broke.