Retry Prompts That Quote the Failing Tool Call Back to the Model

Your scorer already separates failure classes. The retry loop should consume those labels rather than invent its own. The taxonomy has to be small, stable and mapped to an action. Here is the split that works in practice.

The principle behind the table is simple. A class is repairable only if the information needed to fix it is inside the error and inside the model's control. Wrong type, missing required field, disallowed extra property, argument outside a documented range: all of those are visible from the error and fixable by rewriting the call. A timeout tells the model nothing it can act on. A permission error is a statement about your credentials, not about the model's output. A rate limit is a statement about upstream capacity. A not-found error usually means the model invented an identifier, and asking it to invent a different one makes things worse.

There is independent evidence that the argument layer is where a lot of agent pain lives. Analyses of tool calling report that malformed arguments, missing fields and type mismatches are a leading cause of agent breakage in production, distinct from reasoning failures the argument-rot analysis. A loop that repairs exactly those and escalates everything else is aimed at the right surface.

One practical detail: keep class names identical between your scorer, your logs and your retry prompt. If the scorer writes schema_violation and the retry prompt says "invalid arguments," you cannot count repair success by class, and counting by class is how you find out which tools keep failing for the same reason.

The retry prompt is only as good as the error it quotes. Most Python stacks give you a validation exception with structured fields, and you should read those fields rather than regex the human-readable message.

The jsonschema library documents its ValidationError payload in three groups: what happened (message), why it happened (context, cause), and what was being validated (instance, json_path, path, schema, schema_path, validator, validator_value) the jsonschema error handling guide. The validator field is the name of the failed keyword, and validator_value is the associated value in the schema. Those two fields alone give you a sentence the model can act on, and path tells you where in the instance to look.

A community thread on producing better custom messages from the same library shows the common shortcut: attaching readable error text to the schema and reading e.schema.get("error_msg", e.message) when a failure occurs the custom-message thread. That works for user-facing copy. For a retry prompt, prefer the structured fields, because they survive rewording of the message and because you can render the same failure at different levels of detail for logs versus the model.

A worked example makes this concrete. Suppose the tool is create_invoice and the model emits a string where the schema requires an integer, and omits a required field:

{"customer_id": "12345", "amount": "49.00"}

The schema says customer_id is an integer, amount is a number with a stated minimum, and currency is required. The error path is customer_id, the validator is type, the validator value is integer, and the instance value is the quoted string. A bad retry prompt says "validation failed, try again." A good one names the field, the expected type, the value received and the missing required property. The model is now patching two named defects and it keeps everything else in the call it already got right.

If your stack uses Pydantic AI, the framework already distinguishes these situations in its exception types. ModelRetry asks the model to try again with a message, while ToolFailed reports a terminal failure the model should see and adapt to but should not retry, and it does not consume the tool's retry budget the pydantic_ai exceptions reference. That is the same repair-versus-escalate split expressed as two exception classes. You can adopt it directly: raise the retry exception for schema and argument errors, raise the terminal one for timeouts, permission errors and not-found.

There is a known pitfall worth designing around. A Pydantic AI issue reports that framework-generated validation errors can be read by some models as user input rather than as an internal error, which motivates customizable retry text that tells the model explicitly that this is not a user message and what to do next the pydantic_ai retry-formatter issue. Your template should open with that framing.

The template below does four things in order: it marks the message as a runtime-originated repair instruction, it quotes the failing call verbatim, it states the class and the message, and it states the constraint that was broken.

This is a repair instruction from the runtime, not a message from the user.

The tool call you made failed before it executed. Nothing was performed on the
user's behalf.

Tool: create_invoice
Arguments you sent:
{"customer_id": "12345", "amount": "49.00"}

Error class: schema_violation
Message: '12345' is not of type 'integer'; 'currency' is a required property

Constraint broken: create_invoice.customer_id must be an integer per the tool
schema. create_invoice.currency is required and must be a three-letter ISO code.

Rewrite only the failing arguments. Keep every argument that was already valid.
Return one corrected tool call. Do not call any other tool. Do not explain.

Four rules make this template robust.

First, quote the arguments exactly as the model sent them, serialized the way the tool received them. If the model typed a quoted number, show the quoted number. Do not normalize it before quoting, or you erase the evidence.

Second, name the class in one snake_case phrase that matches your scorer's label. This lets you filter traces by class later, and it lets the model see whether it is being asked to fix a structural problem or something else.

Third, state the constraint, not just the error. The message tells the model what the validator objected to; the constraint tells it what the tool actually needs. For a schema violation, the constraint is the property path plus the expected type or format. For an argument error, it is the precondition from the tool's own documentation.

Fourth, constrain the output. "Rewrite only the failing arguments, keep the valid ones, return one corrected tool call" is a narrower instruction than "try again," and narrow instructions are easier to grade. The dev.to loop makes the same move by framing the task as fixing the error in the returned data rather than producing a response from scratch the dev.to retry write-up.

If you are designing the tool schemas in the first place, note that serialization format affects accuracy. Benchmark work summarized in an agent-evaluation review found that the same tasks run under many prompt variations performed better with JSON and Python return formats than with XML, and JSON function documentation beat both the tool-calling evaluation review. A JSON-shaped retry turn is consistent with the format where models already do best.

Here is a small loop that implements the decision rule. It assumes your validator raises an error object carrying a class, message, tool, arguments and constraint.

REPAIRABLE = {"schema_violation", "argument_error"}
MAX_ATTEMPTS = 2  # first call plus exactly one repair

def run_tool_call(model, call, tool, log):
    attempts = 0
    seen_classes = set()

    while True:
        attempts += 1
        try:
            return tool(**call.arguments)
        except ToolError as err:
            klass = err.error_class
            log.record(call, klass, attempts)

            if klass in seen_classes:
                log.escalate(call, err, reason="repeated_class")
                raise
            seen_classes.add(klass)

            if klass not in REPAIRABLE:
                log.escalate(call, err, reason="not_repairable")
                raise

            if attempts >= MAX_ATTEMPTS:
                log.escalate(call, err, reason="repair_budget_spent")
                raise

            call = model.next_call(
                previous_call=call,
                error_class=klass,
                error_message=err.message,
                constraint=err.constraint,
            )

The numbers to keep in mind are about cost, not accuracy. A repair attempt costs one extra model call and one extra tool invocation attempt that fails fast. If the retry turn carries the prior arguments plus a short error, the follow-up input is a few hundred extra tokens rather than a full context rebuild. You spend that only on calls that already failed, which is a small fraction of a healthy run.

The worked trace: a create_invoice call fails with schema_violation because customer_id is a string and currency is missing. Attempt one records the class, writes the retry turn, and attempt two sends the corrected call with an integer ID and a currency code. The tool accepts it. Total overhead is one extra model call and no human involvement.

Now the counterfactual. The same call fails with a timeout on the invoice service. The class is not repairable, so the loop escalates immediately. The human sees the tool name, the arguments, the class and the fact that this was the first failure. Had you retried, you would have spent a call and a timeout window to learn nothing.

The repeated-class rule deserves emphasis. If the first repair attempt produces a second failure of the same class, the repair path is not working for this tool, and a third attempt has the same odds as the second. Escalate with the full trace. The rule also catches a subtle case: a model that alternates between two error classes on one field is not converging, and a class-count check flags it fast.

There is a measurement point behind all this. A review of agent evaluation notes that public benchmark scores often overestimate reliability because they rely on single-attempt success, while consistency metrics measured across repeated attempts better predict production behavior the tool-calling evaluation review. A retry loop is a consistency mechanism, so measure it that way: report first-attempt success and post-repair success separately, and track repair success by error class.

The decision rule is short enough to put in a code comment, and it should be reviewed like any other control flow.

Retry once when the failure class is argument-level or schema-level: wrong type, missing required property, disallowed extra property, value out of range, or an argument that violates a documented precondition. In each case the error names the defect, the model controls the fix, and the tool did not execute, so nothing needs undoing.

Escalate when the class is a timeout, a permission error, a rate limit, a not-found, or a repeat of a class you already tried to repair. Each has a property that makes repair pointless: no actionable information in the error, a cause outside the model's control, or evidence that the repair path has already failed once.

A fourth bucket is worth having: retry once with a different call shape when the model chose the wrong tool entirely. The Berkeley Function-Calling Leaderboard exists to measure both function selection and argument accuracy across single- and multi-turn scenarios, and the same review notes that call accuracy is necessary but not sufficient for an agent to finish a task the tool-calling evaluation review. Wrong-tool failures look different from wrong-argument failures in your logs, and they merit their own class with their own retry text.

For the escalate path, the message back to the model should be a terminal tool result rather than a repair instruction. Say what failed, say it was not repaired, and say the agent should not retry. Pydantic AI's ToolFailed implements exactly this contract, and the docs note that you bound repeated failures at the run level rather than at the tool level the pydantic_ai exceptions reference.

Finally, log enough to audit the rule. For each failure, record the tool name, the class, the attempt number, whether the repair attempt succeeded, and the tokens spent on the follow-up. After a few hundred failures you will have a table that tells you which classes your repair prompt actually fixes and which tools should have their schemas tightened instead. If the same tool appears in the escalate list with schema violations repeatedly, the retry loop is not the fix; the tool's input schema or its documentation is.

When you need to run this loop across several models to check whether the repair rate holds, Quanta Inference puts text, image, video, audio and embedding models behind one OpenAI-compatible API key, which keeps the retry and escalation logic in one place instead of per provider at https://quantaintelligence.ai/inference.