Tool Error Recovery: Structuring Failures for Retries

The first decision any recovery flow needs is a class. Two buckets cover most cases:

• Transient: rate limits, timeouts, temporary network faults, provider overload, intermittent 5xx responses. These will often succeed on a later attempt.
• Terminal: validation errors, permission or authentication failures, missing resources, malformed arguments, unsupported operations. Retrying the same payload reproduces the same failure.

A single agent run also needs a third signal that is not strictly an error: needs clarification. Here the inputs are valid enough to execute but ambiguous enough that the model should ask a person rather than guess. Classifying errors at the tool-definition level into transient, terminal, and needs-clarification is what lets the control loop branch correctly instead of guessing, per Agent Works' recovery patterns.

HTTP status codes map onto this cleanly. A reference implementation from rapidclaw's agent error patterns treats 429, 500, 502, 503, and 504 as transient, plus any error whose message contains "timeout," and everything else as permanent. That is a fine starting point, but the classification must be your own, because the same status means different things in different APIs. A 409 Conflict on an idempotency-key conflict is arguably a success (the operation already happened), while a 422 on a schema mismatch is terminal and needs argument repair, not a repeat.

The table is the decision rule: class drives retryability, retryability drives the action. Do not let the model invent a fifth class.

Give the agent a fixed, machine-readable shape. Four fields carry most of the value: the class, a retryable boolean, a next_action (retry, ask a person, or stop), and the idempotency key that the retry must reuse. Add a human-readable hint, the offending field, and (where you have one) the set of allowed values. The earlier example from Pacholski shows a good enum hint in practice.

{
  "ok": false,
  "error_class": "terminal",
  "retryable": false,
  "next_action": "ask_person",
  "code": "invalid_field",
  "field": "status",
  "received": "done",
  "allowed_values": ["pending", "in_progress", "completed", "cancelled"],
  "hint": "Use 'completed' for a finished task. Ask the requester if unsure.",
  "idempotency_key": "ord-4821-charge"
}

Two details matter more than they look. First, the next_action should be a closed set of strings, so the agent can map it to a control-flow branch rather than paraphrase. Second, include the idempotency key in the error body. When the model decides to retry, it should echo that exact key, not generate a fresh one. If the retry carries a new key, the downstream service treats it as a different operation and the deduplication you designed is defeated, a point formation.dev's retry guide makes in detail.

You can see the same principle at the protocol level. In function calling, the tool result is returned as a distinct message tied to a call_id; that reference is what lets the model correlate an error with the call it made, so keep the correlation explicit in your payload rather than relying on prose, as OpenAI's function calling documentation describes. Providers also support schema-constrained outputs, which cut down on malformed calls in the first place: Cohere's strict_tools mode guarantees the tool name and parameters match the schema, eliminating hallucinated arguments, per Cohere's parameter-type documentation. Fewer malformed calls means fewer terminal errors to recover from.

Once an error is classified transient, the retry policy decides how and how many times. The baseline is exponential backoff with jitter, capped. A concrete, sensible default: base delay of one second, doubling per attempt, with jitter up to half the delay added, and three attempts total, per rapidclaw. Three is usually enough; if a transient fault has not cleared in three tries, waiting longer rarely helps and you should escalate.

The jitter is not decoration. Without it, many agents hitting the same rate-limited endpoint retry in lockstep and turn a brief limit into a thundering herd. With it, attempts spread out and the service recovers. Backoff controls timing and load; it does not establish whether a mutation already happened, which is why backoff and idempotency are separate concerns that both need solving, per formation.dev.

The part most implementations skip is the budget. A single global timeout starves later steps, and setting only per-step budgets lets a long run accumulate too many retries. You want both: a ceiling on retries for any one call, and a ceiling on total retries or total wall-clock time for the whole run. When either is hit, the agent escalates rather than continuing. A worked example:

1. Step 3 (charge the card) times out. Attempt 1 fails, ~1s backoff.
2. Attempt 2 fails, ~2s backoff.
3. Attempt 3 fails. Per-call retry budget exhausted.
4. The run step budget still has room, but this is a write operation, so the policy escalates: next_action: ask_person, status unknown, with the idempotency key attached so the humans (or a later reconciliation job) can check whether the charge landed.

That last point is the one an agent must never get wrong. A timeout after dispatch does not tell you whether the side effect committed, so "retry" for a write is really "re-send the same logical operation with the same key and let the service tell you the truth," per formation.dev.

Idempotency is the load-bearing wall. Without it, every retry is a coin flip on a duplicate side effect: double-charged cards, duplicate tickets, repeated emails. The failure window is simple and brutal. The tool performs the side effect, the write commits, and then the response is lost to a network timeout. The agent retries and the effect happens twice, as Neurobyte's idempotency walkthrough illustrates with an order-creation example.

The fix is a key generated once per logical operation and reused across every attempt at that operation. Three rules make it work:

• Generate the key in orchestration code from a durable execution step, not in the model. A freshly sampled key from the model identifies the retry as a new operation and defeats deduplication.
• Persist the key next to the approved arguments before dispatching, so it survives a crash or restart and can be reused by recovery.
• Enforce deduplication at the service that owns the effect. If the operation and its dedupe record share a transactional store, commit them atomically; if the effect lives in an external provider, carry the key through to that provider's own idempotency support or leave the outcome for reconciliation, per formation.dev.

A wrapper that "remembers successful responses" does not close the gap for an external provider, because a local transaction does not include the provider's commit. The service that owns the effect must enforce the guarantee.

There is a subtle and important case: the model changes the arguments on retry. Say the original charge was for $40 and the model now proposes $50 before the first one is resolved. Do not silently mint a new key and dispatch; both amounts could be charged. Treat the unresolved operation as blocking and reconcile before allowing a changed-argument write, per formation.dev. The same discipline shows up in permission guidance: writes, deletes, and financial operations should each carry different approval requirements, and standing permission to move funds or send email without a human gate is a structural risk, per MLflow.

Three worked flows cover the common shapes. Read them as pseudocode-level templates you can map to your orchestrator.

Flow A, transient read failure that resolves:

1. Tool returns 503 with error_class: transient, retryable: true, next_action: retry.
2. Orchestrator waits 1s + jitter, re-dispatches with the same args.
3. Attempt 2 succeeds; return the data and continue.

Flow B, terminal validation error the model can repair:

1. Tool returns 422 with error_class: terminal, retryable: false, a field, and allowed_values.
2. The agent must not re-send byte-identical arguments. It reads the hint, maps done to completed, and re-issues once.
3. Because the arguments changed, this is a fresh logical operation only if it was a write; if it was a read, no idempotency key is needed. If it was a write and an earlier attempt may have committed, reconcile first.

Flow C, exhausted budget on a high-risk write:

1. Three attempts time out with next_action: retry consumed.
2. Budget exhausted, and the call is above the risk threshold (funds, external comms, irreversible).
3. Pause and request approval with a structured escalation record: what failed, the idempotency key, and the known/unknown status of the side effect. Every gate decision lands in an audit trail, per Agent Works.

The output-format guidance from Pristren's tool-use patterns reinforces why the write path needs this care: tell the model what a tool returns, including empty and partial results, in the description so it handles edge cases rather than hallucinating structure. The same logic applies to errors, so document your error shape in the tool description and make it part of the contract. Hugging Face's function-calling guide shows how that contract looks for success cases, with a typed schema per tool; the error schema is the mirror image for the failure case, per Hugging Face's function calling guide.

Retries fix individual failures. They do not fix a dependency that is fully down. When a third-party API or a specific tool fails repeatedly, a circuit breaker stops sending traffic to it for a cooldown window instead of letting every run hammer a dead service. A small implementation opens the circuit after a failure threshold, holds it open for a recovery window, then lets one trial request through, per rapidclaw. Pair it with a fallback: a read-only lookup instead of a write, or a secondary model for that step. A real orchestrator bug shows the cost of getting this wrong: a platform-transient network error was terminalized instead of routed through the configured bounded recovery path, stopping a partial turn even though the error was recoverable, per this Cloudflare agents issue. The lesson is that your classifier and your recovery router must agree, or recognized transients become false terminals.

The final decision is whether the agent should keep going. Define escalation thresholds up front: after N failed attempts on the same step, on a terminal error with no valid retry path, or before any action above a risk threshold. At that point the agent pauses and asks a person rather than guessing. A short decision checklist:

Every "yes to escalate" branch should produce a structured stop record, not a stack trace. That record is what a person needs to finish the task the agent could not, and it is what makes the difference between an agent that fails loudly once and one that fails silently forever.

When you are building across many tools and models, having a reliable path to call any of them from one interface reduces how much recovery logic is provider-specific; Quanta Inference (https://quantaintelligence.ai/inference) exposes text, image, video, audio, and embedding models through one OpenAI-compatible API key with prepaid, US-dollar pricing, which keeps the error handling in your harness rather than scattered across SDKs.