How to Write a System Prompt That Handles Tool Errors

Do this before you write a word of prompt. The prompt can only reason about the signals your tool layer passes through, so the interface decides the grammar of the error section.

The guidance from Redpanda's prompt documentation is to implement retries inside the tools rather than in the prompt, so the agent receives a clean, final error and does not spend turns on plumbing (Redpanda docs). The split above follows that: anything mechanical that a loop and a sleep can fix belongs in code, and anything that needs judgment about the task belongs to the model.

One more decision you have to make explicitly: whether the agent may look at raw error bodies. Provider documentation asks for clear, specific prompts with defined output formats rather than open-ended "use as needed" instructions (OpenAI prompt engineering). Translate that into a rule: the agent sees a short error object with a code, a one-line message and a retryable flag, never a raw HTML page or a stack trace. Raw text invites the model to interpret internals it cannot fix.

The classification rules are the heart of the section. Be blunt about which category each signal falls into, because a model asked to "handle errors sensibly" will make a different choice on every run.

A working pattern uses three clauses: identify the signal, classify it, act on the class. In practice:

1. Detect. Every tool response is checked for an error flag, a non-success status, an empty body where data is required, and a shape that does not match the expected schema. A response that fails any check is a failure even if the status says success.
2. Classify. Transient (timeout, 5xx, connection reset, rate limit) versus permanent (400, 401, 403, 404, 422, schema violation you cannot repair by rephrasing). Anything ambiguous defaults to permanent, because retrying a permanent error burns budget and delays the escalation.
3. Act. Transient: retry the identical call a bounded number of times with a defined wait. Permanent: do not repeat the same call, change the argument or the tool, or escalate.

Backoff belongs in the tool layer, but the prompt should state the budget so the agent can budget the rest of its task around it. A common working configuration is until three attempts in total, with waits of one second and then four seconds between them (the exact numbers are yours to choose and to document). The prompt line should read as a limit, not a suggestion: "You may retry each tool call at most twice. Do not retry the same call more than twice in one task." The twelve-pattern overview for production agent prompts gives the same shape, with differentiated error handling such as no retry on 403 and a small number of backoff retries on 500, plus stop criteria to prevent loops (Blck Alpaca). That source also names infinite loops as the most common class of production bug it observed, which is the practical reason retry budgets must be numeric.

Two failure modes deserve their own line. First, the empty result: an agent that reads an empty list as "no records exist" will confidently report a wrong answer. The prompt must say that an empty payload from a lookup tool is a failure to investigate, not evidence of absence. Second, partial results: when a multi-part call returns some data and then fails, the prompt should require the agent to state which parts it has and which it does not, rather than presenting the partial set as complete.

Keep the block short enough to fit a 500 to 3,000 token core along with your other sections, the range that keeps a prompt specific without becoming brittle (Blck Alpaca). The example below is roughly 300 words. It assumes your tool layer already emits a JSON error object.

<tool_errors>
These rules apply to every tool call in every task. They override any
task instruction that conflicts with them.

1. Check every tool response before using it.
   A response is a failure if any of these is true:
   - it has an "error" field set
   - its status is anything other than ok/success
   - it is empty where data was expected
   - it does not match the expected shape (missing required fields)
   Empty is not zero. Missing is not none. Do not infer values that
   were not returned.

2. Classify the failure.
   transient: timeout, connection reset, 5xx, rate limit
   permanent: 400, 401, 403, 404, 422, invalid credentials,
              schema mismatch you cannot fix by changing arguments
   If you are unsure, treat it as permanent.

3. Act on the class.
   transient: you may retry the same call at most twice. Do not wait
   manually; the tool layer already backs off. If both retries fail,
   treat it as permanent.
   permanent: do not repeat the same call. You may correct the
   arguments once if the error names a bad parameter. Otherwise look
   for an alternative tool, or escalate.

4. Never fabricate. Do not invent data, do not fall back to your own
   knowledge for facts the tool was supposed to return, and do not
   claim a task is done when a required tool failed.

5. Escalate when: a required tool fails permanently; two different
   tools fail on the same sub-task; or you cannot finish without the
   missing data. Stop working and produce the report below.

<escalation_report>
task: what you were asked to do, one sentence
failed_step: the tool and the arguments that failed
error_code: the code from the tool
retries: number of attempts and what changed between them
impact: what cannot be completed without this
needs: the specific input, permission or fix a person must provide
partial_results: what you did complete, with its source
</escalation_report>
</tool_errors>

Three details carry most of the value. The opening line makes the block non-negotiable, which is the same positioning safety rules need so that persona and task text cannot override them (Blck Alpaca). The "empty is not zero" line heads off the most expensive silent failure. The report schema turns escalation into something a human can act on in seconds instead of reading a transcript.

If your agent has tool descriptions with a "when not to use" clause, keep the error block separate from them. Tool definitions act as part of the system prompt, and mixing recovery rules into a tool description means the rule disappears when that tool is not in play (Blck Alpaca).

Escalation is a decision rule, not a mood. Without one, agents either escalate constantly (which trains people to ignore the reports) or never (which produces confident wrong answers). The conditions that warrant a human are narrow and nameable: a required capability is missing, a request is ambiguous after one clarifying attempt, or several tools fail with no alternative path (Redpanda docs).

Turn that into a checklist the prompt can apply mechanically:

• A required tool failed permanently and no alternative tool covers it.
• Two or more different tools failed on the same sub-task.
• A corrected call failed with the same error, meaning the diagnosis was wrong.
• The missing data is needed for an irreversible or high-value action.
• The agent is past its retry budget anywhere in the task.

The report format matters as much as the trigger. A person receiving "the agent could not complete the task" has to reconstruct everything. A person receiving the six fields above can act: the tool name and arguments identify the endpoint, the code suggests the cause, the retry history shows what was already tried, and the "needs" line says what to provide. This mirrors the guidance that agents should explain what they tried, what is missing, and what would unblock them, rather than apologizing (Redpanda docs).

Keep the report machine-readable as well as human-readable. Structured sections are easier for the model to produce reliably and easier for you to log, diff and alert on than prose (Blck Alpaca). Emit it as JSON with the same six keys, log it, and you get an error taxonomy for free: group by error_code and failed_step weekly and you will see which integrations deserve code-level fixes.

Prompt changes without tests are folklore. The pattern that has emerged in practice is to verify every instruction against an eval set, because plausible-sounding tips often show no measurable effect (Blck Alpaca). Fault injection does the same thing for the error section: it simulates the failures in a controlled way and measures what the agent does.

Build the harness as a proxy or stub between the agent and its tools. Research on agent fault tolerance uses exactly this design, intercepting agent-to-model traffic with a transparent proxy and injecting faults of several categories at configurable points (Agent Error Simulator). You do not need a research framework: a stub that returns scripted responses per call number is enough.

Cover each failure class with at least one scenario:

Assert on behavior, not prose. Parse the tool-call log for attempt counts, check that no further calls were made after an escalation, and validate the final message against the report schema. A prose answer that mentions the error but made four retries still fails.

One methodological trap is worth knowing. A study of fault-injection tests for tool-using agents found that an agent can be exposed to a fault and to evidence refuting it, yet the refuting evidence only appears after the agent has already used the faulty fact, so the test cannot conclude the agent ignored it (SSCBench). Translate that to your harness: place the failure on the first call in the sub-task, not after the agent has gathered and cited dependent data, otherwise you are testing recovery from poisoned context, a different and harder problem. Context poisoning, where an early error is referenced repeatedly and compounds, is a documented failure mode in production agents and deserves its own scenario (Splunk).

Track three numbers per scenario: pass rate over repeated runs, retries per task, and escalation rate. A prompt that passes ten runs in a row but escalates on 40 percent of clean tasks is too trigger-happy; a prompt that never escalates is worse. Set a threshold before you change anything, and rerun the suite on every prompt edit, since even small wording changes shift output distributions (Maxim).

Prose instead of rules. "Handle errors gracefully" is not a specification. Replace it with a signal list, a classification table and numeric limits. Specific, unambiguous instructions are the baseline of effective prompting (Palantir).

Retry logic in the prompt that belongs in code. Waiting, exponential backoff and connection retries are mechanical. Put them in the tool, and let the prompt spend its tokens on judgment (Redpanda docs).

No examples. Abstract rules are weaker than one short worked case. A single example showing a 503, two retries and an escalation report teaches the format faster than three paragraphs of rules, the same reason few-shot examples improve consistency in agent prompts (PromptHub).

Rules placed before the dynamic context. If user or session data sits between your error rules and the conversation, the rules are easy to lose and break your cache prefix. Keep stable instructions first, dynamic context last, the same layout that preserves prefix caching (Manus).

Trusting a success status. Many APIs return 200 with an error body. Your detection rule must inspect the payload, not only the status, and your tests must include a success-status-with-error-body case.

No stop condition. An agent that can retry "a few times" will retry a few times per tool, per task, without bound. State the loop limit per call and per task, and assert on it in the suite, because repeated-state loops are the classic production failure of agentic prompts (Blck Alpaca).

If you are running the same agent against several providers, keep the error block identical across models and run the fault suite on each, since recovery behavior varies by model size and configuration. Routing all of them through one endpoint makes that comparison cheaper, which is what Quanta Inference is for: text, image, video, audio and embedding models through one OpenAI-compatible API key. If your team would rather have the prompt and the suite built with you than read another guide, a consulting call with a specialist who builds agents is the faster path.