Retry Budgets for Subagents: How Many Refusals Before You Stop

Before you can set a number, you need a classification. Write the mapping once and treat it as code, not documentation.

TERMINAL = {"invalid_input", "policy_block", "auth_failure", "schema_violation"}
CONDITIONAL = {"tool_output_truncated", "parse_error", "missing_field", "empty_response"}
TRANSIENT = {"rate_limit", "timeout", "upstream_5xx", "connection_reset"}

Anything not in one of these three sets falls into TERMINAL by default. This default matters: an unknown reason code is an unclassified failure, and retrying it is a coin flip you have not priced.

Why are structured categories the right unit? Because structured output failures are not all the same kind of event. A study of structural reliability on four software engineering tasks found that strict template control nearly eliminates syntax errors while substantial structural and semantic errors persist, so the bottleneck sits beyond syntax (Empirical Study for Structured Output Control). A parse error caused by a missing bracket is fixable with a repaired prompt. A semantic mismatch is not fixed by calling again.

The same logic applies to tool calls. A malformed payload should fail validation and route to retry, repair or escalation rather than straight to the user (MLflow on agent orchestration patterns). Routing is where the taxonomy earns its keep.

Terminal codes share one property: the same input produces the same refusal. Invalid input does not become valid because you asked twice. A policy block is a decision, not a hiccup, and an authentication failure will fail identically until someone rotates a key.

Treat a terminal refusal as a routing event. Send it back with the reason code, the offending field and a suggested repair, and let the caller or the orchestrator decide. This is the cheapest behavior in the system and the most commonly skipped. Many teams wrap every error in a generic retry, which turns a deterministically broken subtask into three billed calls that produce the same refusal.

There is a cost argument too. Azure's architecture guidance is to use the lowest level of complexity that reliably meets requirements, because each added layer brings coordination overhead, latency and failure modes (AI agent orchestration patterns). A retry wrap around a terminal failure adds a failure mode without adding a capability.

One caveat worth writing down: classify on the code, not the message. A policy block and an invalid input can both arrive as "bad request". If your provider surfaces an HTTP status without a structured code, add a normalization step that maps status and body to one of your codes before the retry logic sees it.

Conditional codes are the awkward middle. The request was well formed enough to run, but the reply could not be used. A truncated tool response, a parse failure, a missing required field. Something in the machine broke in a way that a second attempt might not repeat, but nothing tells you it will not.

The rule is one retry, and the retry is not the same request. Change something first:

• For a parse error, attach the validator's error output and the required schema to the retry prompt.
• For a truncated tool response, reduce the requested payload or ask for a narrower field set.
• For a missing field, restate the field name and its type in the instruction.

If the second attempt fails with the same code, stop. Two identical refusals is evidence of a systematic problem, and a third attempt is the point where cost starts compounding without new information. That is the decision rule the first search result rarely gives you: the boundary is two, not five.

Retries also need to be safe to repeat. Anything that changes state needs an idempotency key so a retried call does not double-process, which is the standard guidance for retrying non-idempotent operations (backoff and jitter explained). Subagents that write files, create tickets or post messages need the key on the first attempt, not bolted on after an incident.

Provider-side schema enforcement helps you decide whether an error is conditional or terminal at all. Natives structured outputs and strict schema modes narrow the interface, but provider capabilities, refusals, truncation and fallback still matter, and shape is not truth (TrueFoundry on structured outputs). A schema-valid object can still be wrong. If your validator catches that, the code is semantic, and semantic failures belong in the terminal class.

Transient codes are the only class where patience is a strategy. A rate limit, a timeout, a 5xx from a downstream service: the request failed for reasons that are likely to clear.

The arithmetic of exponential backoff is simple: the delay doubles per attempt up to a cap (PostGrid's retry guidance). With a 250 ms base and a 2 s cap you get delays of roughly 250 ms and 500 ms across two retries, which keeps total added latency under a second. That matters because the caller is blocked the whole time.

Full jitter turns the delay into a random value between zero and the computed backoff, which is the most effective way to stop many clients retrying in lockstep (backoff and jitter explained). Without it, a thousand subagents that all hit the same rate limit at the same moment will all retry at the same moment.

import random, time

def delay_for(attempt, base=0.25, cap=2.0):
    return random.uniform(0, min(cap, base * 2 ** attempt))

Two retries is a deliberate ceiling. Amazon's guidance is to retry with backoff and bound the policy by the caller's deadline rather than copying a fixed attempt count from elsewhere (AWS on timeouts, retries and backoff). If the caller has 1.5 seconds left, a retry with a 2 second cap is not a retry, it is a timeout with extra steps.

Then add the system-level guard. A retry budget caps retries as a fraction of total requests, so when a dependency is broadly down the system stops retrying instead of flooding it (backoff and jitter explained). For agents, denominate the budget in retry tokens per session rather than retry counts, because a retry on a 50,000 token context costs far more than one on a 2,000 token context (retry budgets for LLM agents). A workable starting point is a hard cap of two attempts per call plus a session-level allowance that, once spent, converts all transient codes into escalations.

Take a fan-out step that dispatches 1,000 subtasks, each with a 12,000 token context, and assume a per-subtask refusal rate of 15% before you add any policy. For illustration, assume those refusals split 40% transient, 35% conditional and 25% terminal, and that retries replay the full context.

Two things fall out of the table. First, the classified policy sends roughly 45% fewer retry calls, and the saving is concentrated in the terminal bucket, where retries were pure waste. Second, the residual number is not zero: 150 conditional refusals and about 12 transient second retries remain, and the ones that still fail after their budget are exactly the population that should reach a human.

That last point is the design insight. A retry budget is not only a cost control, it is a triage filter. After the budget is spent, you know three things about the task that you did not know before: the failure is real, it survived a backoff, and it is worth a person's time. Route those to an escalation queue with the reason code, the attempt count and the last validator error attached.

If you are routing across several model providers, keeping the retry policy uniform is easier when the endpoint is uniform. Quanta Inference exposes text, image, video, audio and embedding models through one OpenAI-compatible key, which means one failure taxonomy and one backoff implementation instead of one per vendor (https://quantaintelligence.ai/inference).

Escalation needs a contract, not a hope. Every escalated subtask should carry its reason code, its class, the number of attempts, the last error and the input that triggered it. Without that, a human starts the investigation from zero, and your careful classification buys nothing.

Observability has to sit across the whole run. Multi-agent workflows that stay reliable enforce typed interfaces between agents and centralized observability, and skipping either produces drift and silent failures (MLflow on agent orchestration patterns). Correlated traces are how you answer the only question that matters during an incident: are these refusals independent, or is one downstream dependency failing everything.

Pair the retry policy with a circuit breaker. A breaker stops sending requests to a dependency that is clearly down for a cool-down period, which makes it the natural partner to retries: retry the transient failures, trip the breaker on the sustained ones (backoff and jitter explained). In a subagent fleet, trip on a rolling window of transient refusals per dependency, not per subagent, because the dependency is the shared resource.

Finally, revisit the architecture before tightening the numbers further. If one class of subtask escalates constantly, the fix is usually a narrower task or a better schema, not more attempts. Multi-agent designs buy specialization at a coordination cost that teams often underestimate, and simpler single-agent designs are easier to monitor and debug because there is one trace and one set of tool calls (JetBrains on single vs multi-agent systems). A subtask that needs three retries and a human is often a subtask that was scoped wrong at decomposition.