Retry Budgets for Agent Calls: Backoff and Jitter

The table below is a starting policy, not a law. Storm cost is the extra requests you should expect if every client retries at the same moment with the same settings, expressed as a multiple of the original traffic. With 1,000 concurrent calls all failing at once, a policy that retries once adds 1,000 requests; the multiplier column shows that in relative terms.

Read the last column as your worst-case exposure before the global budget throttles it. A 503 storm that lets every call retry four times can briefly quadruple load on an upstream that is already struggling. The budget in a later section exists to prevent exactly that.

Plain exponential backoff is delay = base * 2^attempt, capped at some maximum. With a one-second base the first retry waits one second, then two, four, eight, and so on Svix on exponential backoff. That schedule is fast enough to catch a blip and slow enough to survive a longer outage, which is why webhook delivery systems use it Svix on exponential backoff.

Without jitter, every client that failed at the same instant retries at the same instant. The AWS Architecture Blog calls this the thundering herd and shows that un-jittered retries produce a sawtooth that overloads the server at every boundary AWS on exponential backoff and jitter. Three jitter formulas are in common use:

# full jitter: best server-side behavior, slightly slower completion
sleep = random(0, min(cap, base * 2 ** attempt))

# equal jitter: at least half the exponential delay, still spreads load
sleep = expo / 2 + random(0, expo / 2)

# decorrelated jitter: upper bound depends on the previous sleep
sleep = min(cap, base + random(0, previous_sleep * 3 - base))

Brooker's measurements found that full jitter does the least total work and puts the lowest load on the server, at the cost of slightly longer time to completion. Decorrelated jitter finishes a little faster but does more total work AWS on exponential backoff and jitter. For agent calls, full jitter is the right default: your upstream is usually the bottleneck, not your own latency.

Cap the delay. Unbounded growth produces absurd waits: after 10 attempts with a 1-second base you would be sleeping over 17 minutes, and after 20 attempts, over 12 days Hookdeck's retry guidance. A cap of a few seconds suits interactive agent calls. Longer caps only make sense for background jobs where nobody is waiting on the response.

Set a deadline as well as an attempt cap. Six attempts at 30 seconds each will burn three minutes you may not have, so treat total wall-clock time as the primary gate and the attempt count as a safety net a TypeScript retry implementation.

Per-call caps stop a single request from looping. A cap of 3 to 5 attempts is the common range for webhook and API delivery: enough to ride out a short blip, short of the point where a loop looks like an attack 2POINT on webhook retries. AWS SDKs expose the same idea through a configurable maximum attempt count and a retry mode that decides how failures are classified AWS SDK retry behavior.

Per-call caps are not enough on their own. If 10 percent of your calls fail and each retries three times, you have added 30 percent to your request volume during exactly the window when the upstream is weakest. That is why you also need a global budget: a counter shared across the process or fleet that limits retries to a fixed share of total calls in a rolling window. A budget of 10 percent means that once retries reach one in ten of the calls you have made, further failures return to the caller immediately instead of retrying.

import random, time

RETRYABLE = {429, 500, 502, 503, 504}
BASE, CAP, MAX_ATTEMPTS = 0.5, 8.0, 4

def call_with_retry(fn, budget, attempt_fn, sleep=time.sleep):
    attempts, waited = 0, 0.0
    while True:
        attempts += 1
        try:
            return fn()
        except Exception as err:
            status = getattr(err, "status", None)
            if status not in RETRYABLE or attempts >= MAX_ATTEMPTS:
                raise RetriesExhausted(status, attempts, waited) from err
            if not budget.take():          # rolling-window counter, e.g. 10% of calls
                raise RetriesExhausted(status, attempts, waited) from err
            expo = min(CAP, BASE * 2 ** (attempts - 1))
            delay = random.uniform(0, expo)   # full jitter
            waited += delay
            sleep(delay)

Two properties make this loop safe to ship. It never retries a status outside the set, and it checks the budget before sleeping, so a storm cannot queue thousands of pending sleeps. The AWS Prescriptive Guidance frames the same pattern as a way to add stability to operations that fail on throttling or temporary unavailability, with the caveat that frequent retries can themselves consume bandwidth and cause contention AWS retry with backoff pattern.

The worst outcome is a bare exception that says "request failed" after 40 seconds of retrying. The caller has no idea whether to fall back, render a cached answer, or tell the user to try again. Return a structured error instead:

{
  "error": "retries_exhausted",
  "last_status": 503,
  "attempts": 4,
  "total_wait_ms": 6320,
  "retry_after": null,
  "trace_id": "01J9..."
}

Four fields do the work. last_status tells the caller whether the upstream was overloaded or unreachable. attempts tells it whether the policy ran its full course or the budget cut it short. total_wait_ms lets it measure user-visible latency. A trace_id lets support reconcile the failure with logs. Include the idempotency key you used, so a later manual replay can be deduplicated rather than repeated Stripe idempotent requests.

Then pick a fallback, in order: another model or provider through the same interface, a cached or previously generated response, a degraded answer with a visible notice, or a hard failure. When you route across several providers, one OpenAI-compatible endpoint makes the fallback a configuration change rather than a rewrite, which is what Quanta Inference provides, covering text, image, video, audio and embedding models behind one key.

Retry policies decay quietly. A budget that is never hit during testing can be saturated in production, and a cap that was right for one workload is wrong for another. Track these:

• Retry rate per error class. If 429s dominate, the problem is volume and pacing, not backoff.
• Budget exhaustion rate. The share of calls where the budget refused a retry. A steady rise means your global cap is too tight or your failure rate is too high.
• Attempts per successful call. A mean above about 1.1 suggests the upstream is unhealthy or your base delay is too short.
• Time to success distribution. Watch p95 and p99, not the mean. Retries inflate the tail long before they move the average.
• Exhausted retries by endpoint. This is your list of calls that need a fallback path.

When a metric crosses a threshold, the response should be a policy change, not a manual intervention: raise the cap for a specific error class, lengthen the base delay, or add a fallback route. If your agents have grown complex enough that retry behavior differs per workflow, map the policy per endpoint and per model first, then decide whether the coordination work is worth doing in-house. Teams that would rather hand the build to a specialist can describe the system they need and have it built with them.