Fitting a Retry Budget to a Tool That Fails Slowly

Take a service that calls a payment or model API with a concurrency cap of 20. Normal operation keeps 16 calls in flight, leaving four slots of headroom. The dependency degrades and 20 percent of calls fail with a timeout or a 5xx. Suppose the retry policy is the common one: retry every failure once, immediately.

The arithmetic is small but easy to get wrong. Of 16 in-flight calls, 3.2 fail on average and are retried. Those retries occupy the four spare slots, taking in-flight from 16 to 19.2, which rounds to about 20 at any instant. Your system now runs permanently at the cap, with zero margin.

The margin is exactly what protects you from the next thing. At the cap, roughly 20 percent of the retries fail too. Each of those is another candidate for a retry, and every one that fires pushes offered concurrency to 21 or 22. Once offered load exceeds the cap, the excess queues for a slot. Queued requests take longer, and longer requests are more likely to hit the client timeout, which produces a fresh failure and a fresh retry. That is the retry storm, and the Azure Architecture Center documents the mechanism plainly: retries within a short period are unlikely to succeed because the service has not recovered, and excessive connection attempts during recovery intensify the original problem (retry storm antipattern).

The last row is the case people hit in production. If you set your client pool to the cap of 20 and then retries draw from a separate pool, the two pools add rather than share, and the dependency sees 24 concurrent requests against a 20-slot limit. The interesting insight is not that retries add load. It is that a retry policy which looks safe at 5 percent error is already at the edge at 20 percent, and past the edge at 40 percent, with no configuration change in between.

A retry budget sets the maximum retries as a ratio of successful or primary traffic, so that the sum stays under the cap. Write it as:

max_retry_fraction = (cap - planned_primary_in_flight) / planned_primary_in_flight

Using the worked example with a cap of 20 and planned primary of 16, the budget is 4/16, or 25 percent. That is the maximum share of traffic that may be retries at any moment. Two familiar implementations default to a similar number. Envoy's retry budget circuit breaker exposes budget_percent, which caps concurrent retries as a percentage of active and pending requests, and it defaults to 20 percent with a floor of 3 minimum retry concurrency (GEP-3388 retry budgets). Linkerd makes budgeted retries the default policy rather than a static max retry count, using the same shape of limit (GEP-3388 retry budgets).

The formula has three inputs you can actually measure. Cap comes from whichever limit binds first: pool size, an upstream concurrency limit, or a rate-limit header. Planned primary in flight is your working set at normal load, not your peak. The fraction falls out, and then you compare it to your observed error rate.

The decision rule that matters:

That last row is why every budget needs a floor. On a low-traffic service, a strict percentage of a small number rounds to zero retries, so a single dropped connection turns into a user-facing error. Envoy's default minimum of 3 concurrent retries exists for this reason (GEP-3388 retry budgets). Set the floor at a number you can afford at your cap, not at your average.

Two related limits deserve the same treatment. If you are calling a third-party API rather than your own database, the provider is already enforcing a ceiling on your behalf, and the shape of that ceiling tells you what to budget for. Concurrency limits return 429 when the slot count is exhausted, and the response carries headers such as X-Concurrency-Limit-Limit and X-Concurrency-Limit-Remaining that you can meter against, with proactive metering recommended well before the limit is hit (concurrency limits). Rate limits keyed on requests or tokens per minute, like those on the Claude Messages API, are enforced differently: limits are defined per usage tier and a 429 comes with a retry-after header telling you how long to wait (rate limits). If your dependency gives you that header, honoring it is not optional politeness, it is the cheapest retry budget you will ever get.

The temptation with a budget is to size it from an average error rate or an average latency. Both averages hide the case you are budgeting for. A mean latency of 120 milliseconds can sit on top of a p99 of four seconds, and at a million requests a day that 1 percent tail is 10,000 slow responses (performance percentiles). The retries are triggered by the tail almost exclusively, because the healthy middle of the distribution never times out.

Percentiles also do not add. You cannot average per-instance p99 values to get a fleet-wide p99, and you cannot average per-minute p99 series into an hourly one; you have to keep the underlying histogram and recompute (p95 latency). For retry budgeting, the practical version of this rule is to measure at the dependency, not at the client, and to look at p95 and p99 concurrency rather than means. A dependency whose p99 in-flight count exceeds the cap once a minute will produce a periodic spike of pool waits that looks random in aggregate dashboards.

There is a compounding effect worth remembering when a single user action fans out to several backend calls. If a page load needs five parallel calls and each has 99 percent of responses under 100 milliseconds, the chance all five finish that fast is about 0.99 to the fifth power, or roughly 95 percent, so about one in 20 loads waits on someone's tail (p99 latency). Add a per-call retry on top and the fan-out multiplies both the offered concurrency and the chance that at least one branch is retrying. Budget at the fan-out level, not per hop, or the per-hop budgets will sum to something you never approved.

The budget only holds if it is enforced somewhere that sees all callers. Three placements work, and the trade-offs are worth stating.

If you go with the in-process route, the counter has to be shared by primary and retry calls. A sketch in Python, using a semaphore sized to the cap and a retry gate that checks the budget before spending a slot:

from threading import Semaphore, Lock

CAP = 20
PRIMARY = 16
BUDGET = CAP - PRIMARY          # 4 retry slots
slots = Semaphore(CAP)
retry_slots = Semaphore(BUDGET)
lock = Lock()
retries_in_flight = 0

def call(fn, is_retry=False):
    if is_retry:
        if not retry_slots.acquire(blocking=False):
            raise RuntimeError("retry budget exhausted")
    try:
        with slots:
            return fn()
    finally:
        if is_retry:
            retry_slots.release()

The important line is the non-blocking retry acquisition. A retry that waits for a budget slot has stopped being a retry budget and become a second queue, and a queue inside a degraded dependency is how the original error turns into a cascade. If the budget is exhausted, fail fast and let the caller (or a higher-level policy) decide.

If your dependency is a database, the same math applies to pool size, and the sizing has to happen across the whole deployment rather than per pod. Each instance carries its own pool, so twenty pods with a pool of twenty demand 400 connections, which can exceed a server-side max_connections even though each pod looks reasonable (connection pool exhaustion). The usable budget is the server limit minus admin, migration, and monitoring reserves, divided by the maximum autoscaled instance count, sized against the peak instance count rather than today's steady state (connection pool exhaustion).

Once retries are part of the picture, a pair of backoff and a jitter is not enough on its own. Exponential backoff spaces retries geometrically so the target gets breathing room, and jitter decorrelates retry waves across callers so they do not arrive in lockstep, but both bound the shape of an individual caller's retries while the budget bounds the aggregate (resilience patterns at scale). Use all three, and put a circuit breaker in front of a dependency that is consistently failing so the retries stop entirely for a window.

Four signals, all cheap to emit:

1. Retry fraction. Retries divided by primary requests, sampled every few seconds. Alert when it crosses 80 percent of the configured budget for a sustained window. This is the leading indicator that the dependency is degrading, and it usually moves before p99 does.
2. In-flight concurrency at the dependency. Count primary and retry together. If p99 in-flight count equals the cap, your headroom is gone regardless of the error rate.
3. Pool or slot wait time. Under 1 millisecond healthy, above 10 milliseconds congested, per the thresholds above (connection pool exhaustion). Rising wait time with a static error rate means retries are consuming the spare capacity.
4. Pool wait time versus query execution time. If waiting for a connection exceeds the time spent executing, the bottleneck is contention, not the query, and adding pool size will only shift the queue downstream (connection pool exhaustion).

For a database-backed service, a quick check of connection states separates the ordinary case from the pathological one. Postgres exposes pg_stat_activity, and a high count of connections in idle in transaction is a hallmark of holding a connection across a slow external call, which is exactly the pattern that starves a pool under retry load (PostgreSQL connection states). A count that climbs steadily and never drops is a leak, not a budget problem, and increasing the budget will delay the discovery without preventing it.

One more habit closes the loop. When you set a budget, write the test that proves it. Drive the service at planned primary concurrency with an injected error rate just above the budget fraction, and assert that dependency in-flight count never exceeds the cap and that retries stop being issued once the budget is spent. The retry policy that looks correct in staging against clean, isolated failures is precisely the one that amplifies load when hundreds of callers hit the same slow dependency at once (the retry storm). Testing at one caller and one error never exercises the correlated case.

If your callers fan out to several model providers rather than one dependency, the budget has to be shared across all of them, or a provider-specific cap becomes invisible when the router fails over. Running every model behind one key with a single concurrency pool is one way to make that budget enforceable in one place: Quanta Inference puts text, image, video, audio, and embedding models behind one OpenAI-compatible API, so retry and concurrency policy is configured once rather than per provider.