Setting a Concurrency Cap from Downstream Latency and Errors

The knee is the utilization point where a small increase in load produces a large increase in latency or errors. Below it, the curve is nearly flat. Above it, queueing takes over and the curve goes vertical.

You find it by stepping load and recording the downstream's own service time and error rate at each step, not your client's end-to-end time. Keep the steps small enough to see the turn. A table makes the decision explicit.

The service time in that table is the downstream's own, so the utilization column is a check on arithmetic rather than a claim about a real system. The shape is what matters: from 2 to 4 concurrent requests, p95 barely moves, and from 4 to 6 it jumps by more than half. That is the knee, and it sits somewhere between 4 and 6.

Saturation looks confusing precisely because compute metrics stay quiet. In the classic pool exhaustion incident, p99 latency jumped from 80 ms to 4.2 seconds while CPU sat at 32%, memory was fine and disk I/O was normal (Radview). The bottleneck was circulation, not capacity. This is why you derive the cap from the downstream's latency and error curve and not from its CPU headroom.

Error rate is the second edge of the knee. Some services degrade in latency before they fail; others return 429s or timeouts once a concurrency limit is reached. ICE Mortgage Technology's API documentation states that when a lender's hard concurrency stop is hit, every process stops and a 429 is returned until remaining concurrency is above zero (ICE concurrency limits). Their guidance asks lenders to keep the remaining ratio under 80% and partners under 20% of the allowed concurrency, which is a vendor telling you to stay clearly below the knee.

The rule is one line:

cap = floor(target utilization * 1000 / p95 service time in ms)

The 1000 converts the target, a fraction, into milliseconds. The result is a count of in-flight requests. With a 70% target and a p95 of 200 ms, the cap is floor(0.7 * 1000 / 200) = 3 concurrent requests.

Run the example properly. Suppose you measured p95 at 200 ms on the flat part of the curve, the knee sits near 80% utilization, and error rate is zero below it. You want headroom below the knee, so you pick a target of 70%:

cap = floor(0.70 * 1000 / 200) = floor(3.5) = 3

Three concurrent requests is not a typo. It feels small, so check the throughput it implies. With service time of 200 ms and a cap of 3, the ceiling is 3 / 0.2 s = 15 requests per second. If the tool needs 40 requests per second, the cap is not the problem: the downstream is the problem, and you either scale it, cache, or batch. Raising the cap to 8 would push utilization well past 1.0, which in queueing terms means a growing queue, and the queue length and wait time grow with utilization until the pool is exhausted (the pool exhaustion guide).

If your downstream reports utilization directly rather than p95, the equivalent allocation rule from the pool world helps. Heroku's PgBouncer guidance and the shared-capacity formula both push you to treat a connection budget as shared: subtract admin, migration and monitoring reserves from the database's maximum, then divide by the maximum number of concurrent instances (the webalert sizing formula). That gives a per-instance ceiling, and the per-tool cap must fit underneath it.

One more check on top of the formula. Little's Law says the number of requests in the system equals arrival rate times time in system (the Little's Law walkthrough). Your cap times p95 gives the arrival rate the downstream can absorb at that cap. If the needed rate is above it, no concurrency setting saves you.

You need a clean measurement loop, because the cap is only as good as the p95 you feed it.

1. Fix the window. Record p95 service time per downstream over a fixed interval, from merged histograms, not averaged percentiles.
2. Step concurrency in small increments and hold each step long enough for the tail to appear. A few hundred requests per step is a starting point.
3. At each step, capture p95 service time, error rate and in-flight count from the downstream itself, along with any retry counts.
4. Find the last step where p95 is within a small band of the flat baseline and the error rate is at its floor. The step after that is the knee.
5. Set the target utilization below the knee. Pick the headroom from the risk in the workload: batch jobs can run closer to the knee than user-facing calls.
6. Record the numbers and the target in the same place as the cap, so the next person can recompute it.

You can watch the same signals live with pg_stat_activity grouped by state, which separates active, idle and idle-in-transaction connections (the PostgreSQL connection states guide). Idle-in-transaction is the state to watch: those connections hold resources without doing work, and a rising count usually means leaks or long transactions rather than a genuinely busy downstream (the webalert diagnosis steps).

If you are measuring a query rather than an API call, read the plan as well as the timing. EXPLAIN ANALYZE shows actual timings, not just estimates, so you can tell whether a p95 that moves is a plan change or a load effect (Datadog's walkthrough). The timing tells you the p95; the plan tells you whether it will stay put.

For anything calling a metered third-party API, read the concurrency semantics carefully, because they differ. monday.com documents multiple limits including a concurrency limit alongside complexity, daily and minute limits (monday.com rate limits). The Claude API enforces rate limits with a token bucket algorithm, continuously replenishing capacity rather than resetting on fixed boundaries, and can enforce a per-minute limit over a shorter interval, so a short burst can trip it (Claude rate limits). A cap learned from the provider's own concurrency headers beats one learned from a guess.

If an agent or worker pool calls several different tools, one global cap will either strangle a fast tool or overwhelm a slow one. The whole point of a per-tool cap is that each downstream has its own service time distribution and its own knee.

The last row is the point: at 1,400 ms and a 60% target, floor(0.6 * 1000 / 1400) = 0, and a cap of zero means you need asynchronous handling, not a smaller number. A long-running tool call that blocks the caller freezes the agent, keeps a slot occupied and prevents any other useful work, which is the same shape of problem as holding a database connection across a slow network call (the blocking tool call analysis, the webalert causes list). Submit the job, receive a task identifier, and free the slot.

Scaling the pool across instances also has a multiplier effect that teams miss. Twenty pods each with a pool of 20 demand 400 connections, which can blow past the database's maximum even though each pod looks reasonable (the pool exhaustion guide). Size per-instance caps against the maximum autoscaled instance count, not today's steady state, and remember to reserve admin, migration and monitoring headroom first (the webalert formula).

Finally, be careful with retries. When requests time out waiting for a slot, retries demand more slots from an already loaded downstream, and each retry makes the saturation worse, which is the timeout cascade (How Tech). A cap without a retry budget is half a policy. Retry only when the error is transient, back off, and count retries against the cap.

Use this to review any proposed number before it goes into production.

The decisions inside that checklist are the real work, and they are where a line-of-business expert earns their keep. A specialist who has run a database pool or an agent platform will read the same table and know whether a 70% target is cautious or reckless for that workload. If your team is building this kind of control loop for an agent or platform, a Quanta expert can take the measurement and sizing work off your hands, and the same skill library installs into a coding agent if you want to run the loop yourself.

A cap is a snapshot of a downstream at one moment. Service time changes with data volume, index health, plan regressions and traffic mix, and the knee moves with it. Reserve headroom for that movement rather than for today's measurement.

Three signals should trigger a recompute:

• p95 service time drifting beyond the band you measured at.
• Error rate rising at a load step that used to be flat.
• Queue wait time, or connection checkout wait time, rising while service time holds.

The last one is the earliest warning. Pool wait time above roughly 10 ms is a sign the pool is congested, and acquisition failures are the canary that you are close to failure (How Tech). If wait time rises before service time does, your cap is too close to the knee and you should lower the target utilization.

Vendors change their limits too, so owning the number is safer than trusting a published one. The PgBouncer project's release notes describe fixes for authentication bypass and denial-of-service issues across a few versions, including an unauthenticated infinite loop and an unbounded SCRAM iteration count (PgBouncer). Running a pooler inside your own service means you control the cap, the version and the update schedule, and you can read wait time directly, while the pooler still gives you transaction-level reuse and low per-connection memory (PgBouncer features, Heroku's pooling modes).

The closing rule: derive the cap from p95 and measured error-free load, express it as a small integer, verify the implied throughput against demand, and recompute whenever p95 or the error curve moves. That is the difference between a number that survives an incident and a number that causes one.