Backpressure Between an Agent and Its Tools: Caps and Queues

You cannot cap what you have not measured. Do this in three steps.

Step 1: Find the hard ceiling. For Postgres, that is max_connections, and you should reserve headroom for admin, migrations and monitoring before dividing the remainder across instances:

usable connections = max_connections - admin reserve - monitoring reserve
max pool per instance = usable connections / maximum autoscaled instances

Size against the maximum instance count, not today's steady state, because autoscaling will find the gap per Webalert's sizing formula.

Step 2: Measure service time under load. For a tool that hits the database, EXPLAIN ANALYZE gives you planning time, execution time and rows removed by filter, which is enough to see whether a query is slow or just wide per the PostgreSQL documentation on EXPLAIN. A query that scans far more rows than it returns inflates service time, and inflated service time inflates the concurrency you need as this thoughtbot walkthrough of reading EXPLAIN ANALYZE plans shows.

Step 3: Compute concurrency, not requests per second. With service time in seconds and target throughput, concurrency needed follows from Little's Law style arithmetic. If a tool takes 50 ms and you want 100 calls per second, you need roughly 5 concurrent calls. If the same tool takes 500 ms, you need 50. That single number explains why a cap that worked at low latency starves under load.

A worked example. A lender API enforces a hard stop of 30 concurrent calls per environment, and calls beyond that are refused outright per ICE Mortgage Technology's concurrency limits documentation. If your agent fleet treats 30 as the target, one retry storm takes the whole environment to zero. The same vendor recommends keeping utilization below 80 percent for lenders and 20 percent for partners, which turns 30 into 24 and 6 respectively. Your cap is a budget, not a target.

A global cap is the wrong abstraction because tools have wildly different costs. A vector lookup, a PDF render and a payments API should not share a semaphore. Model each tool as its own pool with its own limit and its own queue.

Two things in that table matter more than the numbers. First, the cap is per instance, so twenty agent pods each capped at 5 still hit the database with 100 connections. Second, token-bucket smoothing matters because providers enforce limits over short intervals: Anthropic's documentation notes that a 60 requests-per-minute limit may be enforced as 1 request per second, so short bursts trip it per the Claude platform rate limits page. Agent fan-out is bursty by nature, which makes smoothing essential.

There is a further trap with providers that meter concurrency invisibly. Practitioners have reported sending 138 parallel requests, well under published per-minute limits, and seeing latency stretch to about 40 seconds versus under 2 seconds for five requests, meaning the effective constraint was concurrency, not rate per this OpenAI community thread. The lesson generalizes: treat any provider as having an unstated concurrency ceiling, and probe for it with a ramp test rather than trusting the docs.

A queue exists to absorb variance, not to absorb overload. Size it to cover a burst of roughly the duration of one service-time spike, and no more than that. A useful default is queue depth equal to your concurrency cap: enough to smooth a hiccup, small enough that a full queue means something is genuinely wrong.

The failure mode of an unbounded queue is well documented in the pooler world. When PgBouncer cannot create a new server connection without breaching pool_size, max_db_connections or max_user_connections, it queues the request, and only aborts when the client limit is breached per ScaleGrid's PgBouncer walkthrough. The queue is a pressure valve, and it needs a limit.

Bounded queue plus a shedding rule looks like this:

on tool_call(tool, payload):
  if queue[tool].size >= cap[tool].queue_depth:
      return RetryableError(
          code="TOOL_BACKPRESSURE",
          tool=tool,
          retry_after_ms=backoff(tool),
          suggestion="defer or drop, do not resubmit immediately")
  if semaphore[tool].try_acquire(timeout=0):
      enqueue_execute(tool, payload)
  else:
      queue[tool].put(payload)

The critical design choice is that the agent receives a typed, retryable signal rather than a hang or a raw timeout. Anthropic's rate limit behavior is instructive here: some 429 responses carry a retry-after header and some do not, and when they do not, retrying including the SDK's automatic retries fails until access resumes per the Claude platform rate limits page. An agent that cannot distinguish "slow down" from "stop entirely" will burn its entire context window on retries.

Two rules follow. First, never let the agent retry a shed call in the same turn more than once; a shed call is a planning signal, not a transient error. Second, cap total retries per tool per turn explicitly, because retry storms are the mechanism that turns a degraded dependency into an outage: requests time out waiting for a connection, retries demand more connections from an already-empty pool, and the spiral tightens per this systems writeup on exhaustion.

Handling long tool calls as blocking operations is the other half of this problem: a blocked agent cannot plan around a shed call or report progress. The NHI guide on blocking tool calls notes that non-blocking handling separates submission from completion, which is what lets the agent keep working while a slow job finishes.

Most agent dashboards show request rate and error rate. Neither tells you whether you are about to fall over. Watch these instead:

• Queue wait time per tool. Under a healthy system this is milliseconds. Waiting time consistently exceeding execution time is the clearest sign the cap is too tight or the downstream is slow per Radview's symptom list.
• Shed rate per tool. The percentage of calls rejected by your own shedding rule. A nonzero shed rate is fine; a rising one predicts an outage.
• Concurrency utilization ratio. Your own version of the vendor's remaining-ratio formula: divide in-use concurrency by the cap, and alert above roughly 80 percent the same pattern ICE documents.
• Connections stuck in a non-working state. Postgres exposes pg_stat_activity with states including active, idle, idle in transaction and idle in transaction (aborted), and a cluster of connections sitting idle in transaction is a strong signal that something is holding a resource across a slow step per myDBOps on connection states.
• Isolation-related retries. If tool calls write to the same rows concurrently, serialization failures will appear. Postgres repeatable read aborts a transaction that tries to update a row changed by another transaction, preventing lost updates, while serializable adds protection against serialization anomalies per this mkdev explainer on isolation levels. Those aborts are backpressure too, and the agent should treat them as retryable with a fresh read.

A practical alerting split: cap utilization above 80 percent for five minutes is a warning, shed rate above 5 percent for one minute pages someone, and queue wait time above the tool's p99 service time means the cap is mis-sized rather than the load being too high.

One more lever earns its keep: pooling modes. Running PgBouncer in transaction pooling mode lets a server connection return to the pool when a transaction ends, which decouples a long-lived agent session from an idle database connection. The tradeoff is that session-scoped features such as SET, LISTEN and session-level advisory locks stop working under transaction pooling per the PgBouncer feature matrix. If your tools rely on session state, either keep session pooling for that tool only or rewrite the tool to be stateless. Heroku's guidance makes same point plainly: transaction pooling is the recommended mode for typical application traffic per the Heroku Dev Center.

Do this in order, and each step is independently useful.

1. Instrument every tool call with a duration histogram and a queue wait time. No caps yet.
2. Ramp one tool to find where latency bends. That inflection point is your ceiling probe; set the cap at 80 percent of it.
3. Add the bounded queue with depth equal to the cap. Still no shedding; just log what would have been shed.
4. Turn shedding on for one tool. Return a typed retryable error and confirm the agent degrades gracefully rather than retrying.
5. Extend per-tool caps to the rest of the toolset, and add the write-path cap last, because it carries the most risk.
6. Put the three signals above on a dashboard with the alert thresholds you chose.

If you are running agents across several model providers and downstream services, the dispatch layer that applies these caps is the same one that routes requests. Quanta Inference exposes text, image, video, audio and embedding models behind one OpenAI-compatible API key, which lets you put a single cap and queue in front of every model call instead of one per vendor at https://quantaintelligence.ai/inference.