Per-Caller Caps Layered on Top of Tool Tiers

The registry entry is the natural home for the cap because the tool's owner is the person who knows what a call costs upstream. A minimal entry that separates tier from per-caller budget:

{
  "tool": "web_search",
  "tier": "internal",
  "upstream": "https://search.internal/v1/query",
  "cap": {
    "unit": "calls",
    "windows": [
      { "period": "hour", "max": 300 },
      { "period": "month", "max": 15000 }
    ]
  },
  "callers": {
    "team-research": { "month": 15000, "hour": 400 },
    "team-support":  { "month": 9000,  "hour": 250 },
    "team-growth":   { "month": 6000,  "hour": 150 }
  }
}

Three design choices in that shape are worth defending. First, the cap is expressed in a unit (calls, tokens, wall-clock seconds) rather than as a bare number, because token-consuming tools and cheap lookup tools need different meters; agent-specific guidance recommends limiting on tokens per minute and external tool calls per hour precisely because request counts understate LLM cost (NeuralTrust). Second, windows are a list, not a single number, so you can pair a burst ceiling with a monthly budget. Third, per-caller values override the default, so adding a fourth team means adding a key rather than editing shared config.

Keep the schema strict. Schema-constrained output and runtime validation exist because a structurally valid payload can still be wrong, and the same discipline applies to config: a missing unit or a negative max should fail the wrapper's startup check, not surface as an exception on the first agent call (MLflow).

Suppose the upstream search vendor contract allows 30,000 calls per month for the whole company, and the registry has to split it across three callers. The split below is a starting point, not a law of nature; the ratio should follow measured usage from at least one full billing window.

Two counters exist for every call: the per-caller counter and the aggregate counter. Both must have room, or the call is refused. This "all matching limits must allow" behavior is the same AND logic enterprise gateways use across independent named limits, where usage increments against every matching rule and a request proceeds only if none is breached (TrueFoundry rate limiting docs).

Now let support burst. Their monthly share still has 4,200 calls left on day 22, but their agents decide to re-run searches in a loop. The hourly cap of 250 stops them at call 251 of the hour. The refusal returns something like:

{
  "error": "cap_exceeded",
  "tool": "web_search",
  "caller": "team-support",
  "window": "hour",
  "limit": 250,
  "used": 250,
  "reset_at": "2026-05-22T15:00:00Z",
  "retryable": true
}

The important property is where that refusal fires: inside the wrapper, before the upstream HTTP request is built. No vendor call is made, no tokens are spent, and the aggregate counter is never decremented. If the refusal fired after the upstream call, you would have paid for the call and then thrown away the result, which is the most common way cap enforcement leaks money. The same wrapper pattern described for MCP tool calls puts the check between the client and the server so disallowed calls never reach the tool at all (Permission Protocol).

The burst has a second-order effect worth planning for. While support is capped, research keeps using the aggregate ceiling. If the aggregate is calibrated at exactly 30,000 and all three teams spend their full share, the aggregate becomes the binding constraint first; that is fine, but a team that over-bursts in month one will quietly eat another team's headroom. Leaving 5 to 10 percent of the aggregate unallocated is a cheap guardrail.

The order of operations in the wrapper is the whole design. Any step out of order creates either a spend leak or a misleading error. Use this sequence:

1. Identify the caller. Read the caller identity from the authenticated request context, never from a model-supplied argument. A prompt injection that changes the caller field must not change who gets charged; untrusted content steering tool use is a documented risk in agent systems (Checkmarx).
2. Resolve the registry entry and the tier. If the caller's tier does not cover the tool, return a permission error. Do not touch counters.
3. Evaluate every matching cap. Per-caller hourly, per-caller monthly, aggregate. All must allow the call.
4. Increment atomically, then re-check. Increment counters and confirm each is at or below its limit within the same operation, or you will admit more concurrent calls than the budget allows.
5. Call upstream. Only now.
6. Record the outcome. Log the decision, the resolved caps and the counter values so a dispute can be settled from logs, not memory.

A compact implementation of steps 3 to 5 in a wrapper:

def call_tool(tool, caller, args, ctx):
    entry = registry[tool]
    ctx.require_tier(caller, entry.tier)          # access decision

    limits = [
        (f"caller:{caller}:{tool}:hour", entry.callers[caller].hour),
        (f"caller:{caller}:{tool}:month", entry.callers[caller].month),
        (f"tool:{tool}:month", entry.cap.month),
    ]
    for key, limit in limits:
        used = ctx.counters.increment_if_below(key, limit)
        if used is None:
            ctx.log_refusal(tool, caller, key, limit)
            raise CapExceeded(tool, caller, key, limit)

    try:
        result = ctx.http.post(entry.upstream, json=args, timeout=ctx.timeout)
    except Timeout:
        ctx.counters.refund(key)   # see "Refunds" below
        raise
    return result

Two subtleties deserve names. Refunds should only apply when the upstream call provably did not run, because a timeout does not tell you whether the vendor executed the query. And the counter store must be shared across every wrapper instance (Redis, a database, or the gateway's own counter service), since local in-process counters give each replica its own private budget. Gateways that count usage externally are explicitly eventually consistent: a burst of concurrent in-flight requests can overshoot slightly before blocking begins (TrueFoundry rate limiting docs). Plan for a small overshoot rather than claiming exactness.

The window type matters more than the data store. Fixed windows are simple and memory-cheap but allow a caller to spend its full hourly budget at 11:59 and again at 12:01, a 2x burst across the boundary. Sliding windows and token buckets smooth this at the cost of more state; token buckets also let you express a sustained rate plus a burst allowance, which matches agent traffic better than a hard per-minute count (Solo.io).

A practical line: use fixed monthly windows for the budget that finance cares about and a token bucket for the hourly ceiling that protects the upstream. For an agent that may fan out, also cap concurrency per caller, because a thousand queued calls each under the rate limit can still overwhelm a vendor. The general rate-limiter design guidance is to fail in a way that does not take down the whole system when the counter store misbehaves, so decide up front whether an unreachable Redis means "refuse" (safe for spend, worse for availability) or "allow and log" (ByteByteGo).

A refusal that the agent cannot interpret becomes a retry storm, which is worse than the original burst. Return a typed error with the window, the limit, the used count and a reset time, and mark whether a retry could ever succeed. Then give the model a short instruction set: on cap_exceeded with retryable: true, wait until reset_at; on retryable: false, report the limit to the user. Structured, schema-validated payloads are the right vehicle here for the same reason they matter everywhere else in an agent pipeline: they are checked against a contract before downstream code acts on them (TrueFoundry).

Observability closes the loop. Emit a counter for every refusal keyed by tool, caller and window, and a gauge for remaining headroom. If refusals cluster on one caller at the same hour every day, the cap is mis-sized, not misbehaving. Tool-calling optimization work makes the same point about measurement: task efficiency (tokens, calls, retries, dollars) has to be tracked separately from whether the task succeeded, or you cannot tell a budget problem from a quality problem (Paragon).

Registry governance is the other half. An agent registry exists to give a single, auditable inventory of what agents and tools exist and who owns them, which is exactly the object you want to hang caps on (Palo Alto Networks). If your tools already sit behind a gateway rather than hand-rolled wrappers, check whether it supports per-subject partitions and metadata filters before building counters yourself; many do, and the configuration is a few lines rather than a service (TrueFoundry docs). When you would rather spend the engineering time on the product than on counter plumbing, Quanta Inference puts text, image, video, audio and embedding models behind one API key, which is a common place to start the same budgeting discipline.