Per-Caller Caps Over Tool Tiers: Weights and a Floor

Define the tool's total rate R in calls per second (or any window unit). Each caller i gets two numbers:

• Weight w(i), a non-negative number. The caller's fair share of spare capacity is w(i) divided by the sum of all weights.
• Floor f(i), an integer count of calls guaranteed per window. The floor is not a share; it is a protected minimum.

The invariant that makes the design work: the sum of all floors must be less than or equal to R times the window length. If the floors add up to more than the tool can deliver in a window, no allocation rule can honor them, and you have promised more than the tool can serve.

Worked example. A tool serves R = 600 calls per minute. Four callers:

Floors total 220, well under 600, so 380 calls per minute are spare capacity and are divided by weight: A gets 228, B gets 76, C gets 38, D gets 38. A caller's ceiling in a saturated window is its floor plus its share of spare capacity: A = 328, B = 136, C = 68, D = 68. Note that the floors protect the two 30-call agents from being crowded out by Tenant A even though A holds 60 percent of the weight.

Decision rule: set floors first from the minimum service level each caller needs to function, then assign weights from usage and commercial importance. Pick floors so their sum stays under roughly 50 to 70 percent of R. Above that, a single busy caller can push the tool to saturation and the borrowing rule has nothing to lend.

For agents as well as tenants, one more decision: does an agent inherit its tenant's weight, or carry its own? If a tenant runs one interactive agent and one batch agent against the same tool, give the batch agent a small weight and a small floor and the interactive agent a larger floor for latency-sensitive work. Treating every agent as a separate caller is usually right when agents have different latency needs; treating them as one caller is right when they are interchangeable.

One bucket per caller. The refill rate is weight times the tool rate, divided by total weight: refill(i) = R × w(i) ÷ Σw. The bucket capacity should be small, ideally equal to the floor plus a modest burst allowance, so a caller cannot bank a huge burst and then drain the tool.

The floor clamp is the piece most implementations miss. On every check, after computing the refilled balance, raise it to at least the floor value for the current window before deciding. This means a caller that has been throttled still holds enough tokens to serve its guaranteed calls. Concretely:

on request from caller i:
    now = monotonic clock
    elapsed = now - last_refill(i)
    balance(i) += elapsed * refill(i)
    balance(i)  = min(balance(i), capacity(i))
    balance(i)  = max(balance(i), floor(i))   # floor clamp
    last_refill(i) = now
    if balance(i) >= 1:
        balance(i) -= 1
        return ALLOW
    else:
        return THROTTLE retry_after = (1 - balance(i)) / refill(i)

The clamp to floor is safe only if floors are reserved, not spent from the shared balance. Two ways to implement that:

1. Reserve at admission. Track total calls per window. Allow a call if (a) the caller is under its floor, regardless of anything else, or (b) total calls in the window are under R and the caller has tokens.
2. Two-tier tokens. A floor bucket that refills at floor(i) per window and is never borrowed, plus a share bucket that refills at the weighted rate and is borrowable.

The first is simpler and more auditable. The second is more precise for bursty interactive traffic. Either way, the rule that matters is that a floor call is decided before any borrowing decision, because borrowing must never consume a token reserved for another caller's floor.

Rate limiting at the gateway layer typically counts usage against every matching limit, and all matching limits must allow the request TrueFoundry rate limiting. That pattern generalizes cleanly: the per-tool cap and the per-caller bucket are both limits, and the request passes only if both pass and the caller's floor reservation is not violated.

Borrowing is what makes this a shared tool rather than a set of partitions. The rule: when a caller has unused capacity in a window, that capacity is available to other callers, but only to those that are over their weighted share, and only after every other caller's floor has been protected.

In practice:

1. Measure slack. At any moment, slack equals R times elapsed time in the window minus total calls so far, minus the sum of floors not yet served. This is the amount that can safely be lent.
2. Order the borrowers. Among callers that want more than their share, serve them in descending weight order. Ties break by least recently served. This means the highest-weight caller absorbs spare capacity first, which matches the weighting scheme and avoids a low-weight caller capturing a burst that the large tenant paid for.
3. Cap the borrow. A borrower may exceed its share only up to the ceiling (floor plus share), and never beyond the slack computed in step 1. If the borrower's tokens run out, it is throttled with a retry suggestion, not queued indefinitely.
4. Never borrow against a floor. Slack excludes the unserved floor of every caller, so Tenant A's burst cannot consume Agent C's protected 30 calls, even if C has been quiet all window.

A worked borrow. In the example above, at 40 seconds into the minute with a saturated tool, Tenant A has already sent 300 calls (its ceiling is roughly 219 so far if the window is spread evenly, so it is over share), B has sent 90, C has sent 10, D has sent 12. Total is 412 against an allowance of 400 so far, so the tool is at the limit and A is borrowing. C's remaining floor is 20; slack is therefore 400 minus 412 minus 20, which is negative, so A is not allowed to borrow further until C or others catch up. If C's remaining floor were 0 because C already served its 30, slack would be positive and A could continue up to its ceiling.

Two engineering cautions. First, borrowing is where token bucket and fair queuing differ: token bucket caps and does not lend, so a lending policy must be added explicitly, not assumed token bucket rate limiting. Second, counters at gateway scale are eventually consistent, and a burst of in-flight requests can overshoot a limit before blocking begins, so size the floor margin accordingly TrueFoundry rate limiting. Multi-tenant systems that work well in practice combine throttling, quotas and workload isolation, with quotas sized per tenant rather than one global number fairness in multi-tenant systems.

The main operational decision is what counts as one caller. Options, with trade-offs:

For agents that call tools, the identity question is real. Agent catalogs and registries exist largely to answer "which agent is this and what is it allowed to do," which is the same question a rate limiter needs answered to pick the right bucket agent registry. Policy enforcement belongs in the tool-call path itself, before the call reaches the server, which is also the right place for the admission decision MCP policy enforcement. Putting the rate check there gives you one place where identity, policy and quota meet.

Store bucket state in a shared, low-latency store (Redis or equivalent) keyed by caller, and make the read-modify-write atomic with a Lua script or a transactional operation. Non-atomic updates under concurrent requests are how "600 per minute" becomes 680. If you need distributed enforcement, note that many platforms are per-region only or depend on Redis topology for cross-region sync, so confirm the deployment model before relying on a global counter platform comparison.

A repeatable procedure:

1. Measure the tool's true capacity. Find the concurrency and dependency limit of the downstream tool, not the API gateway limit. That is R.
2. Set floors from minimum viable service. For each tenant, ask how many calls per minute its core workflow needs to avoid failing. That is f(i). Sum them. If the sum exceeds 70 percent of R, either raise R or cut floors.
3. Set weights from expected contention. Normalize so weights sum to a convenient number. A tenant paying for a share of capacity gets a proportionally larger weight, but weight affects only spare capacity, never the floor.
4. Size bucket capacity. Capacity = floor + burst allowance. Allow bursts that fit your SLAs, not arbitrarily large ones; a huge bucket is a giant burst waiting to happen token bucket algorithm.
5. Choose the window and token model. Per-minute windows are common and simple; per-second windows react faster but are noisier. Sliding windows avoid the boundary double-spend that fixed windows permit platform comparison.
6. Test with a parasite. Run a load test where one tenant hammers the tool at ten times its share and verify every other caller still receives its floor. If not, the floor clamp or the slack calculation is wrong.

A python sketch of the decision, close enough to port:

def admit(caller, now, total_rate, total_weight, window_seconds, state):
    st = state[caller.id]
    elapsed = now - st.last
    st.balance = min(st.balance + elapsed * total_rate * caller.weight / total_weight,
                     caller.capacity)
    st.balance = max(st.balance, caller.floor)  # floor clamp
    st.last = now
    window_calls = state.window_calls(now)
    reserved = state.unserved_floors(now)
    slack = total_rate * state.elapsed(now) - window_calls - reserved
    if window_calls >= total_rate * state.elapsed(now):
        if caller.is_over_share(now) and slack <= 0:
            return False
    return st.spend_one()

Decision rule for the borrowing order: highest weight first, ties by least recently served. If you instead borrow first-come first-served, the caller that happens to burst at the right millisecond wins, and weights stop meaning anything.

Place the per-caller check in the same enforcement boundary as tool policy, so no call reaches the tool without passing both. Keep the per-tool limit as a backstop: if the denominators are wrong, the aggregate cap still protects the downstream service. Log every decision with caller, weight, floor, share, slack and outcome. Those fields let you reconstruct why a call was throttled and are what you need when a tenant disputes a block.

If you are running many agents against shared tools and do not want to build identity, buckets and borrowing yourself, Quanta Inference provides one API for text, image, video, audio and embedding models, which keeps the tool side manageable while you enforce your own policy layer in front of it. For teams building the caller-side agents, a coding agent skill that knows your limit headers and backoff rules saves each new agent from rediscovering them.

Prove fairness with a three-part acceptance test: one heavy tenant, one quiet tenant, one interactive agent. Heavy tenant at 10x share must not reduce the others below their floors. Quiet tenant's unused capacity must move to the heavy tenant up to its ceiling. Interactive agent must see its floor honored under sustained saturation. When all three hold for a full window with no borrowing against reserved floors, the design is done.