Choosing the Cap Window: Token Bucket, Sliding Log or Fixed

Token bucket. A bucket holds up to N tokens. Tokens refill at a fixed rate. Each call removes one token; if the bucket is empty the call is refused or delayed. Because tokens accumulate while the system is idle, the bucket absorbs a controlled burst up to its capacity, then enforces the average GeeksforGeeks' algorithm reference. AWS API Gateway is a well known implementation of the token bucket approach Zuplo's platform comparison.

Sliding log. You keep a timestamp for every allowed call. On arrival, you drop timestamps older than the window, count what remains, and refuse if the count has hit the limit. Zuplo's primer on the family notes that sliding approaches smooth out the boundary problem at the cost of more computation than a fixed counter Zuplo. The log is the strictest variant: no interpolation, no boundary trick.

Fixed window. Divide time into intervals, count calls in the current interval, and refuse once the counter reaches the limit until the next interval starts Zuplo. It is the cheapest to store and the easiest to explain, and Apigee's Quota policy is built on this fixed window counter model Zuplo.

Take a limit of 60 calls per hour, enforced per caller, and feed all three algorithms the same hostile-but-legal client: a well behaved agent 55 minutes of the hour, then a scripted burst.

Fixed window with hourly boundaries starting at 12:00. The client sends 60 calls at 12:59 and another 60 at 13:01. Both are accepted, because they fall in different windows. The client achieved 120 calls in roughly two minutes against a limit of 60 per hour. Zero refusals. That is the boundary double burst, and it is not a bug in your code, it is the algorithm working as designed Zuplo.

Sliding log with a 60 minute lookback. The 13:01 burst arrives with 60 timestamps still inside the window, so all 60 are refused. The client gets a clean 429 for every attempt and the refusal pattern tracks the real rate. If the client backs off with jitter, retries succeed as timestamps age out, roughly one per minute.

Token bucket, capacity 60, refill 1 token per minute. A client that goes quiet for an hour starts with 60 tokens and can spend all 60 immediately. Then it is refused 59 times over the next 59 minutes before the next token lands. Refusals come in clumps: a burst of permits, then a wall.

The memory numbers are estimates you should size against your own store, but the ratio is the point. A sliding log's footprint is linear in the limit, and the limit is exactly the number you are most likely to raise later. A token bucket and a fixed window are constant size no matter how generous the cap.

For a shared read tool, use a token bucket. Reads are idempotent, so a burst of 60 simultaneous calls is a load problem, not a correctness problem, and the bucket handles it in about 30 bytes of state per caller. The bucket is specifically suited to APIs that need to allow short bursts while holding an average rate over time Zuplo. Your agent's first minute after a long think is bursty by nature, and the bucket rewards that instead of punishing it.

For a write tool, use a sliding log. It stores one timestamp per allowed call, so memory grows with the limit, and that is the price you pay for refusals that are smooth and boundary proof. If the write tool can create, charge, send or delete, a client must never be able to land twice the limit in a two minute span. Refuse to trade that guarantee for bytes.

For everything else, use a fixed window only when the limit is coarse and the action is cheap to repeat or easy to deduplicate. A counter cache that refreshes a public page every 10 minutes at 12 calls per hour can tolerate a boundary double burst. A payment endpoint cannot.

Three secondary rules sharpen the decision:

1. If the tool has an idempotency key and the downstream honors it, a burst is a load issue not a duplication issue, and the token bucket is safe even for writes.
2. If the limit is above a few hundred per caller, a sliding log starts to cost real memory at scale, so consider a sliding window counter or accept a token bucket.
3. If you cannot state the worst case in a sentence, the tool is not ready for a limiter; fix the operation first.

The whole state is two numbers and a timestamp. The refill is computed lazily on read, which is why the per caller footprint stays flat.

import time

class TokenBucket:
    def __init__(self, capacity: int, refill_per_sec: float):
        self.capacity = capacity
        self.refill_per_sec = refill_per_sec
        self.tokens = float(capacity)   # bytes: 8
        self.updated = time.monotonic() # bytes: 8

    def allow(self, cost: int = 1) -> bool:
        now = time.monotonic()
        elapsed = now - self.updated
        self.tokens = min(self.capacity, self.tokens + elapsed * self.refill_per_sec)
        self.updated = now
        if self.tokens >= cost:
            self.tokens -= cost
            return True
        return False

# 60 reads per hour, bursts up to 60
bucket = TokenBucket(capacity=60, refill_per_sec=60 / 3600)

Two refinements before production. First, store the bucket in a shared store (Redis with a Lua script is the usual choice) so that replicas do not each grant a full bucket Solos' rate limiting overview. Second, return a Retry-After header on refusal; the token math gives you the exact wait: (cost minus tokens) divided by refill rate.

The log is a sorted set of timestamps. On each call you remove entries older than the window, count the remainder, and refuse at the limit. The subtlety is that you must prune on every check, or memory grows without bound when a caller goes quiet.

import time
from collections import deque

class SlidingLog:
    def __init__(self, limit: int, window_sec: int):
        self.limit = limit
        self.window = window_sec
        self.stamps = deque()  # one float per allowed call

    def allow(self) -> bool:
        now = time.monotonic()
        cutoff = now - self.window
        while self.stamps and self.stamps[0] <= cutoff:
            self.stamps.popleft()
        if len(self.stamps) >= self.limit:
            return False
        self.stamps.append(now)
        return True

The operational cost is visible in the class: stamps grows to limit and no further. A 20 writes per hour cap on a create-ticket tool costs 20 timestamps per caller, about 160 bytes as raw floats, plus the store's overhead. That is cheap for the guarantee, and a per call policy check sits naturally at the boundary before the tool receives arguments, the same shape as an MCP policy proxy that inspects each tools/call first Permission Protocol's MCP enforcement guide.

If you run the limiter inside an AI gateway rather than in your own code, check how the vendor evaluates rules. TrueFoundry documents that every matching limit is checked and all must allow the request, with usage counted against each, and that enforcement is eventually consistent so a burst of concurrent in-flight requests can overshoot slightly before blocking begins TrueFoundry's rate limiting docs. That overshoot note is a reason to keep write limits well below the capacity of the downstream.

Size the limit from the downstream, not from the agent's feelings. Solo.io's guidance is to start conservatively, monitor, and adjust against observed peak usage and infrastructure capacity Solo.io. For a write tool, add a second, coarser limit on the same caller: 20 writes per hour for the tool, plus 200 per day across all tools. Multiple limits combine as an AND, so the tighter one wins when it matters TrueFoundry.

Two anti-patterns to avoid. First, do not enforce only on the client; client side limits can be forged or bypassed, so the server side check is the one that counts ByteByteGo's rate limiter design. Second, do not set the read limit equal to the write limit just for tidiness. Reads can afford a bucket ten times the write cap; writes should never see a burst they cannot idempotently absorb.

For teams routing many models and tools through one place, the cap window decision is per tool, but the plumbing is shared. Quanta Inference (https://quantaintelligence.ai/inference) exposes text, image, video, audio and embedding models through one OpenAI compatible API key, which keeps the caller identity consistent across models so a per caller bucket means the same thing everywhere. And if you would rather have the cap, the tool policy and the audit trail handled by someone who does this for a living, a Quanta expert (https://quantaintelligence.ai/agents) can review your limits against your actual tool usage.