Turning a Token Estimate Into a Hard Cap: Where the Limit Lives

A gateway sits between your applications and one or more providers. It sees every request regardless of which service sent it, which makes it the natural home for a budget that applies to a team, a customer, a virtual key or an entire organization.

The mechanics are well documented in open-source gateways. LiteLLM enforces personal, team and team-member budgets against spend read from a database, and its documentation is blunt about the consequence: on a deployment with no database, the global budget check is skipped and requests keep being served past the limit per the LiteLLM budget docs. A cap that fails open is not a cap, and this is the most common way gateway budgets quietly stop working.

Other gateways implement the same idea with different accounting. Bifrost describes a four-tier hierarchy of customer, team, virtual key and provider config budgets, each with a maximum limit, a reset duration and a live usage figure, rejecting requests when usage reaches the limit per the Bifrost write-up. Kuadrant takes a more careful approach to the concurrency problem, reserving an estimated token amount when a request arrives and committing the actual usage when the upstream responds, so many simultaneous requests cannot all pass a zero-cost check per the Kuadrant docs.

What the gateway can enforce:

• A ceiling on total tokens or dollars per key, team or tenant, across every service that shares the key.
• Uniform treatment of providers, including normalization of cached input tokens, which providers report inconsistently.
• A single place to change limits without redeploying application code.

What the gateway cannot enforce:

• The cost of the request in flight. Token counts are only known after the response, and as the agentgateway documentation states plainly, it cannot interrupt a response mid-stream, so the request that pushes you over budget succeeds and the next one gets a 429 per the agentgateway rate limit docs.
• Requests that bypass it. Any service holding a provider key directly is invisible.
• A per-request output ceiling. A gateway can refuse the next call, but it cannot make the current completion shorter.

That last point is why gateways also need per-request parameters. The max_output_tokens-style ceiling in the request is the only thing that bounds a single generation, and the production guidance from OpenAI treats request-level limits and retry handling as part of going live, not an afterthought per the OpenAI production best practices guide.

If you own the code that builds the request, you can do something a gateway cannot: count the payload yourself and refuse or shrink it before anything is billable.

The counting has to use the right tokenizer. Tiktoken reproduces the byte-pair encoding used by OpenAI models, so a local count matches what the API charges, while character-based heuristics miss badly on unusual text: a short string with an emoji and a hyphenated word measured 8.0 by a divide-by-four rule against 11 real tokens, a 37 percent miss per the Galileo tiktoken guide. Multiply a 37 percent miss by a million requests and you have a budget that is wrong before anyone spends anything.

Third-party counters exist for other model families, and the general rule from the tokenizer literature is to count with the same tokenizer the model uses, because the same text tokenizes differently across models per the LangChain token splitting guide. Microsoft's own documentation on prompt tokens shows how quickly a payload grows beyond what you wrote: a hidden system metaprompt of roughly 1,200 tokens is added to every prompt, and images below 512 by 512 pixels cost a flat 85 tokens each while larger images are resized and tiled before counting per Microsoft Learn. A service that counts only the user's text will undercount every request with a screenshot in it.

A minimal pre-flight check looks like this:

import tiktoken

enc = tiktoken.encoding_for_model("gpt-4")
LIMIT = 24_000  # input tokens allowed for this feature

def count_messages(messages, tools_text=""):
    total = len(enc.encode(tools_text))
    for m in messages:
        total += len(enc.encode(m["content"])) + 4  # per-message overhead
    return total

n = count_messages(messages, tools_text)
if n > LIMIT:
    messages = trim_oldest_turns(messages, enc, LIMIT)
    if count_messages(messages, tools_text) > LIMIT:
        raise ValueError("request exceeds input budget after trimming")

What the service can enforce:

• A hard input ceiling on every call, before the provider is invoked, including tools, images and retrieved context.
• Truncation strategy you control: drop oldest turns, summarize, or reject with a clear error.
• Per-feature limits that differ by product surface, with the feature label attached to the request for later attribution. Tagging user_id, feature, prompt_version and agent_run_id at the call site is the recommended minimum for cost attribution, and it beats reconstructing context from logs after the fact per Braintrust's cost tracking playbook.

What the service cannot enforce:

• Output tokens, unless the request itself carries an output ceiling. Counting the prompt does nothing about a model that decides to write 3,000 tokens.
• Anything sent by a service you do not own. Each service has to implement the check, and every new client is a new chance to skip it.
• A shared budget. Two services each enforcing 24,000 tokens per call still have no idea what they spent together, unless they report to a shared counter.

This is the placement where the two halves meet. The service bounds the request; the gateway bounds the aggregate. Teams that only build one usually discover the missing half during an incident.

The most common token control in production code is a line in a system prompt asking for brevity. It is also the only one of the three that cannot refuse anything.

The prompt still has a job. It shapes typical length, which lowers the average cost per request, and it reduces how often a request needs a follow-up. What it cannot do is stop a response that is already streaming. A model that has decided to produce a long answer has already produced the tokens you are paying for, and the instruction only influenced the probability, not the outcome. If your output ceiling matters, set it in the request and check it in the response.

Response-side checks are the other half of service enforcement. Read the usage field from every response, including the final event of a stream, and reconcile it against your local estimate. The Usage and Cost APIs exist precisely so you can pull token usage grouped by model, project or API key for a period and compare it with what you predicted per the OpenAI cookbook. A drift threshold is worth setting: MLflow's rate limiting guidance recommends alerting when local estimates drift more than 15 percent from provider-reported usage, because that gap is where silent quota breaches start per the MLflow guide.

Each placement fails in a different way, and the failures are not interchangeable.

Gateway budgets fail open. A missing database, a misconfigured global setting, a deployment without persistence, and the budget becomes a log line. The LiteLLM documentation warns that a budget configured with no database logs a warning once at startup and then serves requests past the limit per the LiteLLM budget docs. Test the failure path deliberately: set a tiny budget, confirm the rejection, then remove the database and confirm you know what happens.

Gateway budgets overshoot by design. Between the response that crosses the limit and the next request that is refused, real spend lands above the cap. Kuadrant's reservation model reduces the race but does not eliminate overshoot, because reservations are estimates and the actual committed usage can exceed them per the Kuadrant docs. A cap of 1,000 tokens per minute with one request returning 1,200 tokens will let that request through and block the next, as agentgateway documents explicitly per the agentgateway docs. Size the cap with headroom, not at the exact number you can afford.

Service checks miss the multi-service sum. A per-call ceiling of 24,000 tokens in five services gives you no protection against 5,000 calls. Attribution layers exist for exactly this gap: provider consoles aggregate by API key or project and cannot tell you which team, feature or agent run spent the money, which is the reason gateway and SDK tagging became standard practice per TrueFoundry's cost attribution write-up.

Service checks mis-count. Wrong tokenizer, missing image tokens, ignored per-message overhead, tool schemas left out of the count. Any one of these turns a hard cap into a soft one with extra steps. The prompt token counter documentation is a useful reminder that the count you see in a playground is only meaningful if the tool uses the provider's tokenizer for the exact text you will send per the prompt token counter reference.

Prompts fail silently. Nobody finds out the instruction stopped working until the invoice arrives, because the model does not report non-compliance.

Retries and tool loops fail everywhere unless you plan for them. Retry storms are one of the recurring causes of unexplained spend spikes: transient provider errors trigger SDK retries with backoff, the application logs only the successful response, and the retry tokens never surface locally per the Braintrust playbook. An agent that calls a tool, receives a malformed result and tries again bills every iteration. Whatever counter you build has to run on every attempt, not on the request your code made once.

A control architecture that handles all of this looks layered, not singular. Hard caps at the gateway for aggregate spend across callers you do not control. Pre-flight checks with the correct tokenizer in each service, plus a request-level output ceiling. Alerts with a drift threshold, never as the only line of defense. And a named owner per budget, because a cap nobody owns is a cap nobody tests. The distinction between a soft alert, a constrained mode that degrades to a cheaper model and a hard cap that stops serving is itself a business decision about whether interruption costs more than the overrun per the NHI Management Group FAQ.

Work through these in order and stop at the first one that matches.

Many callers you do not control, or several services sharing a provider key? Put the hard cap in the gateway, per tenant or per key, and verify the failure path with the database disconnected. Do not rely on the gateway for a per-request ceiling.

A single caller whose worst-case request you need to bound? Put the check in the service, immediately before the client call, using the model's own tokenizer on the full payload including tools and images. Reject or truncate, then send. Set an output ceiling in the request as well.

Revenue-critical path where stopping is worse than overspending? Use constrained mode at the gateway (fall back to a cheaper model or shorter context) and a soft alert with a drift threshold, and keep the hard cap only for non-essential environments such as development and sandboxes.

Agent or multi-step workflow? Count per step, cap iteration count and per-run dollars, and make sure the run identifier is attached to every span so a loop is visible as a loop rather than a large invoice line.

All of the above, as most production systems are? You need two counters, not one: one aggregate in the gateway, one pre-flight in the service, both reconciling against provider-reported usage. The single easiest improvement is making sure both refuse rather than notify, and both fail closed when their backing store is unavailable.

If you would rather not maintain the provider adapters, tokenizers and pricing tables behind that gateway yourself, one option is routing through a single OpenAI-compatible endpoint with prepaid pricing so the spend side is one counter instead of many. That is what Quanta Inference provides: text, image, video, audio and embedding models through one API key, priced in US dollars. It is worth reading the docs for whichever gateway you choose, and specifically hunting for the sentence that describes what happens when its database is down. That sentence tells you whether you have a cap or an alert.