How to Cap the Strong Model in a Per-Task Router

The intuition behind a spend cap is simple: stop calling the strong model once the budget is gone. The failure mode is timing. A counter that increments after each call reflects what has already happened, not what is about to happen. When twenty workers pick up twenty tasks at the same moment, all twenty read the same pre-burst value, all twenty see room in the budget, and all twenty call the strong model. The overshoot equals the number of concurrent callers multiplied by the cost of a call.

Worked example. Suppose the cap is 50 USD per hour for strong-model work, the average strong call costs 0.25 USD, and the service runs 8 workers. At 09:00 the bucket sits at 44 USD. Eight workers each read 44, each see 6 USD of headroom, and each admits one call. That is fine on its own. But if the scheduler releases work in batches of 40 and each task uses two strong calls (draft plus verify), the burst can admit far more than the remaining budget. Within seconds you are at 64 USD, and the counter only learns about it as the responses return.

This is the same problem that billing consoles solve with a two-step check. Provider spend APIs read period-to-date spend that may itself be a lagging reading, as the Claude platform documentation notes: the value can read as zero if the spend reading is temporarily unavailable and should be treated as informational, not transactional (Claude spend limits API). A dashboard is not a gate. You need a synchronous counter on the hot path plus a reserve, and you need to stop admitting work once the reserve is reached, not once the budget is exhausted.

Three decisions determine whether the cap holds.

Pick one budget unit and one window. Dollars per hour is the most defensible choice because it is what you are actually billed for, it absorbs price differences between models, and it survives prompt length variation. Token counts mislead: a 4,000 token prompt on a strong model can cost more than a 40,000 token prompt on a cheap one. Hourly buckets reset at a fixed minute mark, weekly buckets need a rollover job, and monthly buckets are too slow to protect against a bad afternoon. Use two layers: a hard hourly cap (the emergency brake) and a monthly cap (the board-level number).

Use a synchronous counter with reservation. Redis with a Lua script or an atomic increment plus compare is the standard pattern. The rule is important: reserve estimated cost before the call, reconcile to actual cost after. Estimating from token count via the provider's tokenizer is close enough; overshoot on a call is capped at the estimate error, not at the number of concurrent callers. Reconcile by writing the difference back, positive or negative, and keep a running total of unreconciled reservations so a crashed worker cannot permanently inflate the bucket.

# Pseudocode: reserve-then-call with a hard hourly cap
def admit(est_cost_usd, cap_usd, reserve_ratio=0.85):
    key = f"spend:{current_hour_bucket()}"
    script = """
    local spent = tonumber(redis.call('GET', KEYS[1]) or '0')
    local reserve = tonumber(ARGV[2])
    if spent + reserve > tonumber(ARGV[1]) then return -1 end
    redis.call('INCRBYFLOAT', KEYS[1], ARGV[2])
    redis.call('EXPIRE', KEYS[1], 7200)
    return spent + reserve
    """
    return redis_eval(script, key, cap_usd, est_cost_usd)

def route_strong(task):
    admitted = admit(est_cost_usd=task.est_cost)
    if admitted < 0:
        return queue_for_later(task)   # overflow goes to the queue
    try:
        result = strong_model_call(task)
        reconcile(task.est_cost, result.actual_cost)
        return result
    except TransientError:
        release(task.est_cost)          # give the budget back
        raise

Hold back a reserve. Do not let the counter climb to 100 percent of the cap. Set the admission threshold at something like 85 percent, so in-flight calls and estimation error land inside the budget. The remaining 15 percent absorbs stragglers. When spend crosses the reserve line, the router stops admitting new strong calls entirely and everything new goes to the queue.

The distinction between a soft cap and a hard cap matters here. A soft cap triggers alerts and lets traffic through; a hard cap refuses admission. Provider dashboards and seat-level spend limits are useful for governance and per-user fairness, since they resolve limits through a hierarchy of user, group, seat tier, and organization scopes (Claude spend limits API). They are not a substitute for a gate your own router controls, because they operate on a slower clock than a burst.

A queue is not just a buffer. It is the place where you decide what happens to work the cap refused. Three decisions live inside it, and each needs an explicit policy rather than a default.

Which tasks may wait. Not all strong-model work has the same deadline. Email classification can wait until the next hour bucket; a user waiting on a chat answer cannot. Tag every task with a class and a deadline. Interactive classes get a short maximum wait (seconds), asynchronous classes get a longer one (minutes to hours), and batch classes can wait until the next bucket opens.

What happens when the wait expires. Pick one per task class and write it down:

• Downgrade to the cheap model, optionally flagging the output for review.
• Return a clear "try again later" response with a retry-after value.
• Defer to the next hourly bucket.
• Drop, for work that is genuinely optional.

The default should be explicit and observable. Silent downgrades are the most common mistake: users get a weaker answer and nobody records it, so quality regressions never show up in a dashboard.

How the queue drains. Keep the drain rate below the cap. If the hourly cap is 50 USD and average strong calls cost 0.25 USD, the sustainable drain is 200 calls per hour. Draining faster than that just moves the burst one bucket later and creates a sawtooth. Prefer oldest-first within priority class, and add a jitter so a full queue does not drain in one spike when the bucket resets.

Queueing theory has treated this exact problem. Admission control for randomly arriving tasks with classes and prices, where the goal is to accept the tasks that maximize total value over a horizon, is a well-defined optimization problem (Optimal Admission Control for Multiclass Queues). You do not need the full solution. A priority queue plus a drain budget plus a max wait captures most of the value, and it is auditable, which the optimal policy is not.

Set the thresholds from cost per successful task, not per token, because retries, verification calls, and rework all sit inside the real cost. The all-in formula counts uncached input, cache writes and reads, output, initial calls, retries and fallbacks, batch work, and human rework, then divides by successful tasks (Wavect). Two tasks with the same token count can differ many times over in cost per successful task once retries are counted.

Recalculate these when the traffic mix, the prompt templates, or the model assignment in the strong tier change. The router's cost profile moves whenever any of those move, and thresholds set once will drift.

A safety net you cannot see is a safety net you cannot tune. Instrument four things from day one: the counter value by bucket, admission refusals by task class, queue depth and age, and fallback rate. Log the estimate and the actual cost for every strong call so you can measure estimation error and set the reserve from evidence rather than guesswork.

Quality has to be measured alongside cost. A/B testing agents is harder than testing a button because output is non-deterministic and quality spans several dimensions at once, including task success, latency, cost, and hallucination rate; separating a real quality lift from model stochasticity can take large samples of trajectories per arm (Maxim AI). That is the argument for splitting the router's decision from the safety net's behavior: keep the routing rule stable while you evaluate it, and treat cap and queue changes as their own change with its own metrics.

A simple weekly review closes the loop. Read the refusal rate. If it is near zero, the cap may be too loose to protect anything. If interactive refusals are meaningful, the cheap model or the queue is wrong for that class. If estimate error is above a few percent, fix the tokenizer path before tightening the cap.

The combined design costs more engineering than either alone, so it is worth being explicit about when it pays.

The decision rule: if the router runs unattended and strong-model spend is a line item someone defends, you need both. If the router is behind a human who reviews every escalation, a cap with alerts is adequate. If the only worry is latency, a queue is enough.

For teams that would rather not run the counter and queue infrastructure themselves, putting the model access behind one API is a way to centralize spend, since a single gateway is where a cap can be enforced for every call regardless of which model a task lands on. That is the role of Quanta Inference: text, image, video, audio, and embedding models through one API key, OpenAI-compatible and prepaid. The cap and queue logic still belongs in your application, but the accounting surface is one place rather than several.

Beyond the code, someone has to own the thresholds, the fallback policies, and the weekly review. That is a job description more than a library. If the routing work is stuck at the design stage, a free 30-minute discovery call is a reasonable place to pressure-test the cap and queue numbers against real traffic.