Keeping Model Calls Warm: Reserved Capacity

Reserve only when three conditions hold at once. If any one fails, stay on-demand.

1. Your sustained p95 request rate already beats the smallest reservation you can buy. If the floor is bigger than your p95, you are committing to capacity you provably do not use. If you do not have the floor size in hand, ask the provider for its smallest unit before you compute anything else.
2. Your burst ratio is low. Burst ratio is peak request rate divided by p95 request rate, computed over the same window and the same bucket size. A ratio near 1.1 means traffic is almost flat and a reservation will be busy nearly all the time. A ratio of 3 or more means a large share of your minutes would sit idle inside a reservation sized for p95.
3. Your reservation would be busy for most of the review period. As a working threshold, ask whether you can keep a reservation above roughly 60 to 70 percent of guaranteed throughput across a month. Below that, the committed rate is paying for empty minutes, and on-demand plus occasional spill is cheaper even at a higher per-token price.

The rule in one line: reserve for p95, absorb the tail on-demand, re-check monthly.

This mirrors how the serverless world handles the same problem. A systematic review of cold start latency frames the trade as pay-per-use economics versus initialization overhead: scaling to zero is cheap but pays a startup tax on the first request. A dynamic pre-warm study reports that predictive pre-warming cut cold starts by up to 60 percent and improved average response times by 45 percent compared to prior methods, but that result only exists because the authors spent capacity keeping instances warm. That spend is the same decision as a reservation. Pre-warm nothing and every spike costs latency; pre-warm everything and you fund idle capacity.

Take a customer support assistant with these numbers measured over a four-week window, all taken from application logs:

Now convert the steady part into tokens per minute, because that is the unit most reservations are sold in.

• p95 request rate: 4.0 req/s times 60 = 240 requests per minute
• Tokens at p95: 240 times 1,450 = 348,000 tokens per minute
• Peak: 720 requests per minute, or 1,044,000 tokens per minute
• Average load: 1.67 req/s times 60 times 1,450 = about 145,000 tokens per minute

Assume the smallest reservation unit in your region guarantees 200,000 tokens per minute. The reservation then covers 200,000 of the 348,000 tokens per minute your p95 demands, which is 57 percent of p95. You would buy two units (400,000 tokens per minute), which covers p95 fully with 115 percent headroom. That second unit is not free. It is worth buying only if you can keep total utilization above roughly 60 to 70 percent of 400,000, meaning above about 240,000 to 280,000 tokens per minute on an average day.

Here is the check. Average load is about 145,000 tokens per minute, which is well under the line for two units and just under it for one. That tells you the honest answer for this workload: a two-unit reservation would sit around 36 percent utilized, and a one-unit reservation around 72 percent. One unit is defensible if you accept that about 148,000 tokens per minute of your p95 will spill to on-demand. Two units are defensible only if you expect average traffic to grow by roughly 65 percent.

Put that arithmetic in code rather than a slide, so the next review takes minutes:

def size_reservation(rate_avg, rate_p95, rate_peak, tok_per_req,
                     unit_tpm, target_utilization=0.65):
    p95_tpm = rate_p95 * 60 * tok_per_req
    peak_tpm = rate_peak * 60 * tok_per_req
    avg_tpm = rate_avg * 60 * tok_per_req
    units = max(1, round(p95_tpm / unit_tpm))
    guaranteed = units * unit_tpm
    return {
        "p95_tpm": p95_tpm,
        "peak_tpm": peak_tpm,
        "units": units,
        "guaranteed_tpm": guaranteed,
        "p95_utilization": round(p95_tpm / guaranteed, 2),
        "avg_utilization": round(avg_tpm / guaranteed, 2),
        "clears_target": avg_tpm / guaranteed >= target_utilization,
        "burst_ratio": round(rate_peak / rate_p95, 2),
    }

print(size_reservation(1.67, 4.0, 12.0, 1450, unit_tpm=200_000))

Run it and read three outputs in order: p95 utilization, average utilization, and whether average clears the target. In the example, average utilization is 0.36 for two units, which fails the target. The function says buy two units by the p95 rule and the utilization check says do not. When they disagree, the utilization check wins, because p95 is a single percentile and a handful of slow minutes can drag it up.

A second scenario makes the rule concrete. Same request shape, but traffic is flat: peak 4.5 req/s, p95 4.0 req/s, burst ratio 1.125, average 3.8 req/s. Average load becomes 3.8 times 60 times 1,450 = 330,600 tokens per minute. Two units guarantee 400,000, so average utilization is 83 percent. That is a clear reserve: flat traffic, high steady utilization, and a tail small enough to absorb on-demand.

The contrast is the insight. The first scenario has higher peak traffic than the second, yet it is the worse reservation candidate. Burst ratio and utilization decide, not peak.

Providers rarely sell "requests per second." They sell throughput floors in tokens per minute, model units or similar bundles, and the conversion is where sizing goes wrong. Three conversion mistakes account for most bad reservations.

Mixing input and output tokens. Many providers price reserved capacity as separate committed rates for input and output, which the Inventive HQ guide describes. If your prompts are long and your answers short, which is typical for retrieval and extraction work, your input rate dominates and your output commitment can be much smaller. Compute the two separately: input tokens per minute at p95 and output tokens per minute at p95, then compare each to its own floor. In the worked example, p95 input is 240 times 1,100 = 264,000 tokens per minute and p95 output is 240 times 350 = 84,000. Those are very different reservations.

Ignoring prompt caching and batching. If a large share of your input is a fixed system prompt or shared context, cache hits cut the number of billable input tokens, which changes what you should commit to. Both the AWS pricing page and the Inventive HQ guide show lower effective rates for cached or batch work, so a reservation sized on uncached input will be oversized.

Sizing on the wrong window. Compute p95 over one-minute buckets across four representative weeks. Long buckets hide bursts and short ones overreact to single slow minutes. If your product has a weekly cycle, four weeks captures it. If the cycle is longer than a month, treat the reservation as provisional and review it on a schedule that matches.

Finally, write the spill plan down before you sign. Above the reservation, requests fall through to on-demand at the provider's per-token rate. That is fine as long as you have decided how much of the tail you are willing to pay premium for. Use queue depth as the tripwire: GMI Cloud's guidance treats roughly 50 pending requests per GPU as the line where demand has outstripped serving throughput, and below it you have headroom. Pick your own equivalent metric, such as pending requests per worker or queue wait at p99, and alert on it separately from the reservation size, because a reservation masks overload until the queue is already deep.

Cost is the argument people make for reservations. Latency variance is usually the real one. On-demand inference competes for capacity, so your p99 tail moves when someone else has a busy afternoon. A reservation pins a floor you control.

The serverless research makes the same point about warm capacity. A systematic review of cold start latency classifies keep-warm methods, caching and machine-learning-based scaling as separate families, and each exists because the first request to an uninitialized environment is slower than the tenth. A serverless cold start explainer notes that Lambda cold starts range from about half a second to a few seconds depending on runtime and package size, and that warming requests or provisioned concurrency are the standard mitigations. A Supabase performance thread shows what that looks like in practice: a maintainer reported a median cold latency of 400ms against a hot median of 125ms. That ratio is a little over 3x, and it applies to your p99 whether or not you caused it.

For model inference specifically, the equivalent gap is first-token latency while weights load. GMI Cloud puts weight loading for large models at 15 to 30 seconds when scaling from zero, which is why some form of warm floor is close to mandatory for interactive products. That argues for a reservation only when your latency requirement cannot tolerate the tail. If a two-second worst case is acceptable, on-demand plus retries is simpler and cheaper. The table makes the split readable.

Treat the numeric cutoffs as starting points, not law. The burst ratio thresholds are judgment calls. The utilization threshold is the one to take seriously, because it is the direct cost of idle capacity.

Three practices keep a reservation honest.

Review monthly against the same percentiles. Recompute p95 and average utilization, compare to the guarantee, and adjust. A reservation that was 80 percent utilized last month and 40 percent this month is a signal that the workload changed, not that the reservation was wrong when signed.

Split workload by urgency before reserving anything. Interactive chat, agent tool calls and user-facing generation are latency-sensitive and belong on the reservation. Summarization backlogs, embeddings for indexing and evaluation runs are not, and batch inference is priced lower for exactly that reason. If your p95 mixes both, the reservation sizes the wrong thing. Split the traffic first, then compute p95 on the urgent portion only.

Prefer several small adjustments to one big commitment. A long-commitment discount tempts teams into a six-month or annual floor on a workload they measured for three weeks. A reservation you can renegotiate monthly costs slightly more per unit and is far cheaper than twelve months at 30 percent utilization.

Where your models span more than one provider, the p95 arithmetic has to be done per provider, per region and per model, because each has its own reserved unit and its own floor. That is a lot of accounting for one decision. Quanta Inference puts text, image, video, audio and embedding models behind one OpenAI-compatible API with prepaid pricing in US dollars, which collapses several of those ledgers into one line and makes the utilization check a single calculation.