Setting Output Token Limits per Task Without Starving the Model

The rule is short. For each task (summarize a ticket, draft a reply, extract fields, write a commit message), measure completion length from logs, and set the output cap at that task's 99th percentile. Keep a separate, higher cap for reasoning steps and for any agent step that must think before it acts.

Why the 99th and not the average or the 90th? The average tells you about the middle of the distribution, which you probably already satisfy. The 90th tells you where the bulk of ordinary traffic ends. The 99th is the smallest round number that still lets almost every real request complete. Set the cap at the 90th and roughly one request in ten gets clipped; set it at the 99th and roughly one in a hundred does, and you can watch that one.

The distribution is usually long-tailed, not normal, so percentiles beat means. A single 4,000-token generation in a task whose median is 280 tokens barely moves the mean but is exactly the case that a 90th-percentile cap kills. Measuring percentiles rather than averages is the whole point.

Here is a decision table you can keep next to the code:

The bottom row deserves a warning. Leaving the cap at the model maximum does not make the model more verbose, but it removes your safety net on runaway generations, accidental loops in agents, and prompt injection that says "repeat this forever." The cap is a cost and safety control as much as a quality control.

You cannot set the cap from intuition, so start with the numbers you already emit. Every provider SDK returns usage on the response: OpenAI reports completion_tokens on Chat Completions and output_tokens on the Responses API, and Anthropic reports output_tokens alongside cache fields (OpenAI token categories, tracking token usage). Capture those fields, plus a stable task name and the stop reason.

A minimal logging schema gets you 90 percent of the way:

{
  "task": "support_reply_draft",
  "step": "final_answer",
  "model": "your-model-id",
  "input_tokens": 1830,
  "output_tokens": 412,
  "reasoning_tokens": 0,
  "max_output_tokens": 512,
  "stop_reason": "stop",
  "latency_ms": 2140
}

Keep step separate from task. A task like "resolve billing question" may have a planning step, up to five tool-call steps, and a final answer step; each has its own length distribution and needs its own cap. Tracking only a task-level total hides which step is blowing the budget, which is the same argument observability vendors make for per-span token attribution inside agent traces (Braintrust).

Then compute percentiles per task and step over a rolling window. A hundred lines of Python is enough; use a streaming sketch or an exact sort over the last N days:

import math

def percentile(values, p):
    if not values:
        return None
    values = sorted(values)
    k = (len(values) - 1) * (p / 100.0)
    lo, hi = math.floor(k), math.ceil(k)
    if lo == hi:
        return values[int(k)]
    return values[lo] * (hi - k) + values[hi] * (k - lo)

def caps_for(rows, safety=1.0):
    out = {}
    for (task, step), group in group_by(rows, lambda r: (r["task"], r["step"])):
        lengths = [r["output_tokens"] for r in group if r["stop_reason"] != "error"]
        p90 = percentile(lengths, 90)
        p99 = percentile(lengths, 99)
        out[(task, step)] = {
            "p90": round(p90),
            "p99": round(p99),
            "cap": math.ceil(p99 * safety),
            "n": len(lengths),
        }
    return out

Two cautions when reading the output. First, exclude requests that were already truncated at the old cap, because they understate true length; flag them and analyze separately. Second, require a minimum sample size before trusting a percentile. A task with forty observations has a p99 that is really just the longest request you have seen, so treat it as provisional and re-measure when volume grows. The output-length prediction literature exists precisely because length varies enough to matter for scheduling and padding, and it treats accurate length prediction as a real problem rather than a rounding error (output length prediction).

Reasoning steps behave differently enough to deserve their own budget. Internal reasoning is not shown to the user but is generated, billed, and counted against the output limit, so a cap sized for the final answer can cut the thinking short and leave the visible answer confidently wrong (OpenAI token categories). The inspect_ai issue describes exactly this: evaluations calibrated for non-reasoning models silently truncate reasoning traces before the answer is produced (inspect_ai issue 3582).

A workable configuration separates the two budgets:

The reason to keep them separate is that reasoning length is much harder to bound than answer length, and reasoning is not exposed for streaming, so users see nothing while the meter runs. If your provider exposes a reasoning-token field, log it; where it does not, measure the gap between total output tokens and the tokens in the returned text.

One more rule: size the reasoning cap as a fraction of the remaining context, not a fixed number. A planning step in a long agent run may carry a large retrieved context, and context_window - input_tokens can be the binding constraint long before your cap is (Redis: context window overflow). Reserve a floor for the final answer so the reasoning step cannot consume the entire remainder.

A cap you cannot see working is a guess. Log the stop reason on every call and build four signals from it.

1. Truncation rate per task. Share of calls where the stop reason is the output limit. Alert when the rate crosses a threshold you choose, typically a small multiple of what the 99th-percentile design predicts.
2. Root cause attribution. Is the rate rising because prompts grew, because outputs grew, or because inputs are eating the context? Per-call prompt and completion breakdowns separate those causes (Braintrust).
3. Ceiling pressure. For truncated calls, log output_tokens / max_output_tokens. A call that hits the wall at 100 percent of the cap is qualitatively different from one that stopped just short, and a run where 30 percent of samples hit the ceiling is distorted, not just noisy (inspect_ai issue 3582).
4. Empty or malformed answers. An uptick in empty completions, unparsable JSON, or answers that contradict their own reasoning is the downstream symptom of truncation.

Emit the stop reason in structured logs, and if you already run OpenTelemetry, use the GenAI usage attributes so token counts and stop reasons land in the same trace (Braintrust). Then wire the rate into your existing alerting. A reasonable starting threshold is a truncation rate above 2 percent across a fifteen-minute window for any task whose cap was set at the 99th percentile, tuned after a week of observations.

Alert thresholds should key off the percentile you chose, not a universal number. A cap at the 99th percentile should truncate about 1 percent of normal requests; a sustained 3 percent means inputs or outputs shifted, and it is cheaper to widen the cap or adjust the prompt than to let users discover cut-off answers.

Suppose a customer support task, "draft reply," streams from logs over a week. A round 10,000 calls arrive, and after excluding errors you have 9,800 usable rows. The distribution is skewed, as expected:

The mean suggests 305 is plenty. Setting the cap there would truncate roughly four in ten replies, because the mean sits near the median in a long-tailed distribution and hides the tail. Setting it at the p90 of 520 clips one in ten drafts, which for a customer-facing reply means one in ten customers gets a cut-off message. The rule says set the cap at 1,180.

Cost matters here, so do the arithmetic. If the model charges output tokens at five times the input rate, as the published price tables show for current models (token optimization), then raising the cap from 520 to 1,180 does not raise cost on its own: you pay for tokens generated, not for the cap. The cap only governs when generation stops. That is the key insight most teams miss. A higher cap costs nothing unless the model actually uses it, so a tight cap is all downside except as a runaway guard.

Now add the reasoning step. The same task includes an intake triage step that decides routing. Its combined reasoning plus output has a p99 of 2,600 tokens, with the visible answer often under 100 tokens. Give that step its own cap of 2,600 while keeping the draft reply at 1,180. If you had used a single cap of 1,180 for both, roughly 1 percent of triage calls would lose their reasoning before answering, and those failures would appear as empty or wrong routing decisions rather than obvious errors.

Finally, watch the alert. After the change, truncation rate on the draft task should fall to roughly 1 percent. If it settles at 2.5 percent, something changed: prompts got longer, a new template added examples, or a retrieval change pulled in more context. Compare the p99 over the last week against the p99 you used to set the cap; a jump in the p99 is the cause, and the fix is to re-measure rather than to nudge the threshold.

If you are running several tasks across several providers and model families, keeping the caps, the logs, and the usage fields consistent is more work than it looks. A single API surface for text, image, and embedding models, priced in one currency with OpenAI compatibility, removes a layer of that bookkeeping so the token data lines up across tasks. That is the case for Quanta Inference when you scale from one endpoint to many.

A short checklist of the failures you will actually make:

Two more mistakes are worth calling out because they cost money. First, some teams set a low cap to control cost without realizing the bill follows tokens generated, not tokens allowed; a cap is a fence, not a budget. Second, some teams let the model write long preambles before getting to the point, which burns output tokens on "Sure, I can help with that." If a cap is being hit by preamble rather than content, fix the prompt before raising the cap, because a higher cap just makes expensive preamble longer. Prompt-level fixes belong in the same review as the cap; structured prompts that state the required format and length produce more predictable lengths, which in turn make the cap easier to set (Azure RAG prompt guidance).

Finally, confirm your changes against a held-out sample, not the same logs you tuned on. Split the logs by time, set the cap from the earlier period, and check the truncation rate on the later period. That step is what tells you the cap is right rather than fitted.