A Downgrade Path When a Model Overruns Its Output Budget

Length estimates fail because models are poor at tracking their own output length. A study of dynamic length feedback found that while language models measure short texts accurately, that ability declines significantly as text length grows, and the authors propose interrupting generation to feed the running count back to the model as a correction (the length tracking study). In other words, the model does not know it is about to overshoot, so asking it to stay within 800 tokens is a soft signal rather than a guarantee.

The same effect shows up in reasoning tasks, where chain-of-thought output can balloon well past what the question needs. The token-budget work on reasoning finds that the reasoning process is often unnecessarily long and can be compressed by stating a budget, but the choice of budget is critical to whether compression helps or hurts (token-budget-aware reasoning). A budget that is too tight produces a worse answer than no budget at all, which is exactly why a post-call decision is safer than a pre-call guess.

There is a second structural reason estimates miss: the context window is shared between input and output. If the model's window is generous but your prompt, history, and few-shot examples eat most of it, the remaining room for the answer is smaller than the model's headline maximum output. The context window guide spells this out with model-level numbers and shows how to split the total window across system prompt, examples, and user input before spending any of it on output (the context window guide). Budget the whole window, then treat the output share as an estimate you are prepared to revisit.

Each path has a distinct cost and quality profile. The table below is a starting decision aid, not a law; calibrate the thresholds with your own traffic.

The CrewAI community thread on this exact problem is worth reading because it describes the failure honestly: a user hits a model's output ceiling on content that cannot be summarized and asks whether re-prompting from the last lines to continue is a hack (the CrewAI discussion). It is a reasonable hack for prose. It is a poor idea for JSON, where the continuation has to thread back into an open object and most models will re-open or re-close it wrongly.

Worked example: a retry that is worth it

Suppose a call is allowed 1,200 output tokens and comes back at 1,240 with a finish reason of length. Assume the text ends after a complete paragraph and the remaining sentence is filler. Here the excess is 40 tokens, about 3 percent. A retry with the same 1,200 cap and an instruction to be more concise will almost certainly land under the cap, because the model was already close. The cost is one extra call at the same price per output token, and the acceptance criterion is simple: the retried answer parses and contains every required heading. If it fails that test, do not retry a third time.

Worked example: a fallback that pays for itself

Suppose a call is capped at 800 output tokens but the task is a comparison table with five rows and the model needs roughly 2,000 to finish. Retrying at 800 cannot succeed, because the answer is structurally larger than the cap. The right move is a fallback to a tier with a 4,000-token ceiling, or to a model whose output price is low enough that a longer answer still fits the original envelope. That second calculation is where per-token pricing matters. If the primary model charges a high rate for output and the fallback charges a fraction of it, a 2,000-token answer on the cheaper model can cost less in total than a 1,200-token answer on the primary one. Check the published per-token rates for your models, as in the OpenAI pricing page, and compare total cost rather than price per token.

That is the insight most first-pass implementations miss: a downgrade tier is not always cheaper per answer. It is cheaper per unit of work done. A long answer on a cheap model can undercut a truncated answer on an expensive one, and a short answer on the expensive model can undercut a long answer on the cheap one. Decide using total cost for the completed task, not the headline rate.

A retry is only worth spending when it carries new information. The Hermes agent issue tracker documents the failure mode precisely: the first retry re-runs the identical request with no additional context, so it almost always truncates again for the same reason, and the agent then gives up (the truncated tool call issue). The proposed fix is a second retry that injects a hint asking the model to use concise arguments or split large content into smaller writes.

That is the right shape for a retry policy:

attempt 1: original request, original cap
  if finish_reason == stop and schema valid: return
attempt 2: same cap, plus hint
  "Your previous answer was cut at the limit.
   Produce the same content in under N tokens.
   Drop examples, keep required fields."
  if finish_reason == stop and schema valid: return
else: escalate to fallback or refuse

Three rules keep this bounded. First, cap the number of retries at one, or two with a hint. Unbounded retries on the same prompt burn spend with no new signal. Second, never retry with a larger cap and call it a retry; that is a fallback with the same model, and it should be priced as such. Third, require schema validation on the retry, so a shorter but malformed answer does not slip through. The function calling guide recommends exactly this: wrap parsing in try-catch, validate against the original schema before execution, and return structured errors that let the model self-correct on retry (the function calling overview).

Hard truncation has one quality risk worth naming. If the model honors the shorter cap by dropping content rather than compressing it, the answer looks complete but is missing a required section. A word-count check on the retried answer plus a required-fields check catches most of this. If either fails, treat it as a refusal case, not a success.

Fallback is the answer when the correct response simply does not fit the cap. The operational pattern is well established: define triggers, keep a diverse set of backends, bound retries, and monitor every route so you know how often the fallback fires and whether it recovers (the LLM fallback guide).

An overrun is a fallback trigger that most gateways do not model. The usual triggers are errors, timeouts, and rate limits. Adding "finish reason is length and payload invalid" to that list is a small configuration change with a large reliability payoff, and it lets the gateway own the decision instead of every caller reimplementing it.

Cascades are the formal name for this pattern. A cascade sends requests to a cheap model first and escalates to a stronger, more expensive model when the cheap answer fails a quality check. The routing literature notes that frontier models can cost roughly 10 to 50 times more per token than a well-tuned smaller model, and that routing the majority of traffic to the cheaper tier while preserving quality can cut inference spend by 60 to 70 percent (the model routing chapter). An overrun-driven downgrade is a cascade run in reverse: the call started high and moves to a tier that can finish the job within budget.

Two decision rules make this concrete:

• Escalate on necessity. If the task needs the full answer and the current tier cannot produce it within the cap, fall back to a tier with a larger ceiling. Do not negotiate.
• Downgrade on cost. If the task tolerates a longer answer but the current tier's output price makes that answer expensive, fall back to a cheaper model with a larger output ceiling and accept the quality difference if your evals show it is small.

Give the fallback a hard cap of its own, and record the reason for the route change with a marker such as the cause tags used in adaptive routers (the LiteLLM auto-routing docs). Without a recorded cause, you cannot tell a healthy fallback rate from a router that is misconfigured.

Refusal is the least popular path and often the correct one. It applies when the partial answer is dangerous to act on, when no fallback tier is available or acceptable, or when the task is deterministic and a shorter answer would be silently wrong.

A good refusal is not an empty 500. It returns three things: the partial output, a stable error code, and enough metadata for the caller to decide. A workable shape:

{
  "error": "output_budget_exceeded",
  "partial": "...text produced before the cut...",
  "detail": {
    "cap_tokens": 1200,
    "used_tokens": 1200,
    "finish_reason": "length",
    "schema_valid": false,
    "suggested_downgrade": "tier-b",
    "retryable": true
  }
}

The caller then chooses: show the partial answer to a human, re-request with a larger cap, or abandon. This is the same principle behind bounded retries and recovery paths in production orchestration guidance, which stresses that retries need limits, idempotency awareness, and an explicit recovery path (the orchestration guide). A refusal that hands back the previous attempt's state is a recovery path; a refusal that discards it is a dead end.

Refusal is also the right default for anything a downstream system will execute. If a truncated tool call would have completed a write, a delete, or a payment, returning partial arguments as if they were whole is an incident waiting to happen. Treat schema invalidity on a tool call as non-negotiable: refuse, log, and let the agent retry with a hint or escalate to a tier with a larger cap.

The cleanest place for this decision is the gateway layer, because the gateway already sees the finish reason, the token counts, and the price of each model. Centralizing it means every service inherits the same policy, and the observability comes for free.

A gateway can route, retry, and fall back across providers, and the operational guidance for that layer recommends tracking fallback frequency, recovery success, latency, and cost per route (the LLM fallback guide). Extend that dashboard with one more metric: overruns per model per day. That single number tells you whether the caps are too tight, whether a model has started to ramble, or whether a prompt change quietly increased verbosity.

Keep the policy in configuration rather than code. Rules like "if finish reason is length and schema invalid, route to tier B once, then refuse" are readable by anyone on the team and easy to change when a model's behavior shifts. The signal-driven routing paper makes the same case from first principles: opposite deployment constraints should be expressed as different configurations over the same architecture, not as different code paths (the vLLM semantic router paper).

If you are wiring this together across several providers, one API key for every model family removes a full layer of per-provider handling and keeps the overrun policy in one place: Quanta Inference exposes text, image, video, audio and embedding models behind an OpenAI-compatible endpoint, priced in US dollars, so the fallback table does not need a different client for every tier.

The table earlier gives qualitative rules; production needs numbers. Calibrate with four measurements per task type:

1. Cut frequency. The share of calls that finish on a length reason. If it is above a few percent for a given task, the cap is probably too tight for that task.
2. Cut severity. The median number of tokens left unproduced, estimated by re-running a sample without a cap. Small excesses favor retry; large ones favor fallback or refusal.
3. Retry success rate. The share of capped retries that produce a valid answer under the cap. Below roughly half, retries are mostly wasted spend and you should go straight to fallback.
4. Fallback quality delta. How often the fallback answer passes your evals compared with the primary tier. If the delta is negligible, make the cheaper tier the default for that task and skip the overrun entirely.

Run this for a week of traffic, then revisit quarterly. Thresholds drift as prompts change, as models are updated, and as traffic mixes shift toward harder or easier requests. The point of the policy is not to find the perfect number once; it is to make the overrun a decision you have already designed, so the event that used to be an outage becomes a logged route change with a known cost.