Routing by Output Length: A Budget the Model Cannot See

It is tempting to solve this in the prompt: tell the model to "keep it under 400 words" and let it handle the rest. The evidence says that does not hold up. A study of length control found that models can estimate the length of short texts but that the ability degrades significantly as text gets longer, and that prompt-based length instructions are followed imprecisely; the authors built an external feedback loop that interrupts generation and reports the running count back to the model, which improved accuracy without harming quality (can LLMs track their output length). In other words, the model does not know how long its answer is while writing it unless something outside the model tells it.

This is the core problem with treating length as a prompt instruction. The model generates token by token with no counter. Your max_tokens value is not visible to it. When it hits the ceiling, generation stops wherever it happens to be. The only place a length decision can be enforced reliably is your own code: in the router that picks the tier, in the budget you pass with the request, and in the check you run on finish_reason before you parse the response.

That leads to a clean division of labor. The router estimates length before the call and picks a tier. The request carries a hard ceiling with headroom. The response handler checks whether the ceiling was hit and retries or escalates if it was. None of those three steps requires the model to understand anything about tokens.

The estimate does not need to be exact. It needs to be right about the order of magnitude, because that is what separates tiers.

Step 1: Classify the request shape. Sort incoming requests into a small number of shapes: a yes/no or single-value answer, a short explanation, a list of N items, a structured record, a long-form document, or a multi-step agent turn. Shape is usually stated or implied in the prompt itself. "List 10 items" is a shape. "Summarize this" is a shape. "What is the refund window" is a shape.

Step 2: Extract any explicit quantity. If the user gave a number, use it. "Top 20," "five bullet points," "a 500-word post," "one row per order" all convert directly. This is the highest-confidence signal you will get, and it costs nothing to parse with a regular expression before you ever call a model.

Step 3: Apply a per-unit token allowance. Estimate how many tokens one unit of that shape consumes, then multiply. A one-line list item, a JSON object with a fixed schema, a paragraph of prose, and a code snippet each have a different per-unit cost, and your historical logs will give you far better constants than any guess. Start with your own measured median for that shape and adjust quarterly.

Step 4: Add structural overhead. Every answer carries scaffolding: a greeting, a closing sentence, markdown fences, the outer braces of a JSON object, the field names in a schema. Structured outputs make this explicit, because a valid object must include every required field, and schema-constrained decoding spends tokens on keys as well as values (structured output validation). Add a flat allowance per shape.

Step 5: Apply a margin and set the ceiling. Multiply the estimate by a margin, round up to a tier boundary, and pass that as the hard cap. The margin covers the cases where the model explains more than asked, adds a caveat, or formats a list more verbosely than your median suggested.

Here is the whole thing as a small, dependency-free decision function:

import re

# Per-unit token allowances, tuned from your own logs.
UNIT_TOKENS = {"list_item": 35, "bullet": 30, "paragraph": 90, "json_record": 60}
OVERHEAD = {"list": 60, "prose": 80, "json": 120, "answer": 20}
MARGIN = 1.4

def estimate_output_tokens(prompt: str, shape: str) -> int:
    m = re.search(r"\b(\d{1,3})\b", prompt)
    n = int(m.group(1)) if m else 5          # default if no count given
    unit = UNIT_TOKENS.get(shape, 60)
    est = n * unit + OVERHEAD.get(shape, 80)
    if "word" in prompt:                      # roughly 1.3 tokens per word
        w = re.search(r"(\d{2,5})\s*-?\s*word", prompt)
        if w:
            est = int(w.group(1)) * 1.3
    return int(est * MARGIN)

Two things about this function are deliberate. It reads the prompt without calling a model, so it adds sub-millisecond latency and zero cost, which is the same reason heuristic routing exists alongside classifier routing in gateway tooling (LiteLLM auto routing). And it returns a number you can inspect in logs, which matters: a routing decision you cannot explain after the fact is a routing decision you cannot debug, and the same source recommends logging a cause marker per decision. Copy that idea. Record which signal produced the estimate (explicit count, shape default, word target) next to every request.

Suppose a support-tool product sends three kinds of requests, and you route between a cheap tier, a mid tier, and a strong tier. Use the published rates from the pricing page above (OpenAI API pricing): cheap tier at $0.10 input and $0.50 output per million tokens, mid tier at $2.00 and $10.00, strong tier at $10.00 and $50.00. All three requests arrive with roughly 800 input tokens (system prompt, retrieved help-center snippets, and the user question).

Read the second row closely, because it is the insight. That request has a small prompt (800 input tokens) and looks trivial by every input-based measure. But its output is 574 tokens at a tier where output costs five times input: the answer alone is $0.000287, more than the input ($0.00008) by a factor of three and a half. Sent to the strong tier instead, the same request costs a hundred times more for an answer a cheap model can produce, since the task is enumeration rather than reasoning. The first row shows the opposite end: a 28-token estimate means the input dominates and the tier choice barely matters, so the cheapest tier is almost always correct.

Across these three requests the input-based router and the output-aware router agree on rows one and three. They disagree only on the list, and that is where nearly all routable cost usually sits: the middle band of requests with short prompts and medium-length answers. A routing program that classifies only prompt complexity will systematically overpay there or, worse, underprovision the ceiling and truncate.

The row three number also carries a warning. A 2,400-token budget on the strong tier is fine when the user asked for a 1,200-word guide and the work justifies the price. It is not fine when a vague prompt ("tell me about refunds") trips a long-prose shape and sends a $0.13 request down a path that should have cost a fraction of a cent. Watch the distribution of estimates per shape, not just the total bill.

An estimate is only useful if it fails loudly. These are the checks worth building before you trust the numbers in production.

Check the finish reason before parsing. If the response came back with a length-related stop reason, do not parse it. Retry with a larger ceiling, or split the task into smaller calls that each produce a smaller payload. This single check catches silent truncation, which the failure taxonomy calls the most dangerous mode because it looks like success (JSON mode failure modes).

Treat the estimate as a lower bound, not a target. If answers habitually exceed the estimate by 30 percent, your per-unit constants are stale. The literature on structured output pipelines recommends logging raw and parsed outputs and tracking validation failure metrics so regressions surface early rather than at renewal time (structured output observability).

Do not let max_tokens substitute for a budget. A ceiling that is generous enough to fit any answer is not a budget, and a ceiling that is tight enough to be a budget will truncate. Set the ceiling just above the estimate with margin, and let the escalation path handle the misses.

Validate shape after generation, not just before. A schema-valid output can still be wrong in content, and business rules live outside the schema (schemas, not just prose). The same discipline applies to length: check that a "list 10" answer actually contains ten items, because a truncated list is schema-valid JSON.

Track efficiency per request, not per month. Multiple metrics pull in different directions here: tool correctness, input accuracy, task completion, and task efficiency measured in tokens, calls, retries, and dollars should be tracked separately, because a system that eventually gets the right answer by retrying and over-fetching can still be too expensive for production (optimizing tool calling). Output-budget accuracy belongs in that same dashboard.

Put a ceiling on the ceiling. Cap the maximum budget any single request may carry. Long-form and agentic requests can otherwise produce one expensive outlier that wipes out a day of routing savings, and enterprise orchestration guidance is explicit that execution limits are what prevent runaway agent costs (AI agent orchestration).

The estimate fits cleanly into a request path that already has stages. Intent classification comes first, because the shape and any explicit quantity are derived from the same understanding that identifies what the user wants; production systems routinely decompose intent into dimensions like domain, intent, and issue so that each new domain expands only one part of the taxonomy (multi-turn intent classification). The output-length estimate reads the intent, then the router picks a tier, then the call goes out with a ceiling, then the handler checks the finish reason and the validation rules.

That ordering gives you two escalation paths. The cheap one is a same-tier retry with a larger ceiling when the estimate was slightly low. The expensive one is a tier escalation when the answer was cut off or failed validation, which is the cascade pattern: start on the cheapest tier that can plausibly work and only pay for a bigger model when the smaller one demonstrably fails (model routing and cascades). Because you already know the expected output size, you can decide in advance whether a failure is worth escalating: a truncated 30-token answer on the cheap tier is worth a retry anywhere; a truncated 2,400-token guide is worth an escalation only if the user's request clearly demanded the length.

If you are building this across several providers, the plumbing is the boring part and the budget logic is the interesting part. Quanta Inference puts text, image, video, audio, and embedding models behind one OpenAI-compatible API key with prepaid pricing in US dollars, so the tier table in your router maps to one credential rather than one integration per vendor. The routing decision, the estimate function, and the finish-reason checks stay entirely in your code.