When a Provider Lowers the Context Limit: A Migration Checklist

Start from traffic, not from source code. You need a list of features, prompts, and routes, with an owner and a maximum observed input size for each. If you cannot answer "which features send more than the new maximum, and who owns them" within a day, the inventory is the first deliverable.

Two columns carry most of the decision. The maximum input tokens matters more than the average: a feature where 99 percent of calls use 2,000 tokens and 1 percent use 40,000 still has a tail that will break. Truncation behavior matters because many teams discover during this inventory that they were already cutting text and never noticed.

Where the inventory asks for token counts, rely on measured usage. Provider responses return usage fields, and OpenAI documents that Chat Completions reports prompt_tokens, completion_tokens, and total_tokens, while the Responses API reports input_tokens, output_tokens, and total_tokens OpenAI's token counting guide. If you only have logs of text, a rough estimate is one token per four characters for English, but the same guide is explicit that this is an estimate and that counts vary by model, encoding, and language.

The context window is a shared budget, so the inventory needs the output side too. Input tokens plus output tokens must fit within the window, which means a feature that requests a long completion loses more input headroom than its prompt alone suggests the Agenta context management guide. Reasoning models add a third consumer: OpenAI notes that reasoning tokens are not visible as answer text but count toward output usage and are billed as output tokens.

Aggregate token totals answer budgeting questions and hide the diagnosis. A monthly total cannot tell you whether a spike came from a growing prompt template, a request sitting near the context ceiling, or an agent step that retries inside a trace, and each needs a different fix the Braintrust token tracking guide.

Build three views, in this order. Per call, log prompt and completion tokens so prompt bloat shows up across template versions. Per request, log context utilization as a percentage of the limit, which is what a limit reduction actually changes. Per step inside agents, log token counts per span, so a retrying tool call or oversized retrieval step does not hide behind a normal-looking top-level trace.

The percentage view turns a limit cut into a prioritized list. If a route routinely runs at 85 percent of the old ceiling, a 50 percent reduction puts it at 170 percent and it fails. If another runs at 20 percent, it survives and needs no work.

For a worked example, take a support assistant. Suppose its p95 input is 30,000 tokens and its maximum is 70,000, and the provider drops the ceiling to 32,000. Neither the average nor the p95 breaches the new limit, but the tail does, and the tail is where long tickets, pasted logs, and multi-turn conversations live. The right response is not to rewrite the prompt. It is to find what fills the tail, which is usually retrieved documents and accumulated history, and cap those two inputs first. That single change often moves a 70,000-token maximum under 32,000 without touching the instruction text.

A second number worth computing is cost per completed task, not cost per token. OpenAI's guidance is blunt: a lower price per million tokens does not necessarily produce a lower total cost, because models tokenize the same text differently and generate different amounts of output, so you should test representative tasks instead of comparing visible response length OpenAI's token counting guide. A chunking strategy that triples the number of calls can raise total cost even when each call is smaller.

Once you know which call paths breach, three remedies exist and they have different costs. Use a rule, not taste.

Shorten. Removing irrelevant context is not just a size fix. Research on prompt bloat reports that LLMs can be distracted by irrelevant context, with even a small amount leading to inconsistent predictions, and that reasoning performance can degrade well below a model's technical maximum, at lengths around 3,000 tokens in the study cited the MLOps Community writeup. A limit reduction is a useful forcing function to delete context you never needed.

Chunk. Chunking is the process of splitting large text into smaller segments, and it matters for two reasons: embedding models have their own context windows that silently drop excess, and chunks must be small enough to embed well and large enough to carry meaning the Pinecone chunking guide. The same guide notes that long-context models still suffer from the lost-in-the-middle problem, where relevant information buried inside a long document is missed even when it is included.

A practical default appears in comparisons of chunking approaches: recursive character splitting at 400 to 512 tokens with 10 to 20 percent overlap is described as the best default for most use cases, and page-level chunking is reported as the highest-accuracy method in NVIDIA's 2024 benchmarks but only for paginated documents the Firecrawl chunking comparison. Qdrant's course material makes the same point from the retrieval side: whole documents produce a semantic average that a specific query cannot match, and content past a model's limit is simply dropped the Qdrant chunking lesson.

Re-route. If only a thin tail breaches, send that tail somewhere with more room and leave the common path alone. This is also the honest answer when a feature genuinely needs the full document in one call and chunking would degrade it.

Shortening has a trap. Naive truncation can remove the information the model needs, which reduces accuracy and can introduce hallucinations the Agenta guide. Prefer dropping whole retrieved documents, or the oldest conversation turns, over cutting inside a document.

Most production systems already contain ad hoc truncation, added by whoever hit a limit first. A limit reduction is when those rules get centralized, because the alternative is five teams inventing five behaviors.

Make truncation a policy with three parts. Set a budget per input section, for example a fixed share for retrieved context and a fixed share for history, computed against the new ceiling with the output allowance subtracted. Define a deterministic order of removal: drop the oldest history first, then the lowest-ranked retrieved documents, and never cut the system instructions or the user's current message. Emit a signal, a counter or log line tagged by feature, every time truncation fires, so post-migration monitoring has something to read.

Then define fallbacks for requests that still exceed the limit. Three are worth supporting: retry with a shortened input, re-route to a model or endpoint with a larger limit, and an explicit user-facing error that says the input was too long. The error path is better than a silent truncation that returns a confident wrong answer.

def fit_budget(system, history, docs, question, ceiling, output_reserve):
    budget = ceiling - output_reserve
    parts = [("system", system), ("question", question)]
    for label, items in (("doc", rank(docs)), ("history", reversed(history))):
        for item in items:
            if total_tokens(parts) + item.tokens <= budget:
                parts.append((label, item.text))
            else:
                log_truncation(label, item.id)
    return assemble(parts)

Two details make this worth writing down. Compute tokens with the real tokenizer for the target model rather than a character count, since the estimate can be off. And count the structural tokens of the request itself: OpenAI notes that message structure, tools, schemas, images, and files all affect the full input count, and that its input-token counting API includes formatting tokens for roles and boundaries OpenAI's token counting guide.

A limit change touches prompts, and prompt behavior is brittle. A case study on evolving LLM APIs found that an update to the underlying model caused a major performance downgrade on a toxicity classification task and also changed which prompt design performed better, with one prompt outperforming another by 8.7 percent accuracy after the update the regression testing paper. The paper argues that regression testing for LLMs requires changes to traditional approaches because of different correctness notions, prompt brittleness, and non-determinism.

Run three layers, cheapest first.

Record and replay. Capturing real requests and responses and replaying them after a change is an established way to detect unintended behavior differences, and record-replay is specifically recommended for migration and rewrite validation, where the new implementation must produce comparable responses the Keploy record and replay guide. Replay your longest and most representative requests, because those are what the migration changes. The mechanics are simple: intercept, store request and response pairs, then replay against the modified pipeline a record-and-replay walkthrough.

Shadow traffic. Shadow testing duplicates live or replayed traffic to a candidate configuration while the current one stays in the user response path, logging the candidate output separately so you can compare quality, latency, errors, token usage, cost, and format adherence the Yotta Labs shadow testing guide. Two cautions from the same source: shadow testing is not proof of production readiness, and a failed candidate call must not delay or change the user response.

A/B or canary rollout. Once shadow results look acceptable, split traffic and measure user behavior, not just model outputs. A comparison of the two frames the distinction: shadow testing is observational with no user impact, while A/B testing exposes real users and captures satisfaction and task completion signals that shadow traffic cannot the CodeAnt guide.

Timing matters too. A three-stage framework of pre-launch baselining, launch-day verification, and post-launch monitoring argues that a single launch gate is blind to migration errors because those errors accumulate across the timeline rather than at one moment the Siteimprove migration QA framework. Capture a baseline of token distribution and answer quality before the switch, verify at the switch, and keep watching afterwards.

The migration is not done when requests stop erroring. A request that fits within a reduced limit can still produce worse output, because the relationship between input length and quality is not flat. Chroma's evaluation across 18 models found that performance degrades as input length increases even on deliberately simple, controlled tasks, and that degradation is often non-uniform, with real applications likely more affected than the controlled setting the Chroma context rot research.

The utilization view is the one teams forget. Limits get lowered, prompts get extended again, and later the same class of request sits at the edge of the ceiling. Braintrust recommends expressing utilization as a percentage of the context window precisely so teams can see which tenants, features, or routes operate too close to the limit the Braintrust token tracking guide.

One more habit pays for itself: keep a current inventory of models mapped to features, plus a risk tier per feature, so the next limit change is a lookup rather than a scavenger hunt. The Airia guide recommends exactly this, including pre-selecting a replacement candidate for every production model before a deadline arrives the Airia deprecation guide.

If the migration touches several providers at once, routing each call path through one interface turns the re-route step into a configuration change. Quanta Inference exposes text, image, video, audio and embedding models through one OpenAI-compatible API key, which helps when the fix for a breached limit is to point that feature at a model with more room: Quanta Inference.