When a Candidate Model Fails on Long Inputs

You cannot tell a limit problem from a quality problem without knowing how many tokens you actually sent. Count the whole prompt, not just the document: system instructions, retrieved context, few-shot examples, tool schemas, and the user turn all consume the same budget.

A quick preflight check saves hours of guessing:

def preflight(prompt_parts, stated_limit, reserve=512):
    total = sum(count_tokens(p) for p in prompt_parts)
    if total > stated_limit:
        return f"hard limit: {total} tokens vs {stated_limit}"
    if total > stated_limit - reserve:
        return f"near limit: {total} tokens, only {stated_limit - total} left for output"
    return f"within limit: {total} tokens"

Log that number on every request. Input length belongs in your structured logs next to prompt version, model ID, and downstream validation result, because long prompts and edge cases are exactly where failure clusters hide (Yotta Labs' shadow testing guide). When a candidate starts failing, the first question is whether the failing requests are the long ones, and the log answers it without a single new experiment.

Note the subtleties that make manual counting wrong. Tokenizers differ between models, so a 10,000-token prompt for one model can be 11,500 for another. Reserved output budget counts against the same window. And a provider that silently truncates gives you no count at all, so the preflight check is your only warning.

The cleanest experiment for a suspected hard limit is to make the input fit without changing the task. Chunk the document, run each chunk through the same prompt and evaluation, and see whether the outputs are acceptable.

Overlap matters. If you split without overlap at a fixed sentence boundary, an answer that straddles the boundary becomes unanswerable in both chunks. Recursive character splitting at roughly 400 to 512 tokens with 10 to 20 percent overlap is a sound default across document types (Firecrawl's chunking comparison). The overlap is the seam allowance; without it, you will blame the model for a cut you made.

A worked example makes the arithmetic concrete. Suppose your evaluation prompt runs 18,000 tokens and the candidate model's window is 16,000, with 2,000 reserved for output. You are over by 4,000. Split the source document into four chunks of about 4,200 tokens each with 600 tokens of overlap, which is about 14 percent. Each call now runs roughly 4,200 plus 600 plus your 1,200-token instruction block, or about 6,000 tokens, comfortably inside the window with room for the response.

Then replay and read the results as a diagnosis:

• All chunks pass: the failure was a hard limit. Fix the pipeline by chunking, not by switching models.
• Some chunks pass, some fail: you have a quality problem localized to specific content, plus possibly a chunking design problem. Look at what the failing chunks contain.
• All chunks fail the same way: it was never a length problem. The task, the prompt, or the evaluation criteria are wrong.

That last outcome is the most valuable one, and it is the one teams skip. Replaying the same evaluation against split input is the same discipline as shadow testing: send the same production-like input to the candidate, log its output separately, and compare against explicit criteria you set before the run (CodeAnt's shadow traffic guide). The candidate output stays out of the user path while you decide.

The split test cannot distinguish "too long for the window" from "too long for the model's attention," because both reduce input length. To separate them, hold the structure constant and cut to a known safe size.

Keep the beginning and the end of the prompt. Research on long-context degradation consistently finds that models weight information at the start and end of a long context more heavily than material buried in the middle, so truncating from the middle is a fairer test of the remaining prompt than cutting the tail (Chroma's context rot study).

Then re-run the same evaluation on the truncated prompt:

• Truncated prompt passes, full prompt fails: the failure tracks length, not content. You have a soft limit on effective context, and the fix is chunking or retrieval rather than a bigger model.
• Truncated prompt fails on the same step as the full prompt: the model cannot do this reasoning step with this prompt regardless of length. The problem is quality and it will follow you to any model with any window.

Chroma's controlled experiments are relevant here because they isolate input length while holding task complexity constant, and they find performance degrading as input grows even on deliberately simple tasks (the context rot report). In other words, a clean 200 response at 10,000 tokens proves nothing about quality at 10,000 tokens. The commonly used Needle in a Haystack benchmark measures direct lexical retrieval over long inputs and can overstate how well a model reasons across a long document (Chroma's related work summary). Near-perfect retrieval scores are not evidence that your multi-step extraction task will hold up.

If splitting fixed the hard limit but quality is still uneven across chunks, the chunking strategy is now the variable. Change one thing at a time, because a mostly-passing result with slightly different chunk boundaries tells you little.

Three variations are worth running:

1. Overlap only. Same chunk size, larger overlap. If quality improves, you were cutting through answers.
2. Boundary only. Same size, but split on structural boundaries (headings, sections, speaker turns) instead of character count. Meaningful chunks outperform arbitrary ones because each chunk contains a complete idea, which is also the rule of thumb for whether a chunk will make sense to a language model (Pinecone's chunking guide).
3. Order only. Same chunks, shuffled or grouped differently. Order changes are the weakest test for chunking and the strongest for leakage: if results change when you reorder, your chunks are not independent and something is crossing between them.

The same logic applies beyond plain text. The record-and-replay pattern, capturing real requests once and replaying them as fixed test cases, is what makes these comparisons reproducible at all (Keploy's record and replay guide). Replay the identical stored requests against each variant, or you are comparing different inputs and calling it a comparison.

The experiments above are worth almost nothing if you run them once. A prompt that works can regress when the provider updates a model, and LLM APIs do change silently and get deprecated on schedules you do not control, which is why prompt performance needs regression testing the way any API does (the prompt regression testing paper).

Build the replay into a routine:

• Store a fixed set of long-input requests with their expected outputs or acceptance criteria.
• Run them against every candidate model and every prompt change.
• Assert on task-level outcomes (did it extract the right field, did it cite the right clause), not on exact string match, because outputs are non-deterministic (DevOpsBoys on LLM testing in production).
• Alert on the token count of failing requests, not just on failure.

Treat this like any other migration QA: baseline first, verify at the point of change, monitor after. The lesson from web migration QA is that errors accumulate across a timeline rather than appearing at a single launch gate, so validation has to run continuously rather than once (Siteimprove's migration QA framework). Long-input quality regressions behave the same way.

A candidate model rarely fails alone. Teams evaluate it against a current production model, often as a replacement for one on a deprecation timetable, and provider deprecation schedules are published far in advance but still catch organizations without a centralized model inventory (Airia's deprecation guide). The same model can even carry different retirement dates on different platforms, so the candidate's lifecycle is part of the evaluation, not an afterthought (the cross-vendor lifecycle calendar).

When the candidate fails on long inputs, your replay rig should answer three questions at once: does it fail on the same inputs the current model passes, does it fail because of length or content, and does the failure survive the truncation test. If it survives truncation, the candidate is not the problem, and swapping providers will not help. If it does not survive, you have a chunking decision, and you should make it once and document it.

This is also where mixed workloads matter. If you are comparing candidates across several providers or self-hosted models, routing each long-input workload through a single endpoint that exposes text, image, video, audio, and embedding models behind one OpenAI-compatible key keeps the replay rig stable while candidates change underneath it. That is the practical role of Quanta Inference: one interface, so the evaluation harness does not need rewriting every time a candidate does.

Run through this in order and you will usually finish in under an hour:

1. Count the tokens of the failing prompt, including system, context, examples, and reserved output.
2. Compare against the model's stated maximum and confirm whether the provider truncates silently.
3. Split into overlapping chunks and replay the same evaluation.
4. Truncate to a known safe size, preserving the start and end, and replay.
5. If the problem persists, re-chunk on structural boundaries and re-run.
6. Record the passing configuration as a regression test and log token counts from then on.

If the failure survives steps 3 through 5, it is not a length problem at all. It is a task, prompt, or evaluation problem wearing a length problem's clothes, and no amount of extra context will fix it.