What a Model Sees: Assembling Context from Chunks

Here is an assembly order that works for most RAG and agent systems. It is not the only correct order, but each choice has a reason.

1. System message and task instructions. Stable, short, cacheable. This carries the role, the grounding constraint ("answer only from the context below"), the refusal behavior when the context is insufficient, the output format and the citation requirement (Azure RAG prompt engineering).
2. Delimited context block. The chunks, each wrapped in a stable envelope that carries an identifier, a source label and a token count.
3. Conversation history (or a summary of it). Placed after the evidence so that recent turns do not push the evidence around when history grows.
4. The question, last. The final tokens before generation, which is also where attention is sharpest.

The static prefix first, dynamic content last, is also the layout that lets a provider cache the repetitive part of the prompt and bill the cached tokens cheaply on later turns (Firecrawl on token optimization). That is a secondary benefit; the primary one is that the model reads the rules before it reads the evidence.

Here is a minimal assembler that enforces the order and the budget:

def assemble(instructions, chunks, history, question,
             budget=6000, reserve_out=800):
    room = budget - reserve_out
    room -= count(instructions) + count(question)
    kept = []
    for c in chunks:                      # already ranked best-first
        header = f"[{c.id}] source={c.source} tokens={count(c.text)}\n"
        need = count(header) + count(c.text) + count("\n\n")
        if need > room:
            text = trim_to(c.text, room - count(header) - count("\n\n"))
            if count(text) < 40:          # too little to be useful
                break
            kept.append(header + text + "\n[...truncated]")
            break
        kept.append(header + c.text + "\n\n")
        room -= need
    hist = fit_history(history, room)     # summary, then recent turns
    return "\n\n".join([
        instructions,
        "--- CONTEXT ---",
        "\n".join(kept),
        "--- END CONTEXT ---",
        "--- HISTORY ---",
        hist,
        "--- QUESTION ---",
        question,
    ])

Two details matter more than the code style. The assembler trims inside a chunk only when the chunk would otherwise be dropped entirely, and it marks the cut. And it refuses to keep a fragment shorter than some floor, because a 40-token stub of a paragraph mostly adds noise.

Suppose a model window of 8,000 tokens and a reserved output allowance of 800, leaving 7,200 for input. Instructions run 500 tokens, the question 150, leaving 6,550 for history and chunks. A history window of 1,200 tokens (a summary plus the last few turns) leaves 5,350 for evidence.

Now the retriever returns eight chunks at roughly 900 tokens each, 7,200 tokens of evidence against a 5,350 budget. Something gets dropped, and the assembly code decides what.

The recommended rule for most systems is the first: keep chunks whole, drop or defer whole chunks from the bottom of the ranking, and only truncate when a chunk alone exceeds the remaining budget. That last case is real. A single table or a long paragraph can exceed the evidence budget on its own, and a whole-chunk policy then sends nothing, which is worse than sending a marked fragment.

Grading the actual assembly, not just the retrieval, is the way to confirm the rule works. Chunking choices already shift retrieval quality measurably: a large cross-domain study of chunking strategies found paragraph-group chunking at a mean nDCG@5 near 0.459 and Hit@5 around 59%, against nDCG@5 below 0.244 for naive fixed-size character splitting (chunking study on arXiv). Those metrics describe what retrieval hands you. To test what the model received, compare answers generated from the assembled prompt against answers generated from a hand-built ideal context, using the same golden set of queries and labels (Qdrant on retrieval relevance).

The ranking your retriever produces is not automatically the ranking the model should read. Two adjustments are worth making before assembly.

First, fuse lexical and dense rankings properly. BM25 and cosine scores are not on the same scale, and weighted averaging lets one dominate; rank-based fusion sidesteps the problem by combining positions rather than scores, with the reciprocal rank formula scoring each document as the sum of 1/(k + rank) across rankers and a constant k around 60 (OpenSearch on RRF, Abhishek Thakur on reciprocal rank fusion). Documents that both methods rank near the top float up, which is exactly what you want in the top slots of the assembled context.

Second, consider placing the best chunk first and the second-best chunk last among the evidence blocks, with the weaker ones in the middle. This is a direct application of the lost-in-the-middle finding (Weaviate on chunking). It costs nothing and helps when the answer requires combining the top two results.

A reranker belongs to the same stage. Cross-encoders score each query-document pair jointly and cannot scan millions of documents, so the correct architecture is first-stage retrieval of a shortlist, then cross-encoder reranking of that shortlist before assembly (Digital Applied on hybrid search). Whatever order the reranker emits is the order the assembler should consume, best first.

Overflow handling needs a written precedence order, because the default behavior of most truncation utilities is to slice a string at a byte or token offset with no regard for structure. That default is the source of the half-sentence failure: a chunk ending in "the required approval authority is" is worse than no chunk at all, because a model will often complete the thought from its own priors.

A workable precedence order, highest priority first:

1. Instructions and the grounding constraint. Never dropped. If the evidence plus instructions do not fit, shrink the evidence.
2. The question. Never dropped.
3. Top-ranked chunks. Kept whole, in ranking order.
4. Recent conversation turns. Summarized rather than dropped, so the thread survives.
5. Lower-ranked chunks. Dropped whole, lowest rank first.
6. Older history. Compressed to a running summary.

Putting the instruction block last in the priority order, so it can be evicted, is the failure mode worth naming explicitly. If your code fills the window with retrieved text first and appends instructions afterward "if there is room", a long retrieval will silently remove the rule that told the model to cite sources or refuse when the context is insufficient. The model then answers fluently, from memory, with no citations. Nothing errors. That is the expensive bug.

Compression is the pressure valve. Summarizing old turns and condensing low-value chunks keeps more of the relevant material inside the budget than dropping chunks outright, at the cost of an extra model call and some fidelity (Firecrawl on token optimization). It is usually worth it for history and rarely worth it for the top-ranked evidence, which should reach the model verbatim.

Most assembly failures are quiet. Nothing throws, the latency looks normal, and the answers are subtly wrong. These are the patterns to test for.

The version-conflict case deserves a rule of its own. A policy changed in one document and unchanged in a second will both retrieve well, and keyword-vs-vector fusion will happily return both, since BM25 matches rare identifiers precisely while dense search matches paraphrase and each method has a systematic blind spot (pgEdge on hybrid search). If the assembler does not dedupe by source identity and prefer the current version, the model receives a contradiction and resolves it arbitrarily.

Evaluation closes the loop. Retrieval metrics measure the list; assembly failures show up in the answer. Build a golden set of queries and expected documents, then compare pipeline output against it, and inspect failures at the query level rather than relying on a single aggregate score (Unstructured on search quality). When an answer is wrong, log the assembled prompt and read it. Nine times out of ten the answer to "why did the model say that" is visible in the tokens you actually sent.

Run through these before treating the assembly layer as done.

• The instruction block is assembled outside the token budget and can never be evicted.
• Every chunk carries an identifier and a source label in a stable envelope, so the model can cite and you can trace.
• Chunks are kept whole; truncation happens only when one chunk exceeds the remaining budget, and the cut is marked and made on a sentence boundary.
• The precedence order for overflow is written in code, not implied by string order.
• History is capped at a fixed share of the window and summarized rather than dropped.
• Duplicate and stale versions are removed before assembly.
• You log the final assembled prompt for every request, sampled if volume demands it.
• You have a test that deliberately overflows the budget and checks that the instructions and question survive.

The two metrics that matter most for the last two items are straightforward. Precision@K measures how many of the top K results are relevant, and recall@K measures what share of all relevant items appear in the top K (Evidently AI on precision and recall at K); standard IR evaluation rests on exactly this pairing of a test collection, a query set and relevance judgments (Manning, Raghavan and Schütze, Chapter 8). Use them to size the budget: if recall@5 is high but recall@8 adds little, a five-chunk evidence budget loses almost nothing. If recall keeps climbing at rank 8 and 9, either raise the budget or rerank so the extra evidence earns a top slot.

One more discipline: measure the assembled prompt, not the ideal one. Synthetic query sets tend to be easier than real traffic and inflate offline scores (Qdrant on retrieval relevance), and the same bias applies to hand-picked context in assembly tests. Sample real requests, replay them through the assembler, and compare.

If you would rather not hand-roll the per-provider token counting and window arithmetic across models, Quanta Inference exposes text, image, video, audio and embedding models through one OpenAI-compatible API key prepaid in US dollars, which removes one variable from the assembly problem: https://quantaintelligence.ai/inference.