Chunking and Overlap: Passage Size and Retrieval

Precision and recall are the two basic measures for any retrieval system. Recall is the share of relevant items in the corpus that the system actually returned; precision is the share of returned items that are relevant (Evidently AI, Creighton University). For retrieval augmented generation, recall@k is usually the metric that decides answer quality: if the passage holding the answer is not in the top k, the generator cannot answer correctly no matter how good the model is.

Chunking moves recall more than almost any other knob once your query set is fixed, because it changes how many "items" exist and how well each one matches. The Fraunhofer study on chunk size found recall@k varies substantially with chunk size across datasets, with smaller chunks (64 to 128 tokens) winning on concise fact-based answers and larger chunks (512 to 1024) winning where broad context is needed. The same study found different embedding models react differently to the same chunk size, so the right number is partly a property of your model, not a universal constant.

NVIDIA's chunking experiments point the same direction: extreme sizes at both ends underperformed medium ones, and the best size depended on query type. Factoid queries favored 256 to 512 token chunks while complex analytical queries did better with larger or page-level units (NVIDIA). The practical read: there is no single best chunk size, but there is a measurable best size for your corpus and your questions.

Overlap exists to protect against the obvious hazard of splitting. A fixed-size splitter cuts wherever the token count runs out, which can slice an answer in half, for example a procedure where step three lands in one chunk and step four in the next (Meilisearch). Overlap repeats the tail of one chunk at the head of the next so a fact spanning the boundary appears whole in at least one chunk. It costs storage and index size, not answer quality.

Take a support corpus of 40 articles averaging 800 words each, roughly 1,060 tokens per article at 1.33 tokens per word. Build two indexes from the same text.

Scheme A, whole articles: 40 chunks, one per article. Each embedding covers all topics in the article.

Scheme B, 200-token chunks with 50-token overlap: each article yields roughly 7 chunks (step of 150 tokens across 1,060 tokens), so about 280 chunks total. Each chunk covers one or two sections.

Now write 25 real questions from support tickets or search logs, and for each one mark the answer span: the exact sentence or paragraph that answers it, identified by article and character offset. This is the only manual work, and it is what makes the comparison honest.

To score, map the answer span back to chunk IDs under each scheme. Under Scheme A, a chunk counts as a hit if it contains the span. Under Scheme B, same rule. Then compute recall@10 as the share of the 25 queries where at least one hit chunk appeared in the top 10 results.

The 0.60 versus 0.88 gap is the kind of movement you should expect. It is not a guarantee for your corpus: if your articles are short and single-topic, whole-article retrieval may already score well and chunking buys little. The Bhat et al. analysis is a useful reminder that the answer depends on how concentrated the answer is within the document.

Two details make the result trustworthy. First, use the same embedding model for both schemes; changing the model mid-test confounds chunk size with model sensitivity. Second, keep k fixed at the number of chunks you actually pass to the generator. If you feed three chunks into the prompt, score recall@3, not recall@10, because a hit at rank 7 never reaches the model.

The test is a loop: split, embed, index, query, score. Here is the skeleton in Python, using any embedding function you already have.

def fixed_chunks(text, size=200, overlap=50):
    tokens = text.split()  # substitute a real tokenizer
    step = size - overlap
    return [" ".join(tokens[i:i + size]) for i in range(0, len(tokens), step)]

def recall_at_k(queries, spans, chunks, embed, k=10):
    vectors = [embed(c) for c in chunks]
    hits = 0
    for q in queries:
        qv = embed(q["text"])
        ranked = sorted(range(len(chunks)),
                        key=lambda i: cosine(qv, vectors[i]),
                        reverse=True)[:k]
        if any(spans[q["id"]] in chunks[i] for i in ranked):
            hits += 1
    return hits / len(queries)

Three practical rules when you run this.

Split on tokens, not words, once you get to production. A 200-token chunk with a real tokenizer is more consistent than a 200-word chunk. The code above uses whitespace splitting only to keep the example readable.

Carry the article title into every chunk. Zendesk's documentation notes the generator uses the article title alongside the matching chunks (Zendesk help), and prepending the title to each chunk also helps the embedding orient a short passage.

Test one variable at a time. Compare whole versus chunked first. Once chunked wins, test 150, 200, and 300 tokens at a fixed overlap ratio, then fix the size and test 0, 25, and 50 token overlap. The Firecrawl comparison recommends recursive splitting near 400 to 512 tokens with 10 to 20 percent overlap as a general default, which is a reasonable starting point if you have no query set yet.

If you do not have the time to run a full comparison, use this rule instead of a fixed number. The table walks through it.

The decision rule in one line: estimate how many tokens the typical answer needs, then set chunk size at roughly that number, and set overlap at one quarter of it. If the answer is one sentence, you want chunks near 200 tokens; if it is a paragraph of setup followed by a conclusion, you want 500 or more. Overlap at a quarter of chunk size means any fact shorter than that quarter appears intact in at least one chunk.

Write your content so chunking helps. Zendesk's optimization guide asks authors to keep one topic per article, use headings and lists, avoid vague pronouns, and include every step in sequence (Zendesk help center). A well-written article with clear section headings and short paragraphs chunks itself cleanly even with a naive splitter.

Chunking sits upstream of everything, so errors surface as bad answers that look like model problems. Three checks catch most of them.

Look at ten random chunks. If any chunk starts mid-sentence with no subject, or covers two unrelated topics, the splitter is wrong for your content. The Meilisearch guide calls this context preservation and recommends inspecting samples rather than trusting a metric alone (Meilisearch).

Check that no chunk exceeds the embedding model's context window. Tokens beyond the limit are truncated or dropped before vectorization, which silently strips context from the stored vector (Pinecone).

Watch the cost side. More chunks means more vectors, more storage, and more candidates scanned at query time, but fewer tokens per prompt if you pass three short chunks instead of one long one. Amazon Bedrock's Knowledge Bases documentation treats chunking as a managed retrieval setting with tunable size and overlap precisely because the right answer depends on the data (AWS docs).

If your corpus is genuinely mixed and you cannot settle on one size, build two indexes at different sizes and merge their results, or rerank the top 20 candidates from the smaller chunks with a cross-encoder. Both add latency and cost, so only do it after the single-scheme comparison shows recall is still short of your target.

Reliable chunking also requires the same embedding model for index and query. Run both schemes through one API so you are comparing split strategies, not model vendors. Quanta Inference serves text and embedding models through one OpenAI-compatible key, which keeps that variable fixed while you tune chunk size at https://quantaintelligence.ai/inference.

If you have no labeled questions, generate them. Have a model read each article, produce five questions answerable only from that article, and record the source span. This yields a synthetic query set aligned to your real content. Zendesk notes that replies are generated from the title plus the best matching chunks, so write questions the way real users type them, not as paraphrases of headings (Zendesk documentation).

Synthetic sets have a known bias: generated questions tend to reuse the article's own vocabulary, which flatters small chunks that also contain that vocabulary. Mix in at least ten questions from real support tickets or search logs to counter it. The recall@10 number you get from the blended set will be pessimistic compared with synthetic-only and much closer to production.

Measure precision@k alongside recall so you do not trade away answer quality for hits. A scheme that wins recall but returns five marginally relevant chunks per query will produce rambling answers, because the generator treats all retrieved chunks as context.

When the corpus and query set grow past what you can maintain by hand, a dedicated expert can run the comparison and hand back the tuned configuration. Quanta Experts work inside your context and return finished deliverables at https://quantaintelligence.ai/agents.