Chunking Strategies for Retrieval: Size, Overlap and Boundaries

Size is how much text one vector represents. It is usually expressed in tokens or characters, and it caps both what the embedding can encode and how much room one passage takes in the answer prompt.

Overlap is the text repeated at the end of one chunk and the start of the next. It exists to stop a sentence, a condition or a procedure step from being severed at a seam.

Boundaries are the rules deciding where a cut is allowed. Headings, paragraph ends, sentence ends, code blocks and table edges all carry meaning that a character counter cannot see.

The three interact. A generous size with no boundary rules still mangles tables. Tight boundary rules with no overlap still split a two-sentence qualification. Treat them as one configuration, not three independent dials.

Anthropic's Contextual Retrieval write-up states the conventional starting point plainly: break the corpus into chunks "usually no more than a few hundred tokens" before embedding (Anthropic). That is a starting point, not a rule, as the benchmark section below shows.

Small chunks retrieve precisely and fail semantically. The classic example comes from financial filings: a chunk reading "The company's revenue grew by 3% over the previous quarter" answers nothing on its own, because the company, the quarter and the baseline are in the surrounding text. Anthropic uses exactly this case to explain why traditional RAG loses context (Anthropic).

Three failure modes recur:

• Orphaned subjects. A pronoun or definite article ("it", "the policy") points at a noun that no longer appears in the chunk, so the embedding represents a vague statement.
• Lost dates and versions. A rule, price or limit is stated correctly but without the effective date, so retrieval surfaces the wrong era of the document.
• Split logic. In code or procedures, the guard clause sits in one chunk and the operation in another; neither reads as a complete instruction.

NVIDIA's experiments make the size effect measurable. In their end-to-end RAG accuracy tests across several datasets, chunks of 128 tokens generally underperformed medium sizes, and their KG-RAG dataset scored worst at 128 tokens with 0.421 accuracy (NVIDIA). Tiny chunks are cheap to match and expensive to answer from.

There is a fix short of enlarging chunks: prepend context. Contextual Embeddings add a short, chunk-specific explanation of where the chunk sits in the document before embedding it, and averaged across their test sources they cut the top-20-chunk retrieval failure rate by 35 percent (Anthropic cookbook). The chunk stays small; the vector stops being context-blind.

Large chunks fail in the opposite direction. An embedding is one vector for the whole passage, so a chunk covering five topics produces a vector that is the average of five topics and a strong match for none of them. Retrieval gets fuzzy, and also wasteful: Google notes that when material exceeds what fits comfortably, retrieval reduces the number of tokens processed, saving time and cost (Google Cloud).

Two more costs compound:

• Context-window competition. Five large chunks fill the prompt and displace four other passages that might contain the actual answer. Pinecone also points to the lost-in-the-middle problem, where relevant text buried inside a long input is missed even when it is technically present (Pinecone).
• Latency and spend. Every retrieved token is processed on every call.

NVIDIA's numbers show the ceiling. Chunks of 2,048 tokens underperformed medium sizes on both RAGBattlePacket and FinanceBench, while page-level chunking delivered the highest average end-to-end accuracy in their runs at 0.648 with the lowest standard deviation of 0.107 (NVIDIA). Page-level worked because a page in those financial documents is a meaningful unit, not because big is better.

Decision rule: size a chunk so that, read alone by a competent colleague, it would let them answer a typical question on that topic. Pinecone states the same heuristic from the human side: if a chunk makes sense without surrounding context to a person, it will make sense to the model (Pinecone).

Boundary rules are where quality is won or lost, and they differ by document type. The table below is a checklist you can apply during ingestion.

Fixed-length splitting is the simplest method and produces uniform chunks, but it can split sentences and ideas, and critical information can span chunks (Mastering LLM). Recursive splitting, which tries larger delimiters first (paragraphs, then sentences, then words) and only falls back when a segment is still too big, keeps the structure while staying inside a size budget. Chroma's evaluation found heuristic strategies such as the recursive character splitter often perform well in practice when parameterized appropriately, and that default settings of some popular splitters can perform poorly (Chroma). Parameterizing it is the whole job.

A small parsing step, respecting a size ceiling, looks like this:

def chunk_by_boundaries(text, max_tokens, overlap_tokens):
    segments = split_on_headings(text)          # boundaries first
    chunks = []
    for seg in segments:
        pieces = recursive_split(seg, max_tokens)  # sentences, then words
        for i, piece in enumerate(pieces):
            tail = pieces[i - 1][-overlap_tokens:] if i > 0 else ""
            chunks.append(tail + piece)
    return chunks

The important line is the first: structure decides the candidate cuts, and length only decides when a cut is forced.

Overlap is insurance against seams, and like insurance it should be priced. Because overlap duplicates text, it inflates index size, embedding cost and the chance that two near-identical chunks both rank and both consume prompt space; overlapping chunks are a recognized source of redundancy (Mastering LLM).

Use 10 to 20 percent of chunk size as a default only in these situations:

• The splitter is sentence-aware and the risk is a two-sentence argument straddling a boundary.
• The content is procedural and a step depends on the step before it.
• You are using fixed-size splitting because the source has no usable structure.

Skip overlap for code (a function is atomic), for tables (repeating a header inside a table corrupts it), and for FAQ entries (each entry is self-contained). One academic evaluation of fixed-size chunking used 150-word chunks with 15 words of overlap, a 10 percent ratio, which is a reasonable proportion to copy (arXiv).

Suppose you are indexing 2,000 support articles averaging 800 words each, and your chunks average 300 words. That yields roughly 6,700 chunks before overlap. At 15 percent overlap you add about 45 words of duplicated text per chunk, raising the stored corpus by around 15 percent and adding the same proportion to embedding cost.

Now consider the retrieval side. If you retrieve the top 5 chunks and each averages 300 words (roughly 400 tokens), the answer prompt carries about 2,000 tokens of evidence. Drop chunk size to 120 words and you can retrieve 12 chunks for the same budget, which sounds better until questions like "what changed in the returns policy" need the paragraph that names the product line. Page-level chunking scored highest on average in NVIDIA's benchmark, but the same study found factoid queries in some datasets preferred 256 to 512 token chunks while complex analytical queries preferred larger ones or page boundaries (NVIDIA). Chunk size is therefore query-shaped: if your users ask narrow factual questions, bias smaller; if they ask for comparisons and analyses, bias larger.

Hybrid retrieval and reranking change the economics too. Running a reranker over a wider candidate set and then returning a few chunks lets you keep chunks small enough for precise matching while returning enough context; Qdrant's tutorial describes reranking as applying an expensive model to a smaller candidate set already retrieved by a faster method, keeping latency low (Qdrant). If a retrieval failure traces to a chunk that is too small, the cheapest fix is often to retrieve a parent chunk around it. Parent-child chunking keeps large parent segments (about 500 to 2,000 tokens) for context and small child segments (about 100 to 500 tokens) for matching, returning the parent when a child hits (Sandgarden, Medium).

Chunking changes are cheap to make and easy to judge wrongly. Build a small evaluation set first: 50 to 200 real questions, each labeled with the passage that contains the answer, which is the approach Anthropic used with 248 queries and a golden chunk per query (Anthropic cookbook). Then score retrieval with a pass-at-k metric (was the golden passage in the top k results) rather than reading a handful of answers by eye.

Run three configurations and keep the winner: your current settings, a structure-aware split at your target size with 15 percent overlap, and the same without overlap. Change one lever at a time. NVIDIA's results show why per-dataset evaluation matters: optimal size varied significantly even among similar financial document types (NVIDIA).

When the pipeline is in place, model choice becomes the next variable rather than the first. If you want to swap embedding models or compare rerankers without rewriting clients, Quanta Inference exposes text, image, video, audio and embedding models through one OpenAI-compatible API key (Quanta Inference). If chunk design and retrieval evaluation are the bottleneck and you would rather have someone run that loop with you, a consulting call is the shorter path.