Chunking Strategies and Their Effect on Retrieval

Fixed-size chunking splits text at a fixed number of tokens or characters, with optional overlap between consecutive chunks. SurrealDB shows the shape of it: a 512-token window with a 50-token overlap yields chunks whose starting points are 462 tokens apart (SurrealDB). The advantages are real: no parsing, no structure assumptions, no extra model call, and deterministic output. As a first baseline or a high-throughput ingest where speed beats precision, it is hard to beat.

The cost is that the splitter has no idea where a sentence or a topic change happens. A chunk can open mid-clause and close before the qualifying sentence arrives. Firecrawl's survey describes the trade-off plainly: size-based methods are fast and cheap because they split on character count with no API calls, but they ignore context boundaries (Firecrawl). Tahir Saeed's chunking guide adds that smaller chunks may lose overarching context while larger chunks retain more of it but can bury the specific detail a query is after (Tahir Saeed).

Fixed-size is a fine place to start. It is a bad place to stop without measuring anything against it.

Sentence-aware (often called recursive) chunking tries separators in a hierarchy: paragraph breaks first, then sentence ends, then word boundaries, cutting at the nearest natural edge before the target size. SurrealDB describes this as the approach used by common splitters such as LangChain's recursive character splitter, and recommends it for general prose like blog posts, knowledge base articles and support documentation (SurrealDB). Keeping sentences intact matters because a fragment beginning "and therefore the limit is" carries almost no retrievable signal.

Published evidence supports the upgrade. A large cross-domain study benchmarked 36 segmentation methods across six knowledge domains and five embedding models, and reported that content-aware chunking significantly improved retrieval effectiveness over naive fixed-length splitting, with paragraph group chunking reaching the highest overall accuracy and much better top-rank hit rates while simple character chunking performed poorly (arXiv). The gap was large: the best strategies reached mean nDCG@5 around 0.459, while plain character chunking stayed below 0.244 (arXiv).

Semantic chunking goes further. It embeds each sentence, compares adjacent sentences by cosine similarity, and places a boundary wherever similarity drops sharply, merging the segments up to a target size (SurrealDB). The result is the most coherent chunk set, because splits land on real topic changes. The cost is an embedding call per sentence at index time plus a new threshold to tune: too sensitive and you over-split, too loose and you effectively get one giant blob (SurrealDB). Firecrawl's cost tiers make the same point, separating simple and structure-aware methods from semantic methods that must embed every sentence (Firecrawl).

Once the method is fixed, two numbers remain: target chunk size and overlap. Neither has a universal answer, but both have predictable effects.

Smaller chunks produce tighter vectors. A chunk with one clear idea produces a precise embedding that encodes that idea's nuance, which makes it easier for vector search to separate it from near-misses (Weaviate). The failure mode is context loss: a lone sentence from the middle of a paper is hard for a human to interpret, and the model struggles for the same reason. Larger chunks keep the reasoning but blend topics. Averaging several ideas into one vector is like describing a book by averaging all its chapters: the embedding represents no single topic well and retrieval gets muddy (Weaviate). Large chunks also push into long-context generation, where models miss information buried in the middle of a long input, which Pinecone cites as a reason to control how much context you pass downstream (Pinecone).

Overlap recovers facts that straddle a boundary: if a clause starts near the end of one chunk and finishes at the start of the next, overlap ensures one chunk holds the whole thing. Firecrawl recommends recursive splitting at roughly 400 to 512 tokens with 10 to 20 percent overlap as the default for most use cases (Firecrawl). Overlap is not free. Every percent adds to chunk count, and therefore to index size, embedding cost and latency, and it creates duplicate-looking results that can crowd a better result out of the top k.

Price overlap in index terms. With 4,096 tokens of source text and 512-token chunks you get 8 chunks with no overlap. At 20 percent overlap (about 102 tokens) the stride drops to 410 and you get 10 chunks, a 25 percent increase in storage. That is a bargain when boundary-straddling facts are common and waste when they are not, which is exactly what your own query set can answer.

You cannot tune chunk size without something to tune against. The standard apparatus from information retrieval is a test collection with three parts: the document collection, a suite of information needs expressed as queries, and relevance judgments marking each query-document pair relevant or not (Stanford IR book). The same source gives a useful floor: 50 information needs is usually the minimum for stable averages, because results vary widely across queries (Stanford IR book). For a chunking sweep, a few hundred labeled query-chunk pairs is enough to detect large differences, and more is needed to resolve small ranking changes (Qdrant).

Qdrant sets out three ways to build a golden set: human annotation, which gives the highest-fidelity signal; real user queries from production logs with clicks or explicit feedback as the relevance signal; and LLM-based synthetic generation, which scales cheaply but produces queries that are easier to retrieve than real ones, inflating offline scores (Qdrant). A workable blend is a few dozen human-labeled queries for high-value topics, a few hundred log-derived queries for volume, and synthetic queries only to fill gaps.

Two details keep the evaluation honest. Label against the information need, not the query string: a document is relevant if it addresses the stated need, not because it contains all the query words (Stanford IR book). And do not tune on the queries you report results from: keep a development set for tuning and report final numbers on a held-out set, because tuning to one collection overstates expected performance (Stanford IR book).

Precision@K is the fraction of the top K results that are relevant; recall@K is the fraction of all relevant items appearing in the top K (Evidently AI). For chunking, prioritize recall. Chunking failures are mostly coverage failures: the right passage exists but no indexed chunk can match the query. Precision punishes false positives, which a reranker can fix later; recall punishes false negatives that no downstream stage can recover (Unstructured). Unstructured recommends recall when the product punishes false negatives, as investigation and compliance workflows do (Unstructured).

Ranking quality inside the list matters too. MRR focuses on how quickly a user reaches one correct target, and NDCG handles graded relevance (Unstructured). A practical stack is Hit Rate to confirm the right context appears anywhere in the top k, recall plus MRR for chunk and retriever tuning, and NDCG with Hit Rate when a reranker is added (Tarun Jain). Using several metrics matters because a system can be good at finding one result and bad at finding all of them (Tarun Jain).

from dataclasses import dataclass
from typing import Callable, Sequence

@dataclass
class Params:
    size: int          # target tokens per chunk
    overlap: int       # token overlap between neighbors
    splitter: str      # "fixed" | "recursive" | "semantic"

@dataclass
class Chunk:
    doc_id: str
    text: str

def index_chunks(chunks: Sequence[Chunk], embed: Callable) -> list:
    return [(c, embed(c.text)) for c in chunks]

def recall_at_k(queries, retrieve, k: int) -> float:
    hits = 0
    for q in queries:
        got = {c.doc_id for c, _ in retrieve(q.text, k)}
        if got & set(q.relevant_doc_ids):
            hits += 1
    return hits / len(queries)

grid = [Params(size=s, overlap=o, splitter=m)
        for m in ("fixed", "recursive", "semantic")
        for s in (256, 512, 1024)
        for o in (0, 50, 100)]

for p in grid:
    chunks = split(corpus, p)
    store = index_chunks(chunks, embed)
    score = recall_at_k(queries, make_retriever(store), k=5)
    print(p, round(score, 3), "chunks:", len(chunks))

Read the output in two columns, quality and cost. A run that lifts recall@5 from 0.71 to 0.78 while doubling chunk count is a different decision from the same lift at flat cost. The cross-domain study measured this side directly, noting that more, smaller chunks increase index size and latency, and identifying methods that balance effectiveness against efficiency (arXiv).

Answer quality is the second gate, since recall@k is only a proxy. Once a chunk set wins on recall, run a fixed sample end to end through generation, score answers against a rubric, and hold the generation model and prompt constant so retrieval input is the only variable. A clinical decision support study compared advanced chunking for RAG and found the method materially affected retrieval in a domain where a missed passage is a safety issue (PMC).

It is tempting to assume keyword search papers over chunking mistakes. It helps, partly. Hybrid search combines BM25, which scores exact term matches from an inverted index, with dense vector search; the two fail in complementary ways, since BM25 has no notion of meaning and dense retrieval can underweight exact rare terms (Digital Applied). Reciprocal rank fusion merges the two ranked lists by position rather than score, giving each document 1 divided by (k plus its rank) in each list and summing across lists, because raw BM25 and cosine scores sit on incompatible scales (OpenSearch). With the rank constant at 60, a document ranked first by BM25 and second by dense search contributes 1/61 plus 1/62, while one ranked second by a single method contributes only 1/62 (Abhishek Thakur).

All of this operates on chunks. If a rare identifier's surrounding terms are split across two chunks, BM25 may match one fragment and dense search another, and neither returns a readable passage. Running both representations inside the database and fusing them with reciprocal rank fusion keeps them synchronized per chunk, making chunk boundaries the shared unit of quality for both retrievers (pgEdge).

The decision rule: start with recursive sentence-aware splitting at a moderate size and light overlap. Measure recall@5 on a labeled set of at least 50 queries. If recall is high and no query slice fails, stop, because semantic chunking will cost more and move little. If specific slices fail with boundary-straddling or mixed-topic symptoms, test semantic and structure-aware splitting against the same set, and ship only a change that wins on recall without an unacceptable jump in chunk count. When the sweep shows the bottleneck is the model rather than the chunks, testing an alternative embedder on the held-out set is cheap through one OpenAI-compatible endpoint (Quanta Inference).