Choosing Chunk Size and Overlap for a Search Audit Index

The cleanest metric for an audit pipeline is hit rate at k, meaning whether the correct passage appears anywhere in the top k retrieved chunks. It is a recall measure: the number of relevant chunks retrieved within the top k divided by the total number of relevant chunks for that query Rajan Mehta.

Why top five rather than top one? Because a top-page audit usually passes several chunks to the model at once, and the generator can discard what it does not need. What it cannot do is work with a passage it never received. Measuring at five tells you whether the pipeline gives the model a fair chance. Measuring at one tells you how good the ranking is, which is a separate and harder problem.

Two practical cautions. First, hit rate needs ground truth: for each audit question you must know which chunks are correct before you score anything Rajan Mehta. Second, evaluate the whole stack end to end against labels you trust rather than scoring retrieval in isolation and assuming the answer follows Parallel.ai. In practice, label at the passage level and then confirm that the final audit answer was correct. That catches both failures at once.

There is a related trap: mixing query types in one score. Fact-based questions, comparison questions, and summary-oriented questions have different retrieval expectations, and a single averaged number hides which type is failing Rajan Mehta. Keep a separate tally for narrow "does this page do X" checks and for broader "how does this page position itself" checks.

Here is the setup. Take a sample of 2,000 marketing pages averaging 1,400 words each. At roughly 1.33 tokens per word for English prose, that is about 1,860 tokens per page, or 3.72 million tokens across the sample. Assume a fixed overlap of 15 percent, fixed-size token windows, and a labeled set of 80 audit questions, each with one known correct passage on one known page.

The arithmetic for a fixed window with overlap is simple. Each chunk advances by (size minus overlap), so chunks per page is roughly total tokens divided by that stride, plus one. Overlap adds 15 percent to chunk count at every size, and that extra is pure duplication: the same text embedded twice.

Note what does and does not change. For fixed-size windows with the same overlap percentage, total indexed tokens stay roughly constant across chunk sizes, because overlap scales with chunk size. What varies is chunk count and, more importantly, how many tokens each query pulls into the model. Going from 256 to 1,024 tokens per chunk multiplies per-query context by four while returning the same five results. If your model bills by input token, that is a fourfold increase in the input side of every audit query for identical retrieval depth. Token pricing differs sharply between providers and models, and token counts vary by tokenizer, so the same passage can register as materially different counts across platforms Silicon Data.

Now the quality side. In a run of this shape, the pattern that typically emerges is:

• At 256 tokens, hit rate at five is high on narrow factual checks (does the page name a price, a region, a named customer), but the retrieved chunks are thin. The model sometimes gets a sentence without the surrounding claim, and duplication matters because the same 40-word claim appears in several page sections.
• At 512 tokens, hit rate at five is high across both narrow and broader positioning checks. Each chunk usually carries a heading plus its supporting paragraph, which is enough context to recognize what the claim is about.
• At 1,024 tokens, narrow factual checks stay high but the top five starts to include near-duplicate chunks from the same page section, wasting slots. Broad questions improve slightly; precision on specific facts drops.

The mechanism behind the 1,024 regression is the one chunking guides warn about: too large, and a chunk averages unrelated passages into a vector that matches no specific question The AI Engineer. A 1,024-token window on a marketing page frequently spans a feature block, a testimonial, and the opening of a pricing table. Its embedding is a blur of all three.

That is the decision the example is built to expose. Chunk size trades retrieval precision against per-query context cost, and the right point sits where precision is already high and extra tokens buy nothing.

Overlap exists for one reason. It copies the tail of each chunk onto the head of the next so that a sentence straddling a cut survives whole in at least one chunk The AI Engineer. It is a boundary repair mechanism, not a relevance booster. The same guide holds overlap to 10 to 20 percent of the chunk, precisely because every repeated token is a cost with no new content behind it.

For marketing pages, the working rule is:

1. Start at 15 percent. At 512 tokens that is about 77 tokens of carry-over, enough to stop a headline from being separated from the sentence that explains it.
2. Drop to 10 percent if the correct passage appears twice in the top five. Duplicate results are the clearest signal that overlap is doing damage. Two of your five slots are spent on the same text.
3. Raise toward 20 percent only if you see truncation failures, where retrieval finds the right chunk but the sentence stops mid-thought. Cap it there.
4. Never go above 20 percent on marketing pages. Repetitive body copy (the same benefit stated four ways) already produces near-duplicates. Added overlap multiplies them.

Duplicates hurt more in an audit than in a chatbot because of slot scarcity. Five retrieved chunks means a duplicate costs 20 percent of the model's evidence budget. Overlap is also named as a direct cause of duplicate results that increase cost without improving accuracy StackAI.

The short answer at the top is a starting point, not a verdict. Validate it on your own pages. A workable protocol, adapted from general agent evaluation practice: manually review 20 to 50 real traces before building any evaluation infrastructure, because you learn more about failure patterns from reading than from any automated score LangChain. Then define success unambiguously, so two reviewers agree on pass or fail for the same case LangChain.

A concrete version for a top-page audit:

• Build the question set. 50 to 100 questions written the way your audit logic actually asks them, drawn from real checks rather than invented ones. Group them into narrow factual checks, comparison checks, and positioning checks.
• Label the correct passage. For each question, record the page and the specific passage that answers it. Review any machine-generated label before you spend tokens scoring against it Parallel.ai.
• Hold everything else constant. Same embedding model, same prompts, same retrieval depth, same reranker settings. Change only chunk size and overlap Parallel.ai.
• Score hit rate at five per group, not just overall, plus a duplicate rate: how often the same passage appears more than once in the top five.
• Read the failures. For each miss, decide whether the passage was split, diluted, or absent. That classification tells you whether to change size or overlap.

A minimal scoring loop, with the loop variable fixed:

# For each labeled item: one question, one known-correct passage
hits = 0
dupes = 0
for item in labeled_set:
    results = retrieve(item.question, k=5, config=cfg)
    passages = [r.passage_id for r in results]
    if item.correct_passage in passages:
        hits += 1
    if len(passages) != len(set(passages)):
        dupes += 1

hit_rate = hits / len(labeled_set)
dupe_rate = dupes / len(labeled_set)

Then apply the decision rule:

Your labels will surface one more consideration. Summarization-style retrieval performs comparably to long-context reading, while chunk-based retrieval lags behind on some question-answering benchmarks arXiv. For a top-page audit, that argues for keeping chunks aligned to page structure so each one reads as a coherent unit, rather than treating a fixed window as sacred. Recursive splitting on headings and paragraphs first, then on sentence boundaries, is a strong default for Markdown and for reasonably converted HTML StackAI.

Audit indexes invite the question: why chunk at all, when context windows are large enough to hold many pages whole? The honest answer is that it depends on the query mix, and the research is not unanimous. One comprehensive study found long-context models consistently outperform retrieval-augmented approaches when resourced sufficiently, but retrieval remains far cheaper, and the two produce identical predictions for over 60 percent of queries arXiv. That is the practical opening: route the easy majority to retrieval and reserve full-context reading for the hard cases, a split that cut cost by 65 percent for one model and 39 percent for another in that study arXiv.

A separate evaluation adds nuance: long context generally wins on question-answering benchmarks, especially Wikipedia-based ones, while retrieval has advantages in dialogue-based and general question queries arXiv.

For a marketing-page audit, the cost argument dominates. If your corpus is a few hundred pages, feeding whole pages to a large-context model is defensible and removes the chunking decision entirely. If it is 2,000 pages and you run hundreds of queries per crawl, whole-page context scales with corpus size on every query while retrieval stays flat. The worked example gives the numbers: five chunks at 512 tokens is about 2,560 input tokens per query, while five whole pages is about 9,300. Reading the entire corpus on every query is not a per-query option at all.

If you run retrieval and direct context in the same pipeline, an OpenAI-compatible endpoint for text and embedding models removes a layer of glue code; Quanta Inference covers both through one API key at https://quantaintelligence.ai/inference.

Use this before committing a chunking configuration to production.

• Confirm ingestion preserves headings, paragraphs, and tables rather than flattening pages into one text blob.
• Measure tokens, not characters, because embedding and generation limits are counted in tokens Silicon Data.
• Record chunk count and total indexed tokens at each candidate size so the cost of overlap is visible in numbers.
• Label 50 to 100 real audit questions with their correct passages, and review the labels by hand.
• Score hit rate at five per question group, and score duplicate rate separately.
• Apply the size-first, overlap-second rule, changing one variable at a time.
• Log the failure mode for every miss (split, diluted, or absent) so the next change is evidence-driven.
• Re-run the labeled check whenever page templates change, because a new template changes the shape of the text you are cutting.

The last item is the one teams skip. A configuration tuned on last season's landing template quietly degrades when a redesign adds a 400-word hero block above the body copy, because every chunk boundary shifts down the page.