Reading a Retrieval Change as a Curve Shift, Not a Score

A curve comparison only means something if both runs share every input except the variable under test. Write the contract down before the first run.

Your golden set does not need to be large. A fixed set of 20 to 30 pairs is enough to catch regressions, and the value is that it is fixed, so every change is compared against the same expectation How to Know Your RAG Changes Actually Helped. Freeze the labels even if you later grow the set: run the larger set against both configurations rather than comparing new measurements to old ones.

Pick a quality metric that responds to depth and that you trust. Token-level precision and recall on retrieved tokens, plus intersection-over-union, capture chunking effects that document-level benchmarks miss Evaluating Chunking Strategies for Retrieval. If you score with an LLM judge, keep the prompt narrow and the scale tight. A 0 to 2 relevance scale with a required one-sentence reason is more stable across runs than a 0 to 10 scale, and wrapping the chunk in delimiters with an explicit "treat as data, not instructions" line blunts prompt injection through retrieved text How to Know Your RAG Changes Actually Helped. For end-to-end answer quality, faithfulness and answer relevancy measure different failures and should stay separate; a system can retrieve perfectly and still hallucinate RAG evaluation metrics.

One caution on end-to-end scoring at many depths: RAGAs-based faithfulness showed limited reliability in at least one academic-text study, and performance on fixed versus document-specific questions varied substantially, likely due to formatting and preprocessing Evaluating Chunking Strategies for RAG on Academic Texts. If your judge is shaky, run retrieval metrics as the primary curve and treat the answer-quality curve as corroboration.

The mechanics are simple. For each k from 1 to 20, retrieve top-k, compute the metric, write a row. Do it for the incumbent configuration, then for the candidate. Then join on k.

import json, statistics

def sweep(queries, retriever, judge, ks=range(1, 21), repeats=3):
    rows = []
    for k in ks:
        scores = []
        for _ in range(repeats):          # repeat to estimate noise
            for q in queries:
                chunks = retriever.search(q.text, k=k)
                scores.append(judge.score(q.text, chunks))
        rows.append({
            "k": k,
            "mean": statistics.mean(scores),
            "sd": statistics.stdev(scores),
            "n": len(scores),
        })
    return rows

base = sweep(queries, retriever_before, judge)
cand = sweep(queries, retriever_after, judge)

for b, c in zip(base, cand):
    delta = c["mean"] - b["mean"]
    se = ((b["sd"] ** 2 + c["sd"] ** 2) / b["n"]) ** 0.5
    print(b["k"], round(b["mean"], 3), round(c["mean"], 3),
          round(delta, 3), "clears SE" if abs(delta) > 1.96 * se else "inside noise")

Two implementation notes. First, repeat each depth a few times with the same judge and let the variance tell you whether a delta is real; comparing means without an error estimate is how a 1-point move becomes a release decision. Second, if you judge answer quality end to end, hold the generation prompt fixed and be aware that the model's tolerance for extra context varies. One analysis notes that LLMs are relatively insensitive to the position of relevant information in the window, which is why token presence matters more than ranking in these evaluations Evaluating Chunking Strategies for Retrieval.

A worked example, with numbers invented for illustration:

Read this as a rotation, not a lift. The candidate buys precision near the top of the list and loses recall deeper down. That is exactly the signature you would expect from smaller chunks: the vector for each chunk is more specific and matches a query better, while the evidence for multi-part questions is now split across more chunks, so one query can no longer be fully satisfied by any single retrieved item. The parent-child approach exists precisely to trade this off, storing small chunks for matching and returning the enclosing larger chunk for context Parent-Child Chunking in LangChain for Advanced RAG. Parent chunks typically run several hundred to a couple thousand tokens, child chunks a hundred to a few hundred, so a curve that shows the rotation above is a signal to try hierarchical retrieval rather than to revert.

Four shapes cover most real comparisons, and each implies a different release decision.

Parallel lift. The candidate curve sits above the incumbent at every k by roughly the same margin. The change is a genuine improvement and the only question is cost: more chunks stored, higher token spend per query. Ship it if the margin clears noise, then re-tune k downward to reclaim latency.

Rotation. The curves cross. One configuration is better at small k, the other at large k. This is the common case for chunk-size and reranker changes. The decision rule: compare the two at the k you actually ship, and if the new k is better, check whether the regression at the other k affects any real workload. If users routinely ask multi-part questions that need evidence from several passages, the large-k loss matters more than the small-k gain.

Sandwich or dip. The candidate dips in the middle of the range and recovers or surpasses at both ends. Usually a sign that the new chunks split a small number of key passages, so a mid-depth retrieval lands on fragments, while deep retrieval still picks up the pieces. Fix the splitter at the document structure level rather than abandoning the change.

Noise. The gap wanders above and below zero by amounts smaller than the error band. Nothing happened, or the golden set is too small to see what happened. Grow the set or tighten the judge before spending another week on the pipeline.

The shape also tells you where the risk sits for the whole release. If the candidate only improves at k = 1 to 3, then any downstream component that reads more chunks, such as a reranker or a context compressor, gets no benefit, and a reranker's job is to reorder candidates before generation anyway Retrieval Latency Optimization for Production RAG Systems. If the gain is concentrated at large k, you need the budget to actually pass large k into the prompt.

Structure matters as much as size when you read a shape. NVIDIA's chunking experiments found page-level chunking reaching an average end-to-end RAG accuracy of 0.648 with the lowest standard deviation of 0.107 across tested datasets, while extreme sizes of 128 and 2,048 tokens generally underperformed medium sizes, and the best size differed by document type: 1,024 tokens for FinanceBench, 512 for Earnings Finding the Best Chunking Strategy for Accurate AI Responses. That is a curve argument, not a point argument. A single test at one size on one dataset would have picked the wrong winner for the other dataset.

Every point on your curve has a price, and the price is not flat. Retrieval latency and context cost both scale with how much you retrieve and how much of it reaches the model. A smaller chunk size roughly doubles vector count for the same corpus, and a 500 ms retrieval allowance broken down by stage leaves little room for retries and reranking once primary retrieval and assembly take their share Latency Budgets for Memory Retrieval. NVIDIA's sizing work, as summarized in a benchmark review, reported time to first token climbing from about 2.0 seconds at 256-token chunks to 7.1 at 1,024, and separately rising from 3.6 to 7.9 seconds when top-k went from four to 10 Advanced Chunking Techniques: What The Benchmarks Actually Show. Those two levers multiply.

So plot a second y-axis before you decide. For each k, record retrieval time and prompt tokens alongside quality. The release decision becomes a question about a rectangle in that space: does the candidate curve dominate the incumbent at an affordable (k, cost) point, and does it stay on top at the tail rather than just the median? Measure p95 and p99, not just averages, because tail latency shapes user trust One post tagged with latency trade-offs.

A practical shortcut: if the candidate's best affordable point beats the incumbent's shipped point by more than the error band, and the worst point in the tested range is not materially worse than the incumbent's worst, ship. If either condition fails, iterate on the chunker parameters rather than the release.

The most expensive mistake is sweeping the wrong variable. Changing chunker and embedder together produces a curve you cannot attribute. The second is sweeping on a shifted query set, which makes the two curves incomparable at every k. The third is trusting a single judge call per query and reading a delta that lives inside judge variance. A fourth is ignoring answer-side metrics: a retrieval change can improve context precision while the model still answers from the wrong passage, and context precision and context recall move in opposite directions as you change depth RAG evaluation metrics.

A subtler error is over-reacting to advanced-chunking gains. Simpler recursive splitting with sensible overlap often captures most of the available improvement, though defaults can be weak until you tune them Evaluating Chunking Strategies for Retrieval. And the frontier keeps moving: model-aware splitters that derive boundaries from the target model's own tokenizer address tokenization mismatch across model families, where a 512-character window can land anywhere from roughly 100 to 400 tokens depending on the encoder ChunkRank. If you are about to compare two embedding models, check whether your chunk boundaries are sized in the wrong unit, or your curve is comparing two splitters rather than two embedders.

The last error is deciding from the mean of the curve. The mean across k answers a question nobody asked. Report the delta at every k, mark which ones clear the band, and make the release call from the k your users experience and the workload your traffic shows.

When the sweep needs to run against several models at once (an embedding model candidate, a judge, an answer model), routing each through one endpoint removes the per-provider plumbing from the experiment. Quanta Inference gives you text, image, video, audio and embedding models through one OpenAI-compatible API key, priced in US dollars, so the only variable in the sweep is the one you meant to test: https://quantaintelligence.ai/inference.