Indexing for a Retrieval Eval: What to Store So Reruns Are Cheap

A chunk record needs four fields and one derived value. Everything else is convenience.

The chunk_id should be derived, not random. A UUID means the same passage gets a new ID every time you re-split, which destroys the link between an old eval label and the new chunk. Use a deterministic function of the things that define identity:

import hashlib, json

def make_chunk_id(doc_id: str, chunker_version: str, ordinal: int, text: str) -> str:
    material = f"{doc_id}\x1f{chunker_version}\x1f{ordinal}\x1f{text}"
    return hashlib.sha256(material.encode("utf-8")).hexdigest()[:32]

Two consequences follow. First, if the text is byte-identical and the ordinal and chunker version match, the ID is stable across runs, so old labels keep resolving. Second, if you change the splitter, IDs change on purpose, and the manifest tells you exactly which eval labels now dangle. Hashing the text into the ID also means an edited paragraph gets a new ID rather than quietly changing meaning under an old one.

Normalize before hashing. Strip trailing whitespace, collapse runs of spaces, and normalize line endings, or a re-fetch that only changed a CRLF will look like a content change and trigger a pointless re-embed. The freshness playbook makes the same point for production indexing: store a hash of the preprocessed text and skip re-embedding when the hash is unchanged, even if the row's timestamp moved the embeddings freshness playbook.

Store embeddings beside the record but never inside the content hash. The hash describes the text; the embedding describes one model's reading of it. Vectors are keyed by chunk ID plus embedding model version, so re-embedding with a new model writes new vectors instead of destroying the old ones.

The chunk store answers "what text exists." The manifest answers "which text did this eval run see." They are separate objects on purpose.

A manifest is a small, immutable JSON document. It carries a manifest version, the chunker version, the embedding model version and dimension, the ordered list of chunk IDs in scope, and a rollup hash computed over that list. It also records the parent document IDs, so you can trace a removal.

{
  "manifest_version": "retrieval-eval-corpus@2024-11-04T09:12:00Z",
  "chunker": {"name": "markdown-headers", "version": "3"},
  "embedding": {"model": "text-embedding-x", "dim": 1024},
  "documents": ["doc-8812", "doc-8813", "doc-9001"],
  "chunk_ids": ["a1f0...", "9c22...", "77bd..."],
  "rollup": "sha256:5e1b0c..."
}

The rollup is the important part for audit. Compute it by sorting the chunk IDs, concatenating them with a separator, and hashing the result. Any add, remove or reorder produces a different rollup, so a log line like manifest=5e1b0c is enough to reproduce a run. This is the same discipline MLflow describes: log corpora, embeddings, judge prompts and raw outputs as artifacts so any evaluation run can be reproduced from a single ID the RAG evaluation workflow.

Golden queries reference chunk IDs, not file paths. A label that says "the answer is in section 4.2 of the returns policy" is unverifiable once section numbering changes; a label that names two specific chunk IDs is exact. Store per-case evidence labels separately from expected answer wording, because a single passage can support many acceptable phrasings the golden dataset guide.

The manifest also lets you hold out a clean set. Freeze a release manifest, run your tuning against a working manifest, and the two never mix silently.

Keep the store in two layers. The blob layer holds immutable chunk records keyed by chunk ID, one file per record or one file per document, and it never mutates. The index layer is a small SQLite or Postgres table that maps chunk ID to blob path, content hash, chunker version and embedding version, plus a table of manifests. On load, you read the manifest, look up its chunk IDs, and fetch only those records. Nothing else touches the network.

Loading is then a single query plus a bulk read:

def load_corpus(conn, manifest_id):
    ids = conn.execute(
        "SELECT chunk_id FROM manifest_chunks WHERE manifest_id = ?", (manifest_id,)
    ).fetchall()
    order = {row[0]: i for i, row in enumerate(ids)}
    rows = conn.execute(
        f"SELECT chunk_id, text, embedding FROM chunks WHERE chunk_id IN ({','.join('?'*len(ids))})",
        [r[0] for r in ids],
    ).fetchall()
    chunks = [dict(zip(("chunk_id", "text", "embedding"), r)) for r in rows]
    chunks.sort(key=lambda c: order[c["chunk_id"]])
    return chunks

Three rules keep this honest. Never mutate a blob in place: a change writes a new chunk record and updates the index. Never delete a chunk that any manifest still references: mark it superseded instead. Never let the loader fall back to fetching a missing chunk from the source, because a silent fetch is exactly how a rerun stops being reproducible.

If you run many embeddings, keep the embedding model version in the manifest and refuse to load vectors whose recorded model does not match. Mixing dimensions or spaces across a corpus produces retrieval that returns plausible but wrong sources rather than an error the embedding model selection guide.

Change detection is per document, driven by a document-level hash of normalized extracted text, not by a modification timestamp. Timestamps lie when a CMS re-saves a post, and they miss changes when content is regenerated from another system. Compute the document hash, compare it to the last recorded one, and only then decide whether to re-chunk.

The decision table is the whole workflow:

The re-chunk path needs one safeguard. After splitting, hash each candidate chunk and check the store for that ID. Existing IDs are reused with their stored vectors; only genuinely new IDs get embedded. This is delta indexing in miniature, and it is why a one-paragraph edit in a long document usually costs a single embedding call rather than a whole-document rebuild the embeddings freshness playbook.

Deleted content deserves care. If a chunk that a golden query labeled as required evidence disappears, the eval will report a retrieval failure that is really a corpus change. Detect this at manifest build time by diffing the old and new chunk ID lists and checking the removed IDs against your label table. The correct response is to retire or relabel the case, not to let the score fall and blame the retriever.

Sizing is unglamorous but it decides whether the design survives contact with reality. Consider 40,000 chunks averaging 1,200 characters of text (roughly 300 tokens) and 1,024-dimensional float32 embeddings. Raw text runs about 48 MB. Vectors run 40,000 times 1,024 times 4 bytes, about 164 MB. Per-chunk JSON overhead with a 32-character hex ID, source metadata and a short version tag adds perhaps 300 bytes per record, roughly 12 MB. Call it well under half a gigabyte uncompressed for a mid-sized documentation corpus, which is small enough to check into object storage and copy between machines. The vectors dominate, so compress them when cold and accept a slower reload.

The expensive part is regeneration, not storage. Embedding 40,000 chunks in one pass is a few minutes of API time and a real bill; re-embedding the same corpus on every eval iteration multiplies that by the number of experiments. Now the worked example: suppose a team runs the eval ten times a week, and each full rebuild embeds 40,000 chunks. With a store, a typical week touches maybe 200 changed chunks across edited documents. The rebuild path embeds 40,000; the store path embeds about 200. That is a two-order-of-magnitude difference in embedding calls, before counting the fetch and parse time you also skip. Plug in your own chunk count and edit rate, since the ratio is what matters, not these numbers.

One more effect. Because the manifest pins the corpus, metric movement between two runs is attributable to the retriever, the ranker or the query set, not to a corpus that shifted underneath you. Retrieval metrics like recall@k and nDCG only compare meaningfully when the ground-truth passage set is held constant the retrieval metrics overview. And if you care about how the LLM consumes retrieved passages rather than a human reading a ranked list, keep the utility of each passage labeled too, since distractors actively harm generation quality rather than being ignored the utility-based retrieval evaluation paper.

A checklist before you call the store done:

• Every chunk record has a content-derived, deterministic chunk_id and a normalized-text hash.
• Embeddings are stored beside the record and tagged with model version and dimension.
• Every eval run records a manifest version and its rollup hash in the run log.
• Loaders never fetch from source and never fall back to a re-embed.
• The refresh job reports counts: documents scanned, documents changed, chunks added, chunks reused, IDs orphaned from labels.
• A weekly job diffs the current store against the last frozen manifest and flags labels pointing at removed chunks.

Once retrieval is cheap to rerun, the next bottleneck is usually the generation side: judging faithfulness, relevancy and grounding against a fixed set of retrieved passages. Those judges are themselves model calls, and their cost scales with cases times metrics, which is why separate labels for retrieval and generation matter so you can rerun one without the other how RAG evaluation breaks down. Community threads on self-evaluating RAG pipelines show the same pattern: golden sets generated from a small document collection, then replayed against real pipeline outputs, with timeouts and judge instability as the recurring pain the DeepEval and RAGAS discussion. A frozen chunk store removes an entire class of variation from that loop.

When the judge or answer model changes, or when you want to compare several judges on the same fixed corpus, routing those calls through one endpoint keeps credentials and billing in one place. Quanta Inference exposes text, image, video, audio and embedding models behind a single OpenAI-compatible API key, priced in US dollars and prepaid, which is convenient when the eval itself calls several model families (https://quantaintelligence.ai/inference).