Seeding a Vector Store for RAG Tests Without Drift

Before any code, decide how embeddings get produced in tests. Two options compete, and one wins for most teams.

The decision rule: if the test asserts on the retrieval pipeline, hash-derived vectors are correct; if the test asserts on embedding quality, use frozen golden vectors; only call the model live in a separate, non-blocking evaluation job. Mixing live model calls into a unit test tier gives you flaky tests that cost money, and the flakiness is not even the interesting kind.

Hash-derived vectors have a second property that matters: they are cheap enough to regenerate on every CI run. A seeded store with 50,000 chunks rebuilds in seconds, so there is no temptation to keep a long-lived index around and "just append" to it, which is where contamination starts.

The recipe is short. Normalize the chunk text, hash it with a fixed salt, expand the digest into the number of floats your index expects, and map the bytes into a range the store accepts.

import hashlib, struct

DIM = 384  # match the index's configured dimension

def fake_embedding(text: str, salt: str, dim: int = DIM) -> list[float]:
    """Deterministic, stable pseudo-embedding. Same input, same output, always."""
    h = hashlib.sha256((salt + "\x00" + text).encode("utf-8")).digest()
    out = []
    while len(out) < dim:
        h = hashlib.sha256(h).digest()          # extend the digest as needed
        for i in range(0, len(h), 4):
            if len(out) == dim:
                break
            (n,) = struct.unpack(">I", h[i:i+4])
            out.append((n / 2**32) * 2.0 - 1.0)  # map to [-1, 1]
    return out

Notes that save debugging time:

1. Normalize before hashing. Strip trailing whitespace and collapse internal runs of whitespace, otherwise a chunker change that alters only spacing produces new IDs and a spurious drift alert.
2. Keep the salt in configuration, not in the database. The salt is part of the seed's identity. If it changes, every vector changes, and that should be an intentional, reviewed event. Storing it as an environment variable follows the practice of separating config from code described in the twelve-factor summary by DreamFactory.
3. Pick the dimension once. The index is created with a fixed dimension, so a hash function that silently produces 768 floats against a 384-dimensional collection will fail at upsert time. Assert it in the seeding script.
4. Do not use Python's built-in hash(). It is salted per process, so vectors would differ across runs.

The pseudo-vectors are not semantically clustered, which is the point and the limitation. Any test that asserts "the retrieved chunk is about the same topic as the query" will fail. That is fine: split your test suite so that semantic relevance is checked in a nightly evaluation job against a real model, and the fast, deterministic tier checks that the right chunk IDs came back for a query you constructed to be findable by exact match.

Tenant isolation in a vector store is a design decision made once and paid for forever. Pinecone's guide to multi-tenancy recommends the one-namespace-per-tenant pattern and says to prefer it "unless you require the ability to query across tenants." Namespaces physically partition records in an index, queries hit one namespace at a time, and offboarding a tenant reduces to deleting a namespace per Pinecone's multi-tenancy guide.

For the test seed, the namespace name should be a direct function of the tenant ID, prefixed so that seeded data can be distinguished from anything else:

{environment}_{purpose}_{tenant_id}
test_seed_rag_acme
test_seed_rag_globex

A blunter framing appears in Thinking Loop's rules for multi-tenant vector isolation: namespaces are walls, not labels, and a tenant ID stored as a metadata tag on a shared collection is not isolation because one missing filter lets you query across tenants. That rule doubles as a test. Write an assertion that a query issued with tenant A's credentials never returns a chunk whose metadata carries tenant B's ID. If the test passes only because a filter happened to be applied, you have a tag, not a wall.

Two practical notes on namespacing for seeds:

• Keep the tenant list in a fixture file, not scattered through test code. The seeder iterates that list, and the drift check compares the set of namespaces present against the set expected.
• Give one tenant two documents that are near-duplicates of another tenant's, so an isolation regression produces a visible retrieval error rather than a silent pass. The chunk IDs differ, so the hash-derived vectors differ, and the only way the wrong chunk appears is a leak.

The seed job must be safe to run any number of times. Two runs in a row should leave the store byte-for-byte identical in content, and a run after a partial failure should repair rather than duplicate.

The key insight is that a chunk ID should not be an autoincrement counter or a random UUID. It should be a hash over everything that defines the chunk: the source document ID, the chunker version, the chunk's ordinal position, and its span boundaries. When any of those change, the ID changes, which is exactly the signal you want. Guidance on staging data seeding makes the same point in relational terms: snapshots from production are dangerous unless scrubbed, fixtures rarely reflect production scale, and the data needs to be refreshed on a schedule rather than frozen per Cloud Posse's seeding strategy notes.

chunk_id = sha256(doc_id | chunker_version | ordinal | start_offset | end_offset | text_digest)

The seeding loop then becomes an upsert, not an insert:

1. Compute the expected set of chunk IDs from the source documents.
2. List the chunk IDs currently in each namespace.
3. Upsert every expected chunk with its hash-derived vector and metadata.
4. Delete any ID present in the store but absent from the expected set. This is the step that prevents orphan vectors from yesterday's chunker from lingering and skewing retrieval.
5. Write the manifest (next section).

Step 4 is the one teams skip and later regret. In a mutable store, deletion is not cleanup; it is correctness.

A worked example with numbers: suppose 12 source documents produce 3,000 chunks under a 400-token chunker with 50-token overlap. Drop the chunk size to 200 tokens and the count roughly doubles to about 5,600 chunks, and because the chunker version and offsets are in the ID, every one of the original 3,000 IDs becomes stale. The rebuild upserts 5,600 and deletes 3,000. Without step 4, the namespace holds 8,600 vectors and a retrieval test that expects three results for a query silently picks up an old span that no longer reflects the current pipeline.

A manifest is a small JSON file committed to the repo, written by the seeder and asserted by the test run. It records what the store should contain, not what it happens to contain.

{
  "seed_version": "3",
  "hash_salt_id": "salt-2024-a",
  "dim": 384,
  "tenant_set_digest": "9f2c...",
  "namespaces": {
    "test_seed_rag_acme": {
      "chunk_count": 5600,
      "chunk_ids_digest": "b41e...",
      "vector_digest": "77aa..."
    }
  }
}

Two digests do different jobs:

• chunk_ids_digest is a sorted hash of every chunk ID in the namespace. It catches changes to the chunker, the source documents, and the deletion step. A mismatch means the content changed.
• vector_digest is a hash over the ordered list of (chunk ID, first few floats of the vector) pairs. It catches changes to the hash function, the salt, the dimension, or a bug where vectors were computed from the wrong text.

The drift check runs before any retrieval assertion and fails fast with a specific message. chunk_ids_digest mismatch points at the document set or chunker; vector_digest mismatch with matching chunk IDs points at the embedding derivation or the salt. Splitting the check this way turns a 30-minute debugging session into a one-line diagnosis.

This is the same principle as a schema migration check, and it earns the same payoff. Feature flags reduced the risk of releases by decoupling deploy from release as Datadog's overview of feature flags describes; a seed manifest does the equivalent for test data by decoupling "the store changed" from "the code changed." A drift alert means the test's premise is invalid, so the run should stop rather than produce a green or red result you cannot trust.

The seed job belongs in the same pipeline stage that provisions the test database, and it should finish before the first retrieval test starts. If your team already runs per-pull-request preview environments, the seeder can run as an init step for each one, which gives every branch its own fresh namespace and removes cross-branch contamination by construction. AWS Amplify's preview model illustrates the pattern: each pull request gets a unique preview URL and, for apps with backend environments, a temporary backend that is deleted when the PR closes per the Amplify web previews documentation. A vector namespace fits that lifecycle well, and platforms that provision full-stack previews with databases and teardown schedules make it easy to attach a throwaway vector store to each one as Northflank describes its preview environment model.

A concrete CI sequence:

1. Provision the vector store and create the namespace for each tenant in the fixture.
2. Run the seeder against the source documents, writing the manifest.
3. Run the drift check against the committed manifest. Fail the job with the digest name that mismatched.
4. Run retrieval and generation tests, which now assert on chunk IDs with confidence.
5. Tear down the namespace, or let the preview environment's teardown schedule do it.

Steps 2 and 3 are worth timing. On a store with a few thousand chunks, seeding is a handful of seconds and the drift check is a single list operation plus hashing, so the whole thing fits comfortably in the fast tier. Keep the slower AI-judged evaluation in a nightly job, as the RAG gate guidance from Render suggests, so the PR pipeline stays responsive.

One more operational habit: version the seed alongside the code. If the salt changes, bump seed_version in the manifest and note why. If you use a hosted embedding service for the golden-vector tier, the same discipline applies to recording which model produced which file. The twelve-factor discussion of build, release, and run separates those stages for good reason; treating your test fixtures as a release artifact, rather than something generated ad hoc on a developer's laptop, is the same idea applied to data.

If your team is running RAG tests across several model providers or embedding backends, Quanta Inference puts text, image, video, audio and embedding models behind one OpenAI-compatible API key, which lets the nightly evaluation tier point at one interface while the deterministic seed tier stays entirely offline.