Stubbing an Embedding API in CI Without Losing Fidelity

The design is small: canonicalize the input, hash it, expand the digest into the number of floats your real model returns, then normalize to unit length so cosine similarity behaves the way your production code expects. Expanding a 32-byte digest gives you 8 float64 values or 4 float32 values per digest, so for a 1536-dimension vector you need 192 digests; re-hash with a counter suffix, or use a keyed XOF such as BLAKE2b's variable-length output, which is cleaner.

import hashlib, math, struct

DIM = 1536  # match the real model's dimension so downstream code is unchanged

def hash_embed(text: str, dim: int = DIM, salt: str = "ci") -> list[float]:
    digest = hashlib.blake2b(f"{salt}:{text}".encode("utf-8"), digest_size=dim * 4).digest()
    raw = struct.unpack(f"<{dim}f", digest)
    norm = math.sqrt(sum(v * v for v in raw)) or 1.0
    return [v / norm for v in raw]

Three details decide whether this stub is useful or a source of false confidence.

Match the real dimension. If production returns 1536 floats and your stub returns 384, your vector store schema, your index parameters and your dimension-check code all go untested. The Pydantic AI embedding docs show that dimension is a first-class property of the result object, and that models differ in dimension and context length, so pin the stub to the model you actually ship Pydantic AI embeddings guide.

Normalize. Most retrieval setups normalize to unit length so that dot product equals cosine similarity local embedding models overview. If your stub returns unnormalized vectors, code that assumes unit length will pass in CI and misbehave in production.

Salt, and keep the salt stable. With a fixed salt the mapping is reproducible across runs and machines; changing the salt reorders every vector and invalidates any snapshot fixtures you kept. Treat the salt as configuration, not a random value.

A hash stub gives you real coverage of the code you own and zero coverage of the code you rent. Be explicit about the line.

The last four rows are the entire reason a real suite exists. A hash vector is not a semantic representation; two synonyms hash to unrelated vectors, and identical text hashes to an identical vector, so a test that asserts "these two paraphrases are close" will fail with the stub no matter how good your retrieval is. Conversely, a test that asserts "the top result for query Q is document D" is meaningless with the stub, because the ranking is an artifact of the hash.

Quality gates need statistics, not pass/fail. A team writing about RAG release gates makes the point that statistical gating catches regressions that binary tests miss, comparing each change against a rolling baseline, and that ID-based checks should run before expensive judged evaluations RAG CI/CD and release gates. The same split applies here: cheap structural checks in the stub suite, statistical and ID-based quality checks in the real suite.

Write the rule down and enforce it in review. A test belongs in the real-API suite if, and only if, one of these is true:

1. It asserts on ranking. The expected order of results for a query, or the presence of a known-good document in the top k.
2. It asserts on a similarity threshold. "Score above 0.8 means duplicate," "below 0.3 means unrelated." Thresholds are calibrated against a specific model's geometry and do not transfer.
3. It asserts on recall or precision over a labeled set. Any metric computed against ground truth.
4. It asserts on model-specific behavior. Output dimension, maximum input tokens before truncation, behavior of the provider's batch limit, or the effect of a reindex after changing chunk size or model.

Everything else belongs in the stub suite. If someone proposes moving a test to the real suite for a reason not on this list, the likely cause is that the stub suite is asserting too much about what the stub returned, which is a stub problem, not a coverage problem.

A nightly suite that fails every other night teaches the team to ignore it. Two design choices keep it trustworthy.

Pin the corpus and the queries. Retrieval state is mutable: two experiments writing to the same index contaminate each other's results, and rolling back code does not roll back the vectors already written RAG CI/CD and release gates. Give the real suite its own fresh index or namespace, seeded from a committed fixture set. Namespaces are the standard isolation mechanism in vector databases and are the recommended pattern unless you need cross-tenant queries Pinecone on multi-tenancy.

Compare against a recorded baseline, not an absolute bar. Store the previous run's per-query ranks and scores, and fail on a change beyond a tolerance you chose deliberately. A worked example: you have 120 labeled queries and assert that the correct document appears in the top 3. Run one records 108 hits (90 percent). You then change chunk size from 512 to 1,024 tokens. The nightly run records 101 hits (84 percent) and the gate fails, which is the system doing its job, because the drop is what a user would feel. Without a baseline, 101 out of 120 looks fine in isolation and ships.

Split the commands and schedule them differently.

# every commit, no network, seconds
pytest tests/unit tests/integration --embedder=hash

# merge to main and nightly only
pytest tests/quality --embedder=real --baseline=baseline.json --fail-on-drop=0.03

Three operating rules keep this cheap and quiet. Run the real suite on merge to main and nightly, not per commit, so a bad provider day does not block a hotfix. Keep the labeled query set small and curated, on the order of a hundred queries, because its value comes from stability rather than size. And record the model identifier and chunking parameters in the baseline file so that a legitimate model change shows up as an explicit baseline update in review rather than a mysterious red build.

Seeding discipline matters for the same reason it matters in staging databases: fixtures that do not resemble production in scale and messiness produce tests that pass and predictions that fail choosing a seeding strategy for staging. A hundred clean paragraphs from the docs folder will not exercise the chunk boundaries your support tickets create. Anonymized or synthetic-but-realistic text, kept out of the repo if it contains anything sensitive, is the better input handling PII in staging databases.

If the real suite is slow because you are calling several providers, or you want one client that swaps models without rewriting the harness, a single OpenAI-compatible endpoint for text, image and embedding models removes a layer of glue from the test code: Quanta Inference (https://quantaintelligence.ai/inference) exposes embeddings and other model types behind one key, which makes the baseline file portable across providers.

The rule is good, and it has edges worth naming.

Model swaps are not caught cheaply. If a provider retires a model, your dimension test may pass while your retrieval quality collapses. That is a real suite job, and it means running the real suite as part of the upgrade checklist, not just nightly.

Cost and rate limits bite at the edges. A labeled set of a few hundred queries against a paid embedding API is trivial; embedding a 50-million-chunk corpus in CI is not. Keep the real suite limited to queries and a small fixture corpus, and never use it to validate a full reindex. Full reindex validation belongs in a staging job Statsig on staging environments.

Non-determinism inside the model. Some serving stacks quantize or batch differently, which can shift scores slightly between runs. That is why the gate uses a tolerance and a rolling baseline rather than exact equality, and why serving factors like quantization and cache policy are better caught by nightly runs and production monitoring RAG CI/CD and release gates.

Hybrid pipelines blur the signal. If your retrieval fuses embeddings with keyword search and a reranker, a regression in the real suite may come from any stage. Isolate by running the embedding-only stage against the same labeled set when a gate fails, or you will spend the afternoon changing chunk sizes to fix a reranker.