Switching an Embedding Provider Behind a Gateway Route

Before any traffic moves, the new provider must be a first-class citizen in the gateway with the same observability as the incumbent. Four things need to exist:

1. Credentials scoped to the provider, held by the gateway and never by application code. Per-key allow-lists and budgets are the standard control here in gateway designs.
2. A normalized embed interface: input texts in, vectors plus metadata out. Your application should never see the provider's native payload.
3. Capability metadata: vector dimension, whether outputs are unit-normalized, max batch size, max input tokens, and whether the provider supports an explicit output-dimension parameter.
4. Telemetry: latency histogram, error rate, token or character count, and cost attribution per route.

That third item is where swaps die. Record dimension and normalization as declared facts about the provider, and validate them at runtime on the first response. A cheap assertion saves a rebuild:

def assert_contract(resp, expected_dim, expect_normalized):
    v = resp.vectors[0]
    if len(v) != expected_dim:
        raise ContractError(f"dim {len(v)} != {expected_dim}")
    if expect_normalized:
        norm = sum(x * x for x in v) ** 0.5
        if abs(norm - 1.0) > 1e-3:
            raise ContractError(f"norm {norm:.4f} not unit")

Run that check in a preflight call against the candidate provider before the route exists. It costs one request and tells you whether the candidate can serve your index at all.

A route key should be built from dimensions that already exist in your request context, not invented for the migration. The practical minimum is product, environment, and task. Task matters because "embed a document for indexing" and "embed a user query for search" may intentionally use different models later.

Two rules make this durable. First, routes are data, not code: the mapping lives in gateway config with a reviewed change process, so a swap is a config diff. Second, shadow routes never write. A mirrored request goes to the candidate, the response is logged and discarded, and nothing reaches the vector store. That keeps the comparison honest and the rollback instant.

Pin the route key into the request itself, so a response can be attributed even if the config changes mid-flight. This is the same discipline that makes provider failover traceable in a gateway that logs every routing decision per this model gateway description.

A model version stamp is the single most valuable field in this whole exercise. Store it in three places:

• On the gateway response, as a header or metadata field, alongside provider, route key, and timestamp.
• In the vector row, as a column or payload field next to the embedding itself. Every row needs to answer "which model produced this?" without a join.
• In the application's write path, so the code that inserts vectors is forced to read the stamp rather than guess.

Treat the stamp as a composite: provider, model identifier, dimension, and normalization policy. Two identifiers are physically incompatible for search purposes even when the dimension matches, because distance comparisons across embedding spaces are meaningless as the dimension mismatch write-up explains. A bare model name does not capture that; the full stamp does.

Two operational consequences follow. First, your schema must allow multiple stamps to coexist during migration: either a per-row stamp column with the index filtered by stamp, or a separate table or collection per stamp. Second, queries must be routed to the index that matches the query embedding's stamp. If the query is embedded by the candidate and the documents by the incumbent, you get either an error or silent garbage. Enforce the pairing in the query path, not in a code review.

The stamp also gives you a free cost and quality ledger. Grouping stored vectors by stamp tells you how much of the corpus each model has covered, which is exactly the number you need to decide whether a full re-embed is worth it.

Do not evaluate the candidate on public benchmarks alone. Leaderboard rankings lose predictive value once your data has domain vocabulary, long documents, or fine-grained labels that differ from the benchmark corpora per this embedding benchmarking guide. Build a golden set from your own queries and documents.

The mechanics are unglamorous but effective:

1. Freeze a test corpus slice. Pick a few thousand representative documents from the product being moved, including the long, the short, and the ugly.
2. Freeze a query set with known answers. A golden dataset is a fixed set of test cases you run before every meaningful change per this golden dataset guide. For retrieval, the practical minimum is a few dozen query-document pairs drawn from real user queries as this retrieval evaluation guide puts it.
3. Embed everything twice, once per provider, into separate collections stamped with their respective versions.
4. Measure retrieval, not answers. Compute recall@k, MRR, and nDCG@k against the ground truth. Recall@k asks whether the relevant document appears in the top k at all, which matters most when a miss has a real cost per Openlayer. NDCG accounts for graded relevance, MRR rewards putting the best result first again per Openlayer.
5. Mirror live traffic on the shadow route at a small percentage and log both responses. Compare the top-k sets for the same queries as they occur.

There is one subtlety worth knowing. Recall@k measures overlap with the true nearest neighbors of the candidate's own embedding, so a model can have excellent recall and still retrieve worse documents. A recent study argues that 1/Ratio@k, the inverse approximation ratio between retrieved and true distances, tracks downstream utility better than recall and can be computed from standard benchmark inputs per this ANN evaluation paper. If recall differences are small, compute both before you call it.

Set an explicit gate. A defensible starting rule: do not flip unless recall@k on the golden set is at least as good as the incumbent, and the mirrored live traffic shows no drop in top-k agreement beyond a threshold you agreed to in advance. Automate it. Qdrant's tutorial shows the pattern for a CI job that compares approximate and exact top-k results and fails when recall falls below target, which catches regressions from embedding model swaps before production per Qdrant. The same harness, pointed at two collections instead of one, is your side-by-side test.

For a worked example, suppose the golden set is 120 queries and the incumbent's recall@10 is 0.94. The candidate scores 0.91 on the same set. That is within the noise you would expect from 120 queries, but it is not evidence of parity. Widen the set to a few hundred queries, re-run, and only then decide. If the candidate lands at 0.93 or better with a stable confidence interval, and the mirrored traffic agrees, flip the staging route and watch it for a week.

Two contracts decide whether a swap is a routing change or a data migration.

Dimension. If the candidate's dimension differs, you cannot reuse existing vectors in the same index and the choice is binary: a new index per stamp, or a full re-embed. The cost is dominated by re-embedding the corpus, not by the new model's per-token price, so estimate before you commit. OpenSearch's memory documentation gives a concrete per-vector figure to base store sizing on: a 768-dimensional float32 vector occupies 3,072 bytes before index overhead, and compression levels reduce that by a stated factor, down to 96 bytes per vector at 32x per the OpenSearch docs. Use that to size the new collection, not the old one's footprint.

Normalization. If one provider returns unit-normalized vectors and the other does not, cosine and dot-product results will drift even if the dimension matches. Declare the policy per provider, normalize at the gateway if needed, and assert it in the contract check above.

If the dimension matches and the corpus is large, an alternative to a full re-embed is to write new vectors only for new and updated content, tagged with the new stamp, and run two indexes in parallel until content ages out. That is a real option, but only if your query path can search both indexes and merge with a consistent scoring rule, which brings you back to needing the stamp on every row.

Re-embedding is also the moment to revisit index type. HNSW reaches high recall out of the box and absorbs inserts without rebuilds but costs more memory than IVFFlat, which builds faster and uses less memory but drifts as data changes per this index comparison. If the new vectors are written gradually behind a shadow route, HNSW's incremental behavior is usually the safer fit. Parallel HNSW index builds have been reported to cut build time substantially per Neon's writeup, which matters when you are building a second index for the new stamp next to production traffic.

The flip itself should be a one-line config change: point the product's route at the candidate. Two preconditions make that safe.

• The old route stays defined. Do not delete it. Rollback is pointing the route back, which is why the shadow phase wrote nothing.
• The product's data is separable. Documents written under the new stamp go to the new index or the new partition. Queries route by stamp. Old data remains queryable under its old stamp while it drains.

Watch four signals for the first week: retrieval metrics on the golden set run against live data, top-k agreement between the two indexes on sampled queries, embed error rate per route, and cost per thousand requests attributed to the route. A gateway that tracks token usage and upstream model cost per key gives you the last one for free per Chalk's gateway features.

Then repeat for the next product. Each repetition is cheaper because the provider is already registered, the contract check already exists, the golden set harness is reusable, and the stamp schema is already in place. That compounding is the actual goal of the exercise: the second swap should be a config change and a test run, and the third should be almost routine.

If you are still wiring the gateway itself, there is no reason to build provider adapters for text, image, audio and embedding models one at a time. Quanta Inference exposes text, image, video, audio and embedding models through one OpenAI-compatible API key with prepaid dollar pricing, which gives you the route target without maintaining a provider adapter for every candidate you want to test: Quanta Inference.