Deciding Whether a New Embedding Model Earns a Full Reindex

A golden set is a fixed collection of test cases with known good answers that you run every candidate against. For retrieval, that means queries from your real users (or realistic ones), each paired with judged relevance labels for documents in your corpus. Teams commonly start with 50 to 200 query-document pairs drawn from the actual corpus and expected queries the DataAIHub retrieval evaluation guide. For a model-swap decision, aim higher: a few hundred judged queries gives you enough to see a real difference between two models.

Three rules make the set usable:

1. Judge relevance, not similarity. Binary "relevant or not" labels are enough for recall; graded labels (0 to 3) let you compute nDCG. A valid benchmark needs real production queries, ground truth relevance labels and identical inputs across every candidate model the Openlayer guide.
2. Keep it fixed. A golden set is the fixed set of cases you run before every meaningful change, so it must be versioned and never edited to make a candidate look good the Langfuse golden dataset guide.
3. Sample from the workload you serve. A representative sample of query vectors from your workload, embedded with the same model your collection uses, is the standard practice for recall tests the Qdrant recall tutorial.

Split the set. Use roughly two thirds for the go/no-go comparison and hold back the rest to confirm the winner after the rebuild. This protects you from picking a model that happens to fit your sample.

The evaluation procedure is simple to state and easy to get wrong. For each query in the golden set, retrieve the top k results from the incumbent index and from a candidate index built on the same corpus, then compute the metrics. The critical constraint is that k must be the number you actually serve, whether that is the top 8 passages into a prompt or the top 50 into a reranker. Recall at a k you never use tells you nothing.

Recall@k is the fraction of the true exact neighbors that the approximate index returned, and production targets are typically 0.95 or higher the BigData Boutique comparison. For model swaps you want two different recall numbers:

• Index recall, measured against exact search on the same model's vectors. This tells you whether your HNSW parameters are healthy, not whether the new model is better.
• Retrieval recall, measured against the judged relevance labels. This is the number that should drive the decision.

A degraded ANN index never fails loudly. It returns k plausible neighbors and never mentions that it left out several of the good ones, with no error and no log entry, which is why recall has to be measured against exact ground truth rather than inferred from latency or user complaints the pg_recall_guard documentation.

For ranking quality, add nDCG@k, which accounts for graded relevance and gives you a fuller picture when some results are more relevant than others, and MRR if the first result matters most to your product the Openlayer guide. A useful caution: recall@k measures overlap with the exact neighbor set, and studies show downstream task quality can stay stable even when recall@k drops significantly, because closeness in distance matters more than identifier matches the ANN Recall What Matters paper. Treat recall as one input, and let your judged labels and product metrics arbitrate.

A worked comparison, with numbers invented for illustration:

The decision rule this table suggests: Candidate B gains almost nothing for a full rebuild, so drop it. Candidate A gains 8 points of judged recall and 6 points of nDCG, which is large enough to plausibly move answer quality, but it costs a third more vector storage and one extra latency millisecond. Candidate A is worth pricing; B is not.

Set a threshold before you look at results. A reasonable starting point is that the candidate must beat the incumbent by at least 3 to 5 points of judged recall@k, and must not regress nDCG. Write it down so the decision is not negotiated after the numbers arrive.

Now the arithmetic. Three quantities drive the cost: embedding volume, graph build time and evaluation time.

Embedding volume is rows times dimension, in floating point numbers, then converted to bytes. At 4 bytes per float32 value:

volume_bytes = rows * dimension * 4

A 10 million row corpus at 1,536 dimensions is 61.4 GB of raw float32 vectors. At 1,024 dimensions it is 41 GB. If you store a half-precision type instead, halve it; pgvector users facing large builds are advised to consider the halfvec type or binary quantization with re-ranking to reduce index size the pgvector issue thread.

Graph build time is the part teams underestimate. HNSW builds are slower and more memory-hungry than list-based indexes, and raising the build parameters slows the build drastically while primarily improving accuracy the Crunchy Data explainer. Parallel builds changed the picture substantially: a parallel HNSW index build improved build time by a factor of 30 in tests on a 10 million row, 1,536-dimension dataset on a 64 vCPU, 512 GB RAM instance Neon's write-up of the benchmark.

Memory availability decides whether you get near that figure. When the graph no longer fits into maintenance_work_mem, pgvector warns that the build will take significantly time and advises raising that setting; one reported build on 40 million row partitions was still under 30 percent complete after more than a day, and the fix included removing LOCAL from the SET commands so the setting applied outside the transaction, using the default ef_construction of 64 for faster builds, and lowering the vector type the pgvector issue thread.

A simple estimator you can run on a sample:

-- Build a 1 percent sample and time the index creation
CREATE TABLE sample_vectors AS
  SELECT * FROM documents USING sample (1);

\timing on
CREATE INDEX ON sample_vectors
  USING hnsw (embedding vector_cosine_ops)
  WITH (m = 16, ef_construction = 64);
\timing off

Multiply the sample build time by 100 for a first-order estimate, then add 30 to 50 percent for the superlinear behavior that shows up as the graph grows. Do not skip the sample: a naive estimate that ignores the memory-fit warning can be off by an order of magnitude.

Evaluation time is the cheap line item and the one people forget. Rerunning the golden set takes minutes, but re-embedding a few hundred queries and re-judging any results whose label status changes can take a day of human time. Budget it.

The backfill cost table:

Two operational costs often missing from pricing pages are reindexing and embeddings, which can double real production spend when they are not planned the Actian pricing analysis. Put both on your cost sheet before you approve the project.

If you are evaluating several candidates, sending every corpus through one interface saves a lot of plumbing. Quanta Inference exposes text, image, video, audio and embedding models through a single OpenAI-compatible API key with prepaid US dollar pricing, so the candidate comparison and the eventual backfill use the same client code.

A clean backfill lands in a parallel collection or table, never in place. Serve the candidate index to a slice of read traffic or to internal users while the incumbent handles everyone else, then compare judged recall, nDCG and click or answer-accuracy metrics between the two. Gate the switch on the shadow matching or beating the offline numbers.

Practical rules for the shadow period:

• Keep both indexes current. Dual-write new and updated rows to both collections for the duration, or the candidates diverge and the comparison becomes meaningless.
• Automate the recall check. The same helper that compares approximate and exact results can be wired into CI and fail the job when recall falls below your target, which catches regressions from embedding model swaps before they reach production the Qdrant tutorial.
• Record the baseline you accept. Tools exist that approve a recall number and then report when it drifts; a degraded index gives no other signal the pg_recall_guard documentation.
• Watch memory and latency, not just quality. Compression levels and workload modes trade recall and latency for memory, and a candidate that needs 2 to 5 times the memory of a list-based index changes your hosting bill even when quality improves the BigData Boutique comparison, the OpenSearch memory optimization docs.

Decide the rollback in advance: keep the old collection for at least one full business cycle after the switch, and record the exact model, dimension, build parameters and evaluation run that produced the approved index so a future rebuild is a repeatable operation rather than archaeology.

Combine the two halves into a single gate. Run it in order, and stop at the first failure:

1. Golden set exists and is judged. Few hundred queries, relevance labels, held-out split. No set, no evaluation.
2. Candidate beats incumbent on judged recall@k and nDCG@k by your pre-registered margin, measured at the k you serve.
3. Candidate does not regress index recall measured against exact search, and does not blow your latency budget.
4. Backfill estimate is acceptable: embedding volume, sample-based build projection, evaluation time, shadow period, and the storage or memory delta.
5. Shadow run confirms the offline gain on a slice of traffic.
6. Rollback plan and recorded baseline exist before traffic moves.

If step 2 fails, keep the current index and revisit when a candidate clears the bar. Most candidates will not, and that is the system working. It is far cheaper to reject ten models on a few hundred judged queries than to rebuild a production index ten times.

If the work of maintaining the golden set, the CI recall gate and the shadow harness is more than your team can staff, that is a signal about scope rather than quality. Quanta Intelligence builds sites, apps and internal tools with a specialist who works alongside you, which covers the evaluation dashboard and shadow-comparison plumbing without pulling engineers off the product. For the narrower case of specialist retrieval work, an AI expert for your line of business can own the golden set and the reindex runbook.