Choosing an Embedding Model by Dimensions, Cost and Fit

Run the numbers for your own corpus before you look at quality. Suppose you have 800,000 chunks and store float32 vectors.

The bytes-per-vector column is arithmetic (dimensions times 4 bytes for float32). The relative search cost column is a rough scaling of distance computation work per candidate, useful for sizing rather than capacity planning. Two conclusions follow.

First, moving from 1536 to 384 dimensions cuts vector storage by 75% and reduces distance work correspondingly. That is the kind of change that turns a memory-bound index into one that fits comfortably in RAM with room for growth.

Second, dimension alone does not cap you at small corpora. A theoretical analysis in the LIMIT paper estimates that even 1024-dimension embeddings face a breaking point at roughly four million documents, and real models hit the wall sooner the reranker selection guide citing that paper. If your corpus is far below that scale, dimension is a cost lever, not a quality ceiling. Test the smaller option.

Two practical notes. Quantizing float32 to int8 quarters storage again and is usually acceptable for retrieval, but measure recall after quantization rather than assuming it. And if your vector database supports it, ask whether your candidate model produces vectors you can truncate, so you can index at full size and query at a smaller one, or the reverse.

Embedding cost has two shapes and they behave differently.

Indexing cost is proportional to corpus size times average chunk tokens, divided by a million, times the per-million-token price. It is paid once per model, and again whenever you re-chunk or change models. A bake-off with three candidates means three full embeddings of the corpus, so keep the golden set and the evaluation corpus separate from the production corpus during exploration.

Serving cost is query length times per-million-token price times query volume. For short queries this is usually small, but it is not zero at scale, and it interacts with reranking decisions. Adding a cross-encoder reranker on top of retrieval adds 20 ms to 150 ms of latency per query for a 50 to 100 document candidate pool the reranking implementation guide. Reranking is often the better place to spend compute than a larger embedding model, because it is a second-stage operation applied to a shortlist, not a full corpus scan the hybrid search and reranking reference.

A simple decision rule: choose the embedding model that maximizes recall@10 at stage one, then decide separately whether to spend latency budget on reranking. If recall@10 is already above your threshold with the cheap model, a reranker buys precision, not recall, and its value depends on whether your queries are complex. Reported gains from reranking are larger for multi-hop (+47%), complex (+52%) and ambiguous (+41%) queries than for simple fact lookup (+18%) the cross-encoder reranking study.

A golden set is a fixed list of queries, each with the passages that should be retrieved. Without one, every model comparison is an opinion.

Target 100 to 300 queries. Fewer than 100 makes the confidence interval on recall@10 too wide to distinguish close candidates. More than 300 is usually hard to label by hand and slow to review.

Sources for queries, in order of preference: real search logs or support tickets, questions written by the people who answer them, then synthetic questions generated from your documents and reviewed by a human. RAG projects are usually domain specific and built on private data, so public benchmarks rarely substitute for a custom set the path to a golden dataset. Synthetic question generation is a legitimate way to bootstrap the set, but label relevance yourself the RAG evaluation guide.

For each query, record the chunk IDs that genuinely answer it. Graded labels (for example 2 for a direct answer, 1 for useful context, 0 for irrelevant) let you compute nDCG, which rewards putting the best result first. Binary labels are simpler and enough for recall.

A minimal record looks like this:

{
  "query_id": "q014",
  "query": "how do I reset the admin password for the on-prem appliance",
  "relevant": [
    {"chunk_id": "runbook-onprem-12", "grade": 2},
    {"chunk_id": "faq-auth-03", "grade": 1}
  ]
}

Keep the golden set under version control and never tune against it repeatedly without adding fresh queries. It is a test set, not a training set.

Recall@k measures the fraction of queries where at least one labeled relevant chunk appears in the top k. It answers a blunt question: would the right document reach the model at all? In a worked example with 100 queries, 83 of them had their target document in the top 5, giving recall@5 of 0.83 the retrieval metrics tutorial.

nDCG@k adds ordering. Each result contributes its relevance grade discounted by position, then the total is normalized against the ideal ordering. A query where the grade-2 chunk sits at position 1 scores higher than one where it sits at position 4, even though both count for recall.

Run the numbers on a small example:

Recall@10 is 1.0 for both models on all three queries: every labeled chunk appears inside the top ten. nDCG tells a different story. Model B places the grade-2 answer first or second on every query, while Model A buries q1's best answer at position 1 but drops q3's at position 6. Model B wins on nDCG despite tying on recall.

This is the central reason to report both. Recall tells you whether the pipeline can find the answer; nDCG tells you whether the answer arrives early enough to matter after truncation to top 5 for the generation step. Benchmark well, and use the Openlayer summary of metric categories as a checklist: retrieval quality (recall, MRR, NDCG), semantic similarity, clustering quality, and performance metrics like throughput, p95 latency and memory footprint how to benchmark embedding models.

Evaluating retrieval separately from generation is standard practice and makes failures diagnosable: when an answer is wrong, you can tell whether retrieval missed or generation hallucinated the RAG evaluation guide the Redis RAG evaluation guide.

Pick two or three candidates, not eight. A useful slate: one small model (in the 256 to 384 dimension range), one strong general model, and one model that claims domain fit or that you have seen perform on similar data. Add a truncated Matryoshka variant if one of the candidates is Matryoshka.

Hold everything else constant. Same chunking, same chunk IDs, same golden set, same k. If you change chunking between candidates, you are no longer comparing embedding models.

The evaluation loop, in outline:

1. Chunk the corpus once with your chosen strategy and record chunk text and IDs.
2. For each candidate, embed every chunk and build an index in a scratch collection.
3. For each golden query, embed the query with the same model, retrieve top 50, and compute recall@10 and nDCG@10 against labels.
4. Record p95 query latency and an estimate of storage per candidate.
5. Repeat at 2 to 3 dimension settings for any Matryoshka candidate.

A compact scoring harness:

def evaluate(index, embed, golden, k=10):
    recalls, ndcgs = [], []
    for row in golden:
        qv = embed(row["query"])
        hits = index.search(qv, k=50)[:k]
        relevant = {r["chunk_id"]: r["grade"] for r in row["relevant"]}
        recalls.append(int(any(h in relevant for h in hits)))
        dcg = sum(relevant.get(h, 0) / math.log2(i + 2) for i, h in enumerate(hits))
        ideal = sorted(relevant.values(), reverse=True)[:k]
        idcg = sum(g / math.log2(i + 2) for i, g in enumerate(ideal))
        ndcgs.append(dcg / idcg if idcg else 0.0)
    return {"recall@10": sum(recalls) / len(recalls),
            "ndcg@10": sum(ndcgs) / len(ndcgs)}

Read the results with a decision rule, not a ranking. If the small model is within a couple of points of nDCG@10 of the large one on your set, take the small model: it saves storage, lowers latency and simplifies the index. If the small model loses badly, look at the failing queries before concluding the model is at fault. Often the failure is chunking: chunks that mix multiple ideas produce noisy averaged embeddings that retrieval cannot match, and quality collapses below roughly 16 tokens per chunk for every model tested the embedding model decision framework the chunking strategies guide.

One caveat on the decision rule: differences of a couple of nDCG points sit inside labeling noise on a 100-query set. When two candidates are close, add 50 more queries from the same distribution and re-run rather than splitting hairs on the first pass.

If you run hybrid retrieval, the embedding model competes with a lexical ranker, and fusion can rescue a weak dense model. Reciprocal rank fusion combines ranked lists by position rather than raw score, so an unbounded BM25 score and a bounded cosine similarity never need to be normalized onto a shared scale the OpenSearch RRF introduction. A document that ranks near the top of either list gets credit, which means exact-match queries that defeat a dense model can still succeed through the keyword side the RRF explainer the MongoDB RRF guide.

The practical implication for model selection: evaluate candidates inside your actual fusion setup, not in isolation. On the WANDS e-commerce dataset, baseline BM25 and pure KNN were statistically indistinguishable at NDCG 0.6983 versus 0.6953, while a tuned hybrid reached 0.7497, a 7.4% lift over either alone the hybrid search reference. A dense model that ranks second on its own may contribute more inside a fused pipeline than one that ranks first alone.

Run the bake-off twice if you plan to ship hybrid: once dense-only, once with fusion. The winner can differ, and the fusion run is the one that predicts production.

An embedding decision has a shelf life bounded by two events: the corpus changing shape and the query distribution changing.

Corpus drift matters because chunk boundaries and domain vocabulary shift as content grows. A model chosen on 50,000 support articles may behave differently on 200,000 that include tickets, transcripts and forum threads. Sentence-length distributions matter: models tuned for short, clean passages can underperform on long contracts or multi-paragraph tickets how to benchmark embedding models.

Workload drift matters because query mix drives which metric matters. If your traffic shifts toward complex, multi-hop questions, the value of a reranker rises, and the optimal first-stage model may change with it the cross-encoder reranking study.

A simple cadence: re-run the golden set whenever you re-chunk more than 10% of the corpus, when you add a document type you have not indexed before, or when recall@10 on a fresh sample of 50 queries drops. Add each new failing query to the golden set so the set grows toward your real workload.

When the re-test happens, the mechanics are the same as the first bake-off, which is the argument for building the harness early and keeping it in the repository. If you are running candidates across many model types (text embeddings alongside image or audio encoders for multimodal retrieval), putting them behind one API key removes the per-provider glue from the evaluation loop, which is what Quanta Inference is for.