Embedding Model Selection: Dimensions, Cost and Domain Fit

Every dimension in a float32 vector costs 4 bytes. Do the arithmetic on your own corpus before reading any model card:

These are raw vector bytes at 4 bytes per dimension, computed with the same method used in this analysis of the vector dimension tax. Multiply by two or three for index overhead and replicas in a hosted vector database. The same analysis notes that similarity search scales with dimension (a 1536-dimension comparison is roughly four times the work of a 384-dimension one) and that graph indexes such as HNSW get less effective as dimensions climb, because in very high dimensions points become nearly equidistant and the graph gives weaker guidance.

The quality side is flatter than most teams assume. On Matryoshka-trained models, which are trained so that early dimensions carry the most important information, dropping from 768 to 256 dimensions cost less than two points of average MTEB score in the figures reported in that same analysis, and OpenAI found that a truncated 256-dimension embedding still beat its earlier full-size model on MTEB. Elaborate compression is rarely needed either: a study on MTEB found that float8 quantization cut storage 4x with under 0.3% degradation, and that combining moderate PCA (keeping about half the dimensions) with float8 reached 8x compression with less quality loss than int8 alone in this embeddings storage optimization paper.

The decision rule that follows: choose dimensions for your cost ceiling, then check that the model still clears your recall bar. Do not pay for dimensions you have not measured a need for.

Headline per-million-token prices hide the two numbers that set your bill: how many tokens you embed up front, and how often the corpus changes. Work it out on a scratchpad:

initial_tokens   = chunks x avg_tokens_per_chunk
monthly_tokens   = initial_tokens x refresh_rate
query_tokens     = queries_per_month x avg_query_tokens
year_one_tokens  = initial_tokens + 12 x monthly_tokens
cost             = year_one_tokens / 1_000_000 x price_per_million

Worked example. You have 200,000 chunks at 400 tokens each. That is 80 million tokens of initial ingest. If 5% of chunks change monthly, refresh adds 4 million tokens a month, or 48 million over a year. Queries are small: 100,000 queries a month at 30 tokens is 3 million tokens. Year one totals about 131 million tokens. If candidate A costs three times candidate B per token, the difference on 131 million tokens is three times B's cost multiplied by 131 million divided by one million. Note what dominates: refresh volume, not query volume, which is why a provider comparison that models refresh cadence over 12 months reframes the decision away from the one-off ingest price in this vendor cost comparison. That source documents a 3x to 5x spread across vendors, with per-token list prices that change, so read current numbers from each vendor's pricing page rather than repeating them.

Re-embedding is the hidden cost. Embedding spaces are incompatible between models, so switching means re-encoding the whole corpus. On a large corpus that is a multi-day job, as the dimension analysis notes, so treat the model choice as a migration you want to make at most once or twice.

Public benchmarks such as MTEB help you build a shortlist, but their datasets rarely look like your corpus, and an accurate model can still be too slow or expensive to run in Qdrant's model selection guide. Worse, standard retrieval benchmarks score whole-document relevance, not passages, so they cannot account for your chunking at all in Chroma's chunking evaluation.

Domain fit shows up in unglamorous places. Tokenizers matter: the guide above documents a WordPiece tokenizer replacing an unknown word with a single unknown token (so an emoji-heavy post loses its words), lowercasing and accent stripping that make "crème brûlée" and "creme brulee" identical, and byte-level tokenizers that never hit the unknown-token case at all. If your corpus is full of part numbers, product codes, chemical names or non-English text, test a chunk containing them and look at what the model does with it. Also check that the model's stated language coverage matches your data; a demo-friendly English-only model is the wrong pick for a mixed-language support base.

Domain-trained variants exist for code, finance and law, and are reported to beat generalists on recall in those sectors in the vendor comparison above. Treat that as a reason to include one domain-trained candidate in your test, not as a reason to skip the test. Academic work also cautions that the more elaborate strategy does not always win: a study on long academic theses found cluster-based semantic chunking did not outperform simpler strategies under the tested configuration in this arXiv paper. Model choice behaves the same way. The expensive, sophisticated option has to earn its place on your data.

You need 50 to 200 query-chunk pairs. Fewer than 50 is too noisy; beyond 200 the marginal information is small compared with the effort of labeling. Build the set like this:

1. Pull real queries from logs, support tickets or search analytics if you have them. Otherwise have someone who knows the corpus write questions a real user would ask.
2. For each query, mark the chunk or chunks that genuinely answer it. Label at the chunk level, not the document level, and use the chunking you actually plan to ship.
3. Include the hard cases deliberately: exact identifier queries, acronyms, questions phrased in words that do not appear in the source text. Semantic embeddings miss exact matches such as an error code, which is why hybrid search with lexical matching (BM25) exists and why Anthropic reports such large gains from contextual BM25 in the contextual retrieval writeup. Dense-only scores will overstate any model's real-world performance, so decide up front whether you will pair embeddings with a lexical path.
4. Freeze the set before you start scoring, and split off a quarter as a holdout so you do not tune to your test queries.
5. Record, per candidate model: recall at your target k (does the right chunk appear in the top 5 or top 10?), mean reciprocal rank if ranking order matters to you, storage bytes per chunk, ingest cost and cost per 1,000 queries.

Use pass at k as the headline metric. The Claude Cookbook's evaluation measured Pass@k, the share of queries whose golden chunk appeared in the first k results, and it is simple enough to explain to a product owner. Rank each candidate like this:

For "within noise," be strict. If the two recall numbers differ by less than a few percentage points across a 100-pair set, that is close to a coin flip and you should take the smaller, cheaper model. One more lever before you swap models: reranking a wider candidate set with a small cross-encoder often buys more than a model upgrade. Qdrant's tutorial shows reranking a shortlist from a cheap first-stage search, which keeps latency and cost down while improving precision in the Qdrant reranking tutorial. If a cheaper embedder plus a reranker beats your expensive embedder alone, that is your pipeline.

If you are embedding through more than one vendor during the trial, a single OpenAI-compatible endpoint saves you writing two clients and two retry paths; Quanta Inference serves text and embedding models through one API key with prepaid US-dollar billing (https://quantaintelligence.ai/inference). Whatever you use, keep the comparison honest with the same code path:

def evaluate(model, pairs, embed_fn, top_k=10):
    index = build_index(corpus_chunks, embed_fn(model))
    hits = 0
    for query, golden_ids in pairs:
        ranked = index.search(embed_fn(model, query), k=top_k)
        if any(doc_id in golden_ids for doc_id in ranked):
            hits += 1
    return hits / len(pairs)

Run it for both candidates, on the held-out quarter, with the same chunk set and the same top_k. Record the token counts you actually sent; that number, not a pricing page, is your cost input.

Comparing models on different chunk sets equalizes nothing and tells you nothing. Changing the chunker between tests invalidates the comparison, because chunk choice alone shifts retrieval accuracy. Chroma found that default settings for some popular chunking strategies perform poorly, and that a well-parameterized recursive splitter often wins in practice in its chunking evaluation. NVIDIA's chunking experiments found page-level chunking the most consistent across its datasets while the best token size varied by document type, with 512 tokens best for one earnings dataset and 1,024 for another in NVIDIA's chunking study. Lock the chunking, then compare.

Testing only on easy queries hides the difference between models. Testing on five pairs produces a winner that is pure noise. Testing only semantic lookups hides the exact-match failures that hybrid retrieval fixes. And picking the biggest model because it is the biggest ignores that dimension count mostly buys cost, not quality, on typical corpora.

There is also a scale question worth asking early. If your entire knowledge base fits in roughly 200,000 tokens, about 500 pages, Anthropic's writeup suggests skipping retrieval and putting the whole corpus in the prompt with caching, which is a smaller engineering problem than any embedding pipeline in the contextual retrieval writeup. Build the evaluation harness when the corpus outgrows that, not before.