What a Re-Embedding Job Costs in Tokens and API Calls

The two strategies differ in what multiplies against what.

Run the numbers with a 10 million chunk corpus at 480 tokens per chunk. A full rebuild is 4.8 billion tokens. At $0.02 per million, that is $96. An incremental job at 2% monthly churn touches 200,000 chunks, or 96 million tokens, or $1.92. That is 50 times less. At 50% churn the incremental job is 2.4 billion tokens, or $48, and the gap shrinks to 2x. The decision rule follows: incremental wins whenever churn times the cost of one corpus pass stays below the cost of a full pass plus the value of freshness. In plain terms, incremental wins below roughly 100% churn, which is almost always.

The interesting question is not the token bill. It is the quality cost of the stale vectors that incremental leaves behind.

There is one more shell in the cost model that teams forget. Index writes and deletes are not free at the vector database layer, and neither is the compute that turns chunks into embeddings if you run a local model rather than an API. Model size matters as much as token volume: choosing the smallest model that passes your evaluation is a cost decision, not a quality compromise.

An incremental job is only as cheap as its change detection is precise. Too coarse, and you re-embed too much. Too fine, and you miss changes.

Simple approaches are timestamps and content hashes, but a single-byte edit changes a hash, and dynamic pages (a rendered date, a rotating banner) change on every fetch. Practitioners on Stack Overflow describe exactly this problem and the usual mitigation: combine a hash with a content-length check, and strip known-volatile regions before hashing (Stack Overflow discussion on detecting page changes). A syllable-level variant, MinHash or SimHash, keeps the hash stable for small edits so you can decide whether the change is material.

A better change signal comes from chunk boundaries, not documents. Hash each chunk text, not the document. When a paragraph is added in the middle of a document, only the chunks from that point to the next stable boundary change; everything before it keeps its hash. This is where chunking strategy pays back twice. Hierarchical or structure-aware chunking produces boundaries that survive edits better than fixed-size windows, and structure-aware methods have been shown to improve retrieval quality on long documents (research on hierarchical text segmentation chunking).

A useful practice: keep a metadata table with chunk_id, content_hash, embedded_at, and model_version. Three comparison rules then do all the work. A chunk is dirty if its hash changed, if its model_version is behind the current embedding model, or if its embedded_at is older than a policy window for high-volatility sources. The first two rules catch content and model changes; the third catches changes your hashing cannot see.

This is the heart of the article and the part most cost writeups skip. Incremental re-embedding keeps the index cheap, but it also keeps vectors that are semantically wrong for their current text. The cost model needs a quality term, and the quality term needs a measurement.

Build a fixed evaluation set. Roughly 100 to 300 queries with known correct source chunks, refreshed only when the product changes. That set is the ruler. Run it weekly. For each query, record whether the top-k result is a chunk whose source has changed since it was embedded. The fraction of queries where a stale chunk wins is your stale-vector miss rate.

The decision rule uses it directly:

• Under about 2%: keep incremental. The miss rate is noise relative to the cost of a full pass.
• Between 2% and 5%: tighten change detection and re-embed the affected sources; hold the full rebuild in reserve.
• Above 5%: run the full rebuild. At that level, the retrieval system is returning outdated answers with confidence, a failure mode that produces no error and no warning and that standard faithfulness evaluation will not catch, because the answer is faithful to what was retrieved (analysis of the staleness gap).

Where does the 5% come from? It is not sacred. Justify it with your own consequences. If a wrong answer costs a support agent five minutes to correct, and you can measure queries per week, the miss rate converts to hours per week. Compare that to the dollar cost of a full pass at your corpus size and rate. The rebuild wins when the labor cost of stale answers, over the period until your next rebuild, exceeds the rebuild's token bill. Run that comparison with your own numbers; the ratio of the two sides is usually wide enough that a rough estimate is safe.

Measure with a proper metric, too. Precision, recall and F1 on span retrieval against gold labels are the standard frame for attribution-adjacent tasks (citation extraction and evaluation), and the same discipline applies to measuring whether retrieval found the current document rather than the old one. Freshness scoring frameworks split this into content age, embedding lag, stale retrieval rate and coverage drift, which gives you four dials instead of one (Atlan's freshness scoring). The stale retrieval rate is the one that drives the rebuild decision.

Worked example. A 10 million chunk corpus at 480 tokens costs $96 per full pass at $0.02 per million tokens. Your evaluation set shows a stale-vector miss rate of 7%. That is above the 5% line, so the rule says rebuild. The rebuild costs $96 and some downtime. The stale answers had been a silent weekly cost for months. The full pass is the cheap side of the trade.

Token math gets you the bill; call math gets you the job. Two jobs with the same token count can take very different wall clock times and very different retry costs.

Write the scheduler as four parameters you can tune: batch size, concurrency, retries, and timeout. Then measure the empirical average batch fill, because imperfect packing wastes headroom.

def estimate_job(chunks, tokens_per_chunk, rate_per_million,
                 batch_size, concurrency, retry_rate=0.02):
    tokens = chunks * tokens_per_chunk
    token_cost = (tokens / 1_000_000) * rate_per_million
    calls = -(-chunks // batch_size)          # ceiling, perfect packing
    effective_calls = calls * (1 + retry_rate)
    batches_per_wave = concurrency
    waves = -(-int(effective_calls) // batches_per_wave)
    return {
        "tokens": tokens,
        "token_cost_usd": round(token_cost, 4),
        "api_calls": calls,
        "effective_calls": round(effective_calls, 1),
        "waves": waves,
    }

Feed it the full corpus and the churned slice separately. The difference in waves is your wall-clock saving, and the difference in token_cost_usd is your bill saving. Neither tells you whether to run the job. The miss rate does.

Provider pricing often has a batch tier at half the standard rate, visible on the OpenAI pricing page and comparable aggregators. That matters for rebuilds, which are tolerant to hours of latency, and matters less for incremental jobs, where latency has business cost. A rebuild that runs overnight in batch mode is often cheaper than a daytime incremental job that touches a tenth as many tokens, once the rate tier is applied. Check the tier, not just the model.

Most teams run the same job over and over because they never set a cadence. Set one from the miss rate, not from a calendar habit.

1. Pick the smallest model that passes your evaluation. Every extra point of quality at a higher token rate multiplies across the whole corpus on every rebuild.
2. Route embedding calls through one gateway. A single OpenAI-compatible endpoint for embeddings, generation and reranking removes the per-provider key management and lets you compare candidates on real chunks without rewriting pipelines. Quanta Inference (https://quantaintelligence.ai/inference) offers this across text, image, audio and embedding models.
3. Set the cadence. Recompute the stale-vector miss rate weekly. Rebuild when it crosses your threshold, not on a fixed schedule.
4. Keep the ledger. Log tokens, calls, retries and cost per job. The ledger is what makes the next budget estimate one multiplication instead of a guess.

A cost model is only as good as its inputs, and the inputs are cheap to measure. Run one incremental job and one small sample rebuild at your real chunk size, log the tokens and calls, and you will have a number that beats any estimate in an article. That is the last point worth emphasizing: the difference between a $2 monthly bill and a $200 one is almost always the churn rate and the rate tier, not the model's reputation.