When a Reranker Pays for Itself Against a Bigger Cap

Two formulas cover most of it.

Rerank cost per query = (N / 1,000) x P, where N is the number of candidate documents you score and P is the reranker price per 1,000 documents.

Context cost saved per query = (K_large - K_small) x T x C, where K_large is the cap you would need without a reranker, K_small is the cap you keep with one, T is average tokens per retained row, and C is the generator's input token price.

The decision rule: if rerank cost per query is less than context cost saved per query, the reranker pays for itself. If it is more, raise the cap instead.

A third term is worth tracking: latency. Real reranker comparisons report median latency alongside score and cost, for example 0.18 seconds for one reranker and 0.28 for another on the same 50-candidate setup Kapa's reranker benchmark. Reranker latency is a line item, not a rounding error, especially in voice or interactive search.

Worked example 1: short factoid support queries

Suppose you retrieve N = 50 candidates, keep k = 5, and the reranker is priced at $2.00 per 1,000 documents scored. That is 50/1,000 x $2.00 = $0.10 per query.

Now suppose the no-reranker alternative needs a cap of 20 rows to hit the same answer quality. Each row of a short FAQ chunk is about 150 tokens, and the generator charges $0.50 per million input tokens. Saved tokens per query are (20 - 5) x 150 = 2,250. Saved cost is 2,250 / 1,000,000 x $0.50 = $0.001125.

Reranking costs $0.10 to save $0.001. That is roughly 89 times more expensive than the tokens it removes. Raise the cap. For short chunks, the reranker cannot pay for itself on token savings alone.

Worked example 2: long document or code queries

Same N = 50, same $2.00 per 1,000 documents, so rerank cost is still $0.10. But each row here is a 2,000-token code or contract section, and the generator charges $3.00 per million input tokens, which is realistic for larger frontier models.

Saved tokens: (20 - 5) x 2,000 = 30,000. Saved cost: 30,000 / 1,000,000 x $3.00 = $0.09 per query.

Still not quite break-even. Now change one assumption honestly: to reach the same quality without a reranker, you need a cap of 30, not 20, because so many candidates are near-duplicates. Saved tokens become (30 - 5) x 2,000 = 50,000, and saved cost is $0.15 per query. The reranker now nets $0.05 per query before latency.

That is the shape of the answer. Reranking pays when rows are long, when the larger cap must grow disproportionately to compensate, or both.

The two examples are the decision rule in miniature, and the table is the artifact to hand to whoever owns the budget.

The formulas only work with honest inputs, and the hardest input is K_large: the cap you would need without a reranker to match quality. Nobody knows that number from intuition. You get it from an evaluation set.

Use a fixed question set with an answer key, run the pipeline at several caps without a reranker, and record quality at each. Then run the reranked pipeline at k and compare on the same questions with an explicit quality criterion. That is exactly how retrieval latency experiments are framed: treat a smaller candidate set and a reranker as hypotheses, and watch for the specific failure mode of evidence falling outside the set Supermemory's latency budget guide.

Two practical cautions on measurement:

• Do not compare a reranked run at k = 5 against a non-reranked run at k = 5 and conclude the reranker "improves quality." The fair comparison is against a bigger cap that reaches the same quality.
• Keep retrieval and generation time separate in your traces. A reranker's 200 to 500 ms can be invisible in total response time and still blow a voice budget Supermemory.

If you are assembling this across several providers, one API key for the generator and another for the reranker is a second budget to reconcile; Quanta Inference puts text, embedding and other model calls behind a single OpenAI-compatible key priced in US dollars, which keeps the per-query math in one place.

It is tempting to treat these as two ways to "get more recall." They act on different rows.

Raising the cap admits more rows and lets the generator sort it out. Reranking reorders the rows you already retrieved and discards the rest. The difference shows up exactly where bi-encoder retrieval is weakest: negation, contrast, rare identifiers. One reproduction of the NevIR benchmark reports most bi-encoders scoring 5 to 12 percent pairwise accuracy on negation pairs, while the cross-encoder jina-reranker-v2-base-multilingual reaches 65.2 percent and a listwise LLM reranker reaches 77.3 percent Splunk. No cap increase fixes "I love apples" versus "I used to love apples," because both rows are similar to the query in embedding space.

This is where practitioners should stop looking at token math alone. If your failures are ordering failures, a bigger cap adds more of the same confusion. If your failures are coverage failures, the relevant rows were never retrieved and a reranker has nothing to promote; the fix is a wider first-stage retrieval, which is the "raise the cap" path.

A useful diagnostic: sample 50 failed queries and check whether the correct passage was in the candidate pool at N. If it was in the pool but outside the top k, reranking addresses it. If it was never in the pool, rerank and cap both miss, and you need better retrieval or a larger N.

Teams argue about prices per document and per token because they never convert both to the same unit. Convert everything to cost per 1,000 queries. It is the only figure a finance owner will read.

rerank_cost_per_1k   = (N / 1000) * price_per_1k_docs * 1000
context_saved_per_1k = (K_large - K_small) * tokens_per_row * price_per_1M / 1e6 * 1000
break_even_pool      = context_saved_per_1k / (price_per_1k_docs * 1000 / 1000)

Working the factoid example at 1,000 queries: rerank cost is $100.00, context saved is $1.13. The break-even candidate pool, the N at which cost equals savings, is about 1.1 documents. That is not a pool. It tells you no realistic N makes the reranker pay on token savings for that query type.

Working the code example: rerank cost is $100.00, context saved is $150.00. Here the reranker nets $50 per 1,000 queries. If median reranker latency is 0.18s and your query volume is 2 million a month, the added wall-clock time is 360,000 seconds spread across your capacity, which is a throughput question, not a per-query one Kapa.

One more lever that changes the arithmetic: prompt caching. If the retrieved context were placed inside a cached prefix, the input price falls to a fraction on cache hits, which shrinks the context cost saved and makes reranking look worse. Placing the document in the user message rather than the system message keeps the cacheable prefix stable ML4Devs' context budget guide. Count your cache hit rate before you trust the token savings estimate.

Run this once per query type and write the verdict down. Revisit quarterly, not per incident.

1. Sample 50 to 100 real queries of the type and label whether the needed evidence exists in your candidate pool. This separates ordering failures from coverage failures.
2. Set N to the pool size your first stage already produces (a common choice is 50 when a reranker scores one document per call) Kapa.
3. Compute rerank cost per query: (N/1,000) x reranker price.
4. Find K_large empirically: the smallest cap without a reranker that matches the reranked k on your labeled set.
5. Measure average tokens per retained row, not a guess. Long tool outputs spike unpredictably and inflate it Mem0's context engineering guide.
6. Compute context cost saved per query.
7. Decide with the margin, not the tie. If saved cost is less than twice rerank cost, you are near the noise floor of your evaluation and should default to the simpler pipeline.

The last row is the insight worth carrying: reranking's value scales with how inefficient the cap has to be to compensate. When top results are redundant or distractors are common, the larger cap grows faster than the good rows do, and the reranker wins by removing rows rather than adding them.

If the workload splits cleanly by query type, route on it. A short routing step by query class (factoid, code, long document) lets you raise the cap where it is cheap and rerank where rows are expensive. That routing logic, and the per-type break-even table above, is a good fit for a short consulting engagement where someone reviews your traces with you.