Rerankers in Practice: Cross-Encoders, Latency and Cost

The difference explains the cost. A bi-encoder embeds the query and each document separately, so document vectors can be computed offline and indexed once. A cross-encoder concatenates the query and the document into a single sequence, sends them through the transformer together, and outputs one relevance score. Every query token can attend to every document token, which is why it handles exact phrases, conditionals and negation far better, per the Digital Applied reference and the NextGen analysis of cross-encoder mechanics.

That joint pass is also why it cannot scale. A paper on distilling LLMs into rerankers notes cross-encoders need one full forward pass per candidate, which is linear in candidate count at query time, per the arXiv reranking study. The practical rule: run the expensive model on a shortlist of 50 to 100 and nowhere else.

The one-line mental model: the bi-encoder decides what is worth looking at, the cross-encoder decides what to actually use.

RRF is a fusion step, not a reranker. Keeping them separate avoids most integration pain.

Do this: run your existing hybrid retrieval exactly as it is, let RRF produce a single ordered list, then slice the top N and hand that slice to the reranker. The reranker overwrites the final order for those N candidates. RRF still does what it is good at (combining ranks across incompatible score scales), and the cross-encoder does what it is good at (resolving fine-grained relevance within the shortlist).

A minimal implementation in Python shows the shape, following the NextGen integration blueprint:

from sentence_transformers import CrossEncoder

reranker = CrossEncoder("BAAI/bge-reranker-large", max_length=512)

def retrieve_and_rerank(query, top_k_stage1=50, top_n=5):
    # Stage 1: existing hybrid search + RRF, unchanged
    candidates = hybrid_search_with_rrf(query, limit=top_k_stage1)
    if not candidates:
        return []

    # Stage 2: joint scoring on the shortlist only
    pairs = [[query, c["text"]] for c in candidates]
    scores = reranker.predict(pairs, batch_size=32)

    for c, s in zip(candidates, scores):
        c["rerank_score"] = float(s)

    ranked = sorted(candidates, key=lambda c: c["rerank_score"], reverse=True)
    return ranked[:top_n]

Three integration details matter more than the code:

• Do not reuse the RRF score as a feature in the reranker. Feed raw text, let the cross-encoder form its own judgment.
• Cap the document text at the model's max length. The blueprint above uses 512 tokens; truncating mid-answer is a common cause of phantom hits, a failure mode Splunk's reranker selection guide attributes to input handling rather than the model itself.
• Keep the first-stage call async and cache-friendly. The reranker is the latency-critical path; the retrieval call is not.

If you want to test a hosted reranker before committing to self-hosting, an OpenAI-compatible gateway lets you swap scorers behind one key. Quanta Inference (https://quantaintelligence.ai/inference) exposes text, image, video, audio and embedding models through one API, which is useful for benchmarking rerank candidates without re-plumbing auth.

Latency depends on three variables: candidate count, pair length, and hardware. Published figures converge on the same order of magnitude.

The NextGen analysis puts added latency at 20 to 150 milliseconds per query for cross-encoder reranking, with a common 50 to 100 candidate pool. The Digital Applied reference reports a +120 millisecond average, and the Ailog benchmark summary lists 50 milliseconds for 100 pairs on an open MiniLM reranker. Treat the exact number as hardware-specific and measure on your own box.

The most useful published data point is a candidate-count sweep, from the Ailog summary:

Read that table as a diminishing-returns curve. Going from 20 to 100 candidates buys 0.083 MRR for 45 milliseconds. Going from 100 to 200 buys 0.003 MRR for 55 milliseconds. Unless your queries are unusually hard, the sweet spot sits at 50 to 100.

Batching matters more than most teams expect. A cross-encoder scored one pair at a time leaves GPU throughput on the floor. Scoring 32 pairs per batch, as the blueprint above does, is the usual starting point. At 100 candidates, that is four batches, and the GPU stays busy across each one. On CPU-only deployment, expect several times the latency of GPU, so plan to run the reranker on the same node as the retrieval service rather than hopping networks mid-request.

There are two billing models and it pays to know which one you are on.

Hosted rerankers charge per token of the query plus document pairs. You are billed for the input, not the output, because the model returns a float. The token bill per query is:

tokens_per_query ≈ (query_tokens + avg_chunk_tokens) × candidates

A worked example. Suppose your query averages 30 tokens and your chunks average 400 tokens, and you rerank 100 candidates. That is roughly (30 + 400) × 100 = 43,000 input tokens per query. At a hosted rate of $0.05 per million tokens, listed for Cohere Rerank v3.5 in the Ailog leaderboard, the per-query cost is about $0.00215. At 100,000 queries a day, that is roughly $215 a day, or about $6,450 a month. The same table lists cheaper options and open weights, and those prices are of course dated; always read the vendor's current sheet.

Self-hosting inverts the economics. Instead of per-token billing, you pay for a fixed pool of compute. A reranker model in the hundreds of millions of parameters runs on a single mid-tier GPU, and that GPU handles a fixed ceiling of queries per second. Once you approach that ceiling, adding traffic means adding hardware, regardless of candidate count. The arithmetic that matters is utilization: total daily pairs divided by the throughput you measured on your hardware. If you are under roughly 30 percent utilization, hosting is usually cheaper than a per-token API; above 70 percent, the API often wins because you do not have to buy the next GPU.

A simple decision rule that has served well:

• Under 10,000 queries a day and a tight latency budget: start with a hosted reranker API.
• Between 10,000 and 500,000 queries a day: benchmark both, but expect self-hosting to win as utilization rises.
• Above 500,000 queries a day, or strict data residency: plan to self-host and budget one GPU per measured throughput unit.

The token cost of reranking is often trivial next to the token cost of generation. What matters is the latency a user feels. That is the number to protect.

Do not ship a reranker because a vendor says it improves accuracy. Ship it because you measured improvement on your own data.

Build a frozen evaluation set first. That means 200 to 500 representative queries, each with a labeled relevance list. Labeling takes an afternoon and unlocks every later decision. Then compute three metrics before and after adding the reranker:

1. NDCG@10 (normalized discounted cumulative gain). Rewards placing relevant documents at the top.
2. MRR@10 (mean reciprocal rank). Rewards getting the first relevant hit early.
3. Recall@100 on stage one alone, so you know the ceiling the reranker cannot exceed.

Published gains are wide and that should tell you to expect variance, not a fixed number. The Digital Applied reference cites a 7.4 percent NDCG lift from hybrid retrieval over either BM25 or pure KNN on the WANDS e-commerce set (0.7497 versus 0.6983 and 0.6953). The Ailog summary reports average accuracy gains of roughly 33 percent from adding a cross-encoder on top of a bi-encoder across four benchmarks, and the Splunk writeup reports a 12.6-point exact match improvement in a DynamicRAG ablation (47.1 versus 34.5). The NextGen writeup puts retrieval precision gains at 15 to 35 percent with 20 to 150 milliseconds of added latency. These are different tasks and different datasets, which is precisely why your own numbers matter more than the average.

A worked justification. Suppose your frozen set has 300 queries, and adding a 100-candidate cross-encoder moves NDCG@10 from 0.62 to 0.68. That is a 0.06 absolute lift, or about 9.7 percent relative. It costs 125 milliseconds at p50. Your generation step already takes 1.2 seconds. So you are adding about 10 percent to total query time to buy a 9.7 percent ranking-quality gain. That is usually a good trade. If your search is already sub-200 milliseconds end to end and users care about that, it is a bad one. The decision rule is the ratio of ranking gain to perceived-latency cost, evaluated against what your users actually do.

Then run the A/B test. Split live traffic, measure click-through on the top result, reformulation rate (users rewriting the query), and where available, task completion. A reranker that improves NDCG but harms click-through is telling you that your labels disagree with your users. That disagreement is the most valuable signal you will get.

Reranking gains are not uniform. The Ailog query-type analysis reports that fact lookups improved around 18 percent, while multi-hop queries improved around 47 percent and complex queries around 52 percent. That is a strong routing signal.

A practical pattern is to rerank selectively. Score every query with a cheap classifier or heuristic on query complexity, and only rerank the fuzzy cases while skipping exact-match lookups like product codes or error messages. Two considerations make this work:

• Exact-term queries are usually already ranked correctly by BM25, which is what the Digital Applied reference points out about lexical retrieval's blind spot being the mirror image of dense retrieval's.
• Fuzzy and conceptual queries are where cross-encoders show the biggest gains, per Splunk's reranker guide, which reports bi-encoders scoring only 5 to 12 percent on negation-sensitive pairs while a cross-encoder reached 65.2 percent.

The routing rule: rerank any query that contains no exact-match signal, and skip the rest. In practice this cuts reranker load by 30 to 50 percent on many e-commerce and developer-docs corpora, at the cost of leaving the exact-match path untouched.

If selective reranking starts to feel bespoke, an expert who has seen a handful of these pipelines can compress a week of tuning into a call. Quanta Intelligence arranges a consulting review at https://quantaintelligence.ai/: describe the retrieval stack and the current metrics, and a specialist picks it up.

Reranking is worth its cost when your stage-one recall is healthy but your top five are noisy. Signs that this is your situation:

• Users click result number three more than result number one.
• Your LLM answers cite irrelevant chunks even though the answer exists in the corpus.
• Adding more candidates to the shortlist makes the LLM less accurate, which suggests ranking, not recall, is the bottleneck.

Reranking will not help when:

• Stage-one recall is poor. Reranking cannot recover documents that were never retrieved.
• Your chunks are too large and dilute the embedding, a known issue covered in Pinecone's chunking guide and Tiger Data's embedding primer. Fix chunking before you fix ranking.
• Latency is the product's signature. A 125-millisecond addition to a 150-millisecond search is a 83 percent slowdown.
• Queries are predominantly exact-match identifiers, where BM25 is already strong.

One more check worth running: the LIMIT paper cited in Splunk's guide argues embeddings cap the number of distinct top-k result sets a model can return, with a rough breaking point around four million documents for 1024-dimensional vectors. If you are near that scale and seeing saturated ranking quality, a reranker may not be your next dollar; a better embedding model or chunking strategy likely is. Reranking is a precision tool, not a capacity tool.