Hybrid Search and Reranking: When and How to Combine Them

Chunking and retrieval are separate decisions, and conflating them wastes money. Chunking determines what can be found at all: embedding models have context windows and truncate beyond them, and a chunk that makes no sense to a human will not make sense to the model Pinecone's chunking guide. NVIDIA's experiments across multiple datasets found page-level chunking gave the highest average end-to-end RAG accuracy at 0.648 with the lowest variance, while very small (128 token) and very large (2,048 token) chunks generally underperformed NVIDIA's chunking study. Notably, the best chunk size varied by document type even within financial filings, so it is an empirical setting, not a default.

Assuming your chunks are sane, use a short decision rule:

• Queries are mostly natural language, corpus is small, latency budget is loose, and recall is already acceptable. Stay with dense-only retrieval and spend your effort on chunking and evaluation.
• Queries contain product codes, error codes, names, part numbers, parameter names, or regulatory references. Add BM25. This is the strongest single signal you need hybrid.
• Users complain that a document they know exists does not come back. That is a recall problem. Add the second retriever before adding a reranker.
• The right passage is in the top 50 but not the top 5. That is an ordering problem. A reranker is the correct fix, and hybrid may be unnecessary.
• P99 latency budget is under 100 milliseconds end to end. Skip the reranker. Run the two retrievers in parallel and fuse, which costs roughly the slower of the two, not the sum.

Diagnosing which problem you have matters more than any single technique. If relevant chunks never appear in the candidate set, no reranker can recover them, because rerankers reorder what they are given. Rerankers improve ordering, not recall the reranker comparison.

The mechanics are straightforward. Offline, chunk the corpus, generate a dense embedding per chunk, and build a BM25 (or sparse) index over the same chunks. Qdrant's tutorial describes this as storing three representations per document: dense for semantics, sparse for keyword matching, and optionally a late-interaction representation for reranking the Qdrant hybrid tutorial. You can also keep BM25 inside the database you already run: the pgEdge vectorizer computes both a dense embedding and a BM25 sparse vector per chunk inside PostgreSQL and fuses results with RRF pgEdge.

At query time, issue both searches concurrently:

1. Embed the query and run an approximate nearest-neighbor search for the top N candidates (N around 50 to 200).
2. Run the BM25 query for the top N candidates.
3. Deduplicate by chunk ID and fuse the two ranked lists.
4. Optionally rerank the fused shortlist and keep the top 5 to 20 for the prompt.

Two implementation details save trouble. First, retrieve more per retriever than you intend to keep, because fusion discards items that only one method ranks low. Second, when chunks carry metadata such as ticker, fiscal quarter or report type, apply those filters inside both queries rather than after fusion, otherwise you needlessly shrink the candidate pool the advanced retrieval pipeline.

A minimal fusion step in Python looks like this:

def rrf(ranked_lists, k=60):
    scores = {}
    for results in ranked_lists:          # each list: chunk ids, best first
        for rank, doc_id in enumerate(results, start=1):
            scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + rank)
    return sorted(scores.items(), key=lambda x: -x[1])

That is the whole algorithm. The next section explains why it beats a weighted average.

The temptation is to add 0.5 * bm25_score + 0.5 * cosine_score. Do not. BM25 produces unbounded positive scores while cosine similarity is bounded roughly between -1 and 1, so an unnormalized weighted sum lets BM25 dominate regardless of the weight you set the hybrid search reference. Any alpha you pick is then tuning the normalization artifact, not the retrieval behavior.

Reciprocal rank fusion sidesteps this by scoring each document only by its rank in each list: 1 / (k + rank), summed across lists. A document ranked highly by both retrievers wins; a document found by only one still has a chance, just a smaller one pgEdge. OpenSearch adopted RRF as its default fusion path precisely because rank-based aggregation is stable across mismatched score distributions, resistant to outliers, and requires no normalization tuning the OpenSearch RRF announcement. A larger k flattens the score curve, giving more weight to lower-ranked documents the hybrid search reference.

The decision rule: start with RRF at k=60. Only switch to weighted combination if you have a labeled evaluation set large enough to tune the weight and you have verified the scores are calibrated between retrievers. RRF at defaults beats weighted averages often enough that tuning is not worth the engineering time until everything else is measured the hybrid search reference.

One caveat worth knowing: fusion defaults differ by engine, and some products changed their default fusion algorithm between releases. Check what your vector store actually does before you benchmark.

Two reranker families matter. Cross encoders feed the query and a document jointly into one model and emit a relevance score, which makes them accurate and too slow to run over a large corpus. Late interaction models (ColBERT-style) precompute per-token document vectors and score query tokens against them at query time, giving accuracy close to a cross encoder at lower latency and with more involved indexing the Qdrant tutorial.

Latency and throughput figures above are the ranges reported in a survey of twelve reranker patterns the reranker comparison; treat them as planning numbers and measure on your own hardware.

Sizing the candidate set is the main lever you control, and it trades cost for quality linearly. Reranking 50 candidates costs a tenth of reranking 500. Practical ranges:

• Latency-critical search UI: rerank 25 to 50, keep 5.
• Question answering where one miss is expensive: rerank 100 to 200, keep 10.
• Rerank 500 or more only if your retrieval is noisy and you have measured that the extra depth pays.

Anthropic's numbers suggest reranking on top of hybrid is worth the second stage when quality matters: contextual retrieval plus reranking cut failed retrievals by 67 percent versus 49 percent without reranking Anthropic. Their pipeline is the canonical shape: BM25 top chunks plus embedding top chunks, combined and deduplicated by rank fusion, then rated by a reranker before the prompt Anthropic. A segment of the practitioner community treats hybrid plus rerank as the minimum for production, having found pure vector retrieval insufficient outside demos one practitioner's account.

Provider choice is a secondary decision. If you already call several model types, mixing embedding and reranking providers behind one credential removes integration work; Quanta Inference exposes text, image, video, audio and embedding models through a single OpenAI-compatible key at Quanta Inference. Whatever you pick, run the reranker only on the fused shortlist, cache scores for repeated query-document pairs if your traffic repeats, and cap candidate count at the smallest number that preserves your measured recall.

Every one of these techniques is an experiment, and the only defensible way to add them is one at a time against a labeled set. The metrics are standard: recall@k or Pass@k (was the golden chunk in the top k), and NDCG for ordering quality. Anthropic's cookbook uses Pass@k against 248 queries each with a golden chunk, which is a small but workable template the Claude cookbook.

A procedure that works:

1. Build 100 to 300 queries with known correct chunks, sampled from real traffic and including the query types that currently fail (codes, names, and paraphrases).
2. Measure your baseline: recall@50, recall@5, and NDCG@10.
3. Add BM25 and RRF. Measure again. If recall@50 does not move, hybrid is not your problem.
4. Add the reranker over the fused top 50. Measure NDCG@10 and end-to-end P99 latency.
5. Keep only the changes that move the metric beyond run-to-run noise, and check the cost per 1,000 queries before shipping.

Instrument the pipeline with per-stage timings, because the parallel retrieval step and the rerank step fail differently under load. Google's documentation recommends a metrics-driven "RAG Ops" loop, adjusting search configuration, source curation, parsing or chunking, and query rewriting based on measured quality rather than intuition Google Cloud. That is also the right frame for costing: a reranker that adds 40 ms and a few tenths of a cent per query is cheap if it lifts answer accuracy on the queries that generate support tickets, and wasteful if it reorders a list that was already correct.

If the evaluation work itself is the bottleneck, or you need a retrieval pipeline built and instrumented for a specific line of business, that is a scoped engagement: describe what you need at Quanta and a specialist picks it up with you.