When to Return an Empty Result Instead of the Nearest Neighbor

You cannot set a threshold by intuition, and you should not set it from a single bad demo. Start by instrumenting retrieval so you can see the score distribution. Log the top-k scores (k of 2 or 3 is usually enough), the document IDs, and the metadata filters in force for every query. Keep the raw query text and a stable query ID alongside, so a reviewer can reconstruct the decision later. The practice of keeping a dated record of what was collected, what changed and why is standard in qualitative research audit trails, and the same discipline applies to retrieval logs: a decision you cannot trace is a decision you cannot defend per this audit trail checklist.

The specific signal to hunt for is a leak: a query where a neighbor that should have been excluded, because of a permission filter, a deprecated document, an out-of-scope product line, or simple irrelevance, scores above your current cutoff. Two mechanisms produce these:

The filter-after-search case deserves attention because it silently degrades recall. If you ask an approximate nearest neighbor index for 10 results and then remove 6 by permission, you have 4 left, and the best admissible match may not be in the original 10 at all. Filtered queries are precisely where approximate indexes historically fell down, which is why engines added features to fetch complete result sets for filter-heavy queries as AWS describes for pgvector on Aurora. Metadata filtering applied inside the search, rather than after it, both accelerates the query and improves accuracy per this walkthrough of metadata filtering.

Once you can see leaks, label a sample. Pull 200 to 500 queries stratified across topics, including the rare ones, and have a domain expert mark each as: a correct match exists, a correct match exists but is excluded, or no match exists. That labeled set is the only honest basis for a cutoff. Stratified sampling is how you surface tail failures that random sampling misses, the same argument Galileo makes for keeping an LLM judge calibrated on rolling production samples rather than a frozen golden set in its calibration guide.

With labeled queries in hand, the calibration is mechanical. For each candidate threshold t, compute two numbers:

• Leak rate: of the queries where the top admissible score is at or above t, what fraction were labeled "excluded match" or "no match"? These are the cases where you would have admitted bad evidence.
• False empty rate: of the queries labeled "a correct match exists," what fraction have their correct match scoring below t? These are the users you would have turned away with a fallback.

Then sweep t across the observed score range and plot both. Here is a worked example with numbers you can reproduce on your own data. The counts are illustrative, constructed to show the method, not measured results.

Read the table as a menu of trades, not a search for a magic number. Moving from 0.70 to 0.80 cuts leaks from 43% to 15% but triples the false empty rate from 5% to 15%. Whether that is a good trade depends entirely on the cost asymmetry of your application, which is the same decision the Arize evaluation guide asks you to write down before choosing a metric: which error is worse, a false positive or a false negative in their judge alignment guide.

Three decision rules follow.

Rule one: match the threshold to the cost of a wrong answer. In a support bot for billing questions, a wrong tier's policy is a customer-facing error with financial consequences, so you accept a higher false empty rate. In a research assistant where users can tolerate a "not found" and rephrase, a looser cutoff is fine. Write the rule down and get it signed: "leak rate at or below 10%, false empty rate accepted up to 20%."

Rule two: set the cutoff at the point where the leak rate meets your tolerance, not where the curve looks nice. In the table above, a 10% leak tolerance lands between 0.80 and 0.85, so you would test 0.82 and 0.83 next.

Rule three: keep a margin for sampling noise. The calibration set is a sample. A few hundred labeled queries gives you a leak rate estimate with meaningful uncertainty; report it rather than pretending the point estimate is exact, the way the Arize guide insists on reporting uncertainty around main results. If your sample says 15% with a wide interval, pick a slightly stricter cutoff rather than gambling on the optimistic end.

If a reranker sits between the index and the cutoff, calibrate on the reranker's scores, not the raw cosine values. The two live on different scales and a cutoff tuned on one is meaningless for the other. Evaluate retrieval and generation separately before you evaluate end to end, so a retrieval failure is not blamed on the generator as Evidently recommends.

The threshold decision has a second half: what the user sees when nothing clears it. Never return the nearest item with a hedge like "this might be relevant." That reintroduces the leak you just paid a false empty rate to prevent, and it trains users to distrust every answer.

A good fallback does four things:

1. Says plainly that nothing in the knowledge base matches the question.
2. Names what was searched, so the user can tell whether they asked the right system.
3. Offers the nearest specific next step (rephrase, browse a category, contact a human).
4. Requests the missing content in the background, so the corpus gap gets fixed.

No match found in the support knowledge base for: "refund window for enterprise plans".
This assistant covers published policies for standard and enterprise tiers.
Try rephrasing with the plan name, or open a ticket and a specialist will reply.
Your question has been logged so we can add this topic.

Note that the message does not include the weak candidate. That is deliberate. If the top score is 0.71 and your cutoff is 0.82, the candidate is more likely to mislead than help.

Track fallback rate as a first-class metric. A rising fallback rate on a topic is a content request, not a retrieval bug. A fallback rate that spikes for one phrasing but not another is a query-side problem, and query rewriting may fix it. Separating these two is the difference between a threshold that improves the product and one that quietly hides gaps.

A cutoff is a snapshot of a moving system. Query mix shifts, documents are added and retired, embedding models get updated, and each change moves the score distribution. A judge that agreed with reviewers at launch can drift without a single alert firing, which is why continuous calibration on rolling production samples beats one-time validation against fixed gold labels per Galileo's drift analysis. The same logic applies to your retrieval threshold.

Put four things on a schedule:

• Re-score a fresh sample of 100 to 200 production queries against the labeled rubric every quarter, or after any embedding model change, whichever comes first.
• Watch the score histogram for a shift in the median top-1 score. A sudden move usually means a new document batch or a new query pattern, not a threshold problem.
• Audit the excluded neighbors on a sample of fallback cases. If excluded content routinely scores above the cutoff, your filter placement is the problem.
• Version the threshold itself in configuration with the date, the sample size, the measured leak and false empty rates, and who approved it. This is the decision log discipline from research auditing applied to production: when someone asks why 0.82 and not 0.78, the answer is in the record per this decision log example.

One caution about evaluation mechanics. If you use a language model to judge whether a retrieved passage was relevant at scale, validate that judge against human labels before trusting its rate estimates. Precision, recall and F1 against an adjudicated reference will tell you whether the judge is good enough to calibrate on, and class-level counts matter as much as the overall score per this practical measurement guide. Rubric-based judging scales well but needs explicit criteria and calibration against experts to avoid known biases such as verbosity preference per this survey of rubric-based evaluation. Building a small, well-defined eval set from your own domain beats importing a public benchmark whose queries look nothing like your traffic as SuperAnnotate argues.

There is a second drift risk worth naming: few-shot and structured-output conventions in your prompts. If your judge or your generator is steered with example outputs, changes to those examples shift behavior as surely as a model update does, and the community threads on structured-output prompting show how much variation sits in that choice per this OpenAI developer discussion. Freeze prompt templates alongside the threshold version so a prompt edit and a cutoff change can be told apart in the logs.

Not every retrieval path needs a cutoff, and adding one everywhere creates avoidable dead ends.

Skip it, or set it very low, when retrieval is exploratory. "Find me anything about X" and autocomplete-style suggestion lists are fine with weak matches, because the user is browsing and the cost of an odd suggestion is low. Skip it when a downstream reranker plus a hard permission filter already guarantees that only admissible documents reach the model, and you have measured the reranker's precision on your domain. And skip it in pure classification-style lookups where a nearest-neighbor vote is the model itself, such as picking a support macro from a query embedding: there, a wrong label is caught and corrected by an agent, not shown to a customer.

Apply it whenever the retrieved text is shown to a user as fact, whenever excluded content must not surface, or whenever the generator is instructed to answer only from context. Those three conditions overlap heavily in production RAG, which is why the threshold is usually the right default and the exceptions need a reason.

One more practical note on scale. Brute-force comparison against every vector is exact but expensive at volume: a collection of 10 million documents with 768-dimensional embeddings needs roughly 7.7 billion floating-point operations per query for a single distance metric, which is why approximate indexes exist per this vector search overview. Approximate search can miss the true nearest neighbor, and that interacts with your cutoff: a slightly low score may reflect index error rather than genuine irrelevance. When you calibrate, re-run the same queries with exact search on a small subset to see how much of your false empty rate is approximation, not relevance. If approximation accounts for most of it, fix the index parameters before tightening the threshold, and keep the neighbor-search options in mind, since different structures trade recall against speed differently as scikit-learn documents.

Get the mechanics right and the method above gives you a defensible number. Get the evaluation data right and you have a system that can tell a user "nothing matches" without eroding trust. For teams standing up the retrieval service itself, Quanta Inference exposes embedding models and text models through one OpenAI-compatible API key, so you can log scores, run an eval set and swap models without rebuilding the client.