How to Set a Minimum Similarity Score for Retrieval Answers

The score is a function of the chunk that produced it. Fixed-size splitting can cut a sentence or idea in half, which lowers the score a genuinely relevant passage achieves; recursive or sentence-aware splitting tries paragraph breaks first, then sentence ends, producing cleaner boundaries (chunking strategy comparison). Semantic chunking goes further, embedding each sentence and placing a boundary where similarity between adjacent sentences drops, which produces more coherent chunks but adds an embedding call per sentence at index time. In practice that means the same content can score meaningfully higher under semantic chunking than under fixed windows, so a threshold tuned on the old index is invalid after a rechunk. The Bedrock knowledge base documentation distinguishes standard, hierarchical and semantic chunking for exactly this reason (AWS chunking guide).

Freshness matters for a subtler reason. An index is a point-in-time snapshot, and stale embeddings can degrade retrieval accuracy, with one vendor estimating up to 20 percent degradation and no built-in uncertainty signal reaching the user (Atlan on knowledge base staleness). When documents drift, some queries that used to match now score lower. The score distribution moves under a fixed threshold, so recalibrate after major content refreshes, not only at launch. Vendor guidance also suggests re-embedding once roughly 10 to 15 percent of the corpus has changed (Kapa on keeping a RAG index in sync).

A practical sequence: lock the chunking strategy, build the index, then tune the threshold. Tuning on an index you are about to rebuild wastes the work.

You cannot choose a threshold without knowing, per query, whether a correct passage exists in the corpus. That is the ground-truth side of retrieval evaluation: recall at k with known relevant documents, or relevance labels applied by hand or by a judge model (Evidently's RAG evaluation guide).

Steps that work:

1. Sample real queries. Pull from search logs, support tickets, chat transcripts and agent handoffs. Aim for a few hundred. The distribution should match production, not the demo.
2. Include unanswerable queries on purpose. Questions about features you do not offer, competitors, or policies from a different product are essential. Without them you cannot estimate wrong answers at all.
3. Label correctness of the top passage. For each query, mark whether a passage in the corpus answers it and which one. A binary label is enough to start. If two people disagree often, tighten the definition of "correct."
4. Freeze the set. Store queries, labels, the index snapshot identifier and chunking configuration together. When you re-index, re-run the same set.

Manual labeling is slow but trustworthy. LLM-judged relevance can scale the set once your hand labels show the judge agrees with people on a sample; use it to expand coverage, not to replace the first hundred labels. For medical or legal content, the bar for "correct" should be set by a domain reviewer, since an answer that cites the right passage but applies it to the wrong population is still wrong. A retrospective triage study that combined retrieval with multi-evidence clinical reasoning is a useful model of how carefully that domain separates retrieval quality from final answer quality (JMIR Medical Informatics study).

The threshold comparison has to happen on a specific number. Decide which one before sweeping:

• The top-1 score (what the agent would look at first).
• The top-k maximum over what you would actually send to the model.
• The reranked score if you rerank, since rerankers rescore candidates on their own scale.

If any retrieved passage passes the cutoff and is sent to the generator, use the maximum score among the sent passages. Then the two error counts are:

• Miss: a correct passage exists in the corpus but its score falls below the cutoff. The agent hands off or refuses.
• Wrong answer risk: no correct passage exists above the cutoff, but the top score passes. The agent answers from irrelevant evidence.

There is a third category worth counting separately: correct passage exists and passes, but an irrelevant passage also passes and dominates the answer. This is the failure mode the 0.79 example illustrates, and it is invisible if you only look at the top score.

With a labeled set and a fixed score definition, the sweep is mechanical. Compute per-query scores once, store them, then evaluate every candidate cutoff against the stored numbers.

# rows: {"query_id":..., "has_correct": bool, "top_score": float, "correct_score": float|None}
def sweep(rows, thresholds):
    out = []
    for t in thresholds:
        misses = sum(1 for r in rows if r["has_correct"] and (r["correct_score"] or 0) < t)
        wrong  = sum(1 for r in rows if not r["has_correct"] and r["top_score"] >= t)
        answered = sum(1 for r in rows if r["top_score"] >= t)
        out.append({"threshold": t, "misses": misses, "wrong": wrong,
                    "answered": answered, "answer_rate": answered/len(rows)})
    return out

Worked example with numbers. Suppose 400 labeled queries, 300 of which have a correct passage and 100 of which do not. Costs: a miss requires a handoff to a human agent, which you value at 6 minutes of agent time; a wrong answer costs a support ticket and a trust hit, which you value at 15 minutes plus escalation risk. Run the sweep:

At these invented weights, 0.70 wins. Move the wrong-answer cost to 60 minutes (a compliance incident in a regulated workflow) and the table flips: 0.80 costs 504 + 420 = 924 against 234 + 1080 = 1,314 at 0.70, so the stricter cutoff is correct. That is the whole decision rule: the right threshold is the one that minimizes weighted cost, and the weights come from your operation, not from a blog post. The same logic explains why legal and compliance workloads are often tuned to a much narrower band than conversational help desks.

Two refinements:

• Report the operating point, not just the score. State the answer rate, the miss rate among answerable queries and the wrong-answer rate among unanswerable ones, so product owners can argue about costs rather than about the number.
• Evaluate in segments. Split by query type (account questions, policy questions, billing disputes) and check whether one threshold serves all of them. Often you need a per-intent cutoff or a different handoff path for unanswerable queries.

Sweep on a held-out slice. If you pick the cutoff that minimizes cost on the same 400 queries you tuned on, you have fit the noise. Label 500, tune on 300, confirm on 200.

The sweep gives you a starting number. Production tells you whether it holds. Log three things for every request: the top score, the decision (answer or handoff), and an outcome signal. Outcome signals include user thumbs, follow-up tickets within 24 hours, escalation to a human after an answer, and explicit correction phrases in the next turn.

Then watch the score distribution itself. If the median top score for answered queries falls after a content refresh, or if the share of queries near the cutoff grows, you have drift. Store the threshold with the index version in your deployment configuration so that a rollback test cannot silently pair an old cutoff with a new index.

One more control: keep an unanswerable probe set in production monitoring. Send it through the live pipeline periodically and count how many pass the cutoff. That number should stay low, and if it creeps up, wide admission of irrelevant evidence is the likely cause.

The threshold is one parameter in a larger admission policy: chunking, embedding model, retrieval mode, reranking and cutoff interact, and changing any one invalidates the tuning. If you are assembling the pipeline and would rather not maintain separate clients for embeddings, rerankers and generators, Quanta Inference exposes text, embedding and reranking-capable models through one OpenAI-compatible API key with prepaid US-dollar pricing, so the sweep runs against the same endpoint you ship.

If the decision instead needs a second opinion on cost weights (what a handoff actually costs your team, what a wrong answer costs in your market), a short consulting call will map those numbers to a threshold and a monitoring plan.