Measuring Help Center Retrieval: Recall and Precision

You do not need thousands of queries. You need queries that resemble what users actually type, plus labels that say which chunks should answer each one.

1. Pull real queries. Export search logs, chat transcripts, and the subject lines of recent tickets. If you have none, write questions from sales-call objections, not from your documentation table of contents.
2. Sample for coverage, not volume. Aim for a first batch of around 30 to 50 queries spread across every topic area, weighted toward your highest ticket categories.
3. Phrase them like users, not like docs. "Why was I charged twice" not "duplicate charge handling policy." Zendesk's guidance for AI-readable content is to use the wording your customers use, spell out acronyms, and avoid vague pronouns (Zendesk help).
4. Label the gold set. For each query, list every chunk that would count as an answer. This list is your denominator for recall, so missing a valid answer makes your recall look better than it is.
5. Freeze a snapshot. Store the query, the labeled chunk IDs, the top K results, and the date. Re-run the same set after every content change so you can tell improvement from noise.

One practical shortcut: create the gold set by pooling. Take the top 10 results from your current retriever, plus the top 10 from a plain keyword search, plus anything a support agent finds manually, and label that union. This is the pooling approach described in search evaluation literature as a way to approximate the set of relevant records without scanning the whole corpus (Measuring Search Effectiveness). It biases recall slightly upward, because a chunk nothing ever retrieves can never be labeled missing, so treat the number as a floor.

The two metrics at K, the number of results you actually pass to the answer generator, are:

• Precision at K = relevant chunks in the top K, divided by K
• Recall at K = relevant chunks in the top K, divided by all relevant chunks that exist for that query

Evidently AI defines both this way for ranking and recommendations, with relevance as a per-use-case binary label (Evidently AI on precision and recall at K). Meilisearch applies the same pair to RAG chunking, noting that precision is true positives over true positives plus false positives, while recall is the chunks retrieved with the answer over the total chunks containing the answer (Meilisearch on chunking strategies).

Take a query: "Can I get a refund after 30 days?" Say three chunks in your knowledge base are genuinely relevant (call them A, B, C), and you retrieve the top 5: A, D, B, E, F. Of those five, A and B are relevant.

Now do that for every query and average. Suppose you have 40 queries: precision at 5 averages 0.62 and recall at 5 averages 0.71. That is your baseline. Two decision rules follow, and they are what separates a useful measurement from a vanity number:

• Recall below roughly 0.8 is a content problem first. If the right passage is not in the top K at all, no amount of answer-writing polish fixes it. The customer gets a confident non-answer or a deflection.
• Precision below roughly 0.5 is a ranking and chunking problem. Half your context window is filler, which raises cost and gives the generator more chances to anchor on the wrong passage. Databricks and others frame chunk quality as a direct driver of retrieval accuracy, since a chunk that mixes two subjects produces weaker similarity scores than two focused chunks (Databricks on chunking).

A short script to compute both over a labeled set:

def score(query_id, retrieved_ids, relevant_ids, k=5):
    top = retrieved_ids[:k]
    hits = [doc for doc in top if doc in relevant_ids]
    precision = len(hits) / k
    recall = len(hits) / len(relevant_ids) if relevant_ids else 0.0
    return query_id, precision, recall, hits

rows = [score(q, r, rel) for q, r, rel in labeled_queries]
avg_p = sum(row[1] for row in rows) / len(rows)
avg_r = sum(row[2] for row in rows) / len(rows)
print(f"precision@5={avg_p:.2f} recall@5={avg_r:.2f}")

K is a design choice, not a constant. It should equal the number of chunks you feed the model, because that is the point where a bad result costs something.

AI knowledge base platforms routinely let you set how many chunks are retrieved per query, and the tradeoff is explicit: more chunks means more cost per request and more chance of noise, fewer chunks means cheaper but less complete context (a walkthrough of knowledge base settings). That maps directly onto precision and recall at K. Increase K and recall rises while precision usually falls; decrease K and the reverse. The two metrics are inversely related in general because broadening a search to catch more relevant material also pulls in more irrelevant material (Measuring Search Effectiveness).

The useful insight here is a per-query view rather than an average. Compute recall at 5 and recall at 10 for each query and look at the gap:

• If recall at 5 is 0.5 and recall at 10 is 0.9, the right passage is there but ranked too low. Raising K hides the problem at the cost of tokens. The better fix is reranking or better headings.
• If recall at 10 is still 0.4, raising K will never help. The passage is missing or unreachable, so the fix is content.

NVIDIA's chunking experiments found that optimal chunk size varied by dataset and query type, with factoid questions favoring smaller chunks and complex analytical questions benefiting from larger chunks or page-level boundaries, and page-level chunking produced the highest average end-to-end accuracy across the datasets tested (NVIDIA's chunking research). The general lesson transfers to help centers: set K and chunk size by measuring on your own queries, not by picking round numbers.

An average score tells you there is a problem. A per-query failure table tells you which rewrite to do. Sort your queries by recall, take the ten worst, and read the gold chunks against what was retrieved.

Three writing rules do most of the recall work, and all come from vendor guidance on making content machine-readable: make each piece self-contained instead of linking out, repeat the topic from the title in the body, and avoid vague pronouns so a chunk still makes sense when it is read alone (Zendesk help center docs). HelpDocs makes the same point from the agent side: a grounded assistant answers from retrieved passages and cites them, so the quality ceiling is set by your content, not the model (HelpDocs).

Measure a rewrite the same way you measured the baseline. Change one article, re-run the frozen query set, and compare the recall of the five queries that touched it. If a synonym added to a heading moves those five from 0.2 to 0.8, keep it. If it moves nothing, revert it: the problem was elsewhere.

Retrieval quality decays quietly. Products change, policies expire, and the query mix shifts toward whatever broke this month. A maintenance loop:

1. Weekly: re-run the frozen set, log precision and recall at K, and flag any query that dropped.
2. Per content release: re-run the subset of queries that touch the changed articles.
3. Monthly: add ten new real queries from logs and retire stale ones, so the set tracks user language.
4. Quarterly: re-check the relevance rules with a second labeler on a sample, to catch drift in how your team applies them.

Also track a business-facing counterpart. Zendesk recommends resolution-focused measures such as search success and self-service ratio alongside model metrics (Zendesk's AI knowledge base guide). Recall is your leading indicator; deflection is the outcome. When recall at 5 climbs from 0.7 to 0.9 and self-service resolution moves with it, you have evidence that the content work paid.

If the labeling and re-running becomes the bottleneck, the interchangeable part is the retrieval call itself: the same query set can be pointed at a managed retrieval API to compare retrievers or embeddings without rebuilding your pipeline. Quanta Inference exposes text and embedding models through one OpenAI-compatible API key, which keeps that comparison to a config change (Quanta Inference).