Scoring Retrieval With No Labels: Clicks and Citations

The citation signal is simpler than it looks. You give an agent a query, the retriever returns, say, the top 20 chunks, and the agent writes an answer that cites some of them. The agent has effectively told you which chunks it judged useful. Everything it retrieved and did not cite is a weaker candidate.

The mechanics of this signal matter. Anthropic's citations feature returns source locations computed by the API rather than generated by the model, as explained in this walkthrough of agent citations. That distinction is what makes the signal trustworthy for scoring: a cited passage is a pointer the model could not invent, and a citation into a passage that does not say what the answer claims is a failure you can detect. When citations are computed rather than generated, "cited" is close to "used."

A few practical rules make the labels cleaner:

• Log the full retrieved list, not just the cited items. Without the uncited candidates you cannot form negatives.
• Treat sibling chunks from the same parent as one unit. Parent-child chunking returns the parent for context, so that a citation into a child chunk should credit the parent as well; the hierarchy is described in this Sandgarden overview of parent-child chunking.
• Discard answers the agent produced without reading any chunk. A refusal or a general-knowledge answer carries no retrieval signal.
• Cap how much weight a single query gets. One query cited in fifty sessions should not outweigh fifty distinct queries.

The output is a set of pairs. For a query with cited chunk A and uncited chunk B, you get one preference A over B. Three cited chunks against seventeen uncited chunks is 51 pairs from a single query, which is why a few thousand logged sessions produce a workable training set.

One important caveat: the agent's citation behavior reflects the model's judgment, and model judgments drift as models are upgraded. If you retrain a ranker on citations from one model generation and serve it with another, the label distribution shifts. Keep the model name and prompt template in every citation log row so you can re-anchor.

Clicks are the harder signal, because a click is not a vote for relevance. It is a vote for "this looked worth opening." Two corrections make it usable.

First, add a dwell condition. A click followed by a bounce in under a few seconds is closer to a negative than a positive. A click followed by substantial time on page, or a follow-on action, is a much stronger signal. How you set the threshold depends on your content, so calibrate it on a sample you inspect by hand.

Second, correct for position bias. Position bias is the tendency of users to interact with higher-ranked results regardless of relevance, and it is the dominant distortion in click data, as this review of position bias correction documents. Training on raw clicks teaches the ranker to reproduce the existing order, because past winners are always shown first and therefore always win.

Two families of correction are relevant here:

The position bias model at the heart of these methods assumes a click happens only if a user examines a result and finds it relevant, with examination depending on position and relevance depending on the query-item pair. Google's EM algorithm for position bias estimates that bias from regular clicks and reports that pairwise learning-to-rank models exploit the estimated bias better than pointwise ones. That last finding is convenient, because your goal is pairwise labels anyway.

Once you have examination probabilities per position, you can form labels. The safest starting point is within-rank comparisons: two results shown at the same position across different sessions, one clicked and one not. A later refinement is cross-rank comparison after inverse propensity weighting, where a click at position 8 carries more evidence than a click at position 1. A practical primer on measuring position bias notes that on Google Search the first position is clicked far more often than the tenth, which is roughly the size of the correction you are applying.

Numbers make the choice concrete. Start with 10,000 distinct queries. Suppose your retriever returns 20 chunks per query, so you have 200,000 retrieval events.

For citations, assume 70% of sessions produce a usable cited answer and each such answer cites about three chunks. That is roughly 7,000 queries times 3 cited chunks against 17 uncited chunks, about 357,000 preference pairs. Most will be redundant within a query, so down-weight to one pair per cited-uncited combination you keep: a few hundred thousand pairs is a realistic cache.

For clicks, assume the same 10,000 queries but many sessions each, and a click-through rate of 0.3 across 20 results, which is typical of a list interface. You get about 60,000 clicks. Now apply the corrections: drop clicks under your dwell threshold, say half of them, leaving 30,000. Site-level examination probabilities from an EM fit further discount the top positions. You end with roughly 10,000 to 15,000 weighted pairwise labels.

Compare that with manual annotation. A single labeled query-document pair takes a rater a minute or more, so 10,000 pairs is over 160 rater-hours before quality checks. The implicit route gives you a comparable volume of pairs for essentially free, continuously refreshed.

The comparison also tells you which signal to trust at your scale. Below a few hundred clicks total, you cannot fit a position bias model at all and should rely on citations alone. Above tens of thousands of clicks, clicks become stable enough to train a reranker directly.

Once you have pairs, scoring is ordinary pairwise evaluation. For each pair (preferred, dispreferred), run both candidates through the model being evaluated and count how often the preferred one scores higher. That fraction is your pairwise accuracy, and it is robust to class imbalance in a way that raw precision at k is not.

A short evaluation harness looks like this:

pairs = load_pairs(source="citations")  # and separately "clicks_weighted"
correct = 0
for q, better, worse in pairs:
    s_better = model.score(q, better)
    s_worse = model.score(q, worse)
    correct += s_better > s_worse
print(correct / len(pairs))

Three habits keep this honest. Hold out queries, not pairs, so pairs from the same query never straddle train and test; otherwise the model memorizes the query and the score flatters you. Report citation accuracy and click accuracy separately; a model can improve one while hurting the other, and a blend that ignores this hides regressions. And keep a small hand-labeled slice of a few hundred pairs to check that the implicit labels agree with human judgment on your corpus.

You will often find the two signals disagree, and the disagreement is informative. A chunk can be cited by the agent yet rarely clicked by users, because citation measures answer value while clicks measure title appeal. When that happens, check whether your result titles are misleading, or whether citations are clustering on a few chunks per query, which sometimes signals that the chunker split an answer across too many pieces; Anthropic's contextual retrieval post documents how chunking and missing context cause exactly this kind of retrieval failure.

The labels are only as good as the candidate set they are derived from. Two upstream choices quietly reshape the signal.

Chunking first. If chunks are very small, citations cluster on many fragments and uncited neighbors are often semantically identical, which injects noise into your negatives. If chunks are very large, one citation credits a huge block and the pairwise signal blurs. Chroma's technical report on chunking found the choice of chunking strategy has a significant effect on retrieval accuracy and efficiency, and that token-level precision and recall are more meaningful than document ranking for this kind of system. Start from a recursive splitter around 400 to 512 tokens with modest overlap, a default Firecrawl's chunking comparison also recommends, and keep chunk boundaries fixed while you evaluate so label changes reflect the model, not the splitter.

Retrieval strategy second. A pure vector retriever will look worse on identifier queries like error codes or part numbers, because semantic embeddings can miss exact matches; Anthropic's writeup on contextual retrieval shows a query like "Error code TS-999" as the canonical case, and this advanced retrieval pipeline walkthrough builds exactly that hybrid of vector search, full-text search and reranking. Your citation labels will expose this pattern if you segment them by query type. Break your pairwise accuracy down into natural-language queries and exact-match queries and you will usually see the gap immediately.

The decision rule, stated plainly:

• Trust citations when your agent is the consumer. They measure whether the retrieved text supported an answer, which is what the agent needs.
• Trust corrected clicks when humans are the consumer and your list interface is the primary surface.
• Trust neither at small scale. Below a thousand usable events total, collect a hand-labeled set of a few hundred pairs and evaluate against that.
• Never train a ranker on raw uncorrected clicks. The feedback loop is self-reinforcing, which is the core critique in this primer on position bias.

There is one more signal worth logging from day one: whether the answer was abandoned, edited or re-asked. That is a cheap implicit negative and it frequently contradicts the citation labels in productive ways.

If the bottleneck is running several candidate models over the same evaluation pairs without managing a separate integration per provider, Quanta Inference puts text, embedding and reranking models behind one OpenAI-compatible key, which keeps the comparison work about retrieval quality rather than plumbing.