Trimming Retrieved Chunks Before the Cap

The trimming pass has three rules applied in sequence, each cheaper to evaluate than the next:

1. Dedupe: collapse near-identical chunks, keeping the highest-scoring representative of each cluster.
2. Score cut: drop chunks below a relevance threshold, absolute or relative to the top score.
3. Tier cut: from each source tier (primary docs, secondary docs, transcripts, web), keep only the best N chunks, so no single tier monopolizes the window.
4. Cap: enforce the token limit on what remains.

The order is deliberate. Dedupe first because duplicates distort any threshold you set afterwards. A score cut second because it is a per-chunk decision that does not need global knowledge. A tier cut third because it is the only rule that considers provenance, and it should operate on a set that is already clean and relevant.

def trim(chunks, cfg):
    chunks = dedupe(chunks, cfg.similarity_threshold)   # 1. near-duplicates
    chunks = [c for c in chunks if c.score >= cfg.score_floor]  # 2. relevance
    chunks = tier_cut(chunks, cfg.per_tier_keep)        # 3. provenance
    return cap(chunks, cfg.token_budget)                # 4. hard limit

Each rule takes one configuration value, which is the knob your measurement will tune.

Dedupe compares chunks against each other rather than against the query. The practical choice is embedding similarity (cosine distance on the chunk vectors you already store) or a cheap lexical overlap measure such as shingle Jaccard. Keep the representative with the higher relevance score and record the others as suppressions.

Two decisions matter. The similarity threshold controls how aggressive the collapse is; too low and you merge chunks that carry distinct facts, too high and you leave the swarm intact. The representative choice controls what survives; keeping the highest-scoring member biases toward the query, keeping the first biases toward document order.

Dedupe is the one rule where the recall cost is usually near zero, because a suppressed duplicate almost never contains information absent from its representative. The exception is overlap chunks that straddle a boundary. If chunk A ends mid-fact and chunk B begins it, collapsing them can lose the join, so compare on the merged span where your chunker overlaps.

The saving is easy to state. If a candidate set of 40 chunks contains 12 near-duplicates averaging 100 tokens, dedupe removes roughly 1,200 tokens before the cap ever runs. Some of that saving would have been taken by the cap anyway; the difference is that dedupe removes the redundant copies and keeps the diverse ones.

The score cut removes chunks below a relevance threshold. The core tradeoff is the classic precision and recall one: lowering the threshold captures more relevant chunks but pulls in more noise, raising it gives cleaner results but misses relevant documents as this evaluation primer puts it.

Set the threshold empirically, not by taste. Two forms work:

• Absolute: drop anything below score s. Simple and auditable, but scores are not comparable across queries.
• Relative: drop anything below alpha times the top score, plus a minimum floor so a uniformly weak result set does not survive on a single lucky chunk.

The relative form usually wins in production because retrieval scores drift with query type. A definitional query may produce a strong cluster; a multi-hop query may produce a flat, weak one. Cost-aware routing work shows that anchoring every query to the same retrieval depth wastes budget on simple questions and under-serves hard ones, and that a per-query choice beats a static one on both tokens and latency in the CA-RAG results. A relative score cut is the cheap version of the same idea.

The recall risk here is real. A chunk that is essential but lexically dissimilar to the query can score below the cut. Measure it rather than assuming.

The tier cut keeps the best chunks from each source tier rather than the best chunks overall. Tiers might be primary documentation, support tickets, transcripts, and third-party pages, each carrying a reliability weight.

Why keep a tier cut at all if you already have a score cut? Because scoring models are imperfect and provenance is a prior. If your score floor lets eleven chunks from a single verbose tier through, you have spent the budget on one voice. A tier cut that keeps, say, the top three per tier guarantees diversity and stops one source from crowding out another.

The cost is that the tier cut can discard a chunk that the score cut rated highly, purely on provenance. That is the point, but it is also the rule most likely to quietly hurt recall on queries that genuinely need depth from one tier. Measure per-tier recall loss, not just aggregate.

Watch for a mismatch between tiers and query intent. A query about an internal policy should not reserve two slots for third-party web results. Consider letting the tier cut take per-tier allowances that vary by query class, which is the same per-query idea the routing literature applies to retrieval depth in the MDPI version of the CA-RAG study.

You need three assets before you tune anything: a held-out labeled set of queries, a gold label of relevant chunks per query, and a harness that runs the full pipeline with and without each rule.

The labeled set is the expensive part and the one teams skip. Chunk-level recall requires knowing which chunks contain the answer: recall is the number of relevant chunks retrieved divided by the total number of chunks containing the answer per this metric explanation. Build it with human annotation plus an LLM-as-judge first pass, and hold out queries the embedder and chunker tuning never saw.

For each rule, compute four numbers on the held-out set:

The decision rule is a ratio, not an absolute. Compute tokens saved per relevant chunk lost. Keep the rule when that ratio is high relative to your cost of context: if a thousand tokens cost you very little and a missed chunk costs a wrong answer, the bar rises. Prefer rules that save many tokens and break no queries; scrutinize any rule that breaks queries even if its average recall loss looks small, because averages hide the queries that fail outright.

Context precision, which weighs retrieved chunks by relevance and rank position, is a useful companion metric because it captures whether the good chunks sit near the top of what survives trimming as Galileo's documentation describes. A rule can hold recall steady while pushing relevant chunks to worse positions, and the model's answer can still degrade.

A worked example makes the decision concrete. Suppose a held-out set of 120 queries retrieves a mean of 38 chunks at 95 tokens each, about 3,600 tokens, against a 1,500-token cap.

The shape of the result is what matters, not the specific numbers, which belong to your corpus. Dedupe characteristically pays for itself: large saving, near-zero loss. The score cut is the tunable middle, where you sweep the threshold and pick the point where queries broken stays at zero. The tier cut is the risky one, and it is the rule most likely to justify a per-query allowance instead of a global one.

Two operational habits make this stick. First, re-measure when the corpus, chunker, embedder, or retriever changes; a threshold tuned against one embedding model is not portable to the next. Second, log the full trimming trace, including what was suppressed and why, so that a bad answer can be traced back to the rule that removed the answer.

The later stages of the pipeline benefit too. Shorter, cleaner context lowers billed tokens and latency, and the cost of retrieval and generation is what most of the budget conversation is really about as this token cost guide frames it. And the ordering of what survives matters: models use long contexts unevenly, and performance is typically highest when relevant information sits at the beginning or end of the input and degrades when it is buried in the middle per the Lost in the Middle findings. Trimming before the cap gives you the discretion to place the surviving chunks well, instead of inheriting whatever order the retriever produced.

If you are assembling this across several models and vendors, one API key for text and embedding models removes a layer of plumbing from the benchmarking loop: Quanta Inference exposes text, image, video, audio and embedding models through a single OpenAI-compatible endpoint, so the trimming harness can swap scorers without a rewrite.

• Held-out labeled set exists and was never used for chunker or embedder tuning.
• Tokens saved and recall lost are recorded per rule, not just in aggregate.
• Queries broken count is reported separately from mean recall loss.
• The rule's threshold came from a sweep, not a default.
• The trimming trace is logged with reasons and config values.
• A re-measurement trigger exists for corpus and retriever changes.