Setting a Retrieved-Token Cap per Query Class

The cap only helps if you know the class before the retriever runs. You need a cheap, fast, auditable classifier that fires in milliseconds, before any expensive retrieval work.

Three classifier families are practical:

Intent classification. Map the question to a small set of intents (how-to, lookup, policy, escalation, small talk). A fine-tuned small model or a rules-plus-embedding hybrid works. The production-RAG query understanding material treats intent detection, metadata filtering and complexity-aware pipeline routing as the layer that sits between the raw user question and the retriever (Raj kumar, Towards AI). That is exactly the position your cap lookup should occupy.

Expected answer shape. Ask what the answer will look like. A single value or short procedure needs little context. A comparison across several documents, or a "summarize all the policies touching X" request, needs a lot. Answer shape is often easier to detect than intent from the raw question.

Latency tier. Some queries arrive on a path with a hard deadline (a voice turn, an inline autocomplete), others on a background path where a slower, deeper retrieval is fine. The latency-budget guidance separates a live voice turn, a support chat and a background research task as having different constraints, and recommends measuring time to first useful output separately from total completion time (Supermemory, latency budgets). Latency tier is a legitimate routing dimension on its own.

A practical default: classify on intent and answer shape, then let latency tier override the cap downward when the request is on an interactive path. Record the classifier's decision and its confidence on every request. When confidence is low, route to a safe middle class rather than guessing at the extreme.

The routing table is the thing you version, review and tune. Keep it out of code. Store it as a JSON or YAML document keyed by class, with the cap, the retrieval depth and any latency constraint together, because they move as a bundle.

{
  "version": "routing-table-v3",
  "classes": {
    "howto_step": {
      "token_cap": 800,
      "top_k": 4,
      "latency_tier": "interactive",
      "fallback": "howto_step"
    },
    "policy_multi_doc": {
      "token_cap": 4000,
      "top_k": 10,
      "latency_tier": "standard",
      "fallback": "policy_multi_doc"
    },
    "escalation": {
      "token_cap": 6000,
      "top_k": 12,
      "latency_tier": "background",
      "fallback": "escalation"
    }
  }
}

The cap is enforced after retrieval and ranking, by trimming the assembled context to the token budget rather than by dropping whole ranked documents blindly. Token accounting is not portable across providers: a prompt that registers at one count in one tokenizer can exceed it in another, so measure with the tokenizer of the model you actually call (Silicon Data, cost per token).

The paper behind BudgetMem frames this as explicit, query-aware performance-cost control, structuring memory processing as modules each offered in three budget tiers with a lightweight router selecting the tier per query (Zhang et al., BudgetMem). You do not need reinforcement learning to start. A hand-written table captures most of the value, and you can move to a learned router later when you have logs.

Two rules keep the table honest. First, the cap is a maximum, not a target: a small question in a big class should still retrieve only what it needs. Second, every class has a fallback so a low-confidence classification still resolves to a cap.

The table is a hypothesis. The logs decide whether it is right. For every request, record the class, the classifier confidence, retrieved tokens before and after trimming, input tokens sent to the model, output tokens, retrieval latency, total latency, and the cost computed from the per-token prices of the model you called.

Because prices vary by orders of magnitude across models and by an 8x or greater input-to-output ratio within a model family, a single blended "cost" number hides which class is expensive for which reason (Silicon Data, cost per token). Store the input and output token counts separately and price them separately.

A worked example makes the tuning loop concrete. Suppose a support assistant handles 10,000 queries a day, and the class mix is 70% how-to, 20% policy and 10% escalation. Under a flat 4,000-token cap at an illustrative $2 per million input tokens, the input bill is 10,000 × 4,000 × $2 / 1,000,000 = $80 a day. With a routed table (800, 4,000 and 6,000 tokens for the three classes), the per-day input token total is 7,000 × 800 + 2,000 × 4,000 + 1,000 × 6,000 = 19,600,000 tokens, which at $2 per million is $39.20 a day. That is a 51% cut on this illustrative workload, driven entirely by the how-to class dropping from 4,000 to 800. Your prices and mix will differ; the arithmetic is the template.

Do not trust that saving until you have checked quality on the class you trimmed. Compare answers on a fixed gold set before and after the change. The latency-budget guidance is blunt about this: treat optimizations as experiments with a fixed set of questions and an explicit quality criterion, and watch for supporting evidence falling outside a shrunken candidate set (Supermemory, latency budgets).

With per-class logs, each cap becomes a one-dimensional knob you can move on data. The decision rule is simple:

• Raise the cap for a class when its answer quality is below target and long multi-document answers are the failure mode.
• Lower the cap for a class when quality is met but the class's retrieved tokens cluster well below the cap (you are paying for headroom you never use) or when trimming is cutting evidence the model needed.
• Move a class to a different latency tier, not just a different cap, when its P95 latency is the problem.

One caution on reading the numbers: retrieval quality does not rise monotonically with context volume. Beyond a point, adding more retrieved text can dilute attention and bury the useful passage in the middle of a long prompt (Liu et al., Lost in the Middle). That is a reason to keep the how-to and lookup caps genuinely small, not merely smaller than the policy cap.

When you patch a class's cap, bump the table version and keep the old one. If a quality regression shows up two weeks later, you want to be able to point at the table that was live on the bad request, not reconstruct it from memory.

Two things drift. The query mix drifts as the product changes, and the classifier drifts as language and the product's vocabulary change. Both break the table quietly.

Track the class distribution weekly. A class that grows fast is a class whose cap now dominates your bill, and the class that shrinks may be over-provisioned. Track classifier confidence alongside accuracy on a labeled sample: a classifier that is confidently wrong is worse than one that routes uncertain queries to the safe middle class.

The retrieval layer has its own failure modes worth watching for the same reason. The mem0 retrieval guide notes that most agent memory failures happen at retrieval, not at write time, and that a query like a specific account code can be drowned out by semantic similarity while keyword retrieval recovers it (mem0, memory retrieval strategies). If your policy and escalation classes keep missing exact identifiers, the fix may be a hybrid retriever inside the class, not a bigger cap.

For teams that need the tuning loop to survive more than one corpus, the same discipline appears in long-term memory design: decide deliberately what gets retained and retrieved rather than saving everything, because a bigger haystack is not the same as better recall (JetBrains, long-term memory). Per-class caps are the retrieval-side version of that decision.

When the workload spans several model types (text for answers, embedding models for retrieval, occasionally an image or audio step), routing each has its own price sheet. Quanta Intelligence's single API covers text, image, video, audio and embedding models behind one key, which keeps the per-class cost accounting in one place instead of across several vendor bills.