Ordering Retrieved Chunks for Cache Hits

Practitioners use "cache" for four different mechanisms, and mixing them up leads to the wrong fix. The following table separates them.

The first three are exact-match and correctness-neutral: a miss costs money and latency, nothing else. Semantic caching is fuzzy by design and can return a wrong answer with a 200 status (Daily Dose of Data Science). For a retrieval pipeline, the lever you actually control is the prefix: the system prompt, tool definitions, and the chunk block. Of those, the chunk block is the part that churns.

Response caching is a different product. It hashes the request body, model, API key and streaming mode, then returns the stored response without calling the provider, with cached responses arriving in 80 to 300 milliseconds and zero billed tokens (OpenRouter announcement). It is excellent for test suites and agent retries. It does nothing for two similar questions that retrieve overlapping chunks, because the hashes differ. Prompt caching is the layer that helps there, and it only helps if the chunks land in the same place.

Take a support assistant. The static system prompt is 1,200 tokens and tool definitions add 800. Retrieval pulls four chunks of about 500 tokens each, so the variable block is 2,000 tokens and the user question is 30. Total prompt: about 4,030 tokens.

Before the fix, chunks are concatenated in descending similarity score. Query A retrieves chunks 12, 45, 78, 91 at scores 0.91, 0.88, 0.84, 0.80. Query B, twenty seconds later, retrieves 45, 12, 103, 78 at scores 0.89, 0.87, 0.85, 0.81. The shared prefix between the two requests stops after the 2,000 static tokens. Every request pays full input price for roughly 2,000 chunk tokens and gets a cache read on 2,000 static tokens at best.

Now the fix. Sort the retrieved chunks by a stable key before serialization, for example source_id ascending, with score as a tiebreaker. Query A becomes 12, 45, 78, 91. Query B becomes 12, 45, 78, 103. The two serialize identically up to the end of chunk 78, which is 3,500 tokens into the prompt. The model reuses everything up to chunk 78 and prefills only chunk 103 plus the question: about 530 tokens instead of 2,030.

Run the numbers on a representative batch of 10,000 requests with 60% sharing that three-chunk prefix:

• Static tokens per request: 2,000. Chunk tokens per request: 2,000.
• Without stable ordering, assume 20% of requests hit the static prefix only. Cached tokens: 0.2 × 2,000 × 10,000 = 4.0M. Full-price tokens: 24.0M.
• With stable ordering, assume 60% of requests hit through the third chunk. Cached tokens: 0.6 × 3,500 × 10,000 = 21.0M. Full-price tokens: 19.3M.

At a base input rate of $2 per million tokens with cache reads at one twentieth of that (the ratio shown for several models on Anthropic's pricing table), the second configuration costs $0.10 × 21.0 plus $2 × 19.3 = $40.70, against $0.10 × 4.0 plus $2 × 24.0 = $48.40 for the first. That is roughly 16% off the input bill on a modest sharing assumption. Push the shared-prefix share higher and the gap widens. The exact discount depends on your provider and model; substitute your own read and write rates before quoting any figure.

The ordering code is small. Here is the pattern in Python:

def canonical_order(chunks):
    # Stable key first, relevance second. Ties broken by chunk id.
    return sorted(chunks, key=lambda c: (c["source_id"], -c["score"], c["chunk_id"]))

def render_context(chunks):
    # No timestamps, no request ids, no dict key order variance.
    lines = []
    for c in canonical_order(chunks):
        lines.append(f"[source:{c['source_id']}#{c['chunk_id']}]\n{c['text'].strip()}")
    return "\n\n".join(lines)

Two details matter more than the sort itself. First, strip and normalize whitespace in chunk text, because a trailing space reintroduced by a different PDF extractor breaks the chain. Second, do not interpolate a request ID, timestamp or user name anywhere above the chunk block. Those belong after the variable content, or in a header the provider does not tokenize into the prefix. OpenAI's guidance is the same shape: put static content at the beginning and variable content at the end (OpenAI prompt caching).

Even a perfect ordering function can be undone by infrastructure. Prefix caching depends on the serving engine keeping the entry and routing the request to a worker that holds it (Modular). In a distributed deployment, a standard load balancer scatters related requests across pods, destroying cache locality and forcing recomputation. One benchmark of prefix-aware scheduling reports 57 times faster response times and double the throughput on identical hardware compared with naive routing (llm-d). If you self-host, the scheduler is part of your cache strategy.

If you call a provider through a gateway, check whether it does sticky routing. OpenRouter keeps subsequent requests on the same provider endpoint after a cached request, with sessions expiring after ten minutes of inactivity, and it hashes the first system message and first non-system message to identify conversations by default (OpenRouter prompt caching docs). That means a stable first system message does double duty: it anchors the cache key server-side and the routing key gateway-side. You can also pass an explicit session identifier for multi-turn agent flows where opening messages change.

Caching also has a time dimension. Anthropic's cache has a five-minute default lifetime, refreshed free on each use, and the lifetime is measured from the start of the request that writes or reads the entry, so a slow streamed response eats into the window (Claude platform docs). A retrieval corpus that changes hourly is a poor fit for long-lived prefixes unless you accept that the chunk block will miss and only the static head hits. Amazon Bedrock users get the same tradeoff between implicit and explicit caching, with support varying by model and API (Bedrock prompt caching). Decide deliberately which layer you are optimizing.

The following sequence is the order that produces measurable wins fastest.

1. Freeze the head. System prompt, tool definitions, output schema and few-shot examples go first, in a fixed order, with no interpolated variables. Tools should be sorted: static tools alphabetically, per-user dynamic tools last, which is the pattern one production team used to keep a shared cache chain identical across users (ProjectDiscovery).
2. Canonicalize chunk order. Sort by source ID, then score, then chunk ID. Write the comparator into the retrieval module, not into each caller.
3. Version the serialization. Add a single version string to the head and bump it when the format changes on purpose, so you can correlate a hit-rate drop with a deploy. Do not add it per request.
4. Deduplicate and cap. Overlapping chunks from the same document waste prefix space and shift positions. Dedupe by document and text hash before ordering, then take the top N by score and re-sort canonically. Chunk size and boundaries already dominate retrieval quality, so fix chunking before tuning cache behavior (Weaviate on chunking).
5. Pin the route. Use session stickiness or prefix-aware routing so requests with the same head reach the same worker.
6. Measure. Log cached tokens and total prompt tokens per request, then compute the share as cached prompt tokens divided by total prompt tokens. Track response-cache hits separately, because the two mechanisms have different economics and should not be blended into one number (Metabase).
7. Alert on regressions. A prompt template change that breaks prefix stability can silently halve the hit rate and surface later as a cost spike (Metabase). Alert per app, not on the fleet average, which moves too slowly to catch anything.

A quick decision rule for step 4: if your top-N retrieval overlaps by fewer than one chunk across a typical query pair, chunk-ordering work will not pay for itself and you should focus on the static head instead. If overlap is three or more chunks, canonical ordering is the highest-return change available.

Most teams measure the wrong thing. The useful metric is a token-weighted share, not a request count, because a request that hits 100 tokens out of 4,000 is not a success. The SQL pattern below computes the share and an estimated saving per app over a rolling window, adapted from the metric definition at Metabase:

SELECT app_name,
       SUM(cached_tokens)::numeric / NULLIF(SUM(prompt_tokens), 0) AS cached_share,
       SUM(prompt_tokens - cached_tokens) AS full_price_tokens
FROM llm_request_rollups
WHERE window_start >= CURRENT_DATE - INTERVAL '30 days'
GROUP BY app_name
ORDER BY full_price_tokens DESC;

Two caveats. First, cached reads are discounted, not free, and writes can cost extra, so compute savings from recorded costs rather than hits multiplied by list price. Second, expect a ceiling. A workload with a large stable system prompt should see a high cached-token share; a long-tail creative feature with little repeated context may honestly sit near zero. Judge each workload against its own ceiling.

For teams running retrieval across several model providers, the ordering logic should live above the provider boundary, so a model swap does not silently reset your cache behavior. Quanta Inference exposes text, image, video, audio and embedding models through one OpenAI-compatible API key, which keeps the request shape constant while you compare hit rates across models: https://quantaintelligence.ai/inference