When a Cached Brief Goes Stale: Catching Prefix Drift

Provider prefix caching keys on a hash of the literal token sequence up to the cache boundary. One changed byte before that boundary and you miss. That byte-fragility is well documented, including the unglamorous sources: tool description edits, tool ordering from dict iteration, a stray newline in a Jinja template, a session ID injected "for debugging" the byte-stability writeup.

Prefix drift is the inverse problem. The prefix has not changed, or if it has, it changed in a way that still matches the cache entry. The instructions moved underneath it. No cache key covers that, because the cache key is a hash of the prefix, not a hash of the prefix-plus-intent. OpenAI's diagnostics tooling will tell you why a request reused fewer tokens than expected by comparing model, tools, settings and input the diagnostics docs, but it will not tell you that a full hit is semantically wrong.

The decision rule follows directly and is worth stating plainly, because it inverts the usual instinct:

The row that matters is the second one. High hit rate plus rising complaints is the signature of drift, and it is invisible to every cost metric you own.

The rule is short enough to fit on an index card. At prompt assembly time, take the exact bytes of the cached prefix and the exact bytes of the live instruction block. Hash each one. Compare each against the hash recorded when the cache entry was accepted. On any mismatch, fail closed.

"Exact bytes" is load-bearing. Do not hash the parsed object, the template source, or a normalized string. Hash what you are about to send, after serialization, after any strip or join, after the model with its tool array in its final order. If you hash upstream of the final serialization boundary you will miss exactly the whitespace and ordering changes that cause the least visible drift.

import hashlib, json

def sha(b: bytes) -> str:
    return hashlib.sha256(b).hexdigest()

def assemble(prefix_bytes: bytes, instr_bytes: bytes, record: dict) -> dict:
    p, i = sha(prefix_bytes), sha(instr_bytes)
    mismatches = []
    if p != record["prefix_sha"]:
        mismatches.append("prefix")
    if i != record["instr_sha"]:
        mismatches.append("instructions")
    if record.get("pair_sha") != sha(prefix_bytes + b"\x00" + instr_bytes):
        mismatches.append("pair")
    return {
        "prefix_sha": p, "instr_sha": i,
        "pair_sha": sha(prefix_bytes + b"\x00" + instr_bytes),
        "verdict": "fail_closed" if mismatches else "ok",
        "mismatches": mismatches,
    }

The third hash is the one most teams skip and the one that catches the interesting cases. Hashing the prefix and the instructions separately tells you which half moved. Hashing the pair together with a separator byte tells you whether this combination was ever accepted as a working unit. Two blocks can each match their own acceptance record and still be a pair that nobody ever validated, for example when a rollback restores the old prefix while leaving the new instructions in place.

Record three fields on every acceptance: prefix hash, instruction hash, pair hash, plus a template version tag. Write them wherever you already write cache metadata. On a mismatch, the safe default is to rebuild the prompt from current sources with caching disabled for that call, in the same style as keeping dynamic content out of the cacheable region the caching engineering guide. You lose the discount on one call and keep the output correct.

One more detail: salt the hash with the model identifier and the cache scope. Provider caches are keyed per model, and upgrading models swaps cache pools the byte-stability writeup. If your acceptance record does not name the model, a model change will look like drift, and your alerting will cry wolf.

The test set is deliberately small. Six cases, one per root cause, each asserting that the detector fails closed and that the prompt that reaches the model is rebuilt rather than sent half-stale.

Case 1 is the common one and the cheapest to test: take a golden prompt, swap one instruction sentence, assert the verdict is fail_closed and the mismatch list contains instructions. Case 2 is the mirror. Case 3 catches serialization nondeterminism, which is why the byte-stability discipline sorts tools by name before serializing the byte-stability writeup.

Case 4 is the case people argue about. "It is only a newline" is true and irrelevant. The detector hashes bytes, so a newline is a mismatch, and the test asserts that. Case 5 is the sneaky one: a template release that renders identical bytes should not fail, because the rendered bytes are what gets hashed. Test it anyway, because the version tag is what you will use to trace a bad release later. Case 6 is the one that justifies the pair hash. Neither half mismatches its own record; only the combination is unvalidated.

Each case is three assertions: detector verdict, mismatch list contents, and the outbound prompt. That last one matters. A detector that flags the problem and sends the prompt anyway is worse than no detector, because it produces a clean log line next to a bad draft. Run the suite in CI on the prompt assembly module, not on the model, so the whole suite runs in milliseconds and blocks the merge.

Put the check at the single serialization boundary where prompts are built, not at each call site. Large systems fan in: one writeup describes around twenty distinct call sites funnelling through one provider module, and argues the defense belongs at that boundary because the threat is ordinary refactors rather than malicious bugs the byte-stability writeup. The same logic holds for drift. If three teams assemble prompts in three places, you need three detectors and you will maintain two.

The order of operations at the boundary:

1. Render and serialize the prefix to bytes. Freeze it.
2. Render and serialize the instruction block to bytes.
3. Hash both and the pair.
4. Compare against the acceptance record for this prefix, instruction set, model and scope.
5. On match, attach the cache marker and send. On mismatch, rebuild with cache disabled and emit a structured log line with all three hashes and the mismatches.

Step 5 is where the observability lives. Log the mismatch as an event with a bounded set of cause labels (instruction, prefix, pair, model, version), not a free-text message. That gives you a dashboard that answers "which cause dominated this week" instead of "someone saw something."

The economics of failing closed are easy to justify. A rebuilt prompt pays full input price on the shared block for one call. At the $3 per million token example above, rebuilding a 12,000-token prefix once costs about three cents extra versus a cached read. One bad draft that ships to a customer, a partner or a regulator costs orders of magnitude more. Cache engines are also built to tolerate misses: prompt caching reduces time to first token and input cost, and output is recomputed and billed in full either way the caching explainer.

Keep the cache boundary explicit and the drift surface small by ordering content most-to-least stable and keeping everything variable after the boundary the caching engineering guide. A prefix that contains only truly static material drifts less, because there is less in it that could be edited by someone who does not know it is shared.

Five patterns account for most of the failures I have seen described.

Hashing the source, not the bytes. Hashing a template string or a config object means whitespace and ordering changes vanish before hashing. Always hash the serialized payload.

Checking at only one call site. The second team's path skips the check, and that is the path that drifts.

Treating a mismatch as a warning. A warning in a log is a notification that nobody reads. Fail closed or do not bother.

Forgetting the model in the key. Model changes swap cache pools and upgrade cadence is a real source of drift the byte-stability writeup. Salt the hash.

Letting the acceptance record go stale. If the record is written once at deploy and never refreshed after an intentional change, every intentional edit looks like drift and the team switches the detector off. Refresh the record on every deliberate acceptance, and make the refresh a reviewed change.

There is a design question underneath all five: is your cached prefix a product of a versioned template, or of code that runs at request time? The first is checkable. The second is not, because there is no fixed thing to hash. If you find yourself unable to name the bytes of your prefix, that is the drift already.

Prefix drift is one member of a family of prompt-assembly problems: stale retrieval context, mismatched tool definitions, instructions that outlive the policy they were written for. The detection rule generalizes, but only if there is one place where prompts become bytes. That is an architecture decision more than a code decision, and it is the one worth making before the test set.

For teams running multiple model providers or many internal apps, keeping the assembly boundary consistent across all of them is the hard part. A single API surface for every kind of model, with the same request shape for text, image and embedding calls, removes a whole class of per-provider drift Quanta Inference. The hashing rule, the acceptance records and the six-case suite then apply once rather than once per vendor.

If you would rather have someone build the assembly boundary and the test suite with you than start from a blank file, that is the kind of scoped piece of work a specialist can pick up and finish with Quanta. Either way, the rule is the same: hash what you send, compare against what you accepted, and fail closed when the halves disagree.