Running One Spec With the Cache On and Off

The test is easy to describe and easy to get wrong. The procedure:

1. Start the service with the cache disabled. Record every response body for the contract scenarios you care about (a fetch by ID, a fetch after a write, a fetch of a deleted record).
2. Start the same service with the cache enabled, cold. Repeat the scenarios until you also produce a hit: the same read at least twice.
3. Diff the payloads. Structural differences (new fields, missing fields) and value differences (same field, different value) are both findings.
4. Classify each finding as intended (the cache is allowed to add this) or a leak (the contract never agreed to it).

The trap appears at step 3. Suppose the no-cache service returns a balance of 12.40, and the cache-enabled service returns 12.40 from a hit after the balance was updated to 9.10. A schema-level assertion passes, because the shape is legal. A contract test that only checks shape has no opinion about which value is correct. This is the same failure mode contract testing was built to catch, moved one layer down: "schema valid" is weaker than "compatible," because a schema tells you what shapes are legal while a contract test tells you whether the behaviors a consumer relies on are still honored (Steve Kinney).

The worked example with numbers: an account endpoint has a 30-second TTL and a hit rate of 80% during a load test. If each scenario issues 1,000 reads, roughly 800 of them will be served from cache. A contract suite that only exercises a cold cache sees zero hits and therefore zero stale reads. That is why the second run matters and why the first read in each scenario is not the one that fails.

Four fields turn an invisible hit into a checkable one. This is the core decision rule of the article: a hit and a miss must be the same response type, and the hit adds fields rather than replacing them.

Why the freshness stamp and not just a time to live? Because a TTL is a promise, not a proof. The common failure is caching tool output without a freshness contract, and nobody defines how long the result is trustworthy, which is not caching but wishful thinking (Thinking Loop). The stamp puts the actual age in the payload so the consumer can decide.

Why source revision? Because freshness time and data freshness are different things. An entry can be two seconds old and already wrong if a write landed after it was cached. Tying the cached copy to the store's revision number gives the contract something concrete to compare across runs.

Why validation outcome? Because serving stale while revalidating is a legitimate strategy and a dangerous one depending on the endpoint. Teams that serve stale content during revalidation know they need that distinction in the response; consumer engineering teams should ask for it too. If a read path can return stale, the contract must say so, or a consumer will treat a deliberately stale response as fresh.

The contract should be written once and run against both configurations. The mechanics differ by tool, but the shape of the assertions is the same.

If you use a consumer-driven tool like Pact, each interaction declares a minimal expected response, and the provider later proves it can satisfy that interaction. That means the cache metadata has to be in the expectation, not bolted on afterward. The contract for a read becomes:

{
  "given": "an account exists with revision 7",
  "uponReceiving": "a second read of the account",
  "request": { "method": "GET", "path": "/accounts/42" },
  "response": {
    "status": 200,
    "body": {
      "id": 42,
      "balance": 12.40,
      "cache": {
        "status": "hit",
        "cachedAt": "2026-01-01T00:00:00Z",
        "sourceRevision": 7,
        "validation": "fresh"
      }
    }
  }
}

The provider side then has to produce a hit for this interaction, which is the assertion that catches a cache path that silently omits sourceRevision. For spec-driven workflows, the OpenAPI document gains a cache object with required properties, and conformance tests check the running implementation against it. Pact's own model treats the contract as an expected request and a minimal expected response, not just a JSON schema, which is why the cache object belongs inside the expected body rather than in a separate schema file (Steve Kinney).

Two assertion rules do most of the work:

• Exact field set on a miss. A fresh read must return the store fields plus the cache block with status: miss and a null or zero revision. If a miss omits the block, the contract has already drifted.
• Version comparison on a hit. After a write, a read must return either the new revision (correct invalidation) or validation: stale (explicitly allowed staleness). Returning the old revision as fresh is the failure the two-run experiment exists to find.

A comparison of the two configurations shows why a single run is not enough:

The last row is the entire argument. The contract passing on a stale hit is not a tool defect; it is a contract that never asked the question.

Not every endpoint needs this treatment, and adding cache metadata to a write path is noise. The decision rule is the four questions every cache entry should be able to answer: what is the acceptable staleness window, who invalidates the entry and when, what happens when the cache is cold, and what happens when the cache is wrong (Adam the Developer). If a read endpoint cannot answer the fourth question, it should not be behind a read-through cache, and the contract test should assert that no cache fields appear.

The write policy matters here too. A write-through cache writes to both cache and storage, which keeps them consistent but costs performance; a write-back cache writes only to the cache and flushes later, which is faster but risks losing data if the cache fails before the flush (Study.com). For a read-through layer in front of a SQL store, the cache is usually populated on read, so the relevant question is invalidation, not write policy. Cornell's cache notes make the clean statement of the trade-off: write-through is slower but cleaner because memory is always consistent, while write-back is faster but becomes complicated when multiple consumers share memory (Cornell CS 3410). In application terms, the shared memory is your consumer.

The flags analogy holds up. Cache lines carry metadata bits alongside the data: a valid bit that says whether the line holds real data, a dirty bit that says the line is newer than memory, and a lock bit that prevents replacement. When the processor starts up, every line is invalid, and clearing the cache is just setting those bits to invalid (Intermation). Freshness stamp, source revision, hit flag and validation outcome are the same idea at the API layer: the bits that let a reader tell what it is holding. General cache design treats these decisions as trade-offs to be made deliberately, not defaults to inherit (ETH Zürich lecture).

Once the cache fields are in the contract, the provider team owns them. A few practices keep the two-run experiment from decaying into a one-time audit.

Run both configurations in CI. The same contract file runs against a cache-disabled instance and a cache-enabled instance in the same pipeline. If a hit path stops emitting cache.status, one of the two jobs fails.

Test against something that behaves like the real store. Much of the confidence in these tests depends on the store's behavior under caching, and test doubles are usually a pale imitation. The EF Core team's guidance is direct: a database double may compare strings case-sensitively while the production system compares them case-insensitively, and only tests against the real system uncover that (Microsoft Learn). The same is true of a cache double: an in-memory dictionary will not reproduce eviction, TTL expiry or a concurrent invalidation.

Use the real store where the cache interacts with it. Jimmy Bogard's argument against in-memory test databases applies cleanly here: teams that swap in a fake end up writing the integration test anyway, and the fake's API does not match the real one's, so it can return results that do not relate to production behavior (Jimmy Bogard). A containerized real store, spun up by the test harness, gets you realistic invalidation without the fake's failure modes.

Publish expectations, not guesses. Contract tests exist because mocks drift; a consumer-driven contract generated from a running consumer and verified against the provider is the defense against that drift (PactFlow). The cache block belongs in that contract for the same reason every other field does.

Keep the test fast. Contract tests are milliseconds per test because they only touch the interface. Adding a hit path and a post-write read to each scenario multiplies the count but not the runtime, which is why this extra coverage is affordable where a full end-to-end suite with cache flushes would not be (Total Shift Left).

If your service calls several kinds of model as part of the read path (embeddings, a reranker, a summarizer) and the cache sits in front of all of them, the response shape is the only place a caller can see what happened. Quanta Inference (https://quantaintelligence.ai/inference) puts text, image, video, audio and embedding models behind one OpenAI-compatible key, which makes it practical to test the whole read path against a single contract rather than one per provider.