Contract Tests Between a Policy Engine and Its Memory Store

Contract testing treats an interface as a machine-checked agreement rather than a comment. The standard formulation is that a contract captures the interactions a consumer actually relies on, not just JSON shapes, so it verifies compatibility rather than mere schema validity Steve Kinney on API contract testing. The narrowness is the point: contract tests answer one question about the boundary and are fast and deterministic because they touch nothing else Total Shift Left contract testing guide.

Adam Wathan's writeup of preventing API drift makes the specific argument this article builds on: interfaces give a false sense of security, tests make better contracts than interfaces, and the way to enforce it is to run multiple implementations against one shared contract suite Preventing API Drift with Contract Tests. That is the whole pattern. The fake that the policy engine's unit tests use and the production adapter are two implementations of the same interface, so they must pass the same suite.

For a memory store, the suite should describe behavior: values written come back unchanged, deletes remove the key and a second delete on the same key raises the missing-key error, bulk put is all-or-nothing or documented as partial with the exact reporting shape, pagination returns every record exactly once across pages, a TTL'd key is gone after expiry and absent before it, and a transient failure is retried according to the stated policy. Keep implementation details out, because the fake cannot reproduce them and the real store's details are exactly what you are testing.

The practical payoff: a different team changed a schema and nothing watched the boundary, so the break shipped and a mobile client started failing in production Total Shift Left contract testing guide. An if in a code review is not a boundary check; a shared suite is.

Write the suite as an abstract base test class or a table of cases that each implementation's test runner binds to. Then write down what each behavior means, because ambiguity here is where drift hides.

A worked volume rule helps decide whether the suite belongs in CI: a healthy suite of about 40 cases should run in under a second against an in-memory fake and under 30 seconds against a staging store, which is why the CI run needs no isolation tricks. The EF Core team reports over 30,000 tests against SQL Server alone that complete reliably in a few minutes on every commit Microsoft EF Core testing strategy, so "the real store is too slow for CI" is usually an assumption, not a measurement.

Two contract areas are easy to miss and expensive to omit. First, serialization: the fake stores Python or JSON objects directly, while the real store round-trips through a wire format, and a timestamp that survives one and not the other is a classic silent bug. Second, concurrency: the fake is usually single-threaded while staging is not, so make the contract state whatever the policy engine assumes about concurrent writes.

A fake that the policy engine's unit tests use is a test double, and test doubles fall out of sync with the real thing Preventing API Drift with Contract Tests. That is exactly why the fake runs the contract suite in CI. Teams debate fakes versus in-memory databases for DAO testing, and the recurring verdict is that a double is fine for speed but proves nothing about production behavior Testing DAO methods: fakes vs in-memory databases. Microsoft's guidance says the same: passing against a double does not guarantee working against the real resource, because a double may behave in ways the production system does not Microsoft EF Core testing strategy.

A subtler failure is the test double whose API itself diverges from the real implementation's API, and the fix is to run both against shared contract tests Preventing API Drift with Contract Tests. Jimmy Bogard's account of abandoning in-memory providers is the cautionary version: his teams wrote the same tests twice, once against the fake and once against the real database, and asked what the value of two tests was; the answer is one test suite run twice Avoid In-Memory Databases for Tests. That single test codebase run twice is the pattern this article recommends, and it avoids the duplication Bogard objected to.

There is a real limit worth stating plainly: an in-memory fake can never prove the real store's behavior, so the same suite must touch staging before release, not just the fake in CI Microsoft EF Core testing strategy.

Keep the suite in its own module so both test runners can import it. A shape that works:

# contract/memory_store_contract.py
import pytest

class MemoryStoreContract:
    @pytest.fixture
    def store(self):
        raise NotImplementedError

    def test_put_then_get_roundtrips(self, store):
        store.put("k", {"n": 1, "tags": ["a"]})
        assert store.get("k") == {"n": 1, "tags": ["a"]}

    def test_get_missing_raises_not_found(self, store):
        with pytest.raises(StoreNotFound) as exc:
            store.get("never-written")
        assert exc.value.code == "NOT_FOUND"

    def test_second_delete_raises_not_found(self, store):
        store.put("k", {"n": 1})
        store.delete("k")
        with pytest.raises(StoreNotFound):
            store.delete("k")

# tests/unit/test_fake_contract.py  (runs on every CI build)
from contract.memory_store_contract import MemoryStoreContract
from policy_engine.memory import InMemoryStore

class TestFakeStore(MemoryStoreContract):
    @pytest.fixture
    def store(self):
        return InMemoryStore()

# tests/staging/test_real_contract.py  (runs before release)
import os
from contract.memory_store_contract import MemoryStoreContract
from policy_engine.memory import RemoteStore

class TestRealStore(MemoryStoreContract):
    @pytest.fixture
    def store(self):
        yield RemoteStore(os.environ["STAGING_MEMORY_URL"])

Two rules make this durable. Put the fake and the real adapter behind the same small interface so the policy engine takes either one, the repository pattern's reason for existing Designing the infrastructure persistence layer. And decide once, in writing, which failure a user action must never show, so the contract has teeth.

When the suite fails, the test name plus which run failed narrows the diagnosis fast. Use this decision rule:

Intermittent staging failures deserve a special look. If the contract says a key expires after one second and the real store rounds TTLs to a coarser boundary, the test becomes flaky, and a flaky contract test is worse than none because teams start ignoring it. Contract tests exist precisely to replace sprawling, flaky, hard-to-debug end-to-end suites with fast, deterministic boundary checks Steve Kinney on API contract testing, so flakiness is a signal to tighten the specification, not to retry.

The pattern generalizes beyond a memory store. Any dependency behind a policy (a model router, an embedding service, a vector index, a feature store) can have one contract suite run against its fake and its staging instance. That is especially true when the dependency is a model provider, because API behavior changes under you more often than a database's does. Teams running a policy engine over mixed models can standardize the provider behind one interface and attach the same suite, which is the natural use for Quanta Inference as a single OpenAI-compatible entry point for text, image, video, audio, and embedding models.

Before the first release that adopts this pattern, confirm each item:

• The contract suite lives in its own module and imports nothing from the policy engine beyond the interface.
• The fake and the real adapter both satisfy the same interface type, with no extra methods on the fake.
• CI runs the suite against the fake on every build and blocks the merge on failure.
• Staging runs the same suite with the same test names and blocks the release on failure.
• Every error code in the suite is asserted as a code, not just as "raises something."
• Serialization is covered: at least one nested object and one timestamp round-trip.
• TTL cases use margins wide enough that they pass every time, or the contract states the granularity.
• An owner is named for the contract file, the fake, and the real adapter.
• Pagination is tested at a size that forces more than one page.
• Concurrency assumptions, if any, are written down where both sides can find them.

If you would rather not maintain the fake and the adapter yourself, or you need the pattern applied across a line of business, a consultant can help wire it in beside your existing pipeline or a specialist can build the harness with you at Quanta.