Property Tests for Deterministic Keys: Same Input, Same Key

Write the claim as a testable invariant: for any context tuple C, key(C) computed in process A equals key(C) computed in process B, byte for byte. That is the whole property. Everything else in the suite exists to make sure the property is not vacuously true and that the key actually tracks context.

Four properties, not one:

Note the tension between order invariance and sensitivity. You want the key to ignore the order in which fields happen to be stored and to react to every field's value. That combination is what forces canonical serialization rather than relying on whatever order the caller happened to use.

Property-based testing is the right tool because it generalizes over inputs instead of hard-coding one example. Hypothesis, the Python implementation, lets you declare strategies for valid inputs and then searches for counterexamples, shrinking any failure to a minimal case and remembering it for the next run (the author of Hypothesis in Increment). The same article's core critique applies directly to key tests: an example-based test named "keys are stable" tests one tuple and makes a general claim it cannot support.

The test suite only means something if the function under test is written to be deterministic. The pattern is: convert the context to a canonical form, then hash it.

import hashlib
import json
from dataclasses import dataclass

@dataclass(frozen=True)
class AgentContext:
    tenant: str
    intent: str
    params: dict

def canonical_bytes(ctx: AgentContext) -> bytes:
    payload = {
        "tenant": ctx.tenant,
        "intent": ctx.intent,
        "params": ctx.params,
    }
    return json.dumps(
        payload,
        sort_keys=True,          # field order cannot matter
        separators=(",", ":"),   # no whitespace variance
        ensure_ascii=True,       # no locale-dependent encoding
        allow_nan=False,         # reject NaN instead of emitting non-JSON
    ).encode("utf-8")

def idempotency_key(ctx: AgentContext) -> str:
    return hashlib.sha256(canonical_bytes(ctx)).hexdigest()

Four decisions carry the weight. sort_keys=True removes dependence on insertion order. Fixed separators remove whitespace variation between serializers. ensure_ascii=True makes encoding independent of the platform locale. allow_nan=False turns a float that cannot round-trip into a loud error rather than a key that disagrees across platforms.

Then normalize before you serialize. Lowercase the tenant, trim whitespace from intent strings, format decimals as strings with a fixed scale, and render dates in UTC with an explicit format. Every one of those normalizations is a place where two code paths can disagree, so each deserves a mutation test.

Keep an allowlist, not a deny list. Building the payload from three named fields rather than dumping ctx.__dict__ means a new attribute added to the dataclass cannot silently change every key in flight. The same discipline applies to tool parameter schemas: the schema is the contract, and anything outside it should be rejected rather than passed through (Prompt Engineering Guide on function calling).

The single most important detail is -I. Running the child process with sys.executable -I isolates it from the parent's environment and from user site-packages, and combined with PYTHONHASHSEED left unset, each child gets its own randomized hash seed. A key that iterates a set will fail here and pass everywhere else.

# tests/test_key_stability.py
import json
import subprocess
import sys

from hypothesis import given, settings, strategies as st

PROBE = r'''
import json, sys
sys.path.insert(0, sys.argv[1])
from keys import AgentContext, idempotency_key
ctx = AgentContext(**json.loads(sys.stdin.read()))
print(idempotency_key(ctx))
'''

def key_in_fresh_process(project_root, ctx):
    result = subprocess.run(
        [sys.executable, "-I", "-c", PROBE, project_root],
        input=json.dumps({"tenant": ctx.tenant,
                          "intent": ctx.intent,
                          "params": ctx.params}),
        capture_output=True, text=True, check=True, timeout=30,
    )
    return result.stdout.strip()

@given(
    tenant=st.from_regex(r"[a-z][a-z0-9\-]{1,20}", fullmatch=True),
    intent=st.sampled_from(["refund", "cancel_order", "update_address", "resend_invoice"]),
    params=st.dictionaries(
        keys=st.sampled_from(["order_id", "amount", "currency", "reason"]),
        values=st.one_of(st.integers(0, 10**6), st.text(max_size=20)),
        max_size=4,
    ),
)
@settings(max_examples=50, deadline=None)
def test_key_is_byte_identical_across_processes(tenant, intent, params):
    ctx = AgentContext(tenant=tenant, intent=intent, params=params)
    first = key_in_fresh_process(PROJECT_ROOT, ctx)
    second = key_in_fresh_process(PROJECT_ROOT, ctx)
    assert first == second
    assert len(first) == 64 and all(c in "0123456789abcdef" for c in first)

The regex strategy for tenant matters. st.text() will generate unicode, control characters and empty strings, and if your normalizer does not accept those, the test fails for reasons unrelated to determinism. Constrain to the domain the function promises to handle, and test rejection of out-of-domain input separately. Hypothesis strategies describe the range of valid values for an argument, which is how to read from_regex and sampled_from here (Theodo's Hypothesis walkthrough).

Two cost notes. Fifty examples means 100 subprocess spawns, which is roughly a few seconds on a warm machine and is fine on every commit. If your interpreter startup is slow, reduce to 20 examples on the commit hook and keep a nightly run at 500. The deadline is disabled because process spawn time is external noise.

Make the subprocess carry its own imports and nothing else. If the probe imports your agent framework, tracing setup or configuration loader, you are testing those too, and a failure will send you into the wrong file.

Stability alone is satisfiable by a constant. return "abc" is perfectly deterministic. Mutation tests close that hole by asserting the key moves when and only when meaning changes.

import itertools, pytest
from hypothesis import given, strategies as st

@given(
    tenant=st.from_regex(r"[a-z]{3,10}", fullmatch=True),
    intent=st.sampled_from(["refund", "cancel_order"]),
    params=st.dictionaries(
        keys=st.sampled_from(["order_id", "amount"]),
        values=st.integers(0, 10**5), min_size=2, max_size=2),
)
def test_field_order_does_not_change_the_key(tenant, intent, params):
    keys = {
        idempotency_key(AgentContext(tenant, intent, dict(p)))
        for p in itertools.permutations(params.items())
    }
    assert len(keys) == 1

@given(
    tenant=st.from_regex(r"[a-z]{3,10}", fullmatch=True),
    intent=st.sampled_from(["refund", "cancel_order"]),
    base=st.integers(1, 10**5),
)
def test_single_value_change_moves_the_key(tenant, intent, base):
    a = AgentContext(tenant, intent, {"order_id": base, "amount": str(base)})
    b = AgentContext(tenant, intent, {"order_id": base + 1, "amount": str(base)})
    assert idempotency_key(a) != idempotency_key(b)

itertools.permutations over a two-key dict gives two orderings, which is enough to catch insertion-order dependence. For larger parameter sets, cap the permutation count: factorial growth is real, and three or four fields sampled at random covers the failure mode.

Do the same for each field the key is supposed to see. A table of single-field mutations, each asserted to change the key, doubles as documentation:

That last row is worth a comment in your code. If you trim intent strings, say so, because a reviewer will otherwise expect the mutation to change the key.

Sensitivity has a mirror-image risk the retry literature makes concrete: changing a parameter on retry is itself a bug. When a duplicate request arrives with the same key but different data, a correct implementation rejects it rather than silently replaying the cached response (Zuplo). So add one test asserting that the key changes when a field changes, and a second asserting that your resume path refuses to send a request whose recomputed key differs from the stored one. The key proves identity; the resume path must enforce it.

If the downstream API does not offer idempotency at all, the same discipline still applies, and the honest framing is the one the Stack Exchange answer gives: you cannot fake another system's idempotency, so track call state, detect whether the effect landed, and escalate to a person when the outcome is unknown (Software Engineering Stack Exchange). A stable key is the part you control.

The failure mode you are guarding against is narrow and specific: a resume path that recomputes a key and disagrees with the original. That path usually lives behind a webhook or a scheduler, so it is not exercised by normal tests. Two cheap additions fix that.

First, a contract test. Store a handful of fixture tuples and their expected keys in a JSON file, commit it, and assert the current function reproduces them. This is a deliberate exception to "do not hard-code examples": the fixtures change only when you intend to break compatibility, and the diff is reviewable. A change to serialization that silently rotates every key will show up as a one-line assertion failure with a readable fixture diff rather than a mystery duplicate charge.

Second, a resume-path test with a fake key store. Simulate the sequence: create the context, compute the key, store the effect under that key, serialize the paused state, reload it in a fresh process, recompute, and assert the recomputed key is present in the store. If the store lookup misses, the test fails and so would the build.

# .github/workflows/keys.yml
name: keys
on: [push, pull_request]
jobs:
  property-tests:
    runs-on: ubuntu-latest
    env:
      PYTHONHASHSEED: random
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.13"
      - run: pip install pytest hypothesis
      - run: pytest tests/test_key_stability.py -q --hypothesis-seed=random
      - run: pytest tests/test_key_resume.py -q

Leaving PYTHONHASHSEED as random is the point. Setting a fixed seed makes the suite pass forever while production keeps rolling new seeds on every worker.

One more CI check worth the ten lines: run the stability test under a second locale (LC_ALL=C.UTF-8 and, if you support it, a locale with comma decimal separators) and under TZ=Pacific/Kiritimati. Timezone and locale leaks are the second most common cause of key drift after dict order, and neither shows up in default CI images.

If your agent runs across multiple models or providers, the same suite catches a subtler bug: a key built from a provider-specific response field. That is one reason a single API for text, image, audio and embedding models, such as Quanta Inference, keeps the context shape stable regardless of which model answered.

Suppose a refund agent keys on (tenant, intent, params) with params = {"order_id": int, "amount": int}. A run starts, computes a key, calls the payment provider, and the call times out after dispatch. The state is checkpointed and resumes on another worker twenty minutes later.

Without the suite: the resumed worker builds params from a database row returned as an unordered mapping. On 1,000 resumes, suppose 4 percent of resumed runs iterate the two keys in the opposite order. Each of those sends a differently keyed request. The result is 40 duplicate refund attempts per 1,000 resumes, each of which is a customer-facing double credit until reconciliation catches it.

With canonical serialization and the suite: the order-invariance property fails the build on the first commit where someone writes json.dumps(ctx.params) without sort_keys=True. That is one failing assertion, reproduced in shrinking to a two-key dict, caught before merge.

The cost side: 50 property examples with two subprocess spawns each is about a few seconds per commit on a standard runner. The mutation tests add a fraction of a second because permutations over two keys are trivial. Compare that to a single duplicate refund, which is a support ticket, a ledger adjustment, and a trust cost, and the tests pay for themselves at the first prevented incident.

For scale, an agent loop that retries without progress has been documented running 847 reasoning steps at roughly $47 per minute before delivering nothing (the reasoning-loop writeup). Duplicate effects are a different failure, but the economics rhyme: the expensive part is not the compute, it is the incident. The same article's remedy, a debounce hook that fingerprints (tool name, exact input), is structurally the same idea as a key: a fingerprint that must be stable to be useful.

That fingerprint detail is the bridge. If your debounce hook and your idempotency key both fingerprint the same context, they should share the canonical serializer and share the test suite. Two separate serializers is two chances to disagree, and the one that disagrees in production will be the one nobody tested. The escalation and pause-resume patterns make the same structural demand: preserve verified context across the pause and re-check state on resume rather than trusting the paused snapshot (escalation path design, HITL approval workflows).

A short checklist before you trust a green suite:

The decision rule: key on an allowlist of semantic fields, serialize canonically, hash, and test stability across processes plus sensitivity per field. If a field's presence in the key cannot be justified by "two operations that differ here should both execute", it does not belong in the key.

That rule is a design constraint, not just a test. Once you write the sensitivity test for every field, arguments about whether to include a session_id or a retry_count resolve themselves: the test either names the operation boundary or it does not.

For teams standing this up across several agent lines of business, Quanta Skills is the natural place to package the canonical serializer and its test suite so every expert reuses the same one rather than each team writing a fourth version.