A Test Suite for Precedence Rules: Cases That Catch Drift

Drift is not a dramatic break. It is a small reordering that changes outcomes in a narrow band. Someone moves "enrichment vendor" above "system of record" because the vendor is fresher for company size, and then a billing address change from the ERP stops reaching the customer record. The diff is a two-line move, it passes review, and it is invisible until a support ticket or an invoice goes to the wrong place.

The failure mode compounds with agents. An agent writing a test has a context window scoped to one feature, so the cheapest path to green is to reuse whatever fixture already exists. Across many sessions, a shared test world accretes, and a spec that expects one attendee starts failing because another spec quietly added a second to a shared fixture user. Precedence suites have exactly the same hazard: one shared "conflict" fixture reused by five cases, each expecting a different winner.

There is a second, quieter source: undefined ordering. A fixture runner that walks directories in raw readdir order returns files in filesystem-dependent order, and an exact ordered assertion fails at random on unrelated pull requests; the fix is to define the order at discovery, not to loosen the assertion to a set comparison. Precedence fixtures have the same trap when two writes carry the same rank and the same timestamp. Your ladder must define a tie breaker (for example, lowest source ID wins), and your fixtures must pin it.

The third source is scope shadowing. The Archon issue states the invariant plainly: a higher-precedence scope still shadows a lower one, keyed on the scope-relative label. That is exactly the property a scope-override fixture family tests, and exactly the property a careless fixture refactor removes.

Every case is a row set plus one expected outcome. Five fields are mandatory: source, field, value, scope. Add the attributes your ladder reads (rank, updated_at, deleted or stale flag) and one expected winner value per case.

Generate the rank column from the ladder itself, at test run time, rather than hard-coding it. If you hard-code ranks in fixtures, a reorder changes the ladder and the fixtures together, and nothing fails. If fixtures read ranks from the ladder, a reorder changes behavior and the expected winners no longer match, which is the signal you want.

A worked example. Suppose the ladder puts system of record above enrichment vendor above agent. Three writes land on the company_size field of account acct_8841:

• erp writes 1200, rank 5, live, updated 09:00.
• enrichment writes 1400, rank 6, live, updated 11:00.
• agent writes 1600, rank 7, live, updated 12:00.

Expected winner is 1200. The newer, larger values lose on rank. Now mark the ERP write deleted at 12:30. Expected winner becomes 1400, not 1600, because deletion removes a write from consideration but does not promote the agent above the vendor. That single pair of cases catches the most common bug in hand-rolled resolvers: treating deletion as "fall through to newest."

Write fixtures by boundary, not by source. Each pair of sources that can write the same field is a boundary, and each boundary gets at least one conflict case. With eight ladder entries, that is up to 28 pairs, but most pairs never write the same field, so the real count is the intersection of the ladder with your field ownership matrix. Start there; a typical CRM field set yields 30 to 60 cases, which runs in milliseconds.

Family 1: conflicting writes

For each pair, both sources write the same field in the same scope, both live, distinct timestamps. Record the higher-ranked value as expected. Include at least one case per pair where the lower-ranked write is newer, because that is the case a naive "last write wins" resolver gets wrong.

Family 2: scope overrides

A record-scoped manual edit versus an account-scoped admin set versus a tenant default, all writing the same field. The record scope must win even when the tenant default is fresher. Add the shadowing case from the Archon invariant: a tenant sdlc configuration must not pick up the sibling sdlc-ext label, and a record override under acct_1 must not leak to acct_2.

Family 3: deletion and staleness

Three variants per field: the winning write is deleted (the runner-up wins), the winning write is marked stale by an age threshold (the runner-up wins), and every write is deleted (the field resolves to empty or to the schema default, and you assert which). The third variant is the one that produces null-pointer incidents in production and is almost never tested.

Family 4: non-interaction

Two writes to different fields must both survive, with no cross-field precedence. Two writes to the same field in different accounts must resolve independently. Assert the full resolved record, not just the field under test, so that a resolver that mutates the whole row while deciding one field is caught.

Keep three things separate: the ladder definition, the resolver, and the fixtures. The resolver is pure: given a list of writes and the ladder, return the winning value per field. That signature is what makes a table-driven test possible.

# precedence_test.py
import pytest
from decimal import Decimal
from resolver import resolve, LADDER

CASES = load_yaml("fixtures/precedence_cases.yaml")

@pytest.mark.parametrize("case", CASES, ids=[c["case_id"] for c in CASES])
def test_precedence(case):
    writes = [
        Write(
            source=w["source"],
            field=w["field"],
            value=w["value"],
            scope=w["scope"],
            rank=LADDER.rank_of(w["source"]),
            updated_at=parse_ts(w["updated_at"]),
            state=w["state"],
        )
        for w in case["writes"]
    ]
    resolved = resolve(writes, LADDER, now=case.get("now"))
    assert resolved[case["field"]] == case["expected_winner"], case["case_id"]

Four details make this survive contact with a real repository. First, sort the writes by a total order (rank, updated_at, source ID) before resolving, so input order cannot affect the result and the filesystem-ordering flake cannot reach you. Second, set now explicitly for staleness cases so the suite does not change behavior with wall-clock time. Third, assert the full resolved record where the case defines one, not just the tested field. Fourth, fail on the first mismatch and print the case ID and both values, so a red build names the broken boundary.

Run it as a unit test in the same job as your other fast tests, and add a second job that regenerates the fixture ranks from the ladder and fails if the committed fixtures disagree. That job is what makes a ladder edit visible in review: reorder the ladder without updating fixtures and CI goes red with a list of case IDs whose expected winner changed. Playwright's model is a useful reference for scoping: tests in one file run in order in one worker, and parallel workers cannot share state, so isolated fixtures per worker are the safe pattern for any suite that runs in parallel.

A red precedence build has two possible meanings, and teams routinely pick the wrong one. The decision rule is: if the ladder diff in the pull request is empty, the resolver is wrong and the fixture is right. If the ladder diff is non-empty, read the changed pairs out loud. If the change is intentional, update the fixtures in the same commit and require a second reviewer for the fixture diff. If it is not intentional, revert.

The failure this prevents is the one described in the $180 CI post: each rerun looked like a flake, and the successful checks were the tax paid after pushing the suite back into a shape it tolerated rather than proof of health. A precedence mismatch is never a flake. It is either a rule change or a resolver bug, and both deserve a deliberate decision.

Two guardrails keep the rule honest. First, forbid fixture edits from a different author than the ladder edit, except through the reviewer rule above. Second, keep a ladder_version string in the ladder and stamp it into the fixture file header at generation; a mismatch between the two fails a dedicated test with a one-line explanation.

Agentic systems have the same conflict shape. When several agents with overlapping authority write to a shared store, goal conflicts, resource conflicts, policy conflicts and interpretation conflicts all need an explicit resolution order, and the playbook for agentic systems is to design resolution into the architecture rather than improvise it after deployment. Layered safety work in multi-agent control reaches the same conclusion structurally: when constraints from different agents conflict, you need a defined priority order among layers, because the intersection of pairwise safe sets is not the true safe set, a gap the reachability literature calls the leaky corner in multi-agent settings.

Interceptor ordering in request pipelines is the same problem in miniature. If several middlewares can rewrite the same header or body field, their order is a precedence ladder, and an untested reorder is a production incident. Timeouts are another: Playwright has six timeout types with defined fallbacks, and most "fixes" fail because the engineer changed the wrong one instead of the governing one. The fixture families above apply unchanged: pairwise conflict (two timeouts both apply), scope override (test-level beats global), and staleness (a disabled or expired setting must not win).

If you are adding model-written fields to the same records, the resolver is where you enforce that an agent inference never outranks a confirmed human value. Routing those model calls through one endpoint keeps the write path testable in a single place; Quanta Inference offers text, image, video, audio and embedding models through one OpenAI-compatible API key, prepaid and priced in US dollars (https://quantaintelligence.ai/inference).