A Fixture Manifest an Agent Cannot Rewrite: Paths and Hashes

Keep the format dumb on purpose. One entry per line, space separated, no nesting, no globs. A human should be able to read the diff and a script should parse it in twenty lines.

A line looks like this:

testdata/billing/seed_accounts.csv 9f2c8d1e4b7a0c5f3e2d1b0a9c8d7e6f5a4b3c2d1e0f9a8b7c6d5e4f3a2b1c0d 48213 bytes Seed rows for the invoice rounding golden

Five decisions make the schema hold.

Relative paths only. An absolute path or a .. segment is a parsing error. That keeps the manifest valid across machines, containers and CI runners, and stops a fixture from reading a home directory that differs per developer.

SHA-256 rather than a checksum. You need a digest where a one-bit change in a 50 MB file changes roughly half the output bits, so a corrupted or swapped fixture is caught rather than colliding by accident. The Python standard library exposes this through hashlib, and keyed variants through hmac per the Python docs. For fixtures you do not need a key.

Size as a cheap cross-check. The digest catches every change, but size catches the common cases with a number a reviewer can eyeball. If size matches and the digest does not, bytes were edited in place. If size differs, the file was rebuilt or swapped. That distinction saves review time.

A mode field. Hashing raw bytes is correct by default and wrong when a file legitimately varies by checkout: CRLF versus LF, a trailing newline an editor adds, machine-specific paths in a config. In those cases you hash a normalized form and say so, so a reviewer knows the digest does not cover everything.

A reason field. Without it, nobody knows six months later whether testdata/legacy/blob_04.bin is load-bearing. Write why the file is there, not what it contains.

One formatting choice is not obvious: never let a flag rewrite the manifest. The GopherTrunk guidance is blunt that a golden regenerated without review converts a behavior-pinning suite into an it-does-whatever-it-does suite in that same lesson. A script may print proposed entries. The test run may not write them.

The decision rule fits in one line: hash raw bytes unless the file is known to differ between a developer checkout and CI for reasons the test does not care about.

Normalization must be defined, versioned and applied identically wherever the digest is computed. Two lines in the repository beat a paragraph of prose. A workable definition for text: decode as UTF-8, convert CRLF and lone CR to LF, strip trailing whitespace per line, ensure a single trailing newline, re-encode, then hash. For structured fixtures with volatile fields, replace IDs with stable counters and timestamps with a fixed sentinel before serializing, an approach described for golden files generally in this write-up.

The warning that comes with normalization: a normalized digest stops protecting the fields you normalized away. Normalize only what the test genuinely ignores, and say so in the reason field, for example Volatile created_at, normalized before hashing.

Write the verifier yourself. You need error messages that name the file, the expected digest and the observed digest. A minimal version:

import hashlib, pathlib, re, sys, hmac

HEX = re.compile(r"^[0-9a-f]{64}$")

def digest(path, mode):
    raw = path.read_bytes()
    if mode == "bytes":
        return hashlib.sha256(raw).hexdigest()
    if mode == "normalized":
        text = raw.decode("utf-8").replace("\r\n", "\n").replace("\r", "\n")
        text = "\n".join(l.rstrip() for l in text.split("\n")).rstrip("\n") + "\n"
        return hashlib.sha256(text.encode("utf-8")).hexdigest()
    raise ValueError(f"unknown mode: {mode}")

def main(manifest_path):
    root = pathlib.Path(".").resolve()
    failures = []
    lines = pathlib.Path(manifest_path).read_text().splitlines()
    for number, line in enumerate(lines, 1):
        if not line.strip() or line.lstrip().startswith("#"):
            continue
        path_s, want, size_s, mode, reason = line.split(" ", 4)
        if path_s.startswith("/") or ".." in pathlib.PurePosixPath(path_s).parts:
            failures.append(f"line {number}: path must be relative and stay in the repo")
            continue
        if not HEX.match(want):
            failures.append(f"line {number}: digest is not lowercase SHA-256")
            continue
        if len(reason.strip()) < 20:
            failures.append(f"line {number}: reason is too short")
            continue
        target = root / path_s
        if not target.is_file():
            failures.append(f"line {number}: missing fixture {path_s}")
            continue
        if target.stat().st_size != int(size_s):
            failures.append(f"line {number}: size mismatch for {path_s}")
        got = digest(target, mode)
        if not hmac.compare_digest(got, want):
            failures.append(f"line {number}: {path_s}\n  expected {want}\n  observed {got}")
    if failures:
        print("fixture manifest check failed:", file=sys.stderr)
        print("\n".join(failures), file=sys.stderr)
        sys.exit(1)
    print("fixture manifest check passed")

if __name__ == "__main__":
    main(sys.argv[1] if len(sys.argv) > 1 else "fixtures.manifest")

Two details matter more than the rest. Compare digests with hmac.compare_digest, which the Python docs recommend over == to avoid timing attacks per the same documentation; that is overkill against a checked-in fixture, but it costs nothing and removes a review comment. Second, collect every failure in one pass. A log that reports only the first mismatched fixture turns one run into five.

Wire the check in as a pre-commit hook and a CI step. The pre-commit framework is the common starting point for hook wiring in the pre-commit-hooks repository, and a local hook gives developers the failure in seconds. Reaching for a hook is standard practice; a local pytest hook is configured in .pre-commit-config.yaml with the test command as its entry, as this Stack Overflow example does on this thread. CI then catches anything committed with hooks skipped.

Order the CI steps so the manifest check runs before the golden tests. A golden test that runs against unverified fixtures produces a diff that a reviewer may approve by regenerating the golden, which is the wrong fix when the real change was an input.

A manifest entry should never change silently. When it does, four steps follow and none are optional.

1. The check fails and names the path, the expected digest and the observed digest.
2. The author decides whether the input change is intended. If not, the fixture is restored.
3. If it is intended, the commit carries a review record: a Fixture-Change: trailer naming the paths and the reason, or an approval file in the repository.
4. Only after that record exists is the manifest updated and the golden regenerated and re-signed.

Git supports step 3 directly. Trailers live in the last paragraph of a commit message, only there, and git interpret-trailers reads and writes them per the Git documentation. A convention that a checker enforces, with a gate trailer required exactly when a gated file changes and never otherwise, is described in this commits and reporting skill in the skill file. Copy the shape for fixtures:

fix(billing): round invoice totals at the line level

Fixtures for the rounding golden were rebuilt after the schema migration
landed. Totals change only in the multi-currency cases.

Fixture-Change: testdata/billing/seed_accounts.csv purpose=rebuild source=migration

Two rules make trailers work where free-text messages do not. The trailer appears exactly when a manifest path is in the commit and never otherwise, so git log --grep '^Fixture-Change:' lists every input change and nothing else. The reason value has a minimum length, so a stub fails the checker.

If your team does not want a commit-message gate, use an approval file. A small Markdown file under reviews/fixtures/, named after the fixture path, records the date, the old and new digest and the approver. It has one advantage over a trailer: someone other than the author can write it, which is what review is supposed to mean. The trade-off is that it can land in a separate commit, so the checker looks for an approval at the current commit or its parent, not any approval ever written.

The decision rule: trailers where every engineer can review a fixture diff, approval files where fixtures derive from regulated or customer data and a second person must sign off.

The reflexive regenerate. The common end state is a suite that runs an update flag on every failure, or a human runs it and commits without reading the diff. The GopherTrunk lesson states the consequence: whatever the code currently produces, bug included, becomes the new expected output from the same lesson. The manifest does not stop this alone, but it puts the input change and the golden change in one commit, where a reviewer has a chance.

The agent shortcuts the test. ImpossibleBench is built around models weakening tests on impossible tasks and reports large modification rates via Gabriel Anhaia's write-up. The manifest catches a related shortcut: edit a fixture so the output matches and the digest check fails.

The verification gap in agent review at scale. Review asks whether the implementation is sound; regression analysis asks which established behavior the change may have altered, and those are different jobs per this regression guide. Pre-merge verification against a spec catches contract and architectural regressions that pass tests and diff review per Augment Code's pre-merge guide. The manifest is the smallest version of that idea applied to data.

Tests that pass without asserting. A suite can be green with full line coverage and still miss defects, because coverage counts lines run, not assertions that caught something; mutation testing finds those gaps by introducing small changes and reporting survivors as CircleCI explains. The manifest is the input-side analogue.

Advice on the tests themselves overlaps. Keeping assertions focused so a test fails only when behavior is wrong, not when implementation is refactored, is the point of the assertion rule from this walkthrough. Arrange, act, assert keeps each test readable as a statement of intent per these unit test tips. Writing the test first makes it a statement about the requirement rather than the code as described in this TDD write-up. The manifest extends all four to the data.

One billing golden reads 12 fixtures: 4 CSV seeds, 3 JSON payloads, 2 rendered email bodies, 2 config fragments and 1 binary attachment, about 4 MB together. SHA-256 runs at hundreds of megabytes per second on commodity hardware, so the check finishes in well under a tenth of a second. The golden test itself renders a 61 KB document and diffs it. The manifest is not the expensive part.

Now a migration changes how currency codes are stored and a script rebuilds seed_accounts.csv. Size moves from 48,213 bytes to 48,301 bytes and the digest changes.

The manifest turns a silent input change into a visible one with a number attached. After review, the trailer is added, the manifest updated and the golden regenerated and re-signed in one commit.

The same arithmetic argues against listing the whole repository. A 500-entry manifest still runs fast, but every entry needs a reason and a reviewer, and a migration touching 40 fixtures produces 40 reviewed lines. List only what a golden test actually reads. If three goldens read one fixture, list it once and name the tests in the reason.

When fixtures derive from live model outputs, regenerate them through a pinned call so the rebuild is deterministic. Quanta Inference puts text, image, video, audio and embedding models behind one OpenAI-compatible key, which makes regeneration a single call rather than per-model work (Quanta Inference). If a coding agent performs the rebuild, the commit and manifest rules can travel as a skill (Quanta Skills).