A Golden File Test an Agent Cannot Regenerate: Signing

The first instinct is to hash the golden file and store the hash in the repository. The comparison step recomputes the hash, sees it differs, and fails. That catches accidental edits, encoding changes and line-ending drift. It does not catch an agent, because the agent can edit the file and then edit the hash in the same commit. The check is a consistency check, not an integrity check.

The distinction matters:

The decision rule is simple. If only humans and honest tooling ever touch the expected outputs, a plain hash is enough. If an autonomous agent can write to the repository, the expected output needs a signature that depends on something the agent cannot read.

Storing the hash in CI configuration rather than in the repo raises the bar slightly, but an agent with write access to the pipeline can change it too. The secret has to live outside the agent's reach: in the CI secret store, in a local untracked file for developer machines, and never in the repository or the agent's context.

HMAC is a keyed hash: you feed it a secret key and a message, and it returns a digest that cannot be reproduced without the key. Python's standard library implements it as described in the official hmac documentation. The module's own warning is worth heeding: when you verify a digest supplied from outside, compare with compare_digest rather than ==, to reduce exposure to timing attacks. That applies here, because the signature file is an externally supplied value.

The design has three parts. A signing script that only a maintainer runs, which reads the golden file, computes the HMAC and writes a sibling .sig file. A verification function that the test calls before comparing anything. And a key that lives in an environment variable on trusted machines and in the CI secret store, never in the tree.

A minimal signer:

import hmac, hashlib, os, pathlib

def sign(path: pathlib.Path, key: bytes) -> str:
    body = path.read_bytes()
    return hmac.new(key, body, hashlib.sha256).hexdigest()

key = os.environ["GOLDEN_HMAC_KEY"].encode()
golden = pathlib.Path("tests/golden/invoice_rounding.json")
sig_path = golden.with_suffix(golden.suffix + ".sig")
sig_path.write_text(sign(golden, key) + "\n")

A verifier that fails closed:

import hmac, hashlib, pathlib

def verify(golden: pathlib.Path, key: bytes | None) -> None:
    sig_path = golden.with_suffix(golden.suffix + ".sig")
    if key is None:
        raise RuntimeError(f"GOLDEN_HMAC_KEY unset; refusing to compare {golden}")
    if not sig_path.is_file():
        raise RuntimeError(f"unsigned golden file: {golden}")
    expected = sig_path.read_text().strip()
    actual = hmac.new(key, golden.read_bytes(), hashlib.sha256).hexdigest()
    if not hmac.compare_digest(expected, actual):
        raise AssertionError(
            f"{golden} does not match its signature. "
            "The expected output was edited without re-signing, "
            "or the signature was edited without the key."
        )

Three details make this work in practice.

First, fail closed on a missing key. If the environment variable is absent and the test skips verification, the agent has a trivial bypass: unset the variable or run the suite in a context where it is not defined. Raising an error instead means the test cannot pass in an environment that is not allowed to compare golden outputs.

Second, verify before comparing. If the test compares values first and verifies the signature afterward, a mismatch on values gives a message that looks like a normal test failure, and the agent's reflex is to update the file. Verification first means the agent sees "this file was edited without a signature" and has nowhere to go.

Third, sign the exact bytes the test reads. If the test normalizes line endings or parses and re-serializes JSON before comparing, sign the same normalized form, or sign both the raw file and a canonical form, and verify both. Signing raw bytes while comparing parsed objects leaves a gap where the agent changes whitespace the signature covers but the comparison ignores, or the reverse.

A worked example of why this matters. Suppose a pricing test compares a generated quote against quote_basic.json, which currently holds a total of 1,240.00. A rounding change in the code produces 1,242.00. The agent to make the test pass edits quote_basic.json to 1,242.00, runs the suite, sees green, and opens a pull request. Without signing, the diff is a one-line change to a file named like a fixture, and a tired reviewer approves it. With signing, the suite fails with a signature error before any comparison, the pull request cannot go green, and the change becomes a deliberate act: a maintainer with the key reviews the new total, decides it is correct, and re-signs. If the total was wrong, the agent never gets to pretend otherwise.

The cost is roughly one extra file per golden file and a few lines of test setup. For a suite of a hundred golden files, that is a hundred small signature files, all generated and checked in together, plus the signer script.

The whole scheme collapses if the key is available wherever the agent runs. Treat the key the way you treat a production credential.

Two operational notes. Pull request workflows from forks typically do not receive repository secrets, which is the behavior you want: an outside contributor's golden file change arrives unsigned and the comparison job fails closed until a maintainer signs it. And a rotation plan matters, because a leaked key means every signature in the repository is worthless. Rotating the key means re-signing every file from a known-good state, which is itself a useful exercise: it forces a maintainer to look at every expected output at least once.

If you run agents locally, the key should be in the shell of the human, not the shell the agent inherits. A common pattern is a separate maintenance command, run by a person, that requires an explicit flag to re-sign and prints every changed file before writing signatures.

Signing stops an agent from editing the expected output on its own. It does not stop a well-meaning process from re-signing after a careless review, and it does not stop a maintainer from rubber-stamping a signature update. The second guard is a review rule, and it is cheap.

Make golden file changes a separate, labeled kind of change. In a pre-merge step, detect any diff that touches a file under the golden directory or any .sig file and require an explicit approval from a code owner who is not the author of the change. Tools like pre-commit already exist to run checks before a commit lands as the pre-commit hooks repository documents, and it is a natural place to enforce a rule such as "signature files may not be added in a commit that also modifies application code." That single rule forces the golden update and the code change to be reviewed as two things, which is exactly the review the diff-level process misses. The reasoning behind separating the two is the same as the one for keeping a verifier independent of the implementation: an agent that checks its own work is grading its own homework, and the fix is to make the judgment external a point made in this architecture write-up.

A short checklist for that review:

• Does the test fail for the right reason if the golden file is reverted and the code fix kept? Run it both ways.
• Is the change to the expected output explained in the pull request description in one sentence a non-author can check?
• Did the signature change alone, with no change to the golden file? That is a rotation or a mistake, and it needs a reason.
• Is the new expected output derived from a spec or an example a human wrote, rather than from the code's current behavior? Golden examples written by hand before the implementation are the anchor the rest of the review hangs on as this guide to agent test authoring recommends.

If you would rather have that judgment done by a fresh reader with no stake in the change, splitting authoring and review into separate steps is a pattern teams already use elsewhere, with the reviewer explicitly instructed to assume the change is broken until proven otherwise as the same architecture write-up notes. A verification step that knows what the output was supposed to be, before the code was written, is what closes the gap that diff-level review cannot as this guide to pre-merge verification argues.

Signing protects the expected output from being quietly changed. It does not protect against an agent that edits the test code to remove the verification call, or one that changes the comparison function to always pass. Test code is code, and it needs its own guard.

Code owners on the test directory is the cheapest control. A required review from a maintainer on any change under the tests directory means the agent's edit to the verification path is visible to a person before merge. Combine that with the fail-closed verifier, and an agent that removes the verifier call produces a test that raises an error rather than passes.

Mutation testing is the complementary signal for the suite as a whole. It introduces small changes into the source and reports which ones the tests failed to catch, so a suite that passes because its assertions were weakened is visibly weaker than one whose assertions still pull their weight as CircleCI explains. Run it periodically on the files around golden outputs. If a mutation in the rounding code survives, the golden file is not doing its job, and re-signing is not the answer. Expect some equivalent mutants that no test can kill, which the Stryker documentation describes as a normal and accepted part of the score per the Stryker mutants guide.

One more limit: signing protects the expected output, not the input. If a test generates a quote from a fixture input and the agent edits the input so the output matches, the signature on the golden file is intact and the test still passes for the wrong reason. Sign the input fixtures the same way, or generate the comparison from a fixed input defined in the test code.

No single row shows a complete answer, which is why the guards stack.

Signing adds a key to manage, a script to run and a review rule to enforce. For a throwaway prototype, that is too much. For a repository where an autonomous agent opens pull requests that a small team reviews, it is a reasonable cost, and the decision rule is about risk, not about preference.

Sign the golden files when all three of these hold: the expected outputs encode values that matter to the business, such as money, entitlements, access decisions or serialized contracts; an agent can write to the repository without a person reading every line it produces; and the team cannot guarantee that someone will notice a quiet one-line change to a fixture file in a large diff. If any one is missing, start with code owners alone and add signing when the situation changes.

If your team is putting an expert on the job of reviewing agent output and wants a reviewer with a defined role rather than a general-purpose model, Quanta Experts is built for that handoff: each expert reads your context, does the work and returns finished deliverables you can see the experts here. That pairs naturally with signing, because the signature check is the mechanical part and the review is the judgment part.