Fixture Files for a Verification Hop

Log at the boundary, not inside the producer. The fixture should be what the hop actually receives, byte for byte, not a reconstruction from the producer's internal state.

A practical capture procedure:

1. Pick a fixed window: one day, one release cycle, or one representative traffic slice. Long enough to include the rare shapes, short enough that you can process it in one sitting.
2. Log the raw payload plus three pieces of metadata: producer identifier, hop stage name, and a timestamp.
3. Strip what you must not keep (access tokens, personal data) before anything lands on disk, not after.
4. Store the window as a flat directory of files, one payload per file, so the next step can hash them.

On the types of failures worth capturing, the fault-injection literature is a useful guide. MAS-FIRE defines a taxonomy of 15 fault types covering intra-agent cognitive errors and inter-agent coordination failures, and it shows how semantic failures propagate silently without raising runtime exceptions (MAS-FIRE). You do not need all 15 in your fixture set, but the taxonomy tells you which shapes to look for during capture: misunderstood instructions, reasoning drift, malformed arguments handed between stages.

Worked example: suppose the capture window yields 12,000 payloads at the hop. Hashing by content tells you those 12,000 files collapse to 41 distinct hashes. Of those 41, maybe 6 are shapes nobody on the team would have written by hand. Those 6 are your first fixtures. The other 35 are already covered by existing tests, and the remaining 11,959 are duplicates that add nothing but load time.

If the producers are themselves agents, capture the payload after the model has responded, not the prompt. The output is the object that hits the hop, and that is the object the fixture needs to preserve.

Deduplication is mechanical: normalize the payload, hash it, keep the first file per hash. The harder question is which distinct shapes deserve a fixture. That is a sampling decision, and it is where most teams either over-collect or under-collect.

Sample by coverage, not by frequency. A shape that appears 4,000 times a day is almost certainly already handled, because if it were broken you would know by now. A shape that appears once a week and has never thrown an error is more interesting: it passed the hop without being meaningfully exercised. That is the payload that slipped past the hop, and it is the one worth freezing.

The tooling for this reduction is standard in observability work. Trace collection, failure clustering and root-cause analysis take hundreds of individual error events and reduce them to a prioritized list of patterns (Latitude's failure detection guide). You can run the same reduction against your captured window: cluster the 41 distinct shapes, look at which clusters have thin test coverage, and promote one representative from each.

Two decision rules worth writing down:

• Promote on evidence, not on intuition. A shape becomes a fixture when it either (a) caused a production failure, or (b) is a distinct branch the hop has never exercised. Anything else stays in the capture window and gets deleted.
• Keep the smallest payload that reproduces the shape. If the original file is 400 KB because it contains a repeated block, trim it to the smallest input that still reaches the same branch. The fixture's job is to reach the code path, not to be a perfect replica.

A third rule, easy to forget: record the sampling decision. Every fixture should carry a sidecar note saying which window it came from and why it was promoted, so a reviewer a year later can tell a deliberate sample from an accident.

When the hop fails in production, the failing payload becomes a fixture the same day. That is the single highest-value habit in this whole process, because it converts every incident into a permanent regression test.

The mechanics are small:

1. Copy the exact failing payload into the fixture directory.
2. Add or update a test that replays it through the hop and asserts the corrected behavior.
3. Name it (see the next section) and record in a sidecar file which incident it came from.

The fault-injection research supports spending effort here rather than on broader synthetic suites. MAS-FIRE finds that architectural topology plays a decisive role in robustness, with iterative, closed-loop designs neutralizing over 40% of faults that cause catastrophic collapse in linear workflows (MAS-FIRE). A fixture suite is a cheap way to make your linear verification path behave more like a loop: every failure gets fed back in, and the hop gets measurably harder to break.

A second finding is worth internalizing. The Agent Error Simulator, which injects format, logic and tool-call faults into a planner-worker-aggregator workflow, reports a model-size-dependent logic error recovery threshold between 7B and 27B parameters, and universal catastrophic failure under aggregator-level logic injection (Agent Error Simulator). Translated to fixtures: some failures will break the hop no matter how many tests you add, because the design cannot recover from a bad upstream stage. Fixtures tell you which ones those are, so you stop debugging the parser and start fixing the topology.

A minimal CI job looks like this:

verify-hop:
  script:
    - run-hop --fixtures ./fixtures --fail-on-error
  artifacts:
    when: on_failure
    paths:
      - fixtures/failures/

The artifact step matters more than it looks. When the hop fails in CI, you want the offending payload saved as a build artifact so it can be triaged and then promoted into the fixture set without hunting through logs.

A fixture directory becomes useless within a quarter if filenames do not say what they contain. Adopt a fixed three-part name and never deviate:

<producer>_<hop-stage>_<shape-tag>.<ext>

Examples:

The rules that keep this workable:

• Lowercase, underscores only. No spaces, no camelCase, no version suffixes in the name. Versioning belongs in the sidecar metadata, not the filename.
• The shape tag describes the anomaly, not the fix. nested_headers says what the input looks like. fix_nested_headers will lie the moment someone changes the parser.
• One anomaly per file. If a payload is both truncated and nested, split it into two fixtures so a failure points at one cause.
• Stage names are fixed. Pick them once (parse, validate, map, adapt) and enforce them with a lint step in CI that rejects any file whose second segment is not on the list.

This mirrors what mature agent protocols do with identity. The A2A specification publishes an Agent Card at a well-known path defining the agent's identity, endpoint, skills and security schemas, so a caller can discover what it is talking to before delegating any work (A2A communication guide). Your fixture names are the same idea at a smaller scale: predictable enough that tooling and humans can both find what they need without asking.

Unbounded fixture directories are the classic failure mode. They start at 20 files and reach 3,000, and then two things happen: the CI job takes 15 minutes, and nobody dares delete anything because nobody knows what is still needed.

Set a budget with numbers and enforce it. A reasonable starting point:

When you exceed a budget, the fix is retirement, not a bigger budget. Raise the budget only when you can name the new failure class it protects against.

Two operational notes. First, compress the directory if your repo does not already do it; text fixtures compress well and the size budget is about checkout cost, not disk. Second, treat the budget as a contract: a lint step that fails the build on an oversized file is more effective than a wiki page nobody reads.

There is a related discipline in CI systems generally: passing data between tasks should be explicit and typed, not implicit in a shared directory. The SourceCraft CI documentation shows named outputs declared on a producer task and consumed downstream only through explicit needs references, with circular dependencies disallowed (providing data between CI/CD tasks). Apply the same thinking to fixtures: a manifest file listing every fixture and the test that consumes it makes orphaned files visible, and visible orphans get deleted.

Retirement is the half of fixture maintenance everyone skips. A fixture earns its place only while it does at least one of two things: protects against a live regression risk, or exercises a branch that is still reachable.

Write the policy down and run it on a schedule (quarterly is usually enough):

1. Retire on schema deletion. When a producer stops emitting a shape and the schema has removed it, the fixture is dead weight. Retire it with the schema change, not later.
2. Retire on an unreachable branch. If coverage tooling shows the branch a fixture was written to reach is now unreachable, the fixture is noise.
3. Retire duplicates. Every quarter, re-hash the directory. If two fixtures produce identical hop output, keep the smaller one.
4. Age with a prompt, not an auto-delete. Any fixture untouched for a year gets flagged in the manifest and reviewed by its owner. Auto-deletion without review is how you lose the one fixture that mattered.

The rationale for scheduled review rather than continuous pruning comes from the failure literature. Silent failures in agent systems accumulate and are often detectable only through systematic measurement after the fact: in one production study spanning more than 100,000 agent interactions, a failure occurred on the 47th day with no code change, model update or configuration change (Silent Failure in LLM Agent Systems). A fixture set that only grows is a snapshot of old assumptions. A fixture set that is reviewed is a living record of what the hop is actually exposed to.

For teams running the review, a simple rule keeps it short: a fixture survives if a named person can say, in one sentence, which risk it protects. If nobody can, retire it.

A fixture directory is a process artifact before it is a technical one. It only stays useful if capturing, naming and retiring are ordinary steps in the release path rather than a special project.

The strongest version of this: every producer that emits a payload gets its own fixture folder named after it, and every incident that touches the verification hop ends with a fixture, not just a fix. The five-point security checklist for AI agents makes a parallel point about inventory: most organizations cannot answer how many agents have access to production, and shadow AI is the new shadow IT (multi-agent security checklist). The same inventory discipline applies to producers: if you cannot list your producers, you cannot know when a fixture has gone stale.

If the producers are agents running through a shared platform, keeping the fixture set current is cheaper, because you log real outputs at the boundary and replay them without re-running the model. That is one reason a single unified endpoint for text, image and embedding models helps: the payload shapes you capture stay consistent across model families (Quanta Inference).

The final safeguard is a trust-boundary view. Multi-agent systems fail when a downstream stage accepts a payload without a way to verify its provenance or scope, and the missing control is verifiable delegation rather than just least privilege (multi-agent trust boundaries). Fixtures do not solve delegation, but they do give you a replayed, deterministic sample of what your hop will accept, which is the cheapest way to find out that it is accepting too much.