Testing a Tool Contract with Recorded Runs

Recording means running the real tool once, by hand or through a script, and saving everything the contract talks about. That is the request, the exit code, stdout, stderr, and a snapshot of side effects. The practice is well established: record-replay testing captures real traffic and dependency interactions and turns them into replayable test cases, with recordings stored as human-readable files that live in the repository next to the code the Keploy record and replay guide. For a CLI the same idea applies at the process boundary: you run the binary the way a user would and assert only its output, so the tests survive a total rewrite of the internals and change only when behavior changes the Go CLI integration testing writeup.

A fixture that is easy to review and easy to regenerate in a single language looks like this:

{
  "clause": "create rejects duplicate names with exit 5",
  "argv": ["thing", "create", "--name", "duplicate-name", "--json"],
  "cwd_fixture": "state_with_duplicate_thing",
  "env": {"NO_COLOR": "1", "HOME": "$TMP"},
  "exit_code": 5,
  "stdout": "{\"error\":\"conflict\",\"message\":\"resource 'duplicate-name' already exists\",\"existing_id\":\"abc123\"}\n",
  "stderr": "Error: resource \"duplicate-name\" already exists\n",
  "side_effects": {"files_created": [], "files_changed": []},
  "recorded_with": {"commit": "<sha>", "host": "<os>"}
}

Three habits make recordings useful rather than fragile. First, store the fixture as a golden file and give the runner an update flag, so a behavior change is a deliberate diff in review rather than a silent edit the Go CLI integration testing writeup. Second, normalize what the contract does not promise. Timestamps, temporary paths, absolute working directories and random IDs should be masked at record time and at replay time, the same way noise detection excludes varying fields from strict assertions the Keploy record and replay guide. Third, freeze the clock and the environment during replay so token expiry and date-based logic produce identical results the Keploy record and replay guide.

For an MCP server, record at the protocol boundary instead of the process boundary. The official SDK provides an in-memory transport that lets you exercise the server without network overhead, so you can record a ListTools response and a tool call in one pass and replay both the C# Corner guide to MCP tool contracts. Assert the tool name, the required fields and the important types, but do not compare an entire serialized schema string, because a harmless serialization change will break the test and teach the team to ignore failures the C# Corner guide to MCP tool contracts.

The most common reason contract suites rot is that they assert on message text. A developer improves an error message, twenty tests go red, and within a week the suite is muted. The contract usually does not promise the wording; it promises the channel, the type and the code. Assert those, and treat the full text as a snapshot you can review but not as a hard gate.

A practical layering looks like this:

This mirrors the guidance that structured output is not optional for tools that agents consume: JSON to stdout and everything else to stderr, flat structures, consistent types across commands the CLI guide for AI agents. It also matches how a good CLI separates the human view from the machine view; the JSON output and the table are two views of the same data, optimized for different audiences the OpenStatus CLI writeup. Your test should pin down the machine view and only smoke-check the human one.

Two details catch many teams. First, stdout and stderr must be captured separately, because mixing them hides the very bug you are trying to catch. Use a helper that returns both streams plus the exit code, and fail loudly if the process was terminated by a signal rather than exiting, since a shell will report 128 plus the signal number while the kernel reports no exit status at all the Stack Overflow thread on exit statuses. Second, avoid the reserved codes. Codes 1 and 2 carry conventional meanings, 126 through 165 and 255 are reserved, and 127 in particular reads as "command not found", so a user-defined 127 will confuse whoever is debugging the Advanced Bash-Scripting Guide table. The guide suggests restricting user-defined codes to 64 through 113, which leaves fifty codes and keeps them out of the noisy range the Advanced Bash-Scripting Guide table. If you adopt sysexits instead, document it, because the standard is widely referenced and widely ignored, and relying on it without publishing your mapping recreates the problem it was meant to solve the Hacker News discussion of sysexits.

Recorded runs are fast and deterministic, but they cannot prove that your tool still works against the outside world. The answer is a small live set, run less often, behind a flag. Integration testing guidance is consistent on this: prefer the cheaper test when a behavior can be covered either way, and do not write integration tests for every permutation of data access the ASP.NET Core integration testing docs. Integration tests do use the real components and therefore take longer, so they should be limited to the most important infrastructure scenarios the ASP.NET Core integration testing docs.

The live set for a tool contract is short:

The temptation is to mock everything and keep the suite fast. That is a false economy: mocked integration tests do not interact with remote systems, so if the remote system changes the mock data becomes stale and the test provides false confidence the Walmart Global Tech writeup on easy-fix. The better pattern is one set of tests that can run either live, capturing data as it passes, or replayed against captured data the Walmart Global Tech writeup on easy-fix. Keep the live path behind an environment variable and skip it loudly rather than silently when credentials are absent, so a green build never implies coverage you did not get.

For tools that execute code, add one live case that proves the sandbox boundary holds. Agent-generated code fails not at generation but at execution: where the code ran, what permissions it had, and what safeguards sat between the model's output and your infrastructure the Blaxel guide to agent code execution. Run that case in the environment you actually ship, not a laptop.

A contract suite is only useful if it runs where work is decided. The pattern that holds up is a checklist that runs after the agent or developer reports done, with the output recorded on the change. A definition of done is a shared list of the evidence a change needs before the team accepts it, and each item should be a check that runs apart from the author's report and leaves a record the SpecStory definition of done guide. One developer audited 516 "done" claims from his own agent sessions and found no passing test run behind 65 to 69 percent of them, depending on how strictly they were counted the SpecStory definition of done guide.

A minimal contract gate has five items, in this order:

1. Build the artifact once, then test that artifact. No rebuilding mid-suite.
2. Replay every fixture. Compare exit code, channel split, and shape. Snapshot text.
3. Fail if a fixture was deleted, skipped, or its expected values weakened.
4. Run the live set only when the environment flag is set.
5. Publish the fixture diff and the run log on the pull request.

Item 3 matters more than it sounds. Editing expectations so they pass is a known failure mode and should be treated as tampering rather than as a fix the SpecStory definition of done guide. The gate should also reject the inverse: a fixture file that exists but is no longer referenced by any clause, because that is a contract clause quietly dropped.

Keep the gate cheap by choosing the right runner. The forward slash line-runner builds a test binary, runs each case as a user would, and compares against golden files; in the published example it builds the binary, runs it four times, loads four files and asserts behavior in about half a second the Go CLI integration testing writeup. That is the budget you want, because a suite that takes ten minutes per invocation will be skipped under deadline.

Suite cost grows in three dimensions: number of fixtures, cost per fixture, and cost of maintenance. Control all three.

For fixture count, cover each clause once and add a case only when a clause gains a new branch. This is the same discipline as vertical slicing in product work: prefer slices that each deliver independent value over horizontal layers that do not, because small slices are easier to estimate, easier to understand and less risky Visual Paradigm on vertical slicing. A contract clause is the tool-equivalent slice.

For cost per fixture, replay in process where possible. MCP servers can use an in-memory transport, and that removes process startup from the common path the C# Corner guide to MCP tool contracts. CLIs should still be run as processes at least once, because argument parsing and stream buffering are part of the contract the Go CLI integration testing writeup.

For maintenance, re-record instead of editing by hand. When a clause legitimately changes, run the recorder, review the diff of the fixture, and commit both together. If a fixture keeps changing for reasons unrelated to the contract, that is a signal the wrong thing is in the fixture, not that the test is flaky. Apply noise detection rather than loosening assertions across the board the Keploy record and replay guide.

One more lever: make the contract itself a generated artifact. If your exit codes live in a single table in the code, generate both the help text and the test cases from it, and the suite cannot drift from the documentation. Recorded fixtures then only need to prove that the runtime honors the table, not that the table exists.

If your team is spinning up several tools at once, or the tools are built by agents, a shared skill for recording and replaying contract fixtures saves rework. Quanta Skills lets you install that workflow into a coding agent so the same capture, normalize and replay steps are applied to every new command.