Why Multi-Step Agent Runs Need Their Own Mock Fixtures

The core object you need is a call counter plus a table of responses indexed by step. Python's standard mocking library gives you the building block: a side_effect can be an iterable, and each call consumes the next value, as shown in the unittest.mock documentation and in this Stack Overflow answer on returning different results per call.

The iterable is the right primitive but the wrong final design for agents, because a plain list cannot branch. If the agent retries a failed call, or calls a different tool than you expected, a list either raises StopIteration or silently returns the wrong thing. Use a function instead, and keep the step table explicit.

from dataclasses import dataclass, field


@dataclass
class OrderLookupMock:
    """Returns a different payload for each call, keyed by call count."""
    calls: list = field(default_factory=list)
    # step number -> (payload, exception)
    script: dict = field(default_factory=dict)
    default_payload: dict = field(
        default_factory=lambda: {"status": "found", "order_id": "o-1", "total_cents": 4200}
    )

    def __call__(self, order_id: str) -> dict:
        step = len(self.calls)
        self.calls.append({"step": step, "args": {"order_id": order_id}})
        payload, exc = self.script.get(step, (self.default_payload, None))
        if exc is not None:
            raise exc
        return payload


lookup = OrderLookupMock(script={
    0: ({"status": "found", "order_id": "o-1", "total_cents": 4200}, None),
    1: (None, TimeoutError("inventory service timed out")),
    2: ({"status": "found", "order_id": "o-1", "total_cents": 4200, "reserved": True}, None),
})

Two design rules follow from this shape. The fixture records every call with its arguments and step number, so the assertion at the end has something to check. And the step table is data, not code: you can copy it into a test file, read it in a review and diff it when the agent's plan changes.

There is a variant worth knowing when the tool is an HTTP endpoint rather than a Python function. WireMock-style servers can return different responses for successive requests to the same path, and its documentation lists "simulate errors", "control your test environment" and "ensure consistent responses for reproducible testing" among the reasons to mock an agent's APIs, as set out in this WireMock Cloud walkthrough. The same numbered-step table maps to a scenario name.

Most teams write one happy path and one sad path. In a chain of five tool calls, that leaves the interesting failures untested. The useful matrix has three axes: which step fails, how it fails, and what a correct agent does next.

The partial-success row is the one that catches real bugs. Suppose your agent reserves inventory at step two and charges a card at step three. If the fixture fails step three and the agent's retry loop starts from step one, you have reserved inventory twice. A stateful fixture makes that visible: the recorded call list shows three reservations when you expected one, and the assertion fails. The general lesson, that early choices change what is possible later, is exactly why Braintrust calls stateful agents trajectory-sensitive.

A worked example with numbers. Your fixture scripts a five-call plan: look up order, reserve inventory, charge card, write receipt, notify customer. You want to test a single transient timeout. Set the script so call four raises TimeoutError once and call five succeeds. The agent, if it retries correctly, makes six calls in total: calls one to four, a retry of call four that returns a receipt, then call five. Your assertion is on the sequence, ["lookup", "reserve", "charge", "receipt", "receipt", "notify"], and on the fact that exactly one charge appears in the recorded call log with the card token you expect. If the agent retries from the top, the sequence shows two lookups and two reservations and the test fails with a readable message.

Error injection belongs in the fixture, not in the network layer, for the reason the MockServer LLM mocking documentation gives for mocking model endpoints at all: deterministic, instant, free responses without touching the network. It also documents tool calls, streaming and multi-turn conversations as first-class cases, which matters when the agent's own model response is one of the things you need to script. When you script those responses, keep each one byte-for-byte valid for the provider's wire format, or the failure you see will come from your fixture rather than the agent.

The final answer of an agent is a weak signal. A production agent can reach the right answer through an unsafe or wasteful route, which is why the Splunk guide to testing AI agents argues for trajectory-aware evaluation that inspects tool calls and reasoning steps rather than only task completion. It also introduces the Pass^k metric, the probability that all k independent trials succeed, and notes that a single successful trial does not establish production readiness.

In a fixture-driven test you get trajectory assertions almost for free, because your mock already recorded every call. Assert on four things.

1. The ordered list of tool names, which encodes the plan.
2. The arguments of the calls that matter, for example the exact amount charged.
3. The absence of forbidden calls, for example a refund before the order was shipped.
4. The state the agent carries between steps: the order id, the reservation handle, the running total.

Point four is the interesting one, and it is the part a canned mock cannot reach. If the agent has to pass a reservation handle from step two into step three, the fixture proves the handle survives. Give step two a payload with a distinctive handle like res-8812, then assert step three was called with exactly that string. If the agent regenerates a handle or drops it, you get a precise failure.

For HTTP-level fixtures, the same idea lives in contract testing. This overview of API contract testing defines the contract as endpoints, methods, request and response schemas, status codes, headers, authentication and data types, verified on every commit. Checking the fixture's payloads against that contract means a schema-valid response is guaranteed, so a red test really does mean the agent misbehaved.

Determinism has more enemies than randomness. A fixture that embeds a current timestamp, an auto-incrementing database id or a shuffled list will produce different tests on different days. Fix all three: pass a clock into the fixture, use stable string ids, and sort every list you return. Tricentis makes the same point at the tooling level, recommending that teams start AI agent testing with the deterministic test management and test data tools they already run before layering on agentic testing.

The second enemy is drift: the mock slowly stops resembling the real service. This guide to stopping mock drift gives the cleanest formulation. Derive the mock's responses from a versioned contract (an OpenAPI document or a consumer-driven pact) and verify that same contract against the real provider in CI. In the author's words, a contract without provider verification can drift, and provider verification without a consumer-owned expectation can miss what consumers actually use. The article also warns against a permissive catch-all mock that returns 200 for anything, because it teaches the client that invalid requests work. For agent fixtures the rule becomes: your step table should reject an unexpected tool call rather than fall back to a default success.

The PactFlow article on schemas and contracts explains why a schema alone is not enough. Its version of Hyrum's Law is that with enough users of an API, every observable behavior becomes something someone depends on. An agent is a heavy consumer: it will build on whatever your mock returns, including the ordering and the optional fields. If the mock returns reserved on step two only because a leftover field was in the fixture, the agent may learn to require it. Keeping fixture payloads generated from the contract removes that class of accidental teaching.

A short checklist for the fixture itself:

• Every response is generated from or validated against a contract.
• No wall-clock time, no random ids, no unordered collections.
• The step table is stored as data and reviewed like code.
• An unexpected call fails loudly instead of returning a default.
• The recorded call log is printed on failure.

On the last point, Snyk's introduction to Python mocking describes the standard procedure: write the test as if it used the real API, patch the small number of calls that need replacing, configure the responses, then run. When it fails, the recorded calls tell you whether the fault is in the agent or in the fixture. If your mocks are shared across many tests, this cheat sheet of Python mock patterns collects the knobs (spec, side_effect, attribute paths) that prevent a mock from quietly accepting anything you throw at it.

Everything above exists to support one shift in thinking: the unit under test is a sequence, not a call. A single-request mock answers "did the agent call the tool?", while a multi-step fixture answers "given what the tool said at step one, did the agent's step two make sense?" The second question is the one that predicts production behavior, and it is also the one that compounds: with five steps and three branch points you have far more paths than any hand-written set of happy-path tests will cover, so you pick the failures that matter (the partial success, the retry, the empty result) and script those.

Two operational notes for teams running these chains. First, run each fixture several times, not once. Splunk's Pass^k framing applies here: if your agent passes nine runs out of ten on the same deterministic fixture, you have a nondeterminism problem in the agent, not the mock, and the fixture is what lets you see it. Second, keep the model's own responses in the script where it matters. MockServer documents multi-turn conversations and session isolation for exactly this reason: a chain that includes the model's tool-call decisions needs those decisions pinned, or you are re-testing the model's sampling rather than your agent's logic.

When the agent under test is the thing you are building, the fastest route to a working chain is usually to build it against one endpoint that fronts every model, so the fixture only has to pin the tools and not the provider wire formats. That is what Quanta Inference is for: text, image, video, audio and embedding models through one OpenAI-compatible API key, prepaid and priced in US dollars. Point the fixture at it in tests, and the step table stays about your agent's logic.