Mocking Tool Responses for Fast Agent Tests

Before writing a single fake, list every external call the agent can make. Grep for your tool registration code and write the list down. For each tool, record five fields:

This inventory pays for itself twice. Tool names are part of the model's decision signal, so a name that encodes both action and target (get_customer_by_email, not lookup) reduces wrong-tool selection according to these design patterns for tool use. Writing the list forces you to notice overlapping tools and ambiguous parameter names before they cause flaky tests.

Treat the inventory as a specification. A tool specification is the contract between the model and your code, covering the name, the description and the input schema as explained in this walkthrough of tool specifications. Your fake must honor the same contract, otherwise the test passes against a fiction.

Recording is a one-time cost per scenario. The cleanest approach is a thin proxy that sits between your tool implementation and the real API, writes each request and response pair to disk, and then gets switched off. This is the same idea as capturing production traffic through a sidecar or proxy and replaying it later, which is how teams validate services against realistic workloads without touching live systems described in this traffic replay pipeline guide.

Three rules make recordings useful:

1. Sanitize before you commit. Strip auth headers, cookies and personal data. Recorded traffic is valuable precisely because it is realistic, which is also why it needs anonymizing before it lands in a repository per this discussion of replay without production risk.
2. Record the ugly cases, not just the happy path. Capture at least one 4xx, one 5xx, one empty result and one rate-limit response per tool. A fake that only knows success will hide the bugs you most want to find.
3. Store one file per scenario, named for the tool and case (search_contacts__empty_result.json). Flat names beat nested folders when you are debugging a failing test at speed.

Keep recordings small. Save the exact JSON body the API returned, plus the status code and the content type. Do not save timing data: latency is noise in a mock, and you want the fixture to be stable.

A fake is a lookup table. Given an outgoing request, find the recorded response whose request matches. Match in layers, from cheapest to strictest:

import json
from pathlib import Path

FIXTURES = Path("tests/fixtures")

def load_fixtures():
    return [json.loads(p.read_text()) for p in FIXTURES.glob("*.json")]

def fake_call(method, url, body):
    candidates = [
        f for f in load_fixtures()
        if f["request"]["method"] == method and f["request"]["url"] == url
    ]
    for f in candidates:
        if json.loads(f["request"]["body"]) == body:
            return f["response"]["status"], f["response"]["body"]
    raise LookupError(f"No fixture for {method} {url} with body={body}")

Two decisions matter here. First, decide how strict body matching should be. Exact equality catches accidental extra fields, but it breaks on key ordering and on fields like timestamps. Normalize before comparing: sort keys, drop volatile fields, compare the rest. Second, fail loudly on a miss. LookupError is correct. A fake that silently returns an empty object trains your tests to ignore real divergence.

You can go further and use a real HTTP-level mock server, which is the approach WireMock recommends for agent development: mock APIs let you test edge cases, simulate timeouts and rate limits, iterate quickly, and control the environment for reproducible runs per this WireMock guide to testing agent API interactions. A local fake server is worth it when your tool code uses a real HTTP client you do not want to special-case; a plain function fake is simpler when you control the tool layer.

If your agent constructs its own HTTP client deep inside a tool, you cannot swap it cleanly. The fix is architectural: pass dependencies in. Mocking frameworks describe three levels of this. At the simplest, you mock the tool function itself to test tool selection and parameter passing. At the second, you mock the API or service the tool calls. At the third, you design for testability with dependency injection from the start, which is the most flexible option but requires planning as laid out in this guide to mocking external APIs in agent tests.

For fast agent tests, aim for a hybrid: the tool function is real (so you test your code), and its client dependency is injected (so you can swap in the fake). A sketch:

class SearchContactsTool:
    def __init__(self, client):
        self.client = client

    def __call__(self, query: str) -> dict:
        status, body = self.client.get("/contacts", {"q": query})
        if status >= 400:
            return {"error": "contact search failed", "hint": "retry with a full name"}
        return body

In tests, pass a client that calls fake_call. In production, pass the real one. The agent sees no difference, and your test now covers the tool's error handling without a network. This is also where you encode the agent-facing error contract: what failed, why, and how to fix it, in the shape recommended for agent consumers in this error-handling writeup.

Fast tests are only useful if they fail for the right reason. Assert on three things per scenario: the call itself (which tool, which arguments, in which order), the rendered result (how your code turned the response into what the model sees), and the error path (serve a recorded 429 or 500 and assert the agent's next move is sensible rather than a blind retry). Validating the final answer alone misses agents that reach a correct result through an unsafe or wasteful path, which is why trajectory-aware evaluation is recommended over outcome-only checks per this guide to testing AI agents.

Be careful about asserting on the model's exact output text. That is inherently non-deterministic and makes tests flaky. Assert on structure and on tool calls; keep free-text assertions coarse. Also validate tool results in the runtime before acting on them: a search that returns an empty array should trigger a reformulated query, not a final "nothing found" answer, and that check belongs in code, not in the model's reasoning as argued in these tool use patterns.

Mocks are a complement to real calls, not a replacement. Use live calls when the thing you are testing is the contract itself.

A useful split is a two-tier suite. Tier one is mocked, runs on every save, and must complete in seconds. Tier two is a small contract suite that hits the real API, runs on a schedule or before release, and exists to catch schema drift and deprecations. When tier two fails, you re-record the affected fixtures and tier one picks up the change.

There is a measured reason to invest in tool tests specifically. Automated tool testing found that many publicly available tools suffer from underspecification, and the testing method surfaced far more erroneous inputs than prompt-engineering approaches alone per the ToolFuzz paper. An automated pipeline that improved tool documentation raised task accuracy from 22.9% to 35.4% on GitHub tasks and from 29% to 39% on file management tasks according to the ToolFuzz record. Those numbers are about documentation, but the mechanism is the same one mocks enable: cheap, repeatable runs against a stable tool surface.

Suppose your agent suite has 200 tests and each one triggers three tool calls. Against live APIs at 400 ms per call, tool time alone is 200 × 3 × 0.4 = 240 seconds, before any model latency. That is four minutes of pure waiting, and it fails whenever the API hiccups or a rate limit lands. Replace those 600 calls with local fixture lookups at roughly 1 ms each and tool time drops to about 0.6 seconds. The suite goes from "run it before lunch" to "run it on save."

The trade is that you have now frozen the API's behavior at recording time. If the real API renames a field next month, every mocked test still passes. That is exactly the failure mode the scheduled real-call tier exists to catch. Budget maybe 10 to 20 contract tests against 200 mocked ones: enough to detect drift, cheap enough to run nightly.

When you do need real calls across several providers while building the test harness, or when you want to swap model backends without rewriting client code, a single OpenAI-compatible endpoint for text, image and embedding models is the practical path rather than juggling keys. Quanta Inference offers exactly that on a prepaid, dollar-priced basis, which keeps the contract tier's cost predictable.

A handful of mistakes account for most flaky mock suites: patching a private method instead of injecting the client, matchers so loose that the body is ignored, recording only success, silent fallbacks that turn a broken fixture into a green build, and fixtures nobody ever revalidates against the real API. Each one is cheap to fix at design time and expensive to untangle later.