A stub returns predefined responses to specific inputs, while a mock additionally verifies interactions per the WireMock glossary. For a ladder you want mostly stubs and a few mocks. You want the stub to prove that a downgrade produces a parsable answer. You want a mock only when the question is "did the router call rung 2 after rung 1 failed?"
The key upgrade over a naive mock is that the stub should be a small deterministic engine, not a string that says "mocked response". In the sandbox described above, the fake completion function extracts the most relevant sentence from any context block in the prompt, and the fake tool call selects a tool by matching the goal against tool names and fabricates schema-valid arguments from each tool's JSON schema. That is the shape you want: a stub with real structure, so the code downstream gets something realistic to parse.
For a ladder, build one stub per rung, keyed by rung name, and have each stub return the same envelope your real rung returns:
LADDER = [
("primary", {"text": "primary answer", "model": "primary", "tokens": 120}),
("secondary", {"text": "secondary answer", "model": "secondary", "tokens": 110}),
("tertiary", {"text": "tertiary answer", "model": "tertiary", "tokens": 95}),
]
def stub_rung(name, prompt):
for rung, payload in LADDER:
if rung == name:
return payload
raise KeyError(f"no stub for rung {name}")
Two properties matter here. First, the stub returns a dict, not a string, so the test asserts on parsed structure rather than on text you would have to reparse. Second, the token count is included, because the token budget assertion needs a number to check. Real providers report usage on every response, and embedding responses carry token counts the same way per the Pydantic AI embeddings result, so it costs nothing to include the field.
Keep unit and integration tests structurally separate so the stub-based ladder tests run on every commit and the real-call tests run on a schedule. The common pattern is separate directories, with pytest invoked on tests/integration only in the scheduled job as described for pytest projects. That separation is the same technique cloud-init uses: a subset of integration tests carry a @pytest.mark.ci marker and run on pull requests, and most new tests should not use that mark, so a green CI run does not mean every integration test passed per the cloud-init integration testing docs.
The assertion shape per rung
Write one parametrized test that runs every rung through the same four assertions. The rung name is the parameter and the stub supplies the payload.
| Assertion | What it catches | Failure example |
|---|
| Response parses | A downgrade returns a string instead of an object | Rung 2 returns prose, caller calls the JSON parser and gets a decode error |
| Required keys present | A cheaper model drops a field the caller depends on | tokens missing, cost tracking crashes |
| Fits caller timeout | Rung latency budget was never defined | Rung 3 takes longer than the request deadline, caller gets a 504 |
| Fits token budget | Prompt grows and the cheap rung silently truncates | Context stuffing pushes input past the rung's window |
The timeout assertion needs a per-rung budget, not a global one. A cheaper model is usually faster, but a fallback that retries three times before giving up is not, and the caller's deadline covers the whole retry chain. Assert on the summed budget for the worst-case chain, then assert each rung's own observed time from the stub against its slice.
import time, pytest
@pytest.mark.parametrize("rung", ["primary", "secondary", "tertiary"])
def test_rung_contract(rung, deadline_ms=800):
start = time.monotonic()
out = stub_rung(rung, "reset my password")
elapsed = (time.monotonic() - start) * 1000
assert isinstance(out, dict)
assert {"text", "model", "tokens"} <= out.keys()
assert out["text"]
assert elapsed < deadline_ms
assert out["tokens"] <= input_token_budget(rung)
Token counting belongs beside prompt construction, not in the test. In the cost budget guide's Vitest example, the test calls the application's own message-building function and counts tokens with tiktoken before making any live call, so prompt growth is caught without spending money per the LLM cost budget testing guide. The same principle applies here: if your test rebuilds the prompt differently from production, the budget assertion measures the wrong thing.
Antonio Evans
