Splitting Tests by Model Dependency: Stub, Sandbox, Live

Tier 1 is the foundation. Each test replaces the model client with a fixture: a recorded response, a hand-written double, or a rule-based fake. No network call, no API key, and identical output on every run.

What belongs here:

• Prompt assembly: given a user message and retrieved chunks, does the builder produce the expected system message, ordering, and truncation?
• Tool routing: does "reschedule my demo" pick the calendar tool rather than the billing tool?
• JSON parsing: given a response with a trailing comma or an extra newline, does the parser recover or fail loudly?
• Retry and timeout logic: given a timeout on the first attempt, does the wrapper retry once and then surface the right error?
• Cost estimation: does the prompt builder stay under a token ceiling for a representative fixture?

That last item deserves its own note. Cost is a testable property, and the cheapest way to test it is deterministically. Count tokens for the messages your application would send, then multiply by a checked-in rate table labeled as an estimate, because provider prices change and should not be frozen into test code as facts (LLM Cost Budget Testing in CI). The test does not assert an exact cent; it stops unreviewed prompt growth from sneaking into a common route.

The hard constraint is structural: tier 1 finishes in seconds and makes zero network calls. Enforce the second mechanically, not by discipline. A session fixture that patches the HTTP client to raise on any outbound request turns a stray real call into a red test rather than a bill.

# tests/stub/conftest.py
import pytest

@pytest.fixture(autouse=True)
def forbid_network(monkeypatch):
    def boom(*args, **kwargs):
        raise AssertionError("Tier 1 test attempted a network call")
    monkeypatch.setattr("httpx.Client.send", boom)

Record fixtures from real responses wherever you can. A recording teaches you what the provider actually sends; a hand-written double teaches you what you assumed it sends. Keep recordings small by truncating long fields, and check them into the repository so the suite is self-contained.

For RAG tests, a hash-based fake embedder returns the same vector for the same text every time, which removes model download and embedding cost from the fast path. That is the same idea as the deterministic hashed bag-of-words embedder described in the CI agent article (Building an AI Agent You Can Test in CI). The trade-off is that hash vectors are not semantic vectors; they test plumbing, never retrieval quality.

Tier 2 calls a sandbox or replay endpoint and checks the wire, not the words. These tests run on every pull request and should stay under about a minute of wall-clock time.

This mirrors how mature projects separate integration tests. Cloud-init keeps integration tests in a dedicated directory and tags the subset that runs on pull requests with a ci marker, warning that most new tests should not carry that mark (Integration Testing, cloud-init). Your sandbox tier is a deliberately chosen subset, not everything convenient.

pytest offers two clean mechanisms: separate directories run by path, or markers selected with expressions like -m "not live" (Working with custom markers). Structural separation survives a change of test runner, so use directories as the source of truth and markers for ad-hoc selection during development.

What tier 2 must not do is assert on output quality. If the test says "the answer must contain the word 'refund'," it belongs in tier 3. Sandbox output varies between runs, and asserting on it in the pull-request path is how builds turn flaky and then ignored.

A worked example shows why the retry assertion earns its keep. Configure the sandbox to return a 429 on the first call and a valid response on the second, then assert both that the client retried and that it waited at least the configured backoff. The second assertion catches the "fix" that sets backoff to zero to quiet a flaky test, which later hammers the provider under real load.

def test_retries_once_on_429(sandbox, client):
    sandbox.next_responses = [429, 200]
    result = client.ask("hello")
    assert sandbox.call_count == 2
    assert sandbox.elapsed >= 0.2  # configured backoff floor

Sandbox isolation is the other half of the contract. If two developers' experiments write to the same vector index, their retrieval results mix and tests become order-dependent. Isolated, per-pull-request environments with fresh data populated from controlled fixtures are the recommended pattern (RAG CI/CD), and at minimum you should namespace sandbox data by branch or build ID so one experiment cannot poison another (Multi-Tenancy in Vector Databases).

That data should be generated from the schema, not dumped from production. A production dump carries real names, emails, and payment history, and an anonymization script written months ago has almost certainly fallen behind the schema (Staging Without Production Data). Generating connected rows keeps PII out of the sandbox while preserving the shapes your assertions rely on. The Cloud Posse guidance is blunt about the alternatives: fixtures rarely represent production scale, and unscrubbed production snapshots are dangerous (Decide on Seeding Strategy for Staging Environments).

Tier 3 is the only tier that spends real tokens, and it must be the smallest. It calls the real model against a fixed, curated sample (often 10 to 50 cases) and asserts on output quality: did the answer cite a retrieved chunk, did the classifier return the right label, did the JSON parse into the expected shape. Run it nightly or on a label a reviewer applies. Never on every commit.

The cap is the critical design decision, and it belongs in three places:

• Per-run token ceiling, summed across the sample and checked into the repository.
• Per-case budget, so one runaway generation cannot eat the whole run.
• Per-day dollar alarm, paging someone if the daily total crosses a threshold. This catches a sample that quietly doubled.

A worked example makes the cap concrete. Say the sample is 30 cases, each around 2,000 input tokens and 500 output tokens. At a rate of 2.50 dollars per million input tokens and 10 dollars per million output tokens (sample configuration values, not a claim about current provider pricing), one run costs about:

• Input: 30 × 2,000 = 60,000 tokens × 2.50 / 1,000,000 = 0.15 dollars
• Output: 30 × 500 = 15,000 tokens × 10 / 1,000,000 = 0.15 dollars
• Total per run: roughly 0.30 dollars

That number belongs in the pull request template: "Tier 3 run: 0.30 dollars (cap 1.00)." A reviewer who sees 1.20 knows a prompt grew. This is the whole point of cost budget testing: make cost changes visible while they are still small, and let a human accept them deliberately (LLM Cost Budget Testing in CI).

Quality assertions need care. For a RAG pipeline, a good pattern is two-stage: a cheap ID-based check that the correct document was retrieved, then a slower judged evaluation of whether the answer stayed faithful to the retrieved text. Run the cheap check first so you do not pay for a judged evaluation when retrieval already failed (RAG CI/CD). For classification and extraction, exact match or schema validation is often enough.

Keep tier 3 small because it cannot prove quality anyway. A 30-case sample tells you a change did not break the cases you care about; it does not tell you the change is good. Treat a pass as a regression gate, not a certificate.

The decision rule, one line per test:

If you are unsure which tier a test belongs in, use the tie-breaker: can the assertion be written against a fixture without lying? If yes, it is tier 1. Tiers 2 and 3 are only for assertions that need a real endpoint, sandboxed or otherwise.

Wire the tiers to three named targets so everyone types the same command:

test:
	pytest tests/stub
test-contract:
	pytest tests/sandbox
test-live:
	pytest tests/live --maxfail=1

Then guard the live directory so a stray run from a laptop cannot burn tokens:

# tests/live/conftest.py
import os, pytest

def pytest_collection_modifyitems(config, items):
    if os.environ.get("ALLOW_LIVE") != "1":
        pytest.skip("set ALLOW_LIVE=1 to run live tests")

Put the ceilings in a checked-in budgets.toml beside a rate table labeled as an estimate, so a reviewer sees the change in the diff. Record each live run's token totals to a small metrics store, because the daily alarm is useless without a trend.

Two operational notes save pain. Make the tier 1 network ban global, then exempt tiers 2 and 3 via a marker so the ban does not fire on legitimate sandbox calls. And if your team needs one key across text, image, and embedding models for the sandbox and live tiers, that is what Quanta Inference provides, with prepaid dollar pricing so caps map cleanly to invoices (https://quantaintelligence.ai/inference).

For teams that would rather hand the suite design to an expert, the skill library behind every Quanta expert can put someone on the job alongside you (https://quantaintelligence.ai/skills).

Most teams do not start clean. The realistic migration has three steps.

First, measure and classify. Run the current suite and record, per test, whether it touches the network, how long it takes, and whether it asserts on output content. The truth is usually ugly: a handful of tests dominate runtime and cost.

Second, move the cheapest wins into tier 1. Tests that assert on parsing or routing but currently call the model convert fastest and give the biggest speedup. Record fixtures from current successful runs, then replace the live call. If a test cannot be converted without lying, leave it in tier 3 and write down why.

Third, add caps before growing tier 3. Growing a sample under an existing cap is easy; discovering a sample tripled your spend is not.

Watch for two failure modes. Fixtures rot when a provider changes its response schema, and if tier 2 does not catch it, tier 3 will, which is a feature of the split rather than a flaw. And resist marking a flaky tier 2 test as tier 3 to make it pass. If a contract assertion is flaky, the client or the sandbox has a real bug, and moving the test hides it.

The migration is also the moment to fix sandbox data. If it came from a production dump, replace it with generated rows at the same shape and volume, which removes both the compliance exposure and the periodic restore failures caused by schema drift (Staging Without Production Data). If you must use real data, anonymize at the platform layer with deterministic transformations, so hashed identifiers stay consistent across tables and masked emails keep their domains and validation logic still works (Anonymization: The missing link in dev workflows).

Finally, document the rule in the repository. A TESTING.md with the routing table and the caps is the difference between a suite that stays split and one that slowly fills with live tests because it was easier that way.