When to Accept a Live Call in Your Test Suite

Four layers cover almost every case. Move down the list only when the layer above cannot produce the signal you need.

The layer descriptions mirror the LLM testing decision table published by Agiflow, which maps each question to a layer and states what that layer does not prove: a record and replay fixture proves compatibility with a captured exchange, not whether today's model behavior is good 12. A mock server covers error handling but tells you nothing about real provider performance.

The cutoff rule I use is a single sentence: accept a live call only when the assertion cannot be written without the model's actual output. In practice that means three families of test qualify. First, output contract tests where the model's response shape is the thing under test (valid JSON against a schema, required fields present, refusal behavior on disallowed inputs). Second, prompt change tests where you need to observe the effect of an edit on response shape or content. Third, integration seam tests where the live path is the point: auth headers, retry ceilings, streaming chunks arriving in order, timeout handling on a slow provider.

The general mocking-versus-live literature agrees on the shape of the split even where it is not about models. Sparrow's integration testing guidance recommends mocks for fast feedback during development, for backends that are not ready and for error and edge conditions that are hard to trigger against a live API, and live calls for validating the real contract and gaining pre-release confidence 18. Zuplo makes the same point about quota and availability: hit live only when you must, because you are betting on a third party's uptime and your own spend 17. Stack Overflow's long-running integration test debate lands in the same place from the other direction: mock the external parts, or the test stops being a meaningful integration of anything 16.

A useful habit is to write the assertion first, with no call in it, and ask whether it can pass or fail meaningfully. If the answer is "only if the model returns something," you have justified a live call. If the answer is "only if my code does the right thing with whatever comes back," you have justified a fake.

Take a small suite with 12 live tests, each making one call with a 2,000-token prompt and a 500-token response. That is 24,000 input tokens and 6,000 output tokens per run. Run it on every push across five engineers pushing several times a day and the token count grows quietly but steadily, while the failure noise grows faster: a single provider hiccup can red the build.

Now apply the decision table. Of those 12 tests, maybe three genuinely assert model output (schema compliance, refusal behavior, one prompt regression case) and two assert an integration seam (streaming order, retry ceiling). The remaining seven are parser, router and error-mapping tests that can run against fakes and recorded fixtures in milliseconds. The live budget drops from 12 calls to 5, and the seven cheap tests go back to running on every commit where they belong.

The numbers are illustrative, but the framework behind them is published. OpenAI documents rate limits across metrics such as requests per minute, tokens per minute and requests per day, with work throttled by whichever limit is hit first, and recommends separate staging and production projects so development and testing traffic can be isolated with their own rate and spend limits 12. That recommendation is the structural version of the budget: a dedicated test project means a runaway suite cannot touch production quota. The same source notes that prompt caching can reduce latency and input token cost when repeated prefixes qualify, which matters for suites that reuse a long system prompt across many live tests 12.

The practical budget rule: start with 10 to 20 live calls per test file and treat that cap as a test of its own. If a file needs more, the extra tests are usually asserting something a fixture could assert.

Rate limiting is a control plane, not a guardrail. The basic components are the limit (maximum allowed requests), the time window over which it applies, and the identifier that distinguishes one client from another; when a client exceeds its allocation, requests are rejected with HTTP 429 and headers that say when the limit resets 19.

For a test suite, the identifier is you, or more precisely your test project. Three practices keep the suite inside the ceiling.

1. Serialize live tests inside a file and cap concurrency across files with a semaphore or a test-runner flag. A dozen parallel workers each firing live calls will hit the same limit and produce failures that look like regressions.
2. Treat 429 as a signal to back off and retry with jitter, and assert that behavior in a mock-server test so you know the retry path works before you meet a real limit. Zuplo's mocking guide points out that triggering a consistent 429 against a live API is nearly impossible, which is exactly why the mock belongs in the suite 17.
3. Pause between calls using the reset headers the provider returns rather than a guessed interval, and set a hard per-call timeout so a stalled connection does not consume the entire test job's wall clock.

The Stack Overflow question about YouTube's quota on unit tests shows the same failure mode in plain terms: repeated live calls burn a per-endpoint quota, and the advice is to mock only the endpoints you actually use rather than the whole API 20. You do not need to run the real provider to know that your client honors a rate limit.

The gate is the part teams skip, and it is the cheapest to implement. Live tests should be off by default and on in three situations: a scheduled run, a pull request labeled for it, and a manual dispatch.

# pytest: skip live tests unless the flag is set
LIVE_MODEL_TESTS=1 pytest tests/ -m live

# gate in CI: only the scheduled job and labeled PRs export the flag
if [[ "$EVENT_NAME" == "schedule" ]] || [[ "$PR_LABELS" == *"live-model"* ]]; then
  export LIVE_MODEL_TESTS=1
fi

The reason to gate rather than to rely on discipline is that prompt configuration changes behave like code changes but often skip the safeguards code gets. Langfuse's prompt CI/CD guidance makes the point directly: a prompt edit changes production behavior the same way a code change does, but in most teams it skips every safeguard code changes routinely pass 14. The gate is what puts the safeguard back without putting live calls on every commit.

Labeling a pull request is the right ergonomics because the person changing the prompt is the person who should select the live run. A scheduled nightly run catches drift and provider-side changes without blocking anyone. Everything else runs against fakes and fixtures.

Operational limits should live in the same run as quality. The Braintrust comparison places latency and token cost alongside correctness and format compliance precisely because prompt edits can trade one for the other 11. If your live suite records duration and token counts per test, you get the cost conversation for free every time it runs.

A live failure that says "expected 'approved', got 'review'" is useless to a reviewer who cannot see the inputs. Attach the evidence to the work item: the prompt version, the model and parameters, the request and response, the scorer output and the artifact path. Agiflow's task evidence model lists fixture diffs, the reason a cassette was refreshed, the eval command, the threshold, the source link and the artifact path as the things that let a human or an agent prove what changed 12.

Recording the full configuration next to each prompt version is what makes two runs comparable. When the score drops between runs, you can trace it to a model swap or a temperature change as readily as to an edited instruction, which is the whole point of storing configuration with the prompt 11. Without it, a live failure is a shrug.

The same discipline applies to prompt versioning. Langfuse's regression testing material frames the goal as catching the prompt tweak that fixes one complaint and quietly breaks ten other answers before it ships 13. That is a live-eval job, and it is why the live suite exists at all: not to test your parser, but to catch the change nobody expected to matter.

If your team is wiring this into a real product and wants one key for text, image, audio and embedding models with OpenAI-compatible calls and prepaid US dollar pricing, that is what Quanta Inference is for, and a separate test project under that key keeps live suites off your production quota.

• The assertion cannot be rewritten against a fake or fixture without losing meaning.
• The test has a hard timeout and, if it retries, a documented retry ceiling that is itself covered by a mock-server test.
• The call is counted against a per-file budget of roughly 10 to 20 and a total run budget.
• The suite is gated: off by default, on for scheduled runs, on for the labeled pull request, on for manual dispatch.
• The failure artifact includes prompt version, model, parameters, request and response, and the scorer result.
• The live call runs against a dedicated test project, not the credentials your production service uses.

The hardest item is the first. It is also the only one that decides whether the test is worth its cost. If you cannot articulate the behavior the live call proves, the fixture already covers it.