Costing a Contract Test Suite: Stubs Versus Live Runs

Schema-first contract testing is the cheapest useful thing in the suite. Schemas are executable, and both consumer and provider can evaluate them without a network round trip. Schema-based testing has three common forms: generating a schema from typed code and diffing it, asserting inside the unit test framework against input and output, and contract-style verification where the running server is checked against the spec schema-based contract testing. The first two never leave the process; the third needs a server but not a model.

The three approaches trade off differently, which matters when you allocate budget.

Spec-driven tools can generate concrete requests from examples in the specification and assert the response against the same spec, which is why a run against a local server reports counts like four tests, four successes and zero failures for a small employee API Specmatic contract testing docs. Nothing in that class of check needs a model in the loop.

On the consumer side, the right unit of work is a good test for the API client. Pact's guidance is to treat the consumer test as a set of unit tests for the client and let the pact file fall out as a side effect, using loose matchers on the response so the provider can evolve without false alerts Pact consumer docs. Those tests run against a local mock. Mocking external dependencies is what makes scenario tests deterministic and offline-friendly, and the tooling around agent tests distinguishes three levels: mocking the tool function, mocking the API or service boundary, and dependency injection for testability mocking external APIs in agent tests. Level 1 and level 2 mocks cost zero tokens per run. That is the bulk of your suite's assertion surface, and it should stay free.

Some assertions have no honest stub. Three categories are worth paying for.

The first is behavioral conformance that a schema cannot express: does the endpoint actually return the records matching the query, or does it return the right error when the resource is missing? Schema checks confirm the envelope; only a real call confirms the content. Spec-driven execution sits in a middle zone here, because it can exercise real semantics against a test server while still not calling a model.

The second is non-deterministic providers. If the provider is an LLM or an agent, a stub can only replay a response you already captured, so it proves the consumer handles that one shape. It cannot tell you that the provider still returns the field at all under a new prompt, a new model version or a changed temperature. A live run is the only way to watch that behavior drift. The same logic applies to any statistical output, such as a classification score or a ranked list.

The third is provider state. Contract testing in the consumer-driven style works by having the consumer describe an interaction under a named state ("SKU exists"), and the provider replaying it against the real implementation Pact vs Spring Cloud Contract. Verifying those states is inherently a live exercise, because it is a claim about your data and your code at that moment.

The practical decision rule, and the one worth writing into your team's testing doc:

A scenario earns a live call only if you can name a change to the provider that would keep the schema valid, pass the stub, and still break the consumer.

If you cannot name that change, the stub is enough. If you can name several, one representative live call per scenario class usually covers them, because the second live call with the same shape adds little.

Do the count from the endpoint inventory, not from the test files. For each endpoint, list every interaction the consumers depend on: the happy path, each distinct error the consumer branches on, and any pagination or filtering variant that changes the response shape. Then mark each interaction as stub-satisfiable or live-required using the rule above.

A worked example, with numbers you can substitute:

• 40 endpoints in the provider's public surface.
• 32 endpoints have consumers that only branch on schema and status, so they need zero live calls.
• 8 endpoints are model-backed or otherwise non-deterministic, and each needs 2 live interactions: the success path and one error path the consumer branches on.
• Live calls per run: 8 × 2 = 16, not 80.

That is a different number from the one you get if you assume two live calls for every endpoint. The common mistake is to budget as if the whole suite is live, which overestimates spend by a factor of five in this example and tempts you to cut the schedule instead of the calls.

Where a live call is genuinely required, keep it single-purpose. A live call that also exercises three unrelated assertions is harder to attribute when it fails and no cheaper to run. If a live call must set up state (create a record, then read it back), count the setup call too. A chain of three calls that produces one assertion is three calls on the invoice.

If you run the suite in CI per pull request, add the parallel-agent dimension: work that fans out across multiple agents spends tokens per branch of work, so a change that triggers five exploratory agents is five times the model spend of one scaling LLMs to larger codebases. Contract runs are usually serial and small, but if your live checks are agent-driven, count the agents, not the assertions.

Do not estimate from the prompt template. Measure. Instrument the client so every live call logs input tokens, output tokens, total tokens, model, endpoint or feature, environment, request status and retry count setting token budgets for an AI product. Those fields are enough to answer the only question that matters at budget time: which endpoint drives the cost.

Take a representative sample of 20 to 50 live calls per endpoint class, in a staging environment with production-shaped inputs, and record the median and the 95th percentile. The median sets your allowance; the tail sets your alarm.

A worked budget, using a round figure of 1,500 tokens per live call for illustration:

• 16 live calls per run × 1,500 tokens = 24,000 tokens per run.
• Weekly runs: 52 × 24,000 = 1,248,000 tokens per year, roughly 104,000 tokens per month.
• Daily runs on weekdays: about 260 × 24,000 = 6,240,000 tokens per year, roughly 520,000 tokens per month.

If your suite really is 80 live calls at 1,500 tokens, a daily schedule is 3,000,000 tokens per month before retries, which is why the per-endpoint live call count is the decision that governs everything downstream. Price the number from the provider's own pricing page at the moment you set the budget, because published rates move, and record the date you read it in the budget document.

Two cost multipliers are easy to forget.

Retries. Any flaky live call is paid for twice or three times. If 5 percent of live calls need one retry, add 5 percent to the run cost; if you allow two retries, add roughly 10 percent. Instrument retry counts and put the retry multiplier in the formula rather than in someone's head setting token budgets for an AI product.

Replay traffic. If you also run a shadow comparison that sends real traffic to both the old and new implementations and logs differences, that traffic carries the same per-call token cost as the contract run itself. The verification technique of calling both implementations and comparing results is a strong way to prove a migration is safe before cutting over Branch by Abstraction, but it must appear in the budget as its own line, sized from traffic volume rather than endpoint count. The same distinction is why canary rollouts are usually much cheaper per decision than feature-flag rollouts driven by live traffic: the canary tests deployment infrastructure, the flag tests users canary releases and feature flags.

A small helper that turns the ledger into a number:

endpoints = {"search": 2, "summarize": 2, "classify": 1, "lookup": 0}
tokens_per_call = 1500
retry_factor = 1.05

live_calls = sum(endpoints.values())
run_tokens = live_calls * tokens_per_call * retry_factor
print(live_calls, run_tokens, run_tokens * 22)  # per run, per run, monthly at 22 runs

Keep the ledger in version control next to the tests. When someone adds a live call, the diff shows the cost.

The schedule question is really a question about how fast the provider changes. Three patterns cover most teams.

The rule that ties schedule to budget: set the live-call allowance first, then choose the most frequent schedule that fits it, and let everything else default to stubs on every pull request. A provider with a stable, versioned interface and a small number of consumers rarely justifies daily live runs. A provider whose output changes shape when a dependency updates does.

During a migration, the schedule should change shape rather than frequency. Branch by abstraction lets both implementations coexist behind one abstraction layer so you can run them side by side and compare results, which is the migration technique designed for exactly this making large scale changes incrementally. Confluent's walkthrough makes the testing value explicit: because both versions live in production behind the abstraction, you can call either or both and log the differences while always returning the old system's result until accuracy crosses a threshold Branch by Abstraction explained. That comparison traffic is your paid window, and it should be time-boxed like any other budget line. Harness's summary of the pattern adds the operational caution worth planning for: the abstraction layer lets you pause and resume migration cheaply, which is precisely why you can also pause the spend branching by abstraction with feature flags.

If you are doing that comparison work with a mix of model providers, Quanta Experts can run the extraction and comparison passes and hand back a finished difference report, which keeps the analysis out of your contract suite's own token line.

Finally, instrument the run itself. Log tokens per run, live calls per run, retries, and the pass or fail outcome per scenario, then review the ledger monthly. A live call that has never failed and covers a shape the schema already enforces is a candidate for demotion to a stub, and demoting it is the cheapest schedule upgrade available.

1. List every interaction per endpoint, not every test.
2. Apply the decision rule: keep a live call only if you can name a schema-valid provider change that would break the consumer.
3. Move everything else to schema assertions, spec-driven verification against a test server, or recorded stubs.
4. Instrument the client to log input tokens, output tokens, model, endpoint, environment, status and retry count setting token budgets for an AI product.
5. Measure median and p95 tokens for 20 to 50 calls per endpoint class.
6. Multiply, add the retry multiplier, and record the price source and the date you read it.
7. Choose the most frequent schedule that fits the monthly allowance, and default the rest to stubs.
8. Review the ledger monthly and demote any live call that has never caught a real regression.

A note on running this checklist honestly: teams routinely believe their suite is mostly live when it is mostly mocked, and the belief drives the wrong cuts. Measure before you argue. The discussion of AI tooling on brownfield codebases in engineering circles makes the same point from the other direction, that the work is often about managing context and feedback loops rather than one-shotting a problem scaling LLMs to larger codebases, and a contract budget is exactly that kind of discipline.