Choosing a Re-Record Cadence from Vendor Change Volume

Start with one quarter of the vendor's public changelog and classify every entry into one of three buckets. You only count two of them.

Why weight breaking changes double? Because a breaking change that lands in your fixture window does not just make the fixture stale, it makes the fixture wrong in a way that can silently corrupt a passing test. ACCELQ's write-up on contract testing describes the exact shape of that incident: a payment provider adds one field during a routine deploy, nothing is removed, nobody flags it as breaking, and a downstream reconciliation job fails silently for hours. A weighted count keeps you from under-scheduling when a vendor has a quiet quarter for additions and a loud one for removals.

Stripe is a useful example of what a well-labeled changelog looks like. Their release process announcement describes twice-yearly major releases that carry breaking changes plus monthly releases that add features without breaking anything, and their changelog marks entries explicitly as breaking and lets you filter to relevant changes only. That structure makes the count straightforward: major releases count double, monthly entries count once.

The count itself should be reproducible. A short script that pulls the changelog, tags entries, and prints a per-quarter weighted total keeps the cadence decision out of arguments:

entries = [
    {"q": "Q1", "kind": "breaking"},
    {"q": "Q1", "kind": "additive"},
    {"q": "Q2", "kind": "additive"},
]
weights = {"breaking": 2, "additive": 1}
totals = {}
for e in entries:
    totals[e["q"]] = totals.get(e["q"], 0) + weights.get(e["kind"], 0)
print(totals)  # {'Q1': 3, 'Q2': 1}

Track at least two consecutive quarters before you commit to a cadence. One quarter can be an outlier.

The rule: take your weighted change count per quarter and double it to get re-records per quarter, then convert that to an interval.

Two adjacent bands deserve a note. A vendor with a weighted count of 1 per quarter translates to two re-records per quarter, which is roughly every six weeks. In practice most teams round that down to quarterly because the recording cost is not zero and the marginal safety from the extra cycle is small. That is the "one change a year, go quarterly" case: you are deliberately slower than the formula because the vendor is slow.

The formula's real job is the other end of the table. A vendor at six weighted changes per quarter is changing roughly every two weeks, and if you re-record monthly you are guaranteed to run some tests against fixtures older than the changes they are meant to model.

Suppose the vendor's changelog shows six weighted changes in a recent quarter: three additive entries and one breaking entry would already be five under the weighting above, so add one more additive and you are at six.

Six weighted changes per quarter doubles to twelve re-records per quarter, or one every week. That is a lot of recording. Here is how to make it cheap enough to actually happen.

1. Split the fixture set. Only re-record cassettes touched by the changelog entries you classified. A new optional field on one endpoint does not invalidate the other forty.
2. Re-record on a branch, not main. Capture against the vendor sandbox, commit the new cassettes, and let the existing test suite run against them as a gate. Momentic's CI guidance makes the same point about gating merges on test results: a run that only reports is not a control.
3. Use a sandbox, never production. Keploy's sandbox testing overview is blunt about why: the payment gateway costs money per call and the vendor will rate limit you, so recording has to run against an isolated environment with dedicated credentials.
4. Treat a failed re-record as a contract failure. If the new recording does not match your assertions, that is the signal you built the cadence for.

A worked number on cost: if a full re-record takes twenty minutes of wall clock and fifteen of setup, weekly is roughly half a day per quarter of engineer time, which is cheaper than one silent production incident from a stale cassette.

Now the other vendor. Their changelog shows four entries in a year, three of them documentation, one a new optional response field. Weighted, that is 1 per year, or 0.25 per quarter.

Doubling gives 0.5 re-records per quarter. You cannot re-record half a time, so round to the minimum defensible interval: quarterly. That gives you four cycles a year against one real change, which is enough headroom to catch the change within a quarter and cheap enough that nobody argues about the calendar block.

The trap for slow vendors is assuming slow means safe. It does not. DigitalAPI's contract testing guide gives the concrete example: a weather API that documents temperature as a number and starts returning it as text. That is one change, and it breaks every fixture recorded before it. Quarterly captures it inside a quarter. Annual captures it inside a year, if you remember.

One extra control for slow vendors: add a cheap live smoke test that hits the real endpoint on every deploy, in addition to the fixture-based suite. The Smithy smoke test specification describes these as small tests that ensure a client can connect to a live service and get the right kind of response, which is exactly the right scope for catching a slow vendor's rare change without paying the full re-record cost monthly.

The cadence is a calendar decision, but the enforcement is a pipeline decision. Wire it so that a stale or failing re-record cannot merge.

TestCollab's quality gate documentation spells out the exit code contract that makes this work: zero means the criteria passed, one means the gate evaluated results and a criterion failed, two means the gate could not evaluate at all. All nonzero outcomes block the pipeline. That last distinction matters for re-records: if your recording job fails to reach the vendor sandbox, that is a two, and it should not silently pass.

Keep the re-record job separate from the test job. The re-record writes new cassettes; the test job replays them and asserts. Mixing the two means a flaky network during recording looks like a test failure, which trains people to ignore the check.

Recording against production. The recorded response then reflects live customer data and real side effects. GoReplay's record and replay guide is careful to distinguish HTTP response fixtures served from a mock server from raw traffic replay against a live backend, and notes that a capture only represents the traffic observed during its window. Record read-only endpoints or dedicated sandbox accounts.

Re-recording everything every cycle. It inflates the diff and hides the one cassette that actually changed. Scope to touched endpoints.

Not committing the cassettes. A recording that lives on one engineer's laptop is not a fixture, it is a rumor. Commit it next to the test, as the cassette model in vcr-style libraries assumes.

Ignoring additive changes. This is the most expensive mistake. A new field is easy to dismiss, but a fixture recorded before it will not exercise code paths that now run. The example in ACCELQ's incident write-up is precisely an additive change that passed every functional test and still broke a downstream job.

Setting the cadence once and never revisiting it. Recount every quarter. A vendor that doubles its release rate should move you up the table, and a vendor that stabilizes should let you back off.

If your fixtures span several vendors and model providers, the counting gets tedious. Pointing everything at a single API surface cuts the number of changelogs you have to watch, which is what Quanta Inference offers: text, image, video, audio and embedding models through one OpenAI-compatible API, so the re-record calendar tracks one changelog instead of six.