When a Vendor Changes a Tool's Error Codes

Walk through four client behaviors and see what each one depends on.

Retry. Retry policies are written as predicates over the error. In the TanStack Query community thread on testing retry logic, the example retry function returns false for error.code === 403 and otherwise retries up to three failures discussed in the TanStack Query repository. The maintainer's advice in that thread is to extract the function and test it on its own, because the branch on the code is the part that matters. Now imagine the vendor starts returning 401 where it used to return 403. Your predicate no longer matches, and the client retries an authentication failure three times against a rate limited endpoint.

Classification of retriable and non-retriable failures. The retry practices described for LLM systems depend on separating transient errors from permanent ones before backoff starts covered in this retry logic deep dive. Rate limiting is the canonical transient error: a client that receives HTTP 429 should back off and try again, not give up as the LLM Engine documentation states. If a vendor moves its throttle signal from 429 to a 400 with a body code of rate_limited, every client that keys off the status code treats a temporary condition as permanent and drops the request.

Idempotency. The classic failure is the one you cannot see in logs: the first request succeeds at the vendor but times out on your side, your client retries with the same idempotency key, the vendor returns a conflict code the old code was duplicate_request, and your repair logic sends a fresh key. The user is charged twice. Nothing failed at deploy time. The keyword in your repair rule simply stopped matching.

User messaging. A checkout page maps error codes to human text: card_declined becomes "Your card was declined, try another card", insufficient_funds becomes "Add funds and try again". When the vendor consolidates these into a generic payment_failed, the mapping falls through to a default, and customers get an unhelpful message for a recoverable problem.

The same class of drift is listed as a risk in public API governance discussions: an API can stay technically functional while diverging from the contract consumers depend on, with a changed HTTP status code or response field cited among the examples raised in the GitHub community discussion on API contract drift.

For each failure your client repairs, capture and assert the full tuple. Use this table as a template when you write the fixtures.

Two practical rules follow from the table. First, assert the exact code string, not a fuzzy match, because the whole point is to be notified when it changes. Second, keep an explicit list of codes your client treats as retriable, and fail on any code you have not classified, so an unknown error goes to a human instead of a guess.

HTTP fixtures give you a language neutral record of a real response. DNSimple built their client test suites across Ruby, Go, Elixir and Node by recording raw responses from curl output, headers included, and replaying them through stubbing libraries in each language as described in their API client testing writeup. The point of storing the whole response is that the status line and the headers are part of what you are testing, not just the JSON body.

The same idea applies to errors. A minimal capture loop that stores each known failure response, status line included, looks like this:

# capture one fixture per failure scenario, status line and headers included
for case in declined insufficient_funds duplicate_request rate_limited; do
  curl -sS -i -X POST "$VENDOR_BASE/charges" \
    -H "Authorization: Bearer $VENDOR_SANDBOX_KEY" \
    -H "Idempotency-Key: fixture-$case" \
    -d "@fixtures/requests/$case.json" \
    > "fixtures/errors/$case.http"
done

Then run a diff in CI or on a schedule:

# fail the job if the sandbox error surface changed
diff -ru fixtures/errors.expected fixtures/errors || {
  echo "Vendor error responses changed. Review before merging."
  exit 1
}

For test isolation inside your own suite, mocking the vendor keeps runs deterministic. Laravel's Http::fake with recorded responses is the documented approach, and the HttpFixtures package adds generated realistic data so real sensitive values never land in the repo per the package documentation. Framework level interception works the same way: TestCafe request hooks can log outgoing calls and inspect status codes and bodies, or serve a mock response in their place per the TestCafe interception guide. Playwright tests can retry a flaky fetch and fall back to mock data when every attempt fails per this retry and mocking walkthrough.

Two cautions on fixtures. Do not sanitize out the status code or the code string while removing PII, because those are the fields under test. And record the vendor's API version header in the fixture, because that is your anchor when a diff appears.

A snapshot of a single error tells you what the vendor returns once. Your repair logic cares about the sequence. The canonical retry scenario is request one returns 400, request two returns 400, request three returns 200, and the application should have retried exactly twice as described in Beeceptor's retry simulation guide. The same guide lists other sequences worth modeling: a payment gateway that fails twice before accepting a transaction, a job API returning PROCESSING several times before COMPLETED, a webhook endpoint that fails its first deliveries, and circuit breaker and fallback paths.

State based mocking makes those sequences deterministic, because a counter advances per request and resets after success. That mirrors how you should test your own repair code: assert the number of attempts, the delay between them and the final outcome, not just the first response. The TanStack Query contributor example does this by mocking a persistent 429 and waiting for failureCount to reach three before asserting the query enters its error state in the same discussion thread.

Rate limit behavior deserves its own sequence test. Exponential backoff with random jitter is the standard mitigation because it prevents retries from a fleet of clients from synchronizing into a thundering herd per the LLM Engine rate limit guide. Rate limiting exists to prevent server overload and ensure fair usage across clients per Orq's rate limit overview. If the vendor changes the code that signals a throttle, your backoff never engages and you hammer the endpoint until something worse happens.

Snapshotting in CI catches changes when you upgrade, but vendors ship continuously. You need a separate cadence.

Run a scheduled job, daily or on every weekday morning, against the vendor's sandbox. It executes the failure scenarios, captures the responses and compares them to the committed baseline. When the diff is non-empty, it opens a ticket or a pull request with the changed fixtures attached. This is the scheduled-diff pattern in practice: the vendor's sandbox becomes an early warning system, and your repository holds the expected behavior.

Two things make this work over time. Pin the vendor API version explicitly in the request, because the SmartRecruiters guidance recommends pinning integrations to a specific API version rather than "latest" to avoid unexpected issues, and it recommends reviewing the changelog for that version per their breaking changes documentation. Second, treat the diff like a code review: compare old and new code strings, decide whether the change is additive or semantic, and update your classification list in the same pull request.

For broader defensive design, apply the same least privilege thinking to credentials used by the job. Keep sandbox keys scoped to the minimum the failure scenarios need, and rotate them; the same discipline of scoped credentials and short-lived tokens limits blast radius when a dependency is compromised per Sweet Security's AI agent supply chain guidance. If agents or automation call the vendor, inventory that integration alongside models and tools so a change can be traced to every affected consumer per their AI bill of materials guidance.

If you want that scheduled harness to run against the specific model APIs your product depends on, a single endpoint layer across providers makes the diff matrix smaller, and Quanta Inference (https://quantaintelligence.ai/inference) is one way to get there.

Take a payments integration with three failure scenarios your client repairs: declined card, insufficient funds, and duplicate request. Your repair logic is: no retry on declined, prompt the customer on insufficient funds, return the original request id on duplicate.

Suppose these three scenarios account for 400 failed checkouts a day in your traffic. If the vendor renames the duplicate-request code and your matcher stops firing, the fallback path sends a new idempotency key on every retry of a lost response. Assume a lost-response rate of 1 in 200 requests and 2,000 payment attempts a day. That is roughly 10 lost responses daily, of which some fraction are duplicates. If half of those 10 hit the renamed code and get charged twice, that is 5 double charges a day, 150 a month, each one a refund, a support ticket and a trust hit. A scheduled fixture diff that fails on the renamed code costs one engineer an afternoon and catches the change the morning after the vendor's release.

The arithmetic matters because it changes the decision rule. Error contract tests are not worth it for every endpoint. They are worth it wherever the cost of a wrong repair is money, duplicate side effects, or a lost order. Rank your failure scenarios by that cost, write assertions for the top handful, and leave the rest to generic handling with alerting on unknown codes.

Use this rule when you decide what to pin: if a failure scenario has repair logic, has money or data side effects, or appears in a user facing message, it gets a fixture and an assertion on status, code and the fields your code reads. If a failure is logged and abandoned, it gets alerting on unknown codes instead.

A sensible first week:

1. List every branch in your code that reads an error code or a status.
2. Rank them by cost of misfiring, and keep the top ones.
3. Capture real sandbox responses for each, status line included.
4. Write assertions that fail on any change to status, code or consumed fields.
5. Schedule the capture job daily and route diffs to a human.
6. Add an unknown code path that alerts instead of guessing.

When several teams depend on the same vendor, the fixture baseline belongs in one shared place. If that is the shape of your problem and you need a hand mapping which integrations to cover first, a free 30-minute AI discovery call (https://quantaintelligence.ai/calendar) will get you a tailored coverage recommendation.