Simulating Refusals in Staging: Fixtures for Every Code

A fixture is a small handler that inspects a control header or a seed and returns a fixed response for a given code. Determinism is the whole point: if the fixture occasionally returns a different code, your retry-count assertions become flaky and get muted, which defeats the exercise.

The practical pattern is a fixture registry keyed by code. The registry holds a function per code that returns the status, headers and body the real upstream would return. Two details matter:

• Sequence state per test. Counters live in the test scope, not in the server process, so parallel tests do not interfere. Reset the counter at test start.
• Retry-After honored. If a code carries a Retry-After header, the fixture should emit it, because the policy under test must read server guidance rather than only its own schedule. Webalert's analysis of retry storms notes that clients should honor server guidance such as Retry-After and cap attempts and elapsed time.

For codes that represent permanent failures, the fixture should never be configured to "succeed on the third attempt." That variant belongs to the transient codes. Mixing them is how terminal codes sneak into retry loops.

Here is a compact fixture registry in TypeScript:

type Fixture = { status: number; headers?: Record<string,string>; body: unknown };

export const fixtures: Record<string, (attempt: number) => Fixture> = {
  RATE_LIMITED: (n) => ({ status: 429, headers: { "Retry-After": "2" }, body: { code: "RATE_LIMITED" } }),
  UPSTREAM_UNAVAILABLE: (n) => n < 3
    ? { status: 503, body: { code: "UPSTREAM_UNAVAILABLE" } }
    : { status: 200, body: { ok: true } },
  INVALID_INPUT: () => ({ status: 422, body: { code: "INVALID_INPUT" } }),
};

The attempt argument lets a transient code recover after a fixed number of tries. That is what lets you assert "three attempts, then success" without any real clock.

The most valuable fixtures are the ones that assert absence. Terminal codes must produce exactly one call and then escalate, because retrying them wastes time and can amplify load. Temporal's documentation on non-retryable errors makes the case directly: patterns that mark failures as permanent let the system fail fast instead of waiting through exponential backoff for an error that will never change, and it lists invalid input, missing records and authorization problems as the typical cases. Their exercise material on configuring retry policies with non-retryable error types shows both mechanisms: marking the error at the throw site and listing the type name in the policy.

Your fixture test for a terminal code is short and strict:

1. Point the client at the fixture for INVALID_INPUT.
2. Run the operation to completion.
3. Assert the call counter equals 1.
4. Assert the escalation sink received exactly one event with that code.
5. Assert no sleep or timer was scheduled.

That last assertion catches a subtle bug: a policy that classifies the code as terminal but still calculates a backoff before checking the classification, which delays the escalation without any benefit.

Write the expected behavior down before you write the fixtures. The table below is the artifact that the fixtures encode; review it with whoever owns the escalation path. The "attempts" column is the number of calls the test should observe, not a promise about production timing.

The last row is the one people get wrong. A timeout does not tell you whether the operation happened, so retrying is only safe when the request carries an idempotency key. PostGrid's retry guidance recommends idempotency keys for create requests so that a retried call returns the same response rather than creating a duplicate, and it shows a fixed schedule with jitter and a small set of retryable statuses. Fixtures should include a duplicate-creation test: fire the fixture twice with the same key and assert the result is one record.

The general shape of the backoff the fixture exercises is the familiar formula, delay = initialDelay * factor ^ retryNumber, with a cap and jitter applied on top, as described in PostGrid's documentation and in AWS's write-up on timeouts, retries and backoff with jitter, which frames the three tools as timeouts, bounded retries and backoff to avoid converting a small failure rate into a full outage.

Timing is where retry tests become slow and flaky. The standard remedy is to inject a clock abstraction and a scheduler, then let the fixture advance time instantly. Webalert and the AWS builder post both stress bounding retries by attempt count and elapsed deadline, which you can only assert cheaply if time is virtual.

A worked example with numbers makes the assertion concrete. Suppose the policy starts at 1 second, doubles, caps at 30 seconds, and allows 5 attempts. With full jitter, each observed delay is uniform between zero and the capped value, so a strict equality check on the delay will fail. Assert bounds instead:

If you use equal jitter instead, the lower bound becomes half the capped value. Either way, the fixture test asserts three things: the number of calls, the interval bounds, and that the total elapsed time never exceeds the caller's deadline. A test that asserts only the call count will pass happily while a policy waits an hour.

For the rate-limited fixture, one more assertion matters: the policy must use the Retry-After value when the computed delay is shorter than it. That is a decision rule worth encoding: the effective wait is the maximum of the computed backoff and the server guidance, so honoring the server never makes you faster than the server asked.

The last piece is the guardrail. A test that iterates the code list and checks for a registered fixture converts an omission into a build failure. It is a few lines:

import { codes } from "../policy/reason-codes";
import { fixtures } from "./fixtures";

test("every reason code has a fixture", () => {
  const missing = codes.map(c => c.code).filter(code => !(code in fixtures));
  expect(missing).toEqual([]);
});

Add a second guard for classification: every code must declare retryable, and every retryable code must declare max_attempts and an escalate_to. A code with no escalation target will retry until the cap and then vanish, which is exactly the silent failure that MLflow's checklist warns about when it recommends an explicit task state object and replay logging rather than black-box behavior during failures.

One more guard closes the reverse gap: assert that every fixture key exists in the code list. That catches a fixture left behind after a code was renamed, which otherwise gives false confidence that a path is covered.

The reason fixtures deserve exact assertions rather than a smoke test is that retries in modern systems replay more than a single call. In tool-using and agent-style services, a retry often resends the accumulated context, so the cost of the Nth attempt is the cost of the whole history up to that point. An analysis of retry budgets for LLM agents puts numbers on it: a 20% per-step retry rate over three steps yields roughly 1.7 to 1.9 times baseline tokens once context replay is counted, and 2.2 to 2.5 times over five steps, with retries stacked at multiple layers multiplying into much larger fan-out. A single session described there consumed a large token volume because usage-limit errors did not stop the loop.

The decision rule that follows is simple and testable: after exhausting attempts, the policy must stop and escalate rather than reset the counter. Write a fixture that returns the transient code forever, run the operation to completion, and assert both the call count and that no later call occurs. If your test harness cannot express "never called again," add that capability before adding more fixtures; it is the assertion that prevents a retry storm in staging from becoming one in production.

The same replay logic explains why fixtures should be cheap to add and cheap to run. One fixture per code, executed with a virtual clock, keeps the whole suite in seconds and keeps the classification honest as the list grows. When the surface is large (many codes across several upstream providers), a single gateway in front of the models and services under test simplifies the harness: Quanta Inference exposes text, image, video, audio and embedding models through one OpenAI-compatible API key with prepaid US-dollar pricing, which means one fixture layer and one set of reason codes instead of one per provider.