Testing a Model SDK by Its Failure Modes, Not Its Happy Path

The single most useful discipline is to write the expected behavior down as a table before touching the mock server. If you cannot fill in every row, you do not yet know what your error handling does.

The decision rule: retry only transient failures (429, 5xx, timeouts) and only when the request is safe to repeat. Never retry a 400 or a validation error. Never execute a tool call that arrived incomplete. Never persist a partial object as if it were complete.

This mirrors what rate-limit guidance from inference providers recommends. Scale's LLM Engine rate limit guide notes that requests receive HTTP 429 per standard HTTP practice and recommends retrying with random exponential backoff, adding jitter so concurrent retries do not synchronize. It also warns against fixed-interval polling, because poll traffic scales with outstanding tasks and can outpace the endpoint's drain rate. The overview of API rate limits makes the same case: limits exist to prevent overload and ensure fair usage, so a client that hammers a 429 is working against the system it depends on.

You want a local HTTP server that replays a script: response one, response two, response three. Three options work, in increasing order of effort.

The stateful approach is well documented. Beeceptor's retry logic simulation tutorial shows a counter-based mock that returns 400, 400, then 200 in sequence, resetting automatically so every test run starts from the same state. That determinism is the point: you assert the third attempt succeeds without manual cleanup between runs.

For streams and timeouts you will want a real socket, because the interesting failures happen below the HTTP client's abstraction. A compact Python harness:

from http.server import BaseHTTPRequestHandler, HTTPServer
import json, threading, time

SCRIPT = []  # list of callables(handler) run in order

class Handler(BaseHTTPRequestHandler):
    def do_POST(self):
        step = SCRIPT.pop(0)
        step(self)

    def log_message(self, *args):
        pass  # silence test noise

def rate_limited(h):
    h.send_response(429)
    h.send_header("Retry-After", "2")
    h.send_header("Content-Type", "application/json")
    h.end_headers()
    h.wfile.write(b'{"error":{"type":"rate_limit"}}')

def truncated_json(h):
    h.send_response(200)
    h.send_header("Content-Type", "application/json")
    h.end_headers()
    h.wfile.write(b'{"choices":[{"message":{"content":"hello')  # never closed

def half_stream(h):
    h.send_response(200)
    h.send_header("Content-Type", "text/event-stream")
    h.end_headers()
    h.wfile.write(b'data: {"delta":"partial"}\n\n')
    h.wfile.flush()
    h.close_connection = True

Start the server on a thread, point the SDK's base URL at http://127.0.0.1:<port>, and run each test with a fresh script. Keep the harness in the repo next to the tests so the contract and its proof live together.

Two practical guards. First, never let a test reach the real provider: assert the base URL is the mock before the client is constructed. Second, seed the jitter in backoff so timing assertions are stable, then assert bounds (for example, total wait under 10 seconds for three attempts) rather than exact sleeps.

This is where most suites go wrong. A test that asserts response.status == 429 proves the SDK parsed a header. It does not prove your code stopped looping. Assert on your code's outputs: call count, exceptions raised, records written, keys reused.

A worked example with numbers. Suppose your wrapper allows 5 attempts with exponential backoff starting at 1 second, doubling, capped at 8 seconds, with full jitter.

Given this, the correct assertions are: exactly five outbound requests recorded by the mock, total test time under a generous ceiling (say 30 seconds), and a typed RateLimitError raised after the fifth. A Retry-After of 2 seconds must override the nominal delay for that attempt, so the assertion is "delay used was at least 2s", not "delay equals 1s".

This is the same argument the TanStack Query maintainers made in a thread on testing retry functions: there is little value in testing that the library calls your retry function with the right parameters, because the library tests that internally. Extract the policy function and test it directly, and test the integration only for the behavior you own. One reply in that thread shows a 429 test that waits for failureCount to reach 1, 2, then 3, checks the loading state between attempts, and confirms the error state after the final one. That sequence, not the status code, is the assertion.

For timeouts specifically, give the test an explicit budget. A harness that waits on a mocked delay needs its own timeout longer than the backoff ceiling, or the test flakes and someone lowers the retry count to make CI green. That is the failure mode you cannot afford.

Truncated JSON and half-streams are the failures that quietly corrupt state, because many SDKs return whatever parsed before the cut. Your tests must prove three things: no partial object escapes, no partial write commits, and the error is distinguishable in logs from a genuine validation error.

For non-streaming JSON, assert that a completed response whose body ends mid-object raises a parse error and that your code does not cache, store, or return a partially built object. A quick check: after the test, query the store for the record key and assert nothing exists.

For streams, assert the terminal state. If the connection closes after two chunks, your code should either raise or return text marked incomplete, and in either case the downstream consumer must not treat it as final. If your pipeline chunks text into an index, assert the index has no entry for the abandoned message.

For tool calls, the rule is absolute: an incomplete tool call never executes. The threat model is on your side here. A tool call is a request to act, and an agent that executes on a half-delivered payload can be steered by a truncated or tampered response. Vendor guidance for agents keeps returning to the same controls: scoped credentials, short-lived tokens, per-workflow boundaries and runtime guardrails that block calls outside policy, as summarized in Sweet Security's AI agent supply chain overview. A test that asserts the tool was not called is a cheap version of that guardrail.

Two more assertions worth adding. First, that a malformed 200 never becomes a success in your metrics, because a dashboard showing 100% success while every call fell back to defaults is worse than an outage. Second, that the error message names the request and the attempt number, so an on-call engineer can tell a truncation from a timeout without rerunning the job.

If you consume models through more than one provider, this suite becomes a compatibility check: the same scripted failures should produce the same typed errors across providers. A single API such as Quanta Inference reduces that matrix, but you still want the failure tests, because the failure contract is yours, not the router's.

A failure suite passes today and lies tomorrow if the dependency moves underneath it. Two controls fix most of that.

Version pinning. Lock files freeze the entire resolved graph, direct and transitive, into a committed file, and locked-mode restore makes CI fail rather than silently resolve something new. The write-up on NuGet version locking explains this: two builds of the same commit can produce different binaries when a transitive dependency ships a patch over the weekend, floating versions make restore non-deterministic by design, and locked mode turns that drift into a visible error instead of an unreviewed build. The same principle holds across ecosystems.

Integrity checking. Locking tells you which version you get; it does not tell you the artifact is the one you expect. Supply chain taxonomies increasingly separate the two. A systematic review of software supply chain attacks catalogs 19 attack types and maps 25 controls to them, and Apiiro's glossary of supply chain attacks notes that build system compromise, dependency confusion and maintainer account takeover all deliver malicious code through trusted channels. The TryHackMe lab walkthrough shows the operational version: verify checksums against expected hashes, prefer safe serialization formats, and scan artifacts before they enter the pipeline.

The practical rule for SDK testing: every failure scenario you script should be listed in a config file next to the pinned version, and the suite should fail if a dependency change alters an error type. That gives you a second signal, distinct from the test itself.

Treat the failure harness like code you will own, not a one-off script. Minimum viable package:

• A scenarios/ file listing each failure and its expected behavior, reviewed like a spec.
• A mock server module with one function per scenario and deterministic scripts.
• A policy module with the retry decision extracted, so it can be unit tested without HTTP.
• Integration tests that assert call counts, typed errors, storage state and timing bounds.
• A CI job that runs against the mock and fails on any real network egress.

For teams, the natural next step is to share the harness rather than rebuild it per repository. A private library of skills and scenario scripts, with one credit pool and per-person keys, is what Quanta for Teams is for. If your failure suite keeps surfacing policy questions (which failures to retry, which to alert on, who owns the runbook), a free 30-minute discovery call is the faster path to a decision than another round of guessing.