Pinning Model Versions in Tests: Aliases and Snapshots

Retirement is the event your test strategy has to survive. When a model is shut down, requests to that model ID fail immediately rather than degrading gracefully: OpenAI's deprecation page describes retired models as completely turned off and no longer accessible, and states that impacted customers are notified by email and in the documentation (OpenAI API deprecations). A third-party tracker summarizing the same page describes the shutdown stage as the point where any request to the model ID fails with an invalid model error (OpenAI deprecated models and shutdown schedule).

The notice you get depends on the model tier, and this is where test design meets operations. OpenAI's page states minimum notice periods of at least 6 months for generally available models, at least 3 months for specialized variants (chat, codex and deep-research variants are the examples given), and as little as 2 weeks for preview models, which it does not recommend for business-critical production workloads (OpenAI API deprecations). The tracker reports that a specific batch of GPT-4o and GPT-4.1 snapshots was announced with roughly 3 months of notice despite the general 6-month guidance (OpenAI deprecated models and shutdown schedule). The practical reading: treat the published minimum as a floor, not a schedule you can plan sprints against.

This is the decision rule that matters, and it is the opposite of what many teams assume. If a test calls a moving alias, retirement shows up as a loud, immediate failure, which is exactly what you want from a canary. If a test calls a pinned snapshot, retirement shows up as a failure too, but only when that specific snapshot is retired, which may be months after the alias moved. Both fail; they just fail at different times and for different reasons. The mistake is having your main regression suite pinned to a snapshot nobody monitors, so the first time you learn about the retirement date is when CI goes red the morning calls start erroring.

There is also a quieter breakage: an alias can move without any retirement at all. Your test then fails because the provider improved the model, and your assertion on exact output text was the thing that was wrong. That is a false alarm, and false alarms are how suites get muted.

Treat the real-call suite as two layers with different jobs.

The first layer is recorded fixtures: canned request/response pairs captured once and replayed in ordinary unit tests. This is the standard mocking pattern for agent tests, where you mock the external dependency so execution is deterministic and offline-friendly, and you test that your agent calls the right tools with the right parameters rather than testing the provider (Scenario testing guide on mocks). Traffic replay tooling makes the same argument at the system level: synthetic tests miss real sequencing and payload variation, so teams capture production requests, sanitize them, and replay them against a staging environment (Speedscale on traffic replay). Keploy's environment replay documentation describes recording API traffic from production read-only and replaying it in staging, with detected discrepancies reported automatically (Keploy environment replay).

The second layer is a small live suite that actually calls the API, gated behind an environment variable and a budget so it never runs on every commit. Its job is to answer one question: does our request still satisfy the provider's contract today?

A fixture recorder worth committing looks roughly like this. It writes one JSON file per case, records the exact model identifier used, and refuses to overwrite an existing file unless you pass a flag, so re-recording is always a visible, reviewable change.

import json, pathlib, os

FIXTURES = pathlib.Path("tests/fixtures/llm")

def record(case_id, client, model_id, request, force=False):
    path = FIXTURES / f"{case_id}.json"
    if path.exists() and not force:
        raise SystemExit(f"{path} exists; pass force=True to re-record")
    response = client.responses(request, model=model_id)
    path.write_text(json.dumps({
        "model_id": model_id,
        "request": request,
        "response": response.model_dump(),
    }, indent=2, sort_keys=True))

Store the model identifier inside every fixture. When a snapshot is retired, a one-line grep finds every fixture recorded against it, and the re-record diff shows you exactly which outputs moved.

For the live suite, keep the assertions about structure, not prose. Assert that the response parses, that every required tool argument is present and typed correctly, and that no tool name outside your declared list appears. Cohere's documentation is a useful reference point for why this is worth asserting: with strict tool use enabled, the API guarantees that tool names and parameters match the definitions, required parameters are included and data types are respected, which eliminates hallucinated tool names and parameters in exchange for a cap of 200 fields across all tools in a call (Cohere parameter types for tool use). If your provider offers a strict schema mode, your live tests should assert you are actually getting that guarantee.

Tool definitions themselves are worth testing, because they are natural-language contracts and they go wrong in predictable ways. ToolFuzz, an automated tool-documentation tester, found that all LangChain tools and the majority of Composio tools it evaluated were erroneous, and it detected far more erroneous inputs than prompt-engineering-based testing (ToolFuzz on arXiv). The lesson for your fixture set: include malformed and edge-case arguments for every tool, not just the happy path.

Use this rule set and you will not have to relitigate it per test.

The smoke test on an alias is the one place a moving target earns its keep, because you want it to break when the provider repoints the name or retires the model behind it. Keep its assertion shallow: a 200 response, a parseable body, one required field present. Do not assert exact prose there, or you will be chasing the model rather than the contract.

The contract suite on a fixed snapshot is where you put the detailed assertions, because you have chosen those weights and any change is a change you caused. When that snapshot is retired, the suite goes red, and the fix is mechanical: swap the identifier, re-record, read the diff, decide whether each movement is acceptable.

There is one more consideration that shapes the table. Non-determinism means a single trial proves little about reliability; Splunk's writeup on agent testing argues that teams need consistency metrics such as Pass^k, the probability that all k independent trials succeed, rather than one green run (How to test AI agents effectively). If you adopt that idea, apply it to the live suite only. Replaying the same recorded fixture k times proves nothing new.

Drift comes from three places: a snapshot's serving infrastructure changing under you, an alias moving to a new snapshot, and your own prompt or schema edits. The first two are external and the third is yours, so the recording workflow should make the difference obvious.

Record one fixture per (prompt, model, options) combination and never share a fixture across model identifiers. When you re-record, do it in a single commit that touches only fixtures and any assertion text you deliberately changed, so reviewers can see the behavioral movement in isolation. If a diff is enormous, that is a finding: it usually means a prompt or schema change landed without anyone recording it deliberately.

Watch the deprecation channels rather than the tests. OpenAI publishes model shutdown dates programmatically through the List models endpoint, whose optional shutdown_date field carries the date as a YYYY-MM-DD string or null when none is announced (OpenAI API deprecations). A scheduled job that reads that field and opens a ticket when a snapshot you use has a shutdown date is cheap and removes the guesswork. Community threads on specific models show how much confusion dates cause, including users asking whether only a dated variant or an entire family is affected (OpenAI community thread), which is another argument for reading the authoritative page rather than forum summaries.

When you do migrate, run both snapshots side by side against the same fixtures and diff the outputs before you change any assertion. This is the same logic as the old advice for testing external APIs: don't let the test touch live data with side effects, and isolate the layer that constructs requests and parses responses so you can test it independently (Stack Overflow discussion on integration tests for external APIs). Modern practitioners add a pragmatic twist: build mock APIs so the team can exercise error paths, timeouts and rate limits without waiting on anyone (WireMock on testing AI agent API interactions).

If you are testing agents rather than single calls, Trajectory-aware checks belong in the recorded layer, not the live one. Splunk's guidance is that a correct final answer can still come from an unsafe or policy-violating path, so evaluations should inspect tool calls and reasoning steps rather than only the outcome (How to test AI agents effectively). Recorded tool-call sequences are exactly the artifact that lets you assert on that path cheaply. Splunk also reports research finding that a large share of enterprises shipped agentic features that had passed internal evals and still caused a customer-facing failure, which is a good argument for keeping a deterministic baseline before layering on model-specific checks (How to test AI agents effectively).

Suppose you maintain 240 tests that touch a model, across four test files. You do the arithmetic before redesigning.

Today: 240 tests call the alias on every CI run, at a mean of 900 input and 300 output tokens each. That is 288,000 input tokens and 96,000 output tokens per run, plus latency, flakiness and a dependency on the provider's availability during CI. If CI runs 40 times a day, you are making 9,600 live calls a day to prove that your parsers still work.

After redesign: move 228 tests onto recorded fixtures, keep 8 contract tests on a fixed snapshot, and keep 4 smoke tests on the alias. Live calls per run drop to 12, a 95 percent reduction in provider calls and token spend, and 228 of those tests become deterministic and offline. The 8 contract tests run nightly and before each release. The 4 smoke tests run weekly, and their whole purpose is to fail if the vendor default changes or disappears.

The remaining risk is concentrated in exactly the place you can manage it: 12 tests whose failure modes you have named in advance. Add a job that reads the shutdown_date field for every model ID your suite references (OpenAI API deprecations), and you have converted an outage into a ticket.

If your team runs several vendors and that 12-test live set spans more than one API, the plumbing becomes the annoying part: different keys, different request shapes, different error codes, different deprecation endpoints. Quanta Inference exposes text, image, video, audio and embedding models behind one OpenAI-compatible API key, which lets a live contract suite target one set of request conventions while still choosing which underlying model each test pins. That reduces the surface area of your real-call tests rather than the number of models you can evaluate.