How to Read an AI Vendor's Model Change Notice

Before you can judge a change, you need the blast radius. The notice tells you which model is going away. It does not tell you which of your workflows, agents, scheduled jobs and internal tools call it.

Build the list in one working session, not over a week. Grep your repositories for the model identifier string, including configuration files, environment variables, infrastructure definitions and any prompt templates that embed model names in comments. Then reconcile that against the places code does not reach: low-code workflow builders, internal dashboards, spreadsheet automations and the one analyst who runs a side script. If the model is reachable through more than one API route, list each route separately, because each has its own schedule.

A practical shortcut: any system with no owner on the inventory is a system you have already lost. Rather than hunt for more systems, assign an owner to each one you found and note what it produces. Then classify each owner's workload by risk. A model powering internal summarization carries different validation requirements than one driving customer-facing text or financial output, and that risk tier determines how much validation runway each workload gets (the AI model deprecation guide).

This is the step that separates a real migration from a hopeful one. Collect a fixed set of prompts from your production logs, ideally covering your most common cases plus the awkward edges: long inputs, empty inputs, tables, code, unusual formatting requests. Freeze the exact prompts and the exact inputs. Run them against the old model. Save the outputs as your baseline. Then run the same frozen set against the new model and compare side by side.

The reason this matters is that prompt correctness is not a stable property. A case study on toxicity detection found that an API update caused a major performance downgrade and also flipped which of two candidate prompts was better, with the previously weaker prompt outperforming the other by 8.7 percent accuracy after the change (the prompt regression paper). If your prompts were tuned against the old model, part of your quality is leverage on behavior that may not survive the swap.

Replay is not classic regression testing either. LLM tests face different notions of correctness, brittle prompts and non-determinism, which is why the authors of that study argue regression testing for LLMs needs fundamental changes to traditional approaches (the prompt regression paper). In practice that means scoring outputs rather than diffing them, and running each prompt several times to see the spread. A prompt that produces one acceptable answer out of five is a failure even though one sample looked fine.

Keep the frozen set small and boring. A hundred prompts you will actually rerun beats a thousand you will never touch. Store them in version control next to the code, with the baseline outputs committed alongside, so the next notice starts from a working harness instead of a blank page.

A minimal harness looks like this: a file of prompts, a script that runs each one against a named model several times, and a saved result set per model.

prompts = load_jsonl("frozen_prompts.jsonl")

def run(model_id, prompt, attempts=3):
    return [call_model(model_id, prompt) for _ in range(attempts)]

for row in prompts:
    before = run(OLD_MODEL_ID, row["prompt"])
    after = run(NEW_MODEL_ID, row["prompt"])
    record(row["id"], before, after)  # score offline, do not diff raw text

The comment on the last line is the important part. Diffing raw text will drown you in irrelevant variation. Score each output against the criterion that actually matters for that prompt: valid JSON, correct label, number of required fields present, or a quick human review for the cases that carry the most risk.

Side-by-side replay answers "did quality move?" It does not answer the other three questions that decide whether the swap is cheap or expensive.

Behavior and format. Does the new model still obey your output constraints? Look specifically at structured output: JSON keys, date formats, refusal behavior, length limits. These are the first things to break, and they break in ways a casual read of five sample outputs will miss. Compare across the whole frozen set, then read the worst ten by hand.

Latency and retries. If the new model is slower or more likely to hit length limits, your retry logic and timeouts may need to change. Measure rather than assume.

Cost per completed task, not per token. This is where most estimates go wrong. Token price is one input. What matters is the number of attempts, the tokens consumed per attempt, the share of outputs that fail validation and must be retried, and the human review time for failures. A model with a lower unit price that hallucinates valid-looking but wrong structured output can cost more per completed task than a pricier one that rarely fails.

Widen the frame further: the full cost of the change includes integration, retraining, change management, operations and eventual exit, and license or usage fees typically make up only 20 to 40 percent of true AI costs for most implementations (the AI TCO guide). That ratio should recalibrate how much engineering time you are willing to spend validating a swap that looks free on the pricing page. A migration you treat as a one-line configuration change is a migration whose hidden costs you have decided not to see.

If you want to compare the substitution itself rather than argue about it, running both models through a single API is the quickest way to get a like-for-like read. Quanta Inference exposes text, image, video, audio and embedding models behind one OpenAI-compatible key, which makes old-versus-new replay a configuration change instead of a second integration (Quanta Inference).

With the replay results in hand, make one of three decisions for each workload.

Migrate now. Quality holds, cost per task is acceptable, formatting is intact. Move it, keep the old identifier available as a fallback until the retirement date, and watch production error rates for a week.

Remediate first. Quality dropped but the cause is prompt-level, not model-level. Rewrite the prompts against the new model, rerun the frozen set, and only migrate when the scores clear your bar. Remember that a prompt tuned for one model is not automatically the best prompt for its successor.

Escalate or defer. The workload is high risk and the new model does not clear validation. Options include a different model for that workload, a longer parallel-run period, or a reassessment of whether that workload should have been on the retiring model at all.

Whichever path you take, write the outcome into the harness itself. Update the baseline outputs, record which prompts needed rewriting and why, and note the dates you observed. A migration QA process works best as continuous validation across pre-launch baselining, launch-day verification and post-launch monitoring, rather than a single sign-off gate, because errors surface across the whole timeline rather than at launch (the enterprise migration QA framework). The same logic applies here: the value is not in one heroic test run before the deadline, it is in checks placed where breakage actually appears.

Most teams over-invest in flagged changes and under-invest in silent ones. Invert that.

Treat a flagged change with a published window as a scheduled project: run all five steps, but you have time to sequence them. Treat a silent change as a production incident. If you notice outputs drifting with no notice in hand, do not wait for an announcement: pull the current model identifier from your logs, run your frozen set immediately, and compare against the last known-good baseline. The sooner you know, the more options you have.

A simple rule of thumb: if you cannot answer "which model version produced this output we shipped last month?" in under five minutes, you are exposed to the silent case and do not have a harness to detect it. Logging the resolved model identifier and a request fingerprint on every call is cheap insurance, and it turns the next notice, flagged or not, into a solvable problem with evidence attached.

This is also where the build-versus-buy framing earns its keep. Buying a model does not remove the cost of owning its behavior: the new responsibility is probabilistic behavior, evaluation harnesses and model updates, and in some organizations that burden has increased rather than shrunk (the build vs buy framework). The harness you build in step 3 is what converts a vendor dependency into an operational one you can actually run.