Regression Tests for Prompts Without a Live Model Call

The recording step is the only part that costs live tokens. Run a set of real production-shaped inputs through the prompt, review the outputs, keep the ones you would ship, and freeze input plus accepted output plus the full configuration into a versioned file.

Store the configuration with the output, not just the text. The recorded configuration should include the model, parameters such as temperature and reasoning effort, available tools, response format, and any retrieval step that supplies context (Braintrust). Without it, a failing test cannot tell you whether the prompt text, the model, or the retrieval pipeline drifted.

A fixture has four parts, and each one earns its place:

Seed the suite from real traffic rather than imagination. Production failures can be added back to the test set, which keeps regression coverage aligned with the failures users actually encounter (Braintrust). When the prompt lives in a shared library, the library entry and the fixture should share a name, so the test file and the prompt it covers are findable from each other. A naming convention for a team prompt library only has one real job: make a flat list of shared prompts scannable without anyone opening each one, and the same logic applies to fixtures (Prompt Architects).

Sanitising before anything is committed

Fixtures are code. They get cloned, cached, and sometimes pushed to a public remote. Redact before the first commit, not after the first incident.

• Replace names, emails, phone numbers, and account ids with stable placeholders such as [customer_name] and keep the placeholder vocabulary consistent across the suite. Teams that share prompts should state plainly what never goes into a prompt, and customer PII and secrets top that list (Promptly).
• Keep structure, drop content. If the test needs a ten-field invoice, keep ten fields, replace amounts and vendor names with values that preserve the shape.
• Record the redaction step so a reviewer can see what changed. A fixture diff plus a note on why the cassette was refreshed is the evidence a reviewer needs to trust a green build (Agiflow).
• Version the fixtures in the same repository as the prompt. The test should fail when either changes without the other.

A worked number helps here. Suppose you have 40 fixtures. A naive live-call suite with a 2-second median latency per call and three retries on transient failures is roughly 4 minutes of pure wall time at best, and considerably more when a provider rate-limits you. A replay suite over the same 40 fixtures typically finishes in well under a second, because nothing leaves the process. Those are illustrative numbers to size the difference, not measured benchmarks.

Exact-match assertions on model output are the most common way a suite becomes untrustworthy. Sampling means a valid answer is often phrased differently on each run. Format compliance, by contrast, is a good fit for deterministic assertions, because valid JSON, schema adherence, required fields, and length limits can usually be verified in code (Braintrust).

For the fixture you stored, you can make one exact-match assertion legitimately: the replayed output equals the recorded accepted output. That is not testing the model, it is testing your record and replay plumbing. The real prompt tests live in four assertion families.

Structure. Parse the output. Assert the schema, required keys, field types, and any length or count limits. A prompt that used to return a five-item list and now returns four fails here without needing any language understanding.

Key facts. Pull specific values out of the answer and assert on them. The refund note must name the correct order id. The routing decision must be the right queue. The extracted invoice total must equal the value in the input. These are the assertions that catch a prompt edit that "improved" tone while dropping a required field.

Behavioral boundaries. Assert when the model should refuse, escalate, or avoid an action. In domains with policy or compliance requirements, a missed refusal can expose protected data or trigger an action the application was never authorized to take (Braintrust). Record one accepted refusal, and assert that the output declines and names the policy.

Determinism. Run the same fixture three times with the same configuration and assert the outputs are identical. With temperature at zero this should hold. If it does not, you have learned something about the model, and you should move those assertions into the live suite rather than fight the sampler.

import json

def assert_support_reply(output: str, fixture: dict) -> None:
    data = json.loads(output)                      # structure
    assert set(data) >= {"queue", "reply", "confidence"}
    assert data["queue"] == fixture["expected_queue"]   # key fact
    assert len(data["reply"].split()) <= 150            # boundary
    assert fixture["order_id"] in data["reply"]         # key fact

The decision rule is simple. If a human could check the assertion from the input and the output alone, it belongs in the replay suite. If it needs a human to judge quality, it belongs in the live suite or a human review.

Not every test earns its maintenance cost. These four do, and each targets a different failure mode.

Golden set. Ten to thirty representative cases from real traffic, each with an accepted output. This is the backbone. It catches edited instructions that shift tone, change a format, or drop a step. Keep it small enough that a reviewer reads every fixture diff.

Adversarial inputs. Cases where a user tries to pull the model off task: instructions hidden in a retrieved document, attempts to extract the system prompt, requests that conflict with policy. The expected result is a refusal or a bounded answer. A strong test suite covers normal cases, edge cases, known failures, and adversarial inputs so improvements in one example do not hide regressions elsewhere (Braintrust).

Boundary cases. Empty input, maximum-length input, a document with zero matches, a currency amount of zero, a date in the past. These catch crashes and format violations that the golden set, drawn from typical traffic, never touches.

Determinism checks. The repeat-run assertion described above, plus a check that the request the application builds is byte-identical across runs. This is the test that catches broken context assembly, because a context builder that drops a document or reorders it shows up as a changed request, not a changed answer.

The practical split is the one Agiflow states directly: use fakes and recorded fixtures to test deterministic code paths, and use live evals against a golden dataset when the question is output quality, prompt regression, model upgrade risk, safety behavior, or tool-use trajectory (Agiflow).

A prompt test only pays off if it blocks a merge. That means treating the fixture, the prompt, and the expectations as one reviewed unit, and running the suite on every change to any of them.

The gate should be fast and unambiguous. Langfuse's framing of prompt CI/CD is version, gate, and roll out, with prompt edits going through the same stages as code (Langfuse). Concretely:

1. Run the replay suite on every pull request that touches prompts, fixtures, or the context assembly code. Target a runtime measured in seconds so nobody skips it locally.
2. Fail the build on a fixture diff without a matching change note. That single rule prevents the slow drift that happens when accepted outputs are quietly updated to match a regression.
3. Run the live suite on a schedule and before any model or parameter change, separately from the fast gate. Automated evaluation as a CI/CD quality gate works when the live runs are a distinct stage with their own thresholds, not a per-commit dependency (Galtea).
4. Publish the failure with the fixture id, the assertion that failed, and the config diff, so the fix is obvious.

If the suite runs against more than one provider, routing every call through a single OpenAI-compatible endpoint keeps the harness small. That is the case for Quanta Inference, where a model swap is a key change rather than a new client.

For a team that is still centralizing prompts, the library and the fixture set should launch together. An empty library does not get filled, so seed it with prompts that already work (Promptly). The same applies to fixtures: start with the ten prompts the team actually runs daily, not a theoretical catalog. Teams new to this usually treat prompt testing as a separate discipline from prompt storage. It is not: a prompt with no fixture is a prompt with no safety net.

Some questions cannot be answered from a recording, and pretending otherwise produces a suite that is green and useless.

Model upgrades and capability changes. A new model version can change quality and cost together. Fixtures tell you the format still parses; they cannot tell you the answer got better or worse. This is an evaluation question, scored against criteria such as correctness, faithfulness, tone, and safety, and it belongs in the live layer (Braintrust).

Cost and latency budgets. Prompt edits that add examples, context, or reasoning instructions can improve accuracy while also increasing response time or spend, so those changes belong in the same run as quality (Braintrust). Measure tokens and latency on live calls, then record the budget as a threshold.

Safety behavior that depends on sampling. A refusal that holds at temperature zero may not hold across a distribution of samples. If your compliance argument rests on refusal rates, you need repeated live calls, not one recorded answer.

Tool and retrieval trajectories. Whether the model selects the right tool and supplies valid arguments across a multi-step task is a trajectory question. A recorded single response cannot prove it.

Long-horizon task completion. If your system chains steps, the interesting failures appear over sequences. METR's work on task-length horizons found that models succeed almost 100% of the time on tasks taking humans under 4 minutes but below 10% on tasks over roughly 4 hours, which is a reminder that reliability drops with task length (METR). A fixture that replays one step will never surface that.

The decision rule: if the assertion can be evaluated by inspecting one recorded input and output, replay it; if it needs a distribution, a clock, a token counter, or a judgment of quality, run it live on a schedule. Teams that blur this line get slow tests and uncertain quality, and Agiflow's warning is exact on the point: mocks protect fast feedback, evals protect quality, and if one layer tries to do both jobs it usually does neither well (Agiflow).

Most of the work is deciding, not coding. Use these as a checklist.

• What counts as an accepted output? Name who reviews it, so a fixture is a decision, not an accident.
• Which prompts get a fixture first? Rank by blast radius: customer-facing replies and anything that writes to a system of record.
• Where do fixtures live? Same repository as the prompt, versioned with it.
• What is the redaction rule, and who checks it? Write it down before the first commit.
• What is the threshold for the live suite, and who owns it when it fails?
• How does a production failure become a fixture? Whatever the answer, it should take minutes, not a sprint.

The last point is the flywheel. Every incident you turn into a fixture is one more regression the replay suite catches for free next time. Teams that build this habit end up with a suite that grows with real traffic and a live suite that stays small, cheap, and focused on the questions only it can answer.