Keeping a Tagging Prompt in Sync Across Two Models

The minimum viable versioning scheme is a directory of immutable prompt files, each paired with the schema it expects and the model configuration it was validated against. Do not edit a prompt file in place. Create a new one, and let the old one stay on disk because you will need it for rollback and for reproducing old outputs.

A simple layout that works:

prompts/
  tag-transcript/
    v1/
      prompt.md
      schema.json
      config.json
    v2/
      prompt.md
      schema.json
      config.json

The config.json is the part teams forget. It should carry the model identifier, temperature, maximum output length, and any provider-specific settings that affect output. Prompt versioning only works when the recorded configuration travels with the prompt version, because a score difference between two runs can be traced to a model swap or a temperature change as readily as to an edited instruction (Braintrust on prompt testing). If your config lives in an environment variable that someone changes on a Friday, your version history is a fiction.

Three rules keep this honest. First, versions are immutable: once a prompt version is created, it stays unchanged, and any modification produces a new version rather than altering one already referenced in logs (Braintrust on prompt versioning). Second, every production record stores the prompt version, the config, and the model identifier alongside the output. Third, the schema is versioned separately from the prompt text, because a schema change and a prompt change are different kinds of risk.

That separation matters more than it sounds. Adding a required field to the schema can break a parser downstream even if the model still extracts the same information. Changing the prompt can shift label boundaries even if the schema is untouched. If both are bundled in one file, you cannot tell from the version number which kind of change you made.

If you are running several models behind one pipeline, keeping the provider and model settings in a single config file makes the comparison cheap, and routing all of them through one API key with a consistent request format removes a whole class of accidental differences, which is what Quanta Inference is built for.

You cannot gate a deploy without a baseline. The baseline is a set of transcripts that never changes, each with the labels a careful human would assign.

Start smaller than you think. The most common mistake in building these sets is waiting for a "representative" sample of hundreds. A better starting point is twenty to forty transcripts that cover the shapes you actually see: short turns, long monologues, speakers who interrupt, transcripts with missing punctuation from automatic transcription, transcripts where the topic is genuinely ambiguous, and at least three cases where a previous version got it wrong.

Researchers applying language models to interview transcripts have concluded that AI is best positioned to augment rather than replace traditional qualitative methods, and that the balance of human and machine input is what determines whether the analysis holds up (Child Trends case study). The same logic applies to your expected labels: a human assigns them once, they are frozen, and the model is measured against them. If the model disagrees, that is data. If the human revises the expectation to match the model, you have not built a test set, you have built a mirror.

Store each case as a transcript file plus an expected-labels file, and include the reason for any label that a reasonable person might dispute. When a case turns out to be genuinely ambiguous, mark it advisory rather than deleting it, so it still reports a score without blocking a deploy.

Keep the set out of the prompt. Few-shot examples inside the prompt are useful, but the evaluation set must never be the same examples the model sees, or you are measuring memorization. A few-shot approach is a standard prompting strategy for improving response quality (Philip Adu on prompting strategies), and its benefit collapses the moment the examples are the test.

Now you need a harness that runs the fixed set against a candidate version and reports what changed. It does not need to be elaborate. It needs to be repeatable.

The core loop is: load each transcript, render the prompt template with it, call the model, parse the output against the schema, and compare the extracted labels to the expected labels. Record the result per case, per version, so two candidates can be compared side by side rather than argued about in a chat thread.

def run_case(transcript, prompt_version, config):
    messages = render(prompt_version.prompt, transcript.text)
    raw = call_model(messages, config)
    parsed = parse_against_schema(raw, prompt_version.schema)
    return {
        "case_id": transcript.id,
        "prompt_version": prompt_version.id,
        "model": config.model,
        "labels": parsed.labels,
        "raw": raw,
    }

def score_case(result, expected):
    got, want = set(result["labels"]), set(expected.labels)
    return {
        "exact_match": got == want,
        "precision": len(got & want) / len(got) if got else 1.0,
        "recall": len(got & want) / len(want) if want else 1.0,
        "schema_valid": result["parsed_ok"],
    }

Structured outputs via schema enforcement give you a hard guarantee that the response conforms to the format, which is why constrained decoding frameworks have standardized on JSON Schema (the JSONSchemaBench paper). That lets you separate two questions cleanly: did the output parse, and was the content right. Parse failures are rare with constrained decoding but not zero, and they should be treated as hard failures rather than scored.

Run each case more than once if your configuration has any randomness. Regression testing for language models asserts quality scores over a dataset rather than exact output strings, precisely because probabilistic systems cannot be tested by exact matching (Confident AI on regression testing). Three runs per case on a twenty-case set is sixty calls, which is cheap enough to run on every prompt change and expensive enough that nobody does it casually.

A threshold turns a pile of scores into a decision. Without one, every result is arguable, and the person who cares most about shipping wins by default.

A practical gate for a tagging task:

Set your own numbers from your current baseline rather than copying these. The important structural point is the fourth row: a candidate that improves the overall average while breaking a case that previously passed is a regression, and the average will hide it. Prompt testing is designed to show whether a change improved the intended case without breaking behavior that already worked (Braintrust on prompt testing).

A readable diff makes failures explainable. Print one line per case with the expected labels, the candidate labels, and the added and removed tags.

$ python eval.py --prompt tag-transcript/v2 --model candidate -v
[PASS] t-014  exact  expected=budget,staffing  got=budget,staffing
[FAIL] t-031  recall expected=timeline,risk  got=timeline
[FAIL] t-038  precision expected=risk  got=risk,risk_mitigation

Reading that output, you can guess what happened in the second line without opening the transcript: version two dropped a risk tag on a case the previous version caught. That is a block, and it is worth ten minutes before it becomes a dashboard anomaly.

A model switch is not a prompt change, and treating it as one is how teams get surprised. The new model may be stricter about JSON, chattier in its preamble, better at the task, or all three at once. You will not know which until the fixed set tells you.

Run the swap as a formal comparison: same prompt version, same config except the model identifier, same fixed transcript set. If the candidate passes the gate, deploy it and keep the previous pair (prompt version plus model) live as a rollback target for at least a week of production traffic. If it fails, compare the failures to see whether the problem is the model or the prompt.

This matters because the fix for a model swap is often not a rollback but a prompt adjustment that no longer works when you go back. Track that carefully. If model B needs a stricter instruction and you add it, the prompt version is now B-specific, and the rollback target for A is a different prompt version, not the same file with a different model setting. Version the pair, not just the text.

Prompt management, as distinct from versioning, treats the prompt as part of the application infrastructure with runtime updates, monitoring, and access controls (LaunchDarkly on prompt versioning). You do not need a commercial platform to get the critical part: the ability to move a single configuration value and have production return to a known state without a code deploy.

The rollback checklist, run in order:

Step four is the one people skip. Items processed during the bad window have outputs that parsed fine and scored acceptably, so they never trigger an alert, and they sit in your downstream table as quietly wrong labels. Requeue them.

Adding the failing case back to the set (step six) is what makes the fixed set improve over time. Production failures should be added back to the test set so regression coverage stays aligned with the failures users actually encounter (Braintrust on prompt testing).

If you would rather not run this loop yourself for a new line of business, or you want a specialist to own the evaluation set and the rollback runbook with your team, Quanta Experts reads your context and hands back finished deliverables, including the harness and the threshold proposal.