Two Models, One Prompt Library: Keeping Instructions in Sync

Four parts of a prompt are model-specific by nature. Treat them as overrides, never as shared text.

Capability requirements work as hard filters: context window, structured output, tool use and modality can eliminate an otherwise strong model before cost or latency enter the picture (PrompTessor). The prompt library should mirror that logic. If the small model cannot reliably produce a tool call from prose, the prompt must carry an explicit call shape, and that shape belongs in the small model's block.

On task decomposition, a working rule beats a theory. Any task that needs more than one dependent reasoning step, such as look up a record, compare it against a policy, then decide an action, should be written as numbered steps for the small model and as a single described goal for the large one. The same business outcome, two renderings.

Error recovery deserves the most care because it is invisible until it fails. Prompt guidance for production agents puts guardrails in their own section because they must override every other instruction (Vapi's prompting guide). In a two-model library, that section gets two versions: the small model gets a capped retry count and a fixed escalation message, the large model gets permission to repair its own output once before escalating. Document both numbers next to the override, so the next maintainer knows the reasoning.

Four parts of the prompt should be byte-identical across models. Any divergence here is a bug, not an override.

• Fact base. Product names, policy text, prices, dates, and excerpts. If the fact changes, it changes once, and both models pick it up on the next render.
• Policy constraints. Data handling rules, prohibited claims, and refusal triggers. A policy that applies to one model and not the other was never a policy.
• Tone. Voice and formality. The small model can be told to be shorter; it cannot be told to be a different brand.
• Acceptance criteria. What a correct answer contains. This is the contract the tests check, and it must not depend on which model answered.

A useful test: read the shared block aloud. If any sentence names a model, a token limit, or a tool syntax, it is in the wrong block. The shared block is the part a compliance reviewer should be able to sign off on once.

Four steps, each with a check that catches drift early.

1. Move prompts into the repository as structured files. Every artifact gets an immutable version identifier, and every edit produces a new version, so loading a specific version always returns the same text, parameters and metadata (LangWatch). Files that can be diffed in review beat text pasted into a dashboard.

2. Keep each prompt in one file with named blocks. A minimal layout:

prompt_id: refund_decision
version: 7
shared:
  facts: facts/billing_policy_v3.md
  policy: policy/refund_rules.md
  tone: tone/support_neutral.md
  acceptance: tests/refund_acceptance.yaml
models:
  small:
    task: overrides/small/refund_steps.md
    format: overrides/small/refund_schema.json
    tool_syntax: overrides/small/tools.md
    error_recovery: overrides/small/retry.md
  large:
    task: overrides/large/refund_goal.md
    format: overrides/large/refund_schema.json
    tool_syntax: overrides/large/tools.md
    error_recovery: overrides/large/retry.md

3. Render the final prompt at build time. The renderer concatenates shared blocks, then the override for the requested model, and records the rendered hash in the trace. Teams that trace which version produced each output can answer the debugging question that costs hours: which prompt was actually running (LangWatch). Treat prompts as immutable versioned artifacts with environment separation (development, staging, production) so a change is tested before it reaches users (Braintrust).

4. Run the diff check in CI. Two rules are enough to start:

Rule 1: a change touching only one model's override fails
         unless the commit message contains OVERRIDE: <model> <reason>
Rule 2: a change touching the shared block triggers the full
         evaluation suite for both models before merge

Rule 1 is the important one. Most drift happens because someone fixes the small model's output format at 4 pm on a Friday and never revisits the large model, or the reverse. Requiring an explicit override acknowledgment turns a silent divergence into a reviewed decision.

Consider a support assistant that decides whether to refund an order. It runs on a small model for routine requests and escalates to a large model when the case is unusual.

Before the split, the team maintained two near-copies of a 1,200-word prompt. Over six weeks, they made 11 edits. Four of those edits touched only one copy: a tightened refund window, a new prohibited phrase, a changed tool argument name, and a third retry rule. Two of them were policy changes that should have applied to both models.

The cost of that drift is easy to state in tokens, which is why the fix pays for itself. Suppose the library serves 10 million input tokens a day and the small model handles 80 percent of traffic. Cutting the shared block by 300 wasted words per call saves roughly 2.4 million tokens daily on the small model alone, before any quality gain. At high volume, a miscalibrated model choice adds or removes tens of thousands of dollars a month (GMI Cloud). Drift that keeps stale policy text in every small-model call is that same kind of quiet bill.

After the split, the shared block held the refund window, the prohibited-phrase list and the acceptance criteria. The small model's override held four numbered steps, a strict JSON schema and a fixed escalation message. The large model's override held a goal sentence and a repair-then-escalate rule. The next policy change touched one file and appeared in both renders on the same commit.

Validation must run against a representative test set, not single calls, because probabilistic regressions only become visible across many iterations (Vapi's prompting guide). Four common evaluation approaches exist: multiple choice benchmarks, verifiers, leaderboards and LLM judges (Sebastian Raschka). For a two-model prompt library, a verifier for the schema and a judge for the tone and policy checks cover most of the risk.

A practical split of the test suite:

The cross-model row is the one teams skip. Take 50 cases with known correct answers. Run both renders. Score both with the same judge and the same acceptance criteria. If the large model passes and the small model suddenly does not on cases it used to pass, either an override regressed or someone edited the shared block in a way only the large model tolerates. Either way, you want the failure in CI rather than in a customer conversation.

Not all divergence is a bug. Three cases justify an override that the other model does not receive:

• The small model needs a shorter fact base because irrelevant context degrades its accuracy on the target task.
• The large model has a tool the small one lacks, so its error-recovery rules include a path the small model must not attempt.
• Latency budgets differ enough that the small model must answer in one call while the large model may take two.

The decision rule: an override is allowed when it changes how the model reaches the answer, and forbidden when it changes what counts as a correct answer. Format, steps, tool syntax and retries are how. Facts, policy, tone and acceptance criteria are what. When a proposed override crosses that line, it belongs in the shared block or it needs a product decision, not a prompt edit.