From Override to Rule Change: A Pull Request for the Prompt

Pull the cases first. Three to five rows is usually enough, and they must be contiguous: the same input shape recurring, not five unrelated complaints.

The context column is the one people skip and the one that wastes the most effort. When a prompt lacks the information it needs, no rewording fixes it. Build the row set so you can tell an instruction problem from a retrieval problem before spending a week on phrasing.

Overrides cluster, and a cluster is a test category. If you have twelve overrides where the model answered a direct price question with a policy paragraph, you have a category called "direct answer to a direct question" and you write three test cases for it, not twelve.

Prompt versioning tracks every change to a prompt over time so teams know which version runs in production, who made the last change, and can roll back if something breaks; prompts also change more often than code, and non-engineers often need to contribute Agenta. The practical consequence is that the prompt must live in a file git can diff, not in a dashboard text box that only holds current state.

A useful diff is small. One or two sentences changed, with the commit message naming the override cluster:

- Answer the customer's question using the policy context.
+ Answer the customer's question directly in the first sentence.
+ If the answer depends on a policy condition, state the condition
+ in the second sentence only.

Two properties make this reviewable. The change is scoped to the cluster rather than rewriting the whole prompt. And it names a behavior, "first sentence direct answer," which a test can assert. "Make the tone warmer" is not testable and should not ride in the same pull request.

Prompt management goes further than versioning, treating prompts as infrastructure with runtime updates, monitoring, access controls and dependency tracking LaunchDarkly. Dependency tracking is what bites in multi-step systems: if prompt one emits a JSON schema that prompt two consumes, changing that schema breaks prompt two, and the dependency has to be visible in the diff discussion.

There is a governance angle. NIST's AI Risk Management Framework addresses accountability, oversight and human responsibility for AI systems, and NIST CSF 2.0 frames governance and oversight as core security management responsibilities NHI Management Group. A prompt change with a linked override row, a reviewer and a test is the evidence trail those functions ask for. A console edit is not.

This is the decision rule the whole process turns on: run the new test against the current prompt before changing anything. It must fail. If it passes against the unfixed prompt, the override was not reproducible, and you are about to ship a change with no evidence behind it.

Prompt testing runs a prompt and its model configuration against representative inputs, attaches an explicit expectation to each case, and applies the same scoring method across every run, so each result ties to a specific prompt version Braintrust. The same guide separates assertion types that matter: task correctness, format compliance, behavioral boundaries such as when to refuse or escalate, tool and retrieval behavior, and operational limits like latency and token cost.

The recorded configuration must travel with the test. Model, temperature, tools, response format and retrieval step all affect behavior, so a score difference between runs can come from a model swap as easily as from an edited instruction Braintrust. Pin them or your red-to-green proof means nothing.

A worked example on a support prompt that answers product questions:

Step 3 is where people get sloppy. If the failure message does not resemble the logged output, you have written a test that fails for the wrong reason, most often because the test's context does not match production retrieval. Step 6 is the guard against fixing one thing and breaking another: a prompt tweak that fixes one complaint can quietly break other answers Langfuse.

Use deterministic checks wherever you can. For a structured field, a routing decision or a required phrase, an exact check is cheap and needs no interpretation, the same property that makes unit tests fast and unambiguous Braintrust. For open-ended phrasing you need a judge.

One analysis puts LLM judge agreement with human reviewers around 85%, higher than two humans agree with each other on the same task Confident AI. That figure comes from the cited research and is not a guarantee about your rubric.

The known distortions are specific and curable. Position bias makes judges favor outputs presented first or last, and swapping presentation order shifts accuracy by more than ten percent in code evaluation, while self-preference bias pushes judges toward outputs more familiar to their own model family and length bias favors longer, more formal responses tianpan.co.

Rules for a judge that behaves like a test rather than a coin:

• Use a different model family for the judge than for the system under test.
• Grade against a reference answer (the human replacement) rather than open-ended pairwise comparison.
• Keep the rubric text in the repository beside the prompt, so a rubric change is also a diff.
• Pin the judge's temperature and record the judge model in the same config as the prompt config.
• Spot-check twenty graded results by hand when first introducing the judge, and record how many you disagreed with.

If your judge disagrees with your own reading on a meaningful share of the spot-check, the judge needs fixing, not the prompt. Evidently's guide covers judge construction and the offline versus online split, useful when the same rubric should run in CI and on production traces Evidently AI.

Prompt testing asks whether a configuration still behaves correctly on cases you already care about; prompt evaluation asks how good the output is against criteria; A/B testing asks which variant wins with real users; unit testing asks whether deterministic code returns the expected value Braintrust. The pull request needs the first and the last. The other two do not belong in a merge gate.

A test encoding a single override proves one thing. The value comes from the suite growing as your log grows: production failures added back to the test set keep regression coverage aligned with the failures users actually encounter Braintrust. Every time an override cluster appears twice, a new case goes in.

Size by risk. If you have a hundred prompts in production you cannot test them all thoroughly, and versioning pain grows with the number of prompts and the dependencies between them Agenta. Pick the prompts whose failures cost most, usually where a wrong answer reaches a customer, moves money or changes a permission, and give those a real suite. Give the rest a smoke test.

Avoid the tempting shortcut of aggregate metrics. A suite reporting "92 percent pass rate" hides which case regressed. CI needs per-case pass or fail so the red build names the case that broke.

Human review belongs inside the loop for a second reason: when you spot-check graded results you validate not only the judge but the expected behavior you encoded. Reviewers sometimes find the human replacement in the log was itself wrong, which is a more valuable finding than a failed assertion.

If you run several prompts in a chain, check whether disagreement between steps is even visible. Multi-agent designs diverge routinely, with estimates in the twenty to forty percent range depending on task type, and silent failures where an agent returns a confident, well-formatted wrong answer are the most operationally dangerous tianpan.co. A fix for step one will not surface a disagreement in step three, so log the intermediate outputs your test asserts on.

The red-before-green requirement catches the most bad pull requests and attracts the most argument. The argument is always "the case is obviously broken, we all know it fails." That is not evidence. A pasted CI log showing the failing case on the base commit settles the question permanently, because the next reader sees which behavior the commit protects.

Treat prompts with the same care normally applied to application code, since you would not push code to production without version control, testing and proper deployment processes LaunchDarkly. The checklist is what that care looks like when the artifact is a sentence of English.

Three conventions make the pattern cheap to repeat.

Keep the prompt, the test config and the rubric in one repository so a single pull request touches all three, the same idea as referencing the prompt by file in a declarative test config so it lives in version control alongside the code promptfoo. Put the log trace link in the commit body, not just the pull request, so the evidence survives after the pull request closes. And when a model judges the change, record the judge model beside the prompt: judge versions drift, and a judge swap that turns a suite green without a prompt change is a regression in disguise. If you serve the system and the judge through one gateway, that model choice becomes a configuration value rather than a hardcoded SDK call, which is part of what Quanta Inference provides across text, image, video, audio and embedding models.

The final discipline is ownership. Someone has to own each prompt suite the way someone owns a service. Without an owner the suite rots: cases get skipped, the judge drifts, the red-before-green check becomes a ritual nobody reads.