Evaluating Prompt Candidates Without a Human Judge

You cannot gate on "looks good". Each case needs a score in a fixed range, produced by a rule, a judge model, or a mix. Three scorer types cover most needs:

Prefer deterministic checks wherever the requirement is objective. "Must not invent a price" is a regex against your price list. "Must cite at least one source" is a check that the citation field is non-empty and resolves. Reserve judge calls for properties like helpfulness or internal consistency, where no rule can decide.

Judge variance is the thing that breaks naive gates. The same input can produce different scores on repeated runs because of sampling and because the judge itself is a model. Statsig's write-up on prompt regression testing makes the core point well: test slices, not single replies, because what matters is the distribution of outcomes by intent and content type, not one lucky answer. In practice that means running borderline cases two or three times and taking the median, pinning the judge to a fixed model and temperature, and giving the judge a rubric with explicit score anchors (1 = fails, 3 = acceptable, 5 = ideal) rather than "rate 1 to 10".

Version the scorer alongside the prompt. If you change the rubric, old scores are no longer comparable, and a gate that compares new scores to old ones will fire for the wrong reason. Treat the scorer config as part of the prompt asset.

A single pass rate over the whole set is the most common mistake in automated prompt evaluation. It hides exactly the regressions you care about. Consider a support assistant with three intents:

The aggregate moved 4 points, which many teams would wave through as noise. But billing_dispute fell 17 points, and a 61% pass rate on disputes means roughly four in ten customers get a wrong or unhelpful answer on money questions. That is the decision rule: fail the candidate if any slice falls below its absolute floor or drops more than its tolerance, regardless of the overall number.

Set the floor from what the business can tolerate, and the tolerance from measurement noise. A reasonable starting policy:

1. Run the baseline prompt three times against the golden set.
2. Record the per-intent pass rate each time. The spread between runs is your noise band.
3. Set tolerance to roughly twice that spread, so ordinary variance does not fail builds.
4. Set the absolute floor per intent from risk: billing and security intents get a higher floor than chit-chat.

With the numbers above, if baseline billing_dispute runs came out at 79%, 78%, and 77%, the spread is 2 points, so a tolerance of 4 or 5 points is defensible. A drop to 73% would pass tolerance; a drop to 61% would not. The floor might be 75%, in which case even 73% fails. Both checks run, and either can fail the gate.

Weight slices by traffic only when you need a single summary for dashboards. The gate itself should stay unweighted and per-slice, because weighting is exactly how a low-volume intent (which is often the highest-risk one) gets averaged away.

The gate belongs in the same place as your unit tests. GitHub Actions runs workflows defined as YAML files in .github/workflows, triggered by repository events such as pull_request or push: see the GitHub Actions documentation. Limiting the trigger to specific activity types and paths avoids burning runner minutes on every unrelated commit, as covered in the workflow syntax reference and this step-by-step GitHub Actions guide.

A gate job that runs the eval and exits non-zero on regression:

name: prompt-eval-gate
on:
  pull_request:
    paths:
      - "prompts/**"
      - "evals/**"

jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install -r requirements-eval.txt
      - run: python evals/run.py --prompt prompts/support.yaml --golden evals/golden.jsonl --out results.json
      - run: python evals/gate.py --results results.json --baseline evals/baseline.json

Keep gate.py deliberately dumb. It loads results and baseline, compares per-intent rates against floor and tolerance, prints a table, and exits 1 if any slice fails. The eval runner does the expensive work; the gate does the arithmetic. That separation means you can rerun the gate against stored results when you change policy, without paying for model calls again.

Three operational details matter as much as the logic. First, cap the number of judge calls per run and fail closed if you exceed budget, so a runaway loop does not produce a bill. Second, store results.json as a build artifact so a failed gate links to the exact case-level scores that caused it. Third, pin the model and judge versions in the eval config, because an API-side behavior change is itself a regression you want to detect. As Statsig notes, prompts that passed can fail after a provider update with no code change at all, which is precisely why the gate must run on the same inputs repeatedly rather than once at development time.

If you are calling several model families from one eval harness (a fast model for the judge, a stronger one for the candidate), routing them through one API is the difference between three SDKs in your CI image and one. Quanta Inference exposes text, image, video, audio and embedding models through a single OpenAI-compatible key, which keeps the eval runner small.

An automated gate fails a lot in its first weeks, and how you handle those failures determines whether the team keeps it. Three patterns help.

Rollback on failure. In CI, the safest recovery is the one you already understand: the merge does not happen, and the previous prompt stays live. Pipeline tools formalize this. Harness, for example, supports a PipelineRollback failure strategy that reverses a pipeline when a step fails: see the Harness failure strategy documentation. Octopus's writing on modern rollback strategies makes the related point that rollback scripts are often untested and stale, so rehearse the revert path before you need it.

Quarantine noisy cases. Some cases will flap because the task is genuinely open-ended. Move them out of the blocking set into a tracked "watch" list rather than tuning the threshold until they pass. That preserves gate sensitivity and keeps an honest record of known-hard cases.

Decouple release from deploy. If the prompt lives behind a runtime switch, a regression can be turned off without a redeploy. Feature flags exist precisely to turn functionality on or off without changing source code or redeploying, per Unleash's explainer, and Pete Hodgson's feature toggles article categorizes the kinds you need. A kill switch for a prompt family is an ops toggle, not a release toggle, and it should never be deleted.

Treat the prompt as an immutable artifact, and the gate becomes meaningful. Every edit produces a new version with its text, parameters, model ID, and eval results attached, a workflow Microsoft's community answers describe for prompt management: see this Azure AI Foundry thread. Braintrust's survey of prompt versioning tools puts the same idea bluntly: prompts are code, and treating them as configuration files edited in place is how teams lose the ability to roll back.

Putting it together, here is an order that works for a team starting from zero:

1. Harvest. Pull 60 real inputs across your top intents, weighting toward recent failures. Tag intent and failure mode.
2. Write pass conditions. For each case, write the check in one sentence. If you cannot, the case is too vague to gate on.
3. Score the current prompt three times. Record per-intent pass rates to establish noise.
4. Set floors and tolerances per the policy above, and commit baseline.json.
5. Build the runner and gate. Keep them separate. Make the output a table a reviewer can read in ten seconds.
6. Run in shadow mode for two weeks: the gate reports but does not block. Collect false failures and quarantine them.
7. Flip to blocking on the intents that have been stable, then the rest.

Two weeks of shadow mode is the step teams skip, and it is the one that builds trust. A gate that cries wolf gets disabled, and a disabled gate is worse than none because it creates the appearance of coverage.

If your eval harness is part of a larger internal tool, or you want a specialist to stand up the golden set and CI gate with you, Quanta is built for that: describe the pipeline you need and someone picks it up with you.