Thresholds That Fire Too Often: Tuning a Gate

Here is a small but realistic failure. You have labeled 30 candidate swaps by hand: 18 are good (a human would ship them, and shipping improves the artifact) and 12 are bad. You run the current judge and threshold and count the blocks.

The gate blocks 20 of the 30. Of those 20 blocks, 9 were good swaps blocked anyway, and 11 were bad swaps correctly blocked. So the gate catches 11 of 12 bad swaps and blocks 9 of 18 good ones: a block rate on good swaps of 50 percent.

The obvious reading is "the threshold is too high, let it through." But look at what the judge is doing. It only let one bad swap through, so it does discriminate. The problem is that roughly a third of the blocks are false blocks, and the score distribution near the cutoff is smeared. Now compute what your evidence actually supports.

The interval on the good-swap block rate spans more than 40 points. You cannot tell whether the true rate is 30 percent or 70 percent. A threshold sweep on this set will find a cut point that looks wonderful on the 30 cases and then fails on the next 30, because the sweep is fitting the sample, not the pipeline. This is the standard small-sample problem: with so few cases per class, the variance of any estimate dominates, and confidence intervals from the normal approximation are inappropriate below roughly a few dozen cases per class (Cross Validated on evaluating classifiers on tiny samples, Sebastian Raschka on confidence intervals for classifiers).

That is the decision rule this article exists to give you: if the confidence interval on the block rate of good swaps is wider than the change you are considering, stop tuning and go get data. Widening a threshold by 10 points when your estimate wobbles by 40 is not a measurement, it is a mood.

A judge is a prompt, and vague prompts produce blunt scores. "Is this swap an improvement?" invites a hedged 0.6 from almost every case. "Does the new version retain every factual claim that appeared in the original, or explicitly flag any claim it drops?" invites a sharp answer that can be checked.

Your goal is not a harsher judge. It is a judge whose score tracks the thing you actually care about, so that a threshold can later separate cases at all. The method that works in practice is to treat the judge's rubric the way you would treat any prompt asset: versioned, reviewed, and evaluated against a fixed set. Azure AI Foundry, for instance, creates a new prompt asset version on every edit while keeping the full definition and parameters, which is a reasonable model to copy even if you build your own registry (Microsoft Q&A on organizing prompt versioning). The same reasoning applies to recording hyperparameters and evaluation results alongside each version so a quality change can be attributed (Dev community on prompt assets).

Three concrete tightening moves, in order of payoff:

1. Replace holistic questions with enumerated checks. Instead of "is this better," ask for a list: does it preserve every named entity, every numeric claim, every stated obligation, and every commitment in tone. Score each check as pass, fail, or not applicable, then aggregate mechanically.
2. State the boundary between acceptable and unacceptable in the judge's own vocabulary. If a swap that lengthens a section by two sentences is fine and a swap that removes a caveat is not, say both, with an example of each.
3. Add a small set of contrastive pairs. Take five good swaps your judge blocked and five bad swaps it correctly blocked, and include them in the judge prompt as labeled exemplars. This is cheap and it moves the score distribution apart more than any wording change.

The last point matters for the threshold discussion. When you separate the distributions, the same threshold suddenly behaves better, because there is actually a gap to sit in. That is why tightening the judge is a threshold fix that does not loosen anything.

Keep the judge under version control and store the pass rate for each check per version. If a judge version changes the block rate on your fixed labeled set by more than the interval width, you have a regression in the judge itself.

Tightening the judge gives you a sharper score. It does not give you more evidence. That is the second move, and it is the one teams skip.

The core problem is that your labeled set is almost certainly not representative. If your 30 cases came from brainstorming, they encode what an engineer imagined; the cases that matter are the ones production produced. The strongest regression sets are grown from real failures, where each incident becomes a trace, the trace becomes a case, and the case joins the set that gates every future change (Arthur on building regression datasets from production failures). The same flywheel logic appears in golden-dataset guidance: source cases from observed behavior first and use synthetic examples only to cover rare edges you have not yet seen (QASkills on building a golden dataset).

How many cases you need depends on the precision of the estimate you want. A rough planning table helps:

These follow from the usual binomial standard error, and the same caveat about normal approximations applies at the low end; use Wilson or an exact interval rather than dividing by the square root of n (Sebastian Raschka on confidence intervals, Cross Validated on small samples).

Three practical points about widening:

• Grow the set from both directions. Add real cases that got blocked (to measure false blocks) and real cases that shipped with problems (to measure false passes). A set built only from complaints is a set that over-weights the block side.
• Label with the criteria, not against them. Two labelers should work from a written definition of an acceptable swap, or your "good" label is itself a noisy judge. This is the same discipline the golden-dataset literature recommends for expected behavior fields.
• Keep the set fixed once you start comparing. Multiple prompt versions must be evaluated against the same set for comparisons to mean anything (Microsoft Q&A on fixed evaluation datasets, DeepEval on datasets). If you edit the set between runs, you are comparing two different rulers.

Widening is the slower move, and that is the point. It is the only one that converts your block rate from an anecdote into a measurement.

Once the judge separates good and bad swaps cleanly and your labeled set is wide enough that the interval is narrower than the change you are considering, a threshold sweep is legitimate. Run it with a few rules in place.

First, split the set. Sweep on one portion, then confirm on a held-out portion. A cut point that only works on the cases you tuned against is the overfitting problem in miniature, and it is the reason prompt regression practice insists on testing slices rather than single replies or single partitions (Statsig on slices and trends).

Second, decide what each error costs before you look at the curve. A false block costs an engineer an override and a good improvement delayed. A false pass costs a bad artifact in production and a slower, more expensive discovery later. The threshold that minimizes total cost is the right one, and you cannot compute it without assigning weights. This is the same precision-recall trade-off argument that applies to any score-to-label conversion (Evidently AI, Machine Learning Mastery on threshold moving).

Third, consider not moving the threshold at all. If tightening the judge moved the score distribution apart, the existing threshold may now be sitting in the new gap and blocking far fewer good swaps on its own. That is the outcome you want: the gate is no less strict, and it fires less often because it now sees the difference.

A small sweep helper, kept deliberately dull:

def sweep(pairs, weights, cuts):
    # pairs: list of (score, label) where label 1 = bad swap
    # weights: (cost_false_block, cost_false_pass)
    out = []
    for cut in cuts:
        blocked = [(s, y) for s, y in pairs if s >= cut]
        false_block = sum(1 for s, y in blocked if y == 0)
        false_pass = sum(1 for s, y in pairs if y == 1 and s < cut)
        cost = false_block * weights[0] + false_pass * weights[1]
        out.append((cut, false_block, false_pass, cost))
    return sorted(out, key=lambda r: r[3])

Feed it held-out cases, not the ones you tuned the rubric on, and report the interval around the resulting block rate, not just the point estimate.

The whole argument collapses into a short order of operations. Work top to bottom and do not skip.

That last row is where most incidents hide. A quality change can come from the prompt, the app logic, or both, which is why prompts should be tracked alongside application versions rather than separately (Microsoft Q&A on tracking prompts with app versions). Apply the same discipline to the gate: version the judge, version the labeled set, and log the threshold alongside both.

Two failure signatures worth remembering. If tightening the judge makes almost no difference to the score distribution, your rubric is not the problem; your labeled cases may be mislabeled or too easy. If widening the set makes the block rate jump wildly from the small-set estimate, the small set was biased, usually because it was built from imagined rather than observed cases (Arthur on production-sourced tests, QA Wolf on limits of curated sets).

There is a cost angle too. Every judge invocation that scores a swap candidate consumes tokens, and if your gate is rejecting half of everything, you may also be paying twice: once to judge and once to retry. Teams that attach cost and token data to every span can see which branch of a pipeline is expensive, and the same traces let you check whether a cheaper judge configuration preserves the separation you need (Arize on reducing LLM costs without losing quality, Braintrust on where LLM costs hide). A judge that only needs to answer three narrow questions is both more reliable and cheaper than one asked to make a holistic call.

If you are running several judges across several model families, routing them through a single API with consistent pricing makes the cost side of this trade-off easier to compute; Quanta Inference exposes text, image, video, audio and embedding models behind one OpenAI-compatible key.