Turning Refusals into a Labeled Set for Tuning

You rarely need every refusal. You need enough labeled refusals per stratum that a proportion is stable, and you need the strata to line up with the decision you are going to make.

Start by defining strata along two axes.

Axis one: reason code. Whatever your pipeline already records as the cause of the refusal. Typical buckets: no retrieved documents, documents retrieved but below support threshold, conflicting documents, out-of-scope intent, quote requirement not met (no verbatim quote in the sources to anchor the claim), policy block. The quote requirement is the one this article focuses on, because it is the most common source of both over-refusal and silent over-answering in grounded systems.

Axis two: model confidence. Whether you have calibrated scores or only a coarse band (high, medium, low), stratify on it. Refusals near the decision boundary are where threshold changes will move traffic, so they carry more information per label than refusals deep in the safe zone.

A practical allocation: 15% of each stratum, minimum 100 and maximum 400 per stratum per labeling round. With 15% sampling you get a binomial standard error of about 2.5 percentage points at 400 labels, which is enough to see a precision shift of five points. If a stratum contributes fewer than 100 refusals in the window, sample all of them and note the small-n in your report.

import random
from collections import defaultdict

def sample_refusals(events, rate=0.15, floor=100, cap=400, seed=7):
    rng = random.Random(seed)
    buckets = defaultdict(list)
    for e in events:
        if e["outcome"] != "refused":
            continue
        buckets[(e["reason_code"], e["confidence_band"])].append(e)
    picked = []
    for key, items in buckets.items():
        n = max(floor, int(len(items) * rate))
        n = min(n, cap, len(items))
        picked.extend(rng.sample(items, n))
    return picked

Two rules keep the sample honest. First, fix the seed and store the sampled IDs so the set is reproducible and you can re-label the same events later when guidelines change. Second, include a small random control of refusals drawn outside the stratified scheme (say 50 events) so you can detect if your strata have drifted from reality.

The labeling budget is small. A reviewer who can see the question and the evidence bundle typically needs one to three minutes per refusal, faster once the guidelines are familiar. At 1,000 refusals that is a few reviewer-days, which is why refusal sampling is one of the cheapest quality investments available.

The label set is deliberately tiny: correct refusal or incorrect refusal. Resist the urge to add a "partially correct" category. A refusal either was the right call under your product policy or it was not, and the ambiguity usually lives in the evidence, not the label.

The decision rule for grounded answering, stated once and then applied mechanically:

• If the evidence bundle contains a passage that directly supports an answer to the question, a refusal is incorrect.
• If no passage supports an answer, a refusal is correct.
• If passages conflict and neither is clearly more authoritative, a refusal is correct, and you should tag the event with a conflict flag for separate analysis.

The last point is not a judgment call you should leave to reviewers. Research on grounded models shows that misleading or conflicting evidence is a distinct failure mode from missing evidence: prompt-based abstention asks whether context is sufficient, not whether it is correct, and models under explicit abstention prompting still answered 41.6% of misleading questions, with 63% of those answers repeating the planted wrong entity (GRAB-RAG paper). A benchmark that grades evidence sufficiency across five levels, from full support to conflicting, reports the same pattern, with abstention dropping sharply under conflicting evidence (evidence sufficiency benchmark). So make conflict its own reason code and grade it separately.

For the quote requirement specifically, add one clarifying sentence to the guidelines: a refusal is incorrect if a verbatim quote exists anywhere in the bundle that answers the question, even if the quote is in a table, a footnote, or an oddly formatted field. Reviewers who cannot find the quote quickly should escalate rather than guess.

A comparison table helps reviewers calibrate. Populate it with real examples from your first 50 labels:

The fourth row is the one that will generate debate. Decide it once, write it down, and apply it consistently, because inconsistent labels on a stratum that large will swamp your precision estimates.

Inter-annotator agreement is your quality gate. Have two reviewers label the same 100 refusals before the main run. If they disagree on more than 10, the guidelines have a hole; find it and patch it before spending the budget. If they disagree on fewer than 5, you can run single-reviewer labeling with a 10% audit.

Note one hazard: an automated refusal detector is not a labeler. Classifiers that return complied, refused, or partial_refusal are useful for triage and for measuring refusal rates, but they inherit whatever policy the judge model encodes (DeepEval refusal classifier). Use them to route events into strata, then label with humans.

Suppose a month of traffic produces 40,000 grounded answers and 6,000 refusals. You sample 15% per stratum, which lands you at roughly 1,200 labeled refusals after caps. Your labels come back like this:

Overall refusal precision is 993 of 1,200, or 82.75%. Read the table before reading the number. The quote requirement stratum is where the damage is: 119 incorrect refusals out of 340 sampled, meaning the quote check is firing when a usable quote existed. That single stratum accounts for 48% of all your false refusals.

Now you have a decision. If you tighten the quote requirement (accept quotes from tables, accept quotes longer than the current minimum, accept quotes that need light normalization), you expect some incorrect refusals to convert into correct answers. If you also want to hold overall precision above a target, you tighten the other strata slightly to compensate.

This is where threshold selection gets concrete. Suppose your quote check produces a continuous score (passage similarity to the question, plus a verbatim-match flag) and you currently cut at 0.70. You replay the 340 labeled quote-stratum events through the scorer and get:

Moving the cut from 0.70 to 0.78 buys you ten points of precision in this stratum and removes 53 refusals, which become attempted answers. Thirty-seven of those 53 were incorrect refusals, so users gain 37 usable answers, and the other 16 are answers you will want to grade separately. Moving all the way to 0.85 removes 117 refusals but converts only 26 more incorrect refusals into answers, so the marginal return collapses. That is the decision rule: walk the threshold until the next step buys fewer than a handful of false-refusal conversions per hundred refusals released, then stop.

Precision alone can mislead you if the positive class is unbalanced, which is why multi-cut-point analysis and calibration checks matter more than a single ROC operating point (Manokhin on threshold selection). In the table above, precision is computed only over kept refusals, so every row is a meaningful operating point rather than a single global optimum.

Define the four cells of the confusion matrix over the labeled refusal set, with "refused" as the positive prediction:

• True positive: the model refused and the refusal was correct.
• False positive: the model refused and should have answered.
• False negative: the model answered when it should have refused. This cell is not in your refusal log, so it must come from a separate sample of answered traffic.
• True negative: the model answered correctly.

Precision of refusals is TP / (TP + FP). It answers "when the model refuses, how often is it right?" Recall of refusals is TP / (TP + FN), which answers "of everything that should have been refused, how much did we catch?" Both require the answered-side sample, so budget for it: a 2% random sample of 34,000 answers is 680 events, enough to estimate the false-negative rate with reasonable precision.

Coverage is the fraction of traffic that received an answer, correct or not. It moves inversely with refusal rate, so it is the metric your product team will feel. Track it next to precision at every threshold you consider, and keep the ratio in view: a threshold change that raises refusal precision by two points while dropping coverage by eight points is usually a bad trade unless the failure mode is high stakes.

One more metric worth computing is the semantic confusion rate, meaning the share of near-paraphrase question clusters where the model refuses one phrasing and answers another. A large study of paraphrased prompts found that global false rejection rates hide this local inconsistency, and that stricter refusal does not automatically mean more consistent refusal (semantic stability of refusals). If your labeled set includes paraphrased duplicates of the same intent (it should, at least 50 clusters), you get this metric for free.

The choice of which metric to optimize depends on what a mistake costs. When a wrong answer is expensive and a refusal is annoying but recoverable, optimize for high precision on refusals and accept lower recall. When a refusal is the expensive failure (for example, a support bot that sends users to a human for routine questions), flip it and optimize recall while holding precision above a floor. Write the target down before you look at the table, or you will rationalize whichever threshold looks best.

Quanta Inference exposes text, image, video, audio and embedding models through one OpenAI-compatible API with prepaid dollar pricing, which keeps the scoring passes in your threshold sweep on a single key.

The handoff from labels to tuning is a replay, not a retrain. Your threshold sweep should run the exact scoring code that production runs, over the exact evidence bundles captured in the trace, so a measured improvement is a real one. Store the score of each event so future guideline changes can be re-applied without re-labeling.

Sequence the work like this:

1. Freeze the guidelines and the sample (seed, IDs, strata) in a versioned file.
2. Label 1,200 refusals across strata, with a 100-event overlap for agreement.
3. Sample 680 answered events for the false-negative side.
4. Replay every event through the current scorer and record scores.
5. Sweep the threshold, compute precision, recall and coverage at each cut.
6. Pick the cut that meets your precision target with the smallest coverage loss.
7. Shadow the new threshold on live traffic for a week, comparing refusal rates by stratum to the replayed prediction.

Step seven is where surprises appear, usually from strata that were too small to sample confidently. If the shadow run shows a stratum refusing at twice the replayed rate, you have a distribution shift between the labeled window and live traffic, and you re-sample.

Refusals are also trainable, not just tunable. The RefusalBench authors report that selective refusal is a trainable, alignment-sensitive capability and that neither scale nor extended reasoning reliably improves it (RefusalBench). Your labeled set doubles as supervised data. A few hundred correct and incorrect refusal pairs from the quote stratum, with the surrounding evidence, are enough to fine-tune a small model to judge whether a verbatim quote exists. That is often more robust than nudging a similarity score, and it composes with the prompt-level fixes you have already made.

There is an ongoing maintenance cost, and it is worth naming. Re-label 200 refusals every quarter, drawn from the same strata, and compare the incorrect-refusal rate to the baseline. If it drifts by more than three points, the threshold has aged out and needs another sweep. If a new evidence type enters the corpus (scanned tables, images with embedded text, transcripts), add a stratum and sample it from the start rather than discovering it in a customer complaint.

If the labeling and replay work is the part your team cannot staff, an AI expert working inside your account can own the sampling frame, the guidelines, and the threshold report as a recurring deliverable. That is the kind of structured, recurring evaluation work described at Quanta Experts.