Calibrating a Verification Score Against Reviewer Labels

Manual relevance labeling is the standard ground-truth route for retrieval quality, and it is expensive, slow and subjective (Evidently RAG evaluation guide). Plan for that cost rather than discovering it halfway through.

A workable rubric has four parts:

1. The target event, restated in one sentence.
2. Three to five positive examples and three to five negative examples, drawn from your own data, including at least one borderline case.
3. A rule for partial support, citations and hedged language.
4. A rule for out-of-scope items (wrong language, truncated input, missing passage).

Label in two passes. First, have two reviewers label the same 50 to 100 items independently. Measure agreement with a chance-corrected statistic rather than raw percent agreement, because raw agreement overstates reliability when one label dominates; measure choice should match the annotation design, and agreement should be reported with confidence intervals (IAA metric selection paper). If agreement is low, the fix is almost always the rubric, not the reviewers. Rewrite the ambiguous rule, relabel, and only then scale up.

Second, route disagreements to adjudication. A single adjudicator who sees both labels and the rubric resolves them in a short session, and the resolution is recorded as the final label. Do not average or majority-vote binary labels with two reviewers; you will get ties and hidden ambiguity.

Keep reviewers blind to the score during labeling. A reviewer who can see "0.91" anchors on it, and the resulting labels will correlate with the score for reasons that have nothing to do with correctness. Export only the item and its context.

Sample deliberately, not uniformly. If positives are rare, a uniform sample gives you mostly negatives and too few positives per bin. Two acceptable designs: a stratified sample that over-samples from the score's upper range and reweights later, or a uniform sample large enough that every planned bin holds at least 30 positives. Document which one you used, because the reported precision is only valid under that design.

The short answer's threshold, at least 30 positives per bin, comes from the need to estimate a proportion without the interval swallowing the signal. With 30 positives, a precision near 0.9 has a 95 percent Wilson interval of roughly 0.74 to 0.97; with 10 positives it is roughly 0.60 to 0.98, which cannot separate a good bin from a mediocre one.

Quantile bins (equal counts) and fixed-width bins (equal score ranges) answer different questions. Use the table below to choose.

The pragmatic default is equal-count bins, merged from the top and bottom until each holds at least 30 positives. If your labeled set has 400 items and 80 positives, that is at most two or three bins. That is fine. Two well-populated bins tell you more than ten noisy ones.

Plot the reliability diagram alongside the table. A calibration curve bins predictions and plots the fraction of positives per bin against the mean predicted score in that bin, which is the standard way to see whether a score of 0.8 really means 80 percent (scikit-learn probability calibration). Overconfidence looks like a curve below the diagonal; underconfidence looks like a curve above it.

For a score threshold t, every labeled item with score at or above t is predicted positive. Compute the four counts and derive the rates. Precision is the proportion of predicted positives that are truly positive; recall is the proportion of true positives that clear the threshold. These are the same definitions used for retrieval at a cut point, where precision at k measures the relevant share of the top k results and recall at k measures the share of all relevant items found in the top k (Anyscale RAG evaluation docs, retrieval metrics explainer).

from math import sqrt

def wilson(k, n, z=1.96):
    if n == 0:
        return (float("nan"), float("nan"))
    p = k / n
    denom = 1 + z**2 / n
    centre = (p + z**2 / (2 * n)) / denom
    margin = z * sqrt(p * (1 - p) / n + z**2 / (4 * n**2)) / denom
    return (max(0.0, centre - margin), min(1.0, centre + margin))

def bin_report(rows, threshold):
    # rows: list of (score, label) where label is 1 for the target event
    tp = sum(1 for s, y in rows if s >= threshold and y == 1)
    fp = sum(1 for s, y in rows if s >= threshold and y == 0)
    fn = sum(1 for s, y in rows if s < threshold and y == 1)
    tn = sum(1 for s, y in rows if s < threshold and y == 0)
    precision = tp / (tp + fp) if tp + fp else float("nan")
    recall = tp / (tp + fn) if tp + fn else float("nan")
    fpr = fp / (fp + tn) if fp + tn else float("nan")
    return {
        "threshold": threshold,
        "tp": tp, "fp": fp, "fn": fn, "tn": tn,
        "precision": round(precision, 3),
        "precision_ci": tuple(round(v, 3) for v in wilson(tp, tp + fp)),
        "recall": round(recall, 3),
        "recall_ci": tuple(round(v, 3) for v in wilson(tp, tp + fn)),
        "fpr": round(fpr, 3),
    }

labeled = [(0.95, 1), (0.88, 1), (0.81, 0), (0.72, 1), (0.55, 0)]
for t in (0.5, 0.7, 0.9):
    print(bin_report(labeled, t))

The Wilson interval is the right default for proportions near 0 or 1, which precision often is. Report the interval next to every point estimate. A bin showing precision 0.92 with an interval of 0.81 to 0.97 supports a decision; 0.92 with 0.55 to 0.99 does not.

Treat rates computed over thresholds, not disjoint bins. Cumulative rates from the top of the score range down are what a cut point actually uses. Per-bin rates are useful for spotting non-monotonic behavior, such as a middle band where precision dips, which often signals a subgroup the score mishandles.

Suppose you have 500 labeled items with 120 positives. Equal-count bins of 50 items each, merged from the top until each holds at least 30 positives, give four bins. The table below is illustrative, not measured.

Read it top down. Precision falls from 0.83 to 0.33 to 0.14. Recall accumulates as you lower the threshold: 0.33 at 0.90, 0.70 at 0.75, 0.88 at 0.50. The false positive rate climbs from 0.02 to 0.20 to 0.51.

The decision rule falls out of the product requirement. If a wrong verification is expensive (for example, it auto-approves content), you want precision high and the rate of false positives low, so the cut is at 0.90. If missing a true positive is expensive (for example, it drops a legitimate claim from a review queue), you accept the 0.75 cut, where recall nearly doubles and the confidence intervals on precision no longer overlap the 0.90 bin's. That non-overlap is the evidence that the drop is real rather than noise.

A worked number keeps this honest. To raise the 0.75 bin's precision from 0.33 to 0.83 by moving the threshold up, you gain 12 precision points and lose 37 percent of recall in absolute terms (0.70 down to 0.33), while the false positive count falls from 88 to 8. Whether that trade is acceptable is a product question, not a statistics question.

One caution: a score can be well calibrated and still discriminate poorly. A 70 percent prediction that is correct 70 percent of the time is calibrated, but if positives and negatives score similarly, no threshold separates them. Calibration and discrimination are distinct facets, and a model can be excellent at one and weak at the other (Cross Validated discussion). Check both: the reliability curve for calibration, the bin table for discrimination.

Split your labeled set before you look at bins. Use roughly 70 percent for the bin table and 30 percent held out. Choose the candidate cut point on the first slice only, then compute precision, recall and false positive rate on the held-out slice. If the held-out precision lands inside the bin's confidence interval, the cut survives. If it lands far outside, the bin was too small or the sample was not representative, and you go back for more labels.

Run a sensitivity check too. Sweep thresholds in small steps around the candidate and record how much precision and recall change per step. A cut point sitting on a cliff is fragile; reviewers will notice the behavior change week to week.

After launch, keep the loop running. Log a random sample of production items with scores, send them to reviewers, and recompute the bin table on a regular cadence. Selective prediction systems make an explicit trade between coverage (the fraction of items the system handles) and risk (the error rate on those handled), and that trade shifts as inputs drift (selective prediction overview). Reviewers who disagree often reveal a rubric gap, so treat adjudication as a feedback channel into the rubric, not just a tiebreaker.

Two operational checks worth writing into the runbook. First, if any bin drops below 30 positives after a refresh, merge it. Second, if agreement on the held-out slice falls below the level you measured at the start, pause and re-review the rubric before trusting the new numbers.

If the scoring step itself is the bottleneck, running several models behind one interface cuts the plumbing: Quanta Inference (https://quantaintelligence.ai/inference) serves text, embedding and other models through one OpenAI-compatible API key, which keeps the collection and labeling pipeline stable while the scorer changes underneath. For teams that need the whole loop staffed, including reviewers and adjudicators, an engagement through Quanta Experts (https://quantaintelligence.ai/agents) can take the pipeline from rubric to bin table as a finished deliverable.

A final note on scope. Verification scores in question-answering systems are often computed over extracted spans, and span-level answers bring their own labeling quirks: answers can be multi-span, and exact-match definitions differ across studies (MultiSpanQA paper, experimental design study). If your target event is "the right span was extracted," decide how to score partially correct spans before labeling. That single decision will change your precision numbers more than any threshold you pick.