From Inter-Rater Agreement to a Human Label Budget

To convert a kappa target into item counts, you need one number: the probability that two reviewers disagree on a randomly chosen item, which is 1 minus observed agreement. Kappa relates observed and chance agreement through the formula kappa = (observed minus expected) divided by (1 minus expected), so once you assume a chance-agreement level you can solve for the observed agreement your target implies the worked formula example.

A worked example makes it concrete. Suppose your labels are split roughly 50/50 and reviewers are not guessing wildly, so chance agreement is about 0.50. Then:

• Observed agreement = kappa × (1 − chance agreement) + chance agreement.
• At kappa 0.60: 0.60 × 0.50 + 0.50 = 0.80, so 20 percent disagreement.
• At kappa 0.70: 0.70 × 0.50 + 0.50 = 0.85, so 15 percent disagreement.
• At kappa 0.80: 0.80 × 0.50 + 0.50 = 0.90, so 10 percent disagreement.

If your categories are imbalanced, chance agreement rises and the same kappa requires higher observed agreement, which means more disagreement to adjudicate. An imbalanced dataset where both reviewers put 90 percent of items in one class can produce high raw agreement but a modest kappa, because chance agreement is already near 0.80 the kappa calculation example. Run this arithmetic on your own class distribution before you commit to a number.

The disagreement rate tells you the size of the adjudication workload. Ten percent disagreement over 400 double-scored items means about 40 items need a third pass. Twenty percent over the same set means 80. That difference is where budgets quietly blow up.

A small script to solve for your numbers

def items_needed(target_kappa, chance_agreement, ci_half_width, p_agree_guess):
    # Solve binary-search for observed agreement consistent with target kappa
    lo, hi = 0.0, 1.0
    for _ in range(60):
        observed = (lo + hi) / 2
        kappa = (observed - chance_agreement) / (1 - chance_agreement)
        if kappa < target_kappa:
            lo = observed
        else:
            hi = observed
    observed = (lo + hi) / 2
    disagreement = 1 - observed
    # Wilson-style n for a target confidence interval half width
    z = 1.96
    p = observed
    n = (z**2 * p * (1 - p)) / (ci_half_width**2)
    return {
        "observed_agreement": round(observed, 3),
        "disagreement_rate": round(disagreement, 3),
        "double_scored_items": int(n) + 1,
        "adjudications": int(n * disagreement) + 1,
    }

print(items_needed(target_kappa=0.7, chance_agreement=0.5,
                   ci_half_width=0.05, p_agree_guess=None))

This is a planning approximation, not a substitute for a kappa confidence interval, which the NLP annotation guidance recommends reporting alongside the point estimate because sample size and class prevalence both affect how stable a kappa is the inter-annotator agreement review. Reaching that point estimate level of precision is the goal; the script just tells you roughly how many double-scored items get you there.

Estimate the sample size for a stable kappa

Kappa is a noisy statistic. On 100 items, a kappa of 0.70 might really be anywhere from 0.55 to 0.80. Most teams underestimate this. A practical rule for planning:

• 100 to 150 double-scored items gives a rough kappa you can act on for low-stakes triage.
• 300 to 400 double-scored items gives a kappa stable enough to defend in a review.
• Below 80 double-scored items, treat the kappa as an anecdote.

The script above uses a confidence-interval half width as the driver. Tighten it to 0.03 and the item count roughly triples. Start with the widest interval your decision tolerates.

Now you have a count of double-scored items. Multiply by realistic minutes per item. The trap is estimating minutes for a clean, easy item when most of the cost lives in the messy ones.

Task granularity is the single biggest lever. A holistic rubric asks a reviewer for one overall judgment, while an analytic rubric scores several criteria separately, and the analytic version is explicitly noted as more time-consuming to design and apply the rubric best practices guide. Dropping from four criteria to one can halve scoring time. Adding a third reviewer for disagreements adds a full pass on the disputed fraction.

Build the estimate from operations, not gut feel:

Add up per-item time, then add the adjudication cost weighted by your disagreement rate. At 10 percent disagreement and 4 minutes total per item, expected minutes per item are about 4.0 plus 0.10 × 4, or 4.4. At 20 percent disagreement that becomes 4.8, roughly a 9 percent increase in cost for a shift in reliability that may be worth far more than 9 percent of budget.

Include setup work even though it is not per item. Building the rubric takes time. Generating rubrics with an assistant can compress that starting point, but a generated rubric still needs human review and adjustment before it is trustworthy the TCEA rubric guide. Budget a half day for the first rubric draft and one calibration round before the first paid batch.

Convert minutes to dollars with a loaded hourly rate: base pay plus payroll taxes, benefits, tooling, and the share of a supervisor's time. Contract rubrics around 1.4 to 1.6 times the base rate for full-time staff, and use the agency or contractor invoice rate directly if you outsource.

Worked example, 400 double-scored items:

The item scoring line is under a quarter of that label cost. Setup, calibration and audit together are most of the table. That is why cutting the sample from 400 to 200 items saves far less than teams expect: it halves the scoring hours (roughly $400 saved) but leaves the fixed design and calibration cost intact. There is a floor on label cost that sample trimming cannot touch.

Two more hidden costs deserve a line. First, rework: if kappa comes back below target, you rebuild the rubric and rescore, which can double the bill. Budget for one rework cycle. Second, reviewer turnover. Training a new reviewer on a mature rubric typically takes a few hours of shadow scoring against a reference set. A small, stable panel beats a large rotating one on both cost and agreement the adjudication discussion.

Price-setting checklist

• Loaded hourly rate confirmed with finance, not estimated.
• Minutes per item timed on a real sample of 20, not guessed.
• Adjudication rate from a pilot kappa, not a wish.
• One rework cycle built into the budget.
• Audit hours reserved for a 10 percent spot check.
• Calibration time on the calendar before the paid batch.

A reliable label set is only worth what it prevents. Build a simple expected-loss estimate for the judge you intend to calibrate.

Start with the volume and the error rate. Suppose the judge will process 20,000 items per month and you expect it to be wrong on 8 percent of them before calibration, with a 2 percent error rate after calibration. If a wrong call costs $1.50 in rework or support time, the pre-calibration loss is 20,000 × 0.08 × $1.50, or $2,400 per month. The post-calibration loss is 20,000 × 0.02 × $1.50, or $600. The calibration saves $1,800 per month.

Compare that with the $1,514 label bill from the previous section. The payoff point arrives in under one month. This is the calculation that converts a label budget from overhead into an investment, and it is the number to put in front of whoever signs off.

The math changes with a lower error cost or lower volume. At 2,000 items per month and $0.50 per error, the same 6-point error reduction saves only 2,000 × 0.06 × $0.50, or $60 per month. Against a $1,500 label bill, payback takes twenty-five months. In that case, buy a smaller label set with a wider confidence interval, or skip formal kappa and use a single-reviewer spot check.

Judge error is not uniform, either. Calibrating against human labels improves the detection of mislabeled and borderline examples, so the payoff is larger when your item mix contains many ambiguous cases the calibration study. If your data is clean and easy, calibration buys less.

Worked payoff comparison

Set the budget where the marginal cost of another double-scored item equals the expected loss from a wrong judge decision at that sample size. In the high-volume row, that point sits well beyond 400 items. In the low-volume row, it sits below 100.

Once you have a calibrated judge, keep the human labels as an asset. Several paths follow.

Report kappa with a confidence interval and the rater design, not just the point estimate, so that whoever consumes the number knows how stable it is the annotation guidance. Analyze the disagreement patterns rather than only the score; a cluster of disagreements on one subcategory usually points to a rubric gap you can fix cheaply. Keep the disputed and adjudicated items as a standing test set for future rubric changes.

If you plan to run several judges, do not throw out weaker ones by accuracy alone. When you have labeled calibration data, a full panel of judges can produce better calibrated probabilities than keeping only the most accurate ones, because weak judges still carry learnable signal the calibration paper. This means the same label set can support a more sophisticated panel at no extra labeling cost.

If your team would rather hand the modeling and calibration work to a specialist, Quanta Inference puts text, image and embedding models behind one OpenAI-compatible API, which lets you run the judge and the calibration comparisons from a single key (https://quantaintelligence.ai/inference). It is a reasonable next step once your labels are ready and you want the judge in production.

The whole process fits in an afternoon once you have a pilot. Here is the order.

1. Pick the target kappa from the decision table, based on what the judge controls.
2. Estimate chance agreement from a 50-item pilot labeling of your actual class mix.
3. Solve for the observed agreement and disagreement rate your target implies.
4. Choose the confidence-interval half width your decision needs, and size the double-scored set.
5. Time 20 real items across two reviewers, including reasons, and extrapolate minutes.
6. Load the hourly rate and add adjudication, calibration, design, audit and one rework cycle.
7. Estimate the monthly error cost before and after calibration, and compute payback.
8. Stop adding double-scored items where marginal cost meets marginal error saving.

Run the pilot first. Most of the uncertainty in this plan lives in two numbers, minutes per item and disagreement rate, and a single pilot day resolves both. After that the budget is arithmetic.

The habit that separates teams that get value from labeling from teams that burn money is deciding the budget from the decision, not from a textbook kappa. Write the target kappa, the item count, the minutes, the dollars and the payoff point on one page and share it with the person who owns the judge. If the page cannot justify itself, the label project should shrink rather than proceed.