When Two Models Disagree on the Same Input: A Tie-Breaking Rule

A queue needs five fields on every row: input ID, model A output, model B output, the comparison verdict, and the input type. The verdict is computed automatically and should be coarse, such as match, mismatch, or unparseable. Getting reliable outputs that you can compare is a precondition. The well-documented failure mode is a model that returns conversational text around its JSON and crashes the parser downstream (Sharad Bharadwaj). Constrained decoding, which masks invalid tokens during generation against a schema, is one fix, and the JSONSchemaBench work evaluates how well the main frameworks cover real schemas (JSONSchemaBench). Either way, unparseable belongs in the queue as its own verdict, because it is a disagreement about format, not content.

The important design decision is what counts as a mismatch. Exact string equality will flood the queue with trivial differences, like a trailing period or a reordered list. Normalize first: lowercase, strip whitespace, parse structured fields into a canonical form, then compare field by field. Keep a severity column so trivial cosmetic mismatches do not consume human time.

A minimal queue writer looks like this:

def queue_row(input_id, in_type, out_a, out_b):
    va, vb = normalize(out_a), normalize(out_b)
    if va is None or vb is None:
        verdict = "unparseable"
    elif va == vb:
        verdict = "match"
    else:
        verdict = "mismatch"
    return {"input_id": input_id, "input_type": in_type,
            "out_a": out_a, "out_b": out_b, "verdict": verdict}

Everything that is not match goes to review. Everything that matches does not, and that is the whole economy of the method: you spend human time only where the machines already disagree with each other.

You cannot label every disagreement forever, so you label a random subset of the queue. The subset needs to be big enough that the error rate you measure has a usable confidence interval. For a proportion, the standard planning formula is n0 = Z squared times p times (1 minus p) divided by e squared, with a finite population correction n = n0 / (1 + (n0 - 1) / N) when you are sampling a known pool without replacement (Sopact sample size calculator). The same planning arithmetic for finite populations is set out in the PracTools package documentation (PracTools).

At 95 percent confidence and p = 0.5 (the conservative worst case), the large population figure is 384.16, rounded to 385. With a known pool of 1,000 disputed cases, the correction brings it to 278. Here is how the target moves with the size of the queue, using the calculator's own table values:

Two facts matter for planning. First, halving the margin of error roughly quadruples the sample, so ±5 points costs about four times as many labels as ±10 points. Second, once the pool passes a few thousand, the finite correction stops helping and you are capped near 385 for a ±5 point estimate. If your review queue is bigger than that, you do not need more labels, you need a random sample of the queue.

Two honest limits. The formula assumes simple random sampling, and a queue grouped by input type is not one, so stratify and size each type separately. And a precise estimate of a biased sample is still biased (Sopact), so draw the sample by random ID, not by whichever cases look juiciest to the reviewer.

Each label answers one question per disputed input: A correct, B correct, both wrong, or genuinely ambiguous. After the sample is labeled, you compute two numbers per input type. The first is the share of disputed cases where A is right. The second is the share where neither model is right, which is the escalation rate.

Worked example. Suppose a classification pipeline runs two models over 40,000 monthly inputs. They disagree on 1,200 of them, a 3 percent disagreement rate. You sample 278 of those 1,200 at 95 percent confidence and a ±5 point margin. Reviewers label them and you find 154 where A is right, 78 where B is right, 39 where neither is right, and 7 too ambiguous to call.

That gives roughly 55 percent A, 28 percent B, and 14 percent neither, with the remaining few ambiguous. The ±5 point interval means the true A share for that input type plausibly sits anywhere from about 50 to 60 percent. If your decision rule is "route disputed cases to A when A wins more than half," that interval straddles the threshold, and you do not yet have enough labels to set the rule with confidence. Increase the sample or narrow the margin.

If, by contrast, the labels showed A at 72 percent, the interval would clear 50 percent easily and the routing rule would be safe. This is the insight that a bare accuracy number hides: the confidence interval, not the point estimate, decides whether you can act.

Before you trust any of it, check agreement between your reviewers themselves. If two qualified people only agree 80 percent of the time on the same item, a model that "beats" that rate is often fitting one reviewer's quirks, and reported accuracy above human agreement should be treated with suspicion (FiftyOne). Research on annotation practice recommends reporting confidence intervals and analyzing disagreement patterns rather than collapsing them into a single number (Counting on Consensus).

The labels branch into three outcomes per input type.

A stable winner. If A wins on most disputed cases and the interval clears your threshold, write the rule: disputed inputs of this type go to A. That is a routing change, not a model change, and it costs nothing per call beyond what you already pay.

A shared failure. If "neither right" is high, the disagreement is a specification problem, not a competition. Fix the prompt or the schema for that input type, then re-run the queue. Prompt changes should be versioned and tested against a fixed set of cases, because a small edit can quietly break behavior that already worked (Braintrust on prompt versioning). Regression testing against a golden dataset with explicit quality thresholds is how you keep a fix from becoming a new failure (Confident AI). The test set for this is easy: your labeled sample is now a golden dataset.

A genuine tie. If the interval straddles the decision threshold, escalate to a third model on just that slice and label a fresh sample of the three way disputes. Escalation is the most expensive option, so reserve it for the types where neither model clears 50 percent with confidence.

The decision rule in one line: route on the winner when the lower bound of the interval clears your threshold, fix the prompt when the neither-wrong rate is high, and escalate only when the interval crosses the threshold. Every rule is scoped to an input type, never to the whole system.

Use this before you trust the first sample.

The last row is the one teams skip. The labels are not a one-off report; they are the fixture that future changes are measured against.

When the pipeline grows past two models or the queue spans several modalities, running everything through one API key keeps the comparison cheap and the logging consistent. Quanta Inference (https://quantaintelligence.ai/inference) routes text, image, video, audio and embedding models through an OpenAI-compatible interface, which makes a two-model or three-model queue a configuration change rather than an integration project.