A Third Model as Tiebreaker: Cost and Latency

The human column has a hidden requirement: a human is only ground truth if humans agree with each other. Inter-annotator agreement measures how consistently different annotators assign the same label, and low agreement usually points to ambiguous guidelines rather than careless annotators (Voxel51's glossary on inter-annotator agreement). Chance-corrected metrics such as Cohen's kappa and Fleiss' kappa keep random matching from inflating the score, which is why raw percent agreement is a weak signal on its own (the same glossary, and the survey of agreement metrics for NLP annotation in Counting on Consensus).

The practical consequence: if your human reviewers only agree with each other part of the time, a third model that "agrees with humans" is agreeing with one reviewer's idiosyncrasies. Fix the instructions before you spend money on either tiebreaker.

Take a classification task with 10,000 disagreements from a nightly batch. Assume, as planning numbers, that a stronger model costs 2 cents per disagreement and a human reviewer costs 1 dollar per item. These are illustrative rates, not quotes; substitute your own contract and token prices.

The third-model cost is linear and cheap:

10,000 disagreements x $0.02 = $200

Human review of everything is not:

10,000 disagreements x $1.00 = $10,000

Now add the calibration step. Label a sample of 400 disagreements by hand, and have two reviewers overlap on 100 of them so you can compute agreement. Using the sample-size planning formula for a proportion, a 95% confidence level and a plus or minus 5 point margin needs about 385 completed responses for a large population (Sopact's sample size calculator), so 400 is a sound size for a first read. The overlap of 100 is not for precision, it is for quality control.

Labeling 400 items at 1 dollar each plus 100 duplicate labels costs 500 dollars. Run the third model on the same 400 and compare:

Two numbers decide whether the model route is safe. The first is agreement: if the third model matches the human consensus on the 400 items at or above your target, route the rest to it. The second is the human agreement ceiling. If your two reviewers only agree with each other on a portion of the overlapping 100, your "target" is not achievable, because the ground truth is soft. Model accuracy reported above the human agreement rate should be treated with suspicion rather than celebrated (Voxel51).

The latency side is just as lopsided but easier to miss. A model call is fast in the median and slow in the tail. Latency is at least three numbers: time to first token, the gap between tokens, and the 99th percentile that exposes queueing, cold starts and unusually long generations (GMI Cloud's writeup on low-latency inference). The same tension applies to throughput: packing more requests onto a GPU raises throughput and lowers cost per request while making each individual request wait longer (Mathias Lechner's analysis of inference batch economics). For a batch of 10,000 disagreements you want throughput, not interactive latency, so batch aggressively and accept minutes, not milliseconds.

Use three inputs: the expected cost of a wrong call, the volume of disagreements, and the deadline.

Step 1. Price the error. Estimate what a wrong label costs downstream. A misrouted support ticket costs a support hour. A misclassified loan application costs a compliance review. Write the number down.

Step 2. Compute the break-even volume. The model route wins when:

V x Agree x Cost_wrong < V x Cost_human

which rearranges to: route to the model when Agree x Cost_wrong is less than the human cost per item, where Agree is the third model's agreement rate on the calibration sample. With a human cost of 1 dollar, a model agreement of 0.85 and a wrong-call cost of 1 dollar, the model route breaks even. Above that wrong-call cost, humans start to win.

Step 3. Check the deadline. If the human route cannot finish before the deadline at realistic reviewer throughput, the model route is the only executable option, and you should say so explicitly rather than plan a review that will not happen.

Step 4. Break the batch by stake. Nothing forces one rule for everything. Route low-stakes categories to the model, high-stakes categories to humans, and send only the model's low-confidence items to human review.

The rule in one sentence: send disagreements to a stronger third model when the expected cost of a wrong call is low and the deadline is tight; send them to a human sample when the decision is high-stakes, ambiguous, or when you need ground truth to audit the models. Always calibrate first, and keep humans if the third model falls below target on the calibration sample.

One more consideration that the first search result usually skips. Two models that disagree do not have to be judged by a third party at all, and the format of the tiebreaker matters as much as the choice of who judges. If you ask the third model to pick a winner, you are rewarding persuasiveness. Ask it instead to name the specific field, span or clause where the two outputs diverge, and to say what evidence in the input settles it. That turns a vote into a checkable claim, and a reviewer can audit the claim in seconds.

A tiebreaker is a new step in your pipeline, and it needs the same discipline as code. Regressions do not announce themselves. A team changes a prompt to fix one complaint, the fix works, and three unrelated behaviors quietly degrade and surface later as churn with no obvious cause (Confident AI's guide to LLM regression testing). The same failure mode applies to your tiebreaker prompt.

Three practices keep it stable.

Version the tiebreaker prompt and its settings. Prompt behavior depends on the prompt text, the model, parameters, tools and retrieved context, so changing any of them can alter results even when the prompt looks identical (Braintrust's prompt testing guide). Store the model identifier, temperature, response format and prompt version together, because otherwise a score change cannot be traced to a model swap or a temperature change.

Treat the calibration sample as a regression suite. Rerun it every time the tiebreaker configuration changes. The test answers one question: does this configuration still behave correctly on the cases we already care about? That is prompt testing, which is distinct from evaluation, A/B testing and unit testing, and a release process usually needs more than one of them (Braintrust).

Pin the version. Do not edit it. Once a prompt version is created it should remain unchanged, and any modification, however small, should produce a new version rather than altering one already referenced in logs and evaluations (Braintrust's guide to prompt versioning). Immutability is what makes rollback and tracing dependable months later, and prompt management goes further by treating prompts as infrastructure with runtime updates, monitoring and access controls (LaunchDarkly's prompt versioning guide).

One more lever, because latency budgets change: prompt edits that add examples, context or reasoning instructions can improve accuracy while also increasing response time and spend, so latency and cost belong in the same test run as accuracy (Braintrust). If you are comparing several models for the tiebreaker role, Quanta Inference exposes text, image, video, audio and embedding models through one OpenAI-compatible API key, which removes the per-provider plumbing from the calibration step.

The tiebreaker is only trustworthy if you can answer, six months later, why a specific item got a specific label. Log the following for every disagreement you resolve.

Two of these fields earn their keep more than the rest. The input hash means you can rerun the exact case against a new tiebreaker version. The downstream outcome is the only field that tells you whether your estimate of the cost of a wrong call was right. Without it, your decision rule never learns.

Keep the calibration sample in a golden dataset and add production failures back to it as you find them, so coverage tracks the failures users actually hit (Confident AI). The unit of assertion should be a quality score over the dataset, not an exact output string, because a probabilistic system produces several valid answers for the same input (Confident AI). If you want the tiebreaker to read the disagreement, apply the rule and hand back a finished decision with the audit fields filled, Quanta Experts can run that loop for a line of business.