Building a Tie-Breaker Agent: Rules and Risks

Before prompting anything, classify the conflict. This is the step that saves the most time, because each class maps to a different response, and only two of them belong in a model judge.

Stylistic disagreement. Both outputs say the same thing with different phrasing, length, or format. No judge needed. Pick with a deterministic tiebreak (shorter output, schema-conforming output, lower token count) or merge. Sending this to a model judge wastes money and invites the length bias described below.

Reasoning disagreement. The candidates reach different conclusions. This is where a tie-breaker helps, provided your rubric is explicit.

High-confidence disagreement. Both candidates are confident and mutually exclusive. A judge can still pick, but this is the signature of a genuinely ambiguous case, and a confident verdict here may be a coin flip dressed up. Route a portion to humans and use the judge's confidence score as the routing signal.

Adversarial disagreement. One candidate was steered by injected content. A model judge can be steered too, so the defense is external ground truth (a verifier checking against a source of record), not another opinion (consensus and arbitration patterns). Keep adversarial-looking cases out of the judge path and send them to a rule check plus a human.

The practical reason to classify is that the classifier is cheap. A regex for output shape, a diff on structured fields, or a small embedding similarity between the two outputs will separate stylistic from substantive conflicts for a fraction of a model call. Only the substantive bucket should hit the judge, which is often the majority of your savings.

The working pattern, drawn from production arbitration designs and the LLM judge literature, has four blocks in a fixed order: rule, evidence, context, output schema.

A useful arbitration example comes from medical imaging, where a small LLM receives a table of expert predictions with confidence scores, a compact decision rubric, and a constrained output format (MELLMA multi-expert arbitration framework). The design principles transfer directly: compact evidence, explicit rule, fixed output shape.

A minimal prompt skeleton:

You are a tie-breaker. You must pick exactly one candidate or escalate.

RULE (apply in order, stop at the first that decides):
1. If one candidate violates a hard constraint in the request, pick the other.
2. If the request names a specific format, pick the candidate that conforms.
3. If one candidate cites a source given in CONTEXT and the other does not, pick the citing one.
4. If neither rule decides, return "escalate".

EVIDENCE: You may use only the REQUEST and the two candidates below. Do not use outside knowledge.

REQUEST:
{verbatim user request, no prior turns}

CANDIDATE A:
{full text}

CANDIDATE B:
{full text}

OUTPUT: JSON only, matching this schema:
{"verdict":"A"|"B"|"escalate","rule_fired":1-4|0,"reason":"one sentence","confidence":0.0-1.0}

Three details matter more than they look:

Cap the context to the request and both candidates. Including earlier conversation turns, prior drafts, or the agents' internal reasoning drags in noise and lets one candidate's framing leak into the verdict. The same error shows up in human-facing evidence arbitration: models over-rely on recent or salient sources even when they conflict with direct evidence, and they follow recency more consistently than explicit reliability cues (Evidence Arbitration in LLMs). Keeping the judge's evidence minimal and symmetric is the defense.

Require a rule identifier. Asking the judge to say which rule fired forces it to walk the decision procedure and produces a verdict you can audit. A verdict with rule_fired 0 should be treated as "no rule matched", and routed to a human.

Do not ask the judge for the answer. Ask it to choose between the candidates and say why. A judge that is allowed to write its own answer becomes a third opinion, and now you have three candidates.

The output schema also gives you free measurement: verdict, rule fired, reason, and confidence are all fields you can score against labels, and reason is the field a human reviewer reads first in the fallback queue.

The judge pattern inherits documented biases. A comprehensive benchmark catalogued twelve distinct bias types in LLM judges, with position bias, self-preference bias, and length bias among the most impactful in production (consensus and arbitration patterns). Position bias means swapping candidate order shifts verdict rates by more than ten percent in some evaluations, so you should either run each conflict twice with swapped order and treat disagreement as a tie, or randomize order per request and log the order used. Self-preference bias means the judge favors outputs from its own model family, which is a reason to run the tie-breaker on a different family than at least one of the two agents. Length bias means verbose candidates win, which is why the prompt above caps output and asks for a rule, not a discussion.

The deepest failure is correlated error. Majority voting and judge selection both amplify shared blind spots when agents come from the same training distribution; the single highest-leverage architectural fix is model heterogeneity (consensus and arbitration patterns). If both agents and the judge are the same model, you are asking one model to catch its own mistake. Assume it cannot, and design the fallback accordingly.

A practical checklist before you trust the judge:

Any check that fails is a fix, not a footnote. The labeling work is the same work that decides whether you need the judge at all.

A tie-breaker is a component in a distributed system, and it should inherit the boring properties of one: retries, timeouts, logging, and a fallback.

First, run the tie-breaker out of band. Do not block a user request on a model that may be slow. Queue the conflict, return the more conservative candidate (or hold the output) while the judge runs, and revise when the verdict lands. Job queue patterns in distributed databases exist exactly to move slow work out of the critical path while keeping status and retries visible (YugabyteDB job queue data model). If the user must wait, set a hard timeout on the judge and fall back deterministically.

Second, persist the conflict. Store request, both candidates, verdict, reason, rule fired, confidence, candidate order, judge family, and latency. This is the identical discipline as multi-agent observability: log the disagreement and its resolution so you can tell not just which agent was wrong but that they disagreed at all (Tianpan on conflict patterns). Without the log, the disagreement silently disappears and debugging takes several times longer than a single-agent failure.

Third, route low-confidence verdicts to humans. Define a threshold on the confidence field, send those to a review queue with both candidates and the judge's reason, and feed the human's choice back as a label. If your judge does not emit calibrated confidence, measure and fix its calibration before you rely on the score, because an LLM's self-reported confidence is not a probability and confidently wrong agents will report high confidence (Tianpan on conflict patterns).

Fourth, add a bucket-level safeguard: circuit breakers on token and time budgets. Splunk's multi-agent guidance warns that coordination failures include budget-draining debate loops and recommends token and time caps as circuit breakers (Splunk coordination strategies). A tie-breaker that calls another tie-breaker is exactly the loop to prevent.

Finally, keep the judge out of the critical write path when the action is destructive. If the verdict triggers a merge, a payment, or a deployment, the verdict should be a recommendation, and a rule or a human should be the gate. The judge proposes, the pipeline enforces: a code review step, a schema check, or a release gate is the real boundary, and the model is just another input to it.

Suppose a code review pipeline has two review agents, A and B, over the same pull request. You sample 400 conflicts (PRs where A and B disagree) and label each as A better, B better, or both wrong.

Results: A is better on 150, B is better on 150, both wrong on 100. Picking A at random gives you 150 out of 300 decidable cases, a 50% pick rate. You build a tie-breaker and, on a held-out 200-conflict set with a rubric based on hard constraints, you get 130 correct, 30 wrong, and 40 escalations, with 25 of the escalations decided correctly by humans. Of the 30 wrong, 20 came from judge position bias (same high-flip subset) and 10 from correlated error (both candidates wrong but the judge picked one anyway).

The useful conclusion is not the accuracy number. It is where the errors come from. Position-bias errors are fixable by swapped-order runs; correlated errors are not fixable inside the judge and must go to a human or a deterministic check. This is the difference between treating the judge as a classifier you improve and treating it as an oracle you trust.

To get the 200-conflict labeled set, budget human review time rather than more model calls. Two labels per conflict on a random 10% overlap lets you estimate inter-reviewer agreement, and anything below reasonable agreement is a rubric problem, not a judge problem. Build the rubric from the cases where humans disagree first.

If you need to run the judge across several model families to test heterogeneity, one API with several providers behind it cuts the integration work, which is what Quanta Inference (https://quantaintelligence.ai/inference) gives you. For teams that want the expert pattern packaged rather than assembled, Quanta Experts (https://quantaintelligence.ai/agents) is the product shaped like this article's advice.