When Two Agents Disagree: Resolving Conflicting Outputs

The short answer is a fixed order of checks. Do not ask the agents to debate the point again: that burns tokens, and LLM judges carry their own biases. Instead, walk down this tree:

1. Is there an objective test? Code that runs, a query that returns rows, a schema that validates, a math answer that checks, a build that compiles. If yes, run the test and take the passing answer. If exactly one answer passes, you are done.
2. If several pass, is one cheaper or safer? Same output but fewer lines, no new dependency, smaller blast radius: pick the simpler one and log the comparison.
3. If there is no test, compare evidence. Who attached tool output, a cited source, a reproduction step, or a file path they actually read? The answer with reproducible evidence wins.
4. If evidence is equal, check calibration. Look up each agent's historical accuracy on this task type rather than trusting its self-reported confidence.
5. If you still cannot separate them, or the action is irreversible, escalate to a person.

The order matters because each step is cheaper and more reliable than the one below it. A test that runs costs seconds and is authoritative. Evidence comparison costs a human or judge a couple of minutes. Escalation costs hours. Never start at step 4 or 5 when step 1 is available.

Microsoft's cloud adoption guidance makes a related point for architecture: a single-agent design is more predictable and has lower operational overhead, and you should only move to multiple agents when something specific forces separation, such as compliance boundaries or distinct teams Microsoft's single versus multi-agent guidance. The same logic applies to resolution. The cheapest disagreement is the one you did not need to resolve because only one agent was ever running.

Worked example. Two agents are asked to fix a failing invoice total calculation. Agent A edits calculate_total() and adds a regression test. Agent B edits the caller and adjusts a rounding constant. You run the test suite. Agent A's test passes and the original failing test goes green; Agent B's change passes lint but the original test still fails. The tree stops at step 1: Agent A wins, Agent B's branch is closed with the failing test output attached as the reason.

Not every task has ground truth, but more do than people think. Coding is the obvious case: worktree-isolated agents each run their own tests, and the branch that keeps the suite green is the candidate for merge Augment Code's multi-agent workspace guide. Data extraction has a schema. API answers have status codes and response shapes. Classification has a labeled sample you can score.

A cheap technique is to force objectivity into the prompt. Ask each agent to return its answer plus the exact command it ran and that command's output. Two agents that both claim the parser is correct are unhelpful; one that shows pytest tests/test_parser.py passing on its branch and one that does not is decidable.

If you run agents in sandboxes, keep the test command identical across sandboxes. Windmill, for example, runs each agent execution in its own NSJAIL sandbox with filesystem and network limits, so results are only comparable if both sandboxes mount the same repository state and the same test command Windmill's sandbox write-up.

Once you drop below objective tests, you are scoring evidence quality. Four signals, in descending order of trust:

• Tool output that can be re-run. An agent that says "the endpoint returns 500" and attaches the curl output is stronger than one that says "the endpoint looks fine." Re-run the tool call yourself if it is cheap.
• A source you can open. A citation that resolves beats a confident sentence with no link every time. Check that the link actually says what the agent claims.
• Reproduction steps. Concrete steps another person or agent can follow and land in the same place are stronger than an argument from plausibility.
• Breadth of files read. An agent that touched and quoted the relevant files is more likely to be right than one that answered from general knowledge.

The failure mode to guard against is the confidently wrong answer with a polished format. Silent failures, where an agent returns a well-formed answer that is simply wrong, are the most operationally dangerous kind in multi-agent systems tianpan's conflict resolution analysis. Format quality is not evidence.

Do not settle evidence disputes with an LLM judge unless you know its biases. A benchmark of judge models catalogued twelve distinct bias types, including position bias that shifts accuracy by more than 10% on code evaluation when you swap the order of two responses, and self-preference toward outputs that resemble the judge's own model tianpan's consensus and arbitration analysis. If you use a judge, randomize order, use a model family different from both agents, and keep the judge out of decisions where an objective test exists.

Confidence scores feel like a quick tiebreaker. They are also poorly calibrated by default. An agent that is confidently wrong reports high confidence, and a 2025 study cited in the arbitration literature found that much of the gain from confidence-weighted selection disappears without empirical calibration data tianpan's consensus and arbitration analysis.

Majority voting has a similar trap. It helps on reasoning tasks with multiple valid paths, but it fails on correlated errors: agents trained on similar data share blind spots, so five voters can agree on the same wrong answer and manufacture false consensus tianpan's consensus and arbitration analysis. The highest-leverage fix is model heterogeneity: pair agents from different model families so their errors are less correlated.

A practical decision rule: use confidence as the fourth tiebreaker, never the first. Keep a small table of past decisions for each agent and task type, and replace self-reported confidence with measured accuracy once you have a few dozen examples. If you need one API key across several model families to get that heterogeneity, Quanta Inference provides text, image, video, audio and embedding models behind an OpenAI-compatible endpoint (https://quantaintelligence.ai/inference).

A support-team migration gives you a realistic arithmetic case. Two agents respond to "rewrite this query to run under 200 ms on the reporting replica." Agent A returns a rewrite with an index recommendation. Agent B returns a rewrite with a materialized view.

1. Test: run both on a copy of the replica and measure. A is 640 ms, B is 140 ms. B passes; A fails. Time spent: about ten minutes of query time.
2. Suppose both hit 150 ms. Then compare cost: A adds an index (fast writes, no new object), B adds a view that needs refreshing (a scheduled job). The cheaper-to-operate answer wins and the comparison is logged.
3. Suppose there is no replica and no measurement. Then compare evidence: A attached EXPLAIN ANALYZE output; B attached a general article about materialized views. A wins on evidence, not on eloquence.
4. Suppose both attached EXPLAIN ANALYZE and both claim correctness with no way to measure. Now escalate. A database owner should decide, because the action changes production.

Step 4 is the point people skip. Escalation is not a failure of the system; it is the designed outcome for genuinely ambiguous, irreversible decisions.

Resolution only pays off if the disagreement is recorded. Three habits make that cheap:

• Log the losing branch, not just the winner. Store the rejected answer, its evidence, and the reason it lost. That log is what lets you spot a systematically overconfident agent before it costs you a week.
• Standardize branch naming so the agent is identifiable. Conventional Branch defines prefixes for AI sources such as claude/, codex/, copilot/, and cursor/ alongside purpose prefixes like feature/ and fix/ Conventional Branch specification. When two branches compete, you can tell at a glance which agent produced which.
• Cap the debate. If you do allow agents to exchange arguments, set a maximum number of turns before a supervisor or a person is brought in, and log the full thread for audit MTM's disagreement protocol write-up.

A coordination layer that owns a shared queue also helps. Temporal's worker-specific task queue pattern shows how to route follow-up activities to the same worker that holds local state, with a non-retryable schedule-to-start timeout that fails rather than retrying into a queue that may never be served Temporal's worker-specific task queue pattern. The same idea works for resolution: route the adjudication to a fixed, known decider and time out cleanly instead of looping.

For teams that want the coordination and the adjudication to happen without hand-built scripts, Quanta Experts are a natural next step (https://quantaintelligence.ai/agents). Each expert reads your context, does the work and hands back finished deliverables, which gives you a consistent decider to route conflicts to.

Escalation is a decision, so give it rules rather than a mood. Escalate when any of these is true:

• Both answers pass every test and the cost difference is small. A coin flip between two working answers is wasted compute.
• The action is irreversible or has external effects. Sending an email, charging a card, deleting rows, deploying to production.
• The evidence conflicts at the source. Two cited documents disagree, and neither agent checked further.
• The domain has a compliance boundary. Microsoft's guidance starts multi-agent architectures from separate agents precisely when regulations or policies require data isolation or review Microsoft's single versus multi-agent guidance.
• The same agent has lost the same dispute three times on the same task type. That is a prompt or context bug, not a one-off.

Build the escalation into the run, not into a person's memory. The escalation should arrive with the two answers, the test results, the evidence, and the specific question you need answered.