How to Test a Dispute Agent on Contradictory Evidence

You cannot test conflict handling on a packet that hides its contradictions. Most payment data arrives normalized and deduplicated, which flattens exactly the signal you need.

The fix is to keep each document as a separate, source-tagged block, and require the agent to answer per field with a list of candidate values rather than a single value. Something like this:

{
  "field": "delivery_address",
  "candidates": [
    {"value": "14 Oak Street", "source": "carrier_tracking_record", "retrieved_at": "..."},
    {"value": "41 Oak Street", "source": "order_fulfilment_record", "retrieved_at": "..."}
  ],
  "resolution": "followed carrier_tracking_record",
  "resolution_rule": "carrier delivery evidence outranks internal fulfilment data",
  "conflict_named": true
}

This shape does three things. It preserves both values, so the reviewer can see what the agent saw. It forces a conflict_named flag, which becomes your test assertion. And it carries the source identifier, so the precedence rule can be checked mechanically.

The underlying claim about evidence quality supports this. Delivery confirmation matters because issuers decide on the full package, and a basic delivered scan may not establish where a package was left or who received it, while photos, address information and detailed tracking can per this guide to delivery confirmation. If two delivery records disagree about where the parcel went, the disagreement is materially relevant to a merchandise-not-received claim, not noise to be smoothed away.

The precedence order

Write the order into the system prompt as an explicit, numbered list. A defensible default for card disputes:

1. Network rule and reason code requirements. The code determines what evidence is needed and which party bears liability per the reason code reference. If the code requires signature proof and only one source has it, the code's requirement governs which source is even admissible.
2. Authenticated third-party records. Carrier tracking, signed proof of delivery, 3-D Secure authentication results, and bank-side authorisation data. These originate outside the merchant's own systems.
3. Merchant system of record. Order, fulfilment, billing and subscription records.
4. Customer communications. Support tickets, emails, chat transcripts. Useful for demonstrating resolution attempts, weaker as proof of fact.
5. Summaries and derived documents. Anything a human or model wrote about the primary evidence. Never outranks the primary evidence.

Two refinements earn their place.

Time. If a source has been superseded, the later version of the same source type wins over the earlier one. Two order records from the same system are not a conflict; one is stale.

Explicit instability. If the conflict crosses levels that the order cannot separate, for example two authenticated third-party records that disagree, the order has no answer. That is the handover trigger, covered below.

Tell the agent what to do when the order resolves the conflict, and what to do when it does not. A prompt that says "prefer the stronger source" without defining stronger is a prompt that will invent a hierarchy per case. The dispute-resolution literature is blunt on this point: AI tools can produce inaccurate, misleading or fabricated content, and users bear responsibility for oversight and independent verification per this analysis of prompting in dispute resolution practice.

If you are running this across several evidence types and want one interface rather than a separate integration per vendor, Quanta Inference exposes text, image and document models behind a single OpenAI-compatible endpoint at https://quantaintelligence.ai/inference.

The central test is simple to state and hard to pass: the agent must name the conflict. If it produces a confident single value without flagging that a second source disagreed, the test fails, even if the chosen value turns out to be the one a human would have picked.

That rule sounds harsh until you consider what the alternative measures. A test that only checks the final answer rewards lucky guessing and punishes nothing. A test that checks the disclosure rewards the behavior you actually need in an audit trail.

Build the test set around injected conflicts. The legal domain has a structured method for this: generate documents, inject six structured contradiction types, and model both self- and pairwise inconsistencies, with human validation to confirm plausibility per this framework for contradiction detection. You do not need the full academic apparatus, but the discipline transfers. Each test case should carry a known conflict, a known intended resolution, and a known correct handover decision.

A practical conflict test matrix:

Run each case against a keyword or structural assertion first, then a graded assertion. Assertions in a test harness compare output against expected values or conditions, and include equality checks, structural checks, similarity checks and custom functions per the assertion documentation. Use the structural check for conflict_named being true and for both source identifiers appearing in the output. Use a graded rubric for whether the stated reasoning is coherent, not whether it matches a preferred sentence.

Worked scoring example

Take 200 synthetic cases with 60 seeded conflicts. Suppose the agent names the conflict in 51 of the 60, and in the other 9 outputs a single clean value. Suppose further that of the 51 named, 44 pick the value the precedence order dictates and 7 pick the other one, and that 2 of the 140 conflict-free cases get falsely flagged as conflicting.

Read the numbers in three parts rather than as one score.

• Detection rate: 51 of 60, so 85% of seeded conflicts surfaced. The 9 misses are the dangerous set, because they look like confident correct answers.
• Resolution accuracy given detection: 44 of 51, so about 86%. The 7 wrong picks usually trace to an ambiguous rule in the prompt, not to model capability.
• False flag rate: 2 of 140, about 1.4%. Low enough that human reviewers will not start ignoring the flag.

Two lessons fall out. First, fix the prompt before swapping models: resolution accuracy given detection is a prompt problem, and detection rate is the only part that is mostly a model problem. Second, weight detection far above resolution in your release gate, because a missed conflict is invisible downstream while a wrongly resolved one is at least on the record.

Second, the reverse also happens: models merge conflicting evidence rather than flagging it. The contradiction-detection work notes that when contradictions in input evidence go unresolved, generation models often merge them into a single output, producing unsound results per the same framework. That is precisely the behavior your test must catch, and it is why the assertion has to be about disclosure rather than about the final value.

The handover rule is the third leg. State it in the prompt as a condition with a required action, not as guidance.

The rule: if two or more sources at the same precedence level conflict and no temporal rule separates them, the agent must stop, produce no merit decision, and return a structured handover containing the field, both candidate values with source identifiers, both timestamps, the precedence level, and the reason no rule applied.

Then make the handover part of the passing criteria. A test case with an unresolvable conflict passes only when the agent hands over. An agent that picks a side "because the tracking data usually wins" fails that case even if the pick looks reasonable.

Three implementation notes.

Make handover cheap for the reviewer. A handover that dumps the whole packet back to the queue gets ignored. Send only the conflicting field, both values, both sources, and the agent's reason for stopping. Reviewers can resolve a two-value conflict in seconds; they will not reconstruct it from a 40-page packet.

Log handovers separately from errors. A handover is a correct outcome. If your dashboard counts it alongside failures, the team will tune the agent to stop handing over, which is the opposite of what you want.

Set a review window that fits the network deadline. Since merchant response windows are measured in weeks and processors often want submissions before the network cutoff per the evidence packet checklist, a handover sitting in a queue is a deadline risk. Route handovers to a named reviewer with the deadline attached, not to a shared inbox.

The same pattern holds in regulated settings where multi-agent dispute handling has reached production: escalation calibration is named as one of the challenges that determines whether a build survives contact with operations per this multi-agent dispute architecture guide. Calibration means the rate of handovers is tuned to a level reviewers can absorb, and the threshold is tested rather than left to the model's mood.

It is easy to assume that once conflicts are handled, gaps are covered. They are not, and the tests prove it.

A field gap test asserts three things: the output marks the field as absent, the agent does not substitute a plausible value, and the downstream decision accounts for the absence. The third is where most implementations quietly fail. An agent that correctly writes "tracking_number": null and then proceeds to argue that delivery is proven anyway has passed the first assertion and failed the purpose.

Gap handling connects to reason codes more tightly than it first appears. The code determines the evidence the issuer must assemble to defend the chargeback per the reason code guide, and merchants who engage in representment win roughly 30% to 45% of contested cases, with those numbers dropping when evidence is incomplete, mismatched or late per the same checklist. Incomplete evidence is a gap, and the agent should be able to recognize that a required evidence type is missing for the code in question, not just that a JSON field is null.

Build the gap test set accordingly: for each supported reason code, one case where the decisive evidence type is absent, one where it is present but conflicting, and one where it is present and clean. That gives you a three-state matrix per code, and the contrast between the three states is what teaches you whether the agent is distinguishing them at all.

There is a useful measurement approach here as well. Scoring structured outputs per field, rather than per document, points reviewers directly at the specific fields that warrant attention, and in document-processing settings the lowest-trust documents are the ones likely to contain an error while per-field scores reveal where it sits per this benchmark of structured output error detection. Map that idea onto your own outputs: score the conflicting and missing fields specifically, not the whole packet, so the review queue sorts by field risk.

Three prompt-level habits move the pass rate more than model choice.

Separate thinking from answering. Ask the agent to first list candidate values per field with sources, then apply the precedence rules, then produce the answer. A prompt that splits the output into a reasoning section and a response section keeps the model grounded, because if it works out in the reasoning section that it cannot answer, the response stays factual per this account of hallucination-resistant prompting. In dispute work the reasoning section is also your audit record.

Give the precedence rules as data, not prose. A numbered list with explicit source-type names, matching the source tags in your packet, is checkable. Prose like "use the most reliable source" is not, because the model decides what reliable means per case, and your test can then only grade the outcome, never the rule application.

Ask for the handover decision explicitly. End the reasoning section with a forced choice: resolved by rule, or handover. When the model must answer that question, it cannot slide past the conflict.

Teams building this into a product usually keep the orchestration thin: one agent per role (triage, evidence retrieval, resolution, documentation), with a conflict layer that every role writes to. That mirrors the multi-role dispute architecture pattern and, more importantly, keeps the conflict record in one place rather than scattered across agent logs.

If the build is inside a payments or fintech line of business and you want the dispute work handled as a deliverable rather than a side project, Quanta Experts assemble an expert that reads your context and returns finished work at https://quantaintelligence.ai/agents.