How to Use an LLM to Assemble a Dispute Evidence Pack

Before a model sees a single record, the agent needs a join key. In practice the transaction ID or order ID is the spine, and everything else hangs off it: the payment authorization, the fulfillment record, the carrier tracking events, and the support conversations. Chargeback Gurus describes representment as a package containing a rebuttal letter plus compelling evidence, and notes the fee is not refunded even when you win, which is why assembling the package correctly the first time matters more than assembling it fast.

Build a collector that pulls, for one dispute:

• The original transaction record: amount, currency, timestamp, authorization result, AVS and CVV responses.
• The order record: line items, shipping address, billing address, email, account creation date.
• Fulfillment and carrier events: dispatch time, scan events, delivery confirmation type, signature or photo availability.
• Every support message touching the order, including pre-purchase and post-purchase, in timestamp order.
• Prior order history for the same customer, which is strong context for a claim of unauthorized use.
• Policy and product-page snapshots as they appeared at purchase time.

Store each item with an immutable reference (document ID, URL, export timestamp). The model will later cite these references, and your checks will verify the citations resolve. If a source system cannot be reached, the agent should mark the pack incomplete rather than proceed. A partial pack that looks complete is the worst outcome in this workflow.

The core insight is to split the job in two. The model's first task is extraction into a schema you control; its second task is drafting prose from that schema. Prose generated directly from raw logs is where invented tracking numbers and paraphrased amounts come from. Prose generated from a validated table can be checked line by line.

Define the schema explicitly and ask for JSON. A minimal version:

{
  "dispute_id": "string",
  "reason_code": "string",
  "claimed_amount": {"value": "string", "source_ref": "string"},
  "transaction": {
    "timestamp": {"value": "string", "source_ref": "string"},
    "amount": {"value": "string", "source_ref": "string"},
    "avs_result": {"value": "string", "source_ref": "string"},
    "cvv_result": {"value": "string", "source_ref": "string"}
  },
  "authentication": {
    "eci": {"value": "string|null", "source_ref": "string|null"},
    "cavv_present": {"value": "boolean", "source_ref": "string|null"}
  },
  "fulfillment": [
    {"event": "string", "timestamp": "string", "source_ref": "string"}
  ],
  "delivery_confirmation": {"type": "string|null", "source_ref": "string|null"},
  "customer_messages": [
    {"timestamp": "string", "author": "customer|merchant", "summary": "string", "source_ref": "string"}
  ],
  "evidence_gaps": ["string"]
}

Every scalar is wrapped with a source reference, which forces the model to point at the document it read. Fields it cannot populate must be null plus listed in evidence_gaps. Tell the model in the system prompt that null is the correct answer for anything absent, and that inventing a value is the only unacceptable failure. Structured-output modes make the schema enforceable; this benchmark write-up on structured outputs shows the failure mode you are guarding against, where a model returned a valid-looking JSON object with a subtly wrong amount and an impossible date, and argues that per-field trust scoring lets reviewers focus on the small fraction of fields that warrant attention.

Then draft. The drafting prompt receives only the evidence table, the reason code, and a template. Its instruction set has three rules: every sentence containing a fact must cite at least one source_ref; no number may appear that is not in the table; and the letter opens by naming the specific claim and closes with a list of attachments matching the table's source references.

This is the part most teams under-specify. Some values are verifiable in a millisecond and some are catastrophically wrong if wrong, so treat the second group as locked fields. The agent may copy them; it may never compute, infer or complete them.

Two adjacent rules save a lot of grief. First, the model may summarize a customer message, but any text in quotation marks must be a verbatim substring of the transcript; your checker can test this mechanically. Second, the model may not state a conclusion about the cardholder's intent. A sentence such as the cardholder appears to have forgotten the purchase is an argument the human reviewer makes, not one the model asserts.

Delivery evidence deserves its own caution. A basic delivered scan does not establish where a package was left or who received it, and delivery confirmation does not guarantee a reversal; it works as part of a package alongside order records, invoices and communications, as this guide to delivery confirmation in disputes explains. Instruct the model to describe exactly what the carrier recorded and nothing more. If the scan says left at front door, the letter says left at front door, not signed for by the recipient.

Automation should catch the cheap errors so reviewers spend their attention on judgment calls. Run these in order and block on failure:

1. Schema validation. All required fields present or explicitly null with a gap entry.
2. Reference resolution. Every source_ref resolves to a stored document.
3. Locked-field exactness. Every locked field matches its source byte for byte.
4. Numeric containment. Every number in the draft appears in the evidence table.
5. Quote verification. Every quoted string is a substring of a source transcript.
6. Timeline sanity. No event precedes the order, and no date is impossible.
7. Address consistency. Billing and shipping are stated accurately and the discrepancy, if any, is described rather than smoothed over.
8. Evidence coverage. The reason code's required evidence types are all present or listed as gaps.

The timeline and address checks catch a specific, common hallucination: a model that produces a tidy narrative by quietly reconciling a shipping address that does not match the billing address. That discrepancy is often the most important fact in a not-received dispute, in either direction, and it must survive into the letter. Chargeback Gurus on compelling evidence frames the standard as documentation that directly contradicts the claim, which means discrepancies are evidence, not noise to be cleaned up.

Track your checks. Two numbers tell you whether the system is working: the rate at which drafts fail numeric containment (high early, falling as prompts tighten) and the rate at which human reviewers still correct a factual error that passed all checks. The second number is your real error rate and should drive prompt and schema changes.

The gate is not a rubber stamp. Give the reviewer a side-by-side view: the draft on the left, the evidence table with resolved source links on the right, and every locked field highlighted. The reviewer's job is to confirm four things and nothing else: the letter answers the reason code, every claim traces to a source, nothing is overstated, and nothing is missing that would strengthen the case.

A decision rule worth stating plainly: do not fight disputes where the available evidence does not actually disprove the claim. Chargeback Gurus on representment makes the same point, that merchants should assess whether the dispute is valid and whether escalation makes financial sense, and a dispute package checklist stresses that all dates, transaction IDs and names must match across documents, because a mismatch causes confusion or delay. An LLM makes it cheap to draft a letter for every dispute; resist that. A weak, well-formatted argument costs you reviewer time and does nothing for your win rate. Route cases with unresolved evidence gaps to accept-or-refund instead.

Friendly fraud context matters for that routing decision. Visa's material on friendly fraud notes it represents around 20% of fraudulent disputes globally and up to 30% for high-volume online merchants, while a BBC report on chargeback scams describes a restaurant owner who found chargebacks arriving after card payments taken over the phone, with a fixed fee on top of the lost sale. Both point to the same operational conclusion: the evidence pack wins when it shows participation and delivery, so keep the artifacts as a matter of routine rather than reconstructing them under deadline.

One more structural point: a soft chargeback, also called a retrieval request, arrives before a formal dispute and asks for more information about a transaction. Shift4's overview of chargebacks notes that responding without delay is advisable because a retrieval request is a chance to avoid a chargeback entirely. Your agent should handle retrieval requests with the same collector and checks, because the evidence assembled there is reused if the dispute follows.

Keep a standing list, because models fail in recognizable patterns:

• A tracking number with a plausible carrier prefix that matches no shipment in your export.
• A delivery date one day off the carrier's scan, or in the wrong timezone.
• A paraphrased policy line that softens or hardens the actual terms.
• A prior-order count that rounds up.
• A quoted customer message with tidied grammar.
• A summary of a chat thread that omits the message where the customer asked to cancel.
• An inferred discount, partial refund or adjustment the payment record does not show.
• A confident statement that authentication succeeded when the ECI is absent.
• A named recipient for a signature that was never collected.
• A currency symbol applied to an amount from a different currency.

When you route this through one API, keep extraction and drafting as separate model calls with separate prompts, even if both hit the same endpoint. Mixing them reintroduces the failure mode you just removed. Quanta Inference serves text, image and embedding models through one OpenAI-compatible key, which keeps the extractor, the drafter and any classifier you add on the same billing and auth path.

The sign-off itself should be a named person, a timestamp, and a checklist of confirmations recorded against the draft version. That record is what you show internally when a letter goes wrong, and it is what keeps the model where it belongs: assembling and drafting, never deciding.