Verifying Agent Tool Calls: Tests That Catch a Doubled Refund

The single most useful design decision is separating intent identity from attempt identity. A run ID, trace ID, or tool-call ID usually changes when work is retried, so none of them can answer "did this refund already happen?" One intended refund can have five attempts and five traces while still having one operation key (AI Programming Manual).

Good keys come from the business decision, not the payload. refund:invoice_842:approval_17 is stable across retries of one approval and different for a second legitimate refund on the same invoice. Do not derive keys by hashing tool arguments alone, because two customers may legitimately receive the same template message, and one customer may legitimately receive it twice months apart.

The runtime should generate the key before asking the model to call the tool, then inject it into the tool context. The model supplies business arguments; the platform supplies identity. A small operation table makes that explicit:

CREATE TABLE tool_operations (
  operation_key   TEXT PRIMARY KEY,
  payload_hash    TEXT NOT NULL,
  status          TEXT NOT NULL CHECK (
    status IN ('in_progress','completed','needs_reconciliation')
  ),
  lease_until     TIMESTAMPTZ NOT NULL,
  result_json     JSONB,
  downstream_id   TEXT,
  updated_at      TIMESTAMPTZ NOT NULL DEFAULT now()
);

The distinction between "safe to try" and "unknown whether it happened" is the whole point. Collapsing both into failed is how duplicate effects get born (AI Programming Manual). Webhook receivers reached the same conclusion: dedupe on the event's unique ID before doing any work, store that ID durably, and inside the same transaction as the side effect apply a unique constraint, so a conflict is your signal to acknowledge and do nothing (Devlume). Your refund tool should behave like that receiver.

A fixture for duplicate processing has to be boring and exact. Four elements matter:

1. A seeded identity. One customer, one invoice, one approved refund intent, all with fixed IDs. Tests that invent IDs at runtime cannot assert counts reliably.
2. A fixed operation key. Not a fresh UUID per run. The point is to replay the identical key, and the identical arguments, twice.
3. A clean effect surface. A sandbox ledger, an outbox table, and a captured email list, all keyed by the test run ID so cleanup is a single delete.
4. A failure injector. A switch that makes the first call commit and then drop the response, which is the only way to reproduce the real failure rather than a fake one.

The mock, sandbox, live decision is about which layer you are trying to prove, not about realism. Mocks are fast and deterministic; sandboxes catch real schemas, auth, and state transitions; live scoped checks only when nothing else can represent the production surface (Hamming). Duplicate-write tests belong in the sandbox almost every time, because a mock that just returns 200 twice proves nothing about whether your unique constraint fires.

A practical fixture looks like this in a test file:

fixture = {
    "run_id": "eval_dupe_refund_014",
    "customer_id": "cus_fixture_07",
    "invoice_id": "in_842",
    "operation_key": "refund:in_842:approval_17",
    "amount_cents": 4200,
    "drop_first_response": True,
}

Then the test harness runs the agent or the tool wrapper twice with that fixture and inspects the ledger directly. If you want one API key across the models driving the agent and the model used to judge failures, that is what Quanta Inference (https://quantaintelligence.ai/inference) is for, but the assertions themselves should be plain code.

The assertion that matters is a count on the external system, not a string in a transcript. A passing transcript is useful evidence, not proof that the workflow succeeded; the ordinary failure is the agent saying "you're booked" while the system holds zero or two appointments (Hamming).

Take a refund of 4,200 cents, about $42. A single-intent run should leave these numbers in the sandbox:

That last row matters for triage. Audit entries may legitimately record two attempts, because attempts are real events. The ledger may not. Writing both assertions into the same test teaches reviewers the difference between "we tried twice" and "we paid twice", which is the bug you are hunting.

Webhook practice gives you a sanity check on the same numbers. The recommended verification for an idempotent receiver is that two deliveries with the same ID produce only one record, both requests receive HTTP 200, and the business rule runs once (Asaas docs). Apply that shape to your agent: two tool calls, one effect, both calls report success to the model so it stops retrying.

Not every tool deserves the full operation table. Spend the guard where a duplicate costs money, trust, or compliance, and skip it where a repeat is harmless.

Two refinements. First, prefer absolute state changes: SET status = 'paid' is retry-safe while balance += amount is not, and keep any dedup window longer than the sender's retry window, because a one-hour TTL against a 72-hour retry period protects only the first hour (Prismatic). Second, for multi-step workflows that reserve inventory, charge, and email, treat each step as its own guarded operation rather than wrapping the sequence in a retry, since a retried sequence re-enters steps that already committed (TianPan.co).

If you want the guard designed once for a line of business rather than per tool, Quanta Skills (https://quantaintelligence.ai/skills) packages these patterns as installable skills for coding agents, and an expert can be put on the job for a specific workflow.

Run the duplicate-write suite on every change to prompts, tool schemas, and orchestration code, because all three can alter retry behavior without touching the tool. Gate merges on it. Keep the suite small; a dozen tools times three scenarios is plenty and finishes in minutes.

When a test fails, classify it before fixing it:

• Two effects, both calls returned success. The tool has no deduplication. Add the operation key and constrain it.
• Two effects, model retried after a misleading response. Make the tool return an explicit "already applied" success, which removes the model's reason to retry (RunGuard).
• One effect, but the run hung. The lease expired and the status flipped to needs_reconciliation; reconcile against the downstream system before retrying, never infer success from an exception.
• Zero effects and one success message. The worst case. Assert on external state, not on the agent's claim.

Signature and replay checks on inbound webhooks belong in the same suite if the agent's effects feed back through webhooks, and the Standard Webhooks spec is the reference for consistent signing conventions across providers (standard-webhooks.md), with the project's own overview describing the goals of security, interoperability, and reliability (Standard Webhooks).

One more habit: keep the fixtures and assertions as reviewable files, so a change to deduplication logic shows up as a diff in a test that a reviewer can actually read.