Idempotency at the Agent Tool Interface

The Idempotency-Key request header exists precisely to make non-idempotent methods fault tolerant, and it is still a non-standard, experimental feature, so you own the semantics (MDN, IETF draft). That freedom is useful: you can be stricter than the public APIs.

The decision rule for your tool schema is simple.

Monday's API documentation makes the reuse rule explicit: never generate a new key on every retry attempt, because a fresh key identifies the retry as a different operation and defeats deduplication (monday.com). The same page recommends deriving keys deterministically for automated mutations, for example from the triggering event ID.

Where the key comes from matters more than its format. The orchestration layer should generate it from the durable execution step, such as {workflowRunId}:{stepId}, before dispatch, and persist its association with the logical operation and approved arguments (Formation). Do not ask the model to produce it. A newly sampled value can label the retry as a different operation. A fresh tool-call ID from the provider's response is not a stable business-operation identifier either, which is visible in how each provider mints those IDs, from Anthropic's toolu_ prefixes to Gemini's function-name-derived IDs (SylphAI).

A practical tool contract looks like this:

{
  "name": "create_ticket",
  "description": "Create a support ticket. Requires Idempotency-Key header.",
  "input_schema": {
    "type": "object",
    "required": ["title", "account_id"],
    "properties": {
      "title": {"type": "string"},
      "account_id": {"type": "string"}
    }
  }
}

The key stays outside the model-controlled arguments, in the header, so a regenerated plan cannot silently change it. Storing it in the arguments is acceptable only if your runtime guarantees it is attached after the model returns and never overwritten.

The server side has one correct shape: claim the key atomically, do the work, store the response in the same transaction, and replay it on every repeat. The atomic claim is what makes the whole thing race free. A SELECT then INSERT leaves a window where two concurrent retries both believe they are first, which is the classic time-of-check to time-of-use pattern OWASP documents for shared state (OWASP). A unique constraint on the key plus INSERT ... ON CONFLICT DO NOTHING closes that window in one statement (backendbytes.com).

A schema that supports the full contract:

CREATE TABLE idempotency_keys (
  key            VARCHAR(128) PRIMARY KEY,
  request_hash   VARCHAR(64)  NOT NULL,
  status         VARCHAR(16)  NOT NULL DEFAULT 'processing',
  response_code  INT,
  response_body  JSONB,
  created_at     TIMESTAMPTZ  NOT NULL DEFAULT now(),
  expires_at     TIMESTAMPTZ  NOT NULL
);

request_hash is the fingerprint. If the same key arrives with a different body, reject it rather than replaying a response for different arguments; MDN describes servers storing a hash of the request alongside the key exactly for this purpose (MDN).

The response shape should be stable and self-describing, so the agent runtime can tell a replay from a first execution and the test can assert on it.

{
  "ok": true,
  "idempotent_replay": true,
  "original_response": {
    "ticket_id": "T-4471",
    "created_at": "2026-02-11T09:14:03Z"
  }
}

Monday returns the cached body with an Idempotency-Replayed: true header and a 409 Conflict plus Retry-After when a duplicate arrives while the first is still processing (monday.com). Copy that split. A repeat after completion gets the stored body and the replay marker. A repeat during processing gets a conflict, so the caller polls instead of piling on work.

Two decisions deserve a rule of their own.

First, what to cache. The failure mode that bites teams is caching every response, including 500s, so a transient internal error becomes the permanent answer for that key (Mathildaduku). For agent tools, cache definitive outcomes (success, validation failure, business rejection) and leave an unresolved technical failure marked processing or failed_retryable so a later retry can re-attempt without creating a second side effect. Stripe's documented behavior is the stricter public reference point: it saves the resulting status code and body of the first request regardless of success, and subsequent requests with the same key return the same result (Stripe). Pick one and document it, because your test will encode the choice.

Second, expiry. Keys must outlive your longest realistic retry chain. Monday caches for 30 minutes, Stripe keeps keys for at least 24 hours, and general guidance lands between 24 hours and 7 days (monday.com, backendbytes.com, Zuplo). For agent workflows, size it to your slowest recovery path, including a human restarting a crashed orchestrator the next morning. Storage cost is small next to a duplicate refund.

If the tool calls an external provider, the guarantee must reach that provider's own idempotency contract. A local ledger proves intent; only the service that owns the effect can prevent a second charge, and where the mutation and its deduplication record share a transactional store they should commit atomically (Formation, backendbytes.com).

A contract you cannot test is a wish. The test has three assertions and one of them is the one teams skip.

1. Send the call with key K.
2. Send the identical call with key K again.
3. Assert the downstream effect count is exactly one (one row in charges, one sent message, one created resource).
4. Assert both response bodies are identical after normalization, including the idempotent_replay flag and the original identifier.
5. Assert the second call did not advance any business counter, such as an inventory decrement.

The third assertion is the important one. Two 200 responses with the same body prove the response layer is consistent; they do not prove the side effect happened once, because a tool that executes twice and returns a cached body passes a body-only check. Count effects at the sink.

A minimal test, framework agnostic:

def test_same_key_twice_one_effect(client, sink):
    key = "wf-8812:step-3:create_ticket"
    body = {"title": "SSO broken", "account_id": "A-1029"}

    first = client.post("/tools/create_ticket", json=body,
                        headers={"Idempotency-Key": key})
    second = client.post("/tools/create_ticket", json=body,
                         headers={"Idempotency-Key": key})

    assert first.status_code == 200
    assert second.status_code == 200
    assert sink.tickets_created == 1
    assert first.json()["original_response"]["ticket_id"] == \
           second.json()["original_response"]["ticket_id"]
    assert second.json()["idempotent_replay"] is True

Then add the variants that catch real bugs. Send the same key with a different body and assert a 4xx fingerprint rejection. Send two requests with the same key concurrently, from separate connections, and assert exactly one effect plus one conflict or replay; OWASP's parallel-request guidance describes firing requests through a release gate to widen the collision window, which is the right trick for making this deterministic (OWASP). Delete the first response row and resend to confirm behavior after expiry. Finally, run the test with the tool's real latency, not a stub, because the next section is where stubs lie.

Here is the failure the same-key-twice test will not catch if your stub is instant.

The client's retry deadline (say 5 seconds) is shorter than the tool's server-side processing window (say 12 seconds for a provision step that waits on a downstream queue). The first attempt times out at the client. The retry arrives at second 5. Your tool has not finished, so the key row is still processing, and depending on how you wrote the code, one of two things happens: the retry gets a conflict (correct) or it re-enters the work (wrong). If the handler treats "no stored response yet" as "proceed," both attempts execute. The key existed the whole time and bought nothing.

The arithmetic is the rule: client retry deadline must exceed the tool processing window plus your storage write. If the p99 processing time is 12 seconds and the stored-response write takes 200 ms, a 5-second deadline guarantees at least one retry that lands mid-flight, and a 15-second deadline makes that retry impossible for almost every call. This is the same at-least-once reasoning that applies to any unreliable channel, where the ack can always be lost, so the receiver must deduplicate rather than the sender guess (backendbytes.com).

Two fixes, and you should use both.

Fix the deadline. Set the client retry deadline above the tool's p99 processing window and publish that number in the tool description so the agent runtime and any wrapper honor it. Backoff with jitter controls timing and load but does not establish whether a mutation already happened, so it is not a substitute for the deadline rule (Formation).

Reserve the key at entry. Insert the processing row before any side effect, in its own commit. Any later state, including a crash, is then visible to the next attempt as "work in flight or done," never as "new request." This is the only ordering that survives a process restart between claim and effect, and it matches the atomicity requirement that the mutation and its deduplication record commit together where they share a store.

There is a third case worth writing into the runbook: the tool succeeds, the response write fails. Now the key says processing forever while the side effect exists. A reconciliation job that compares in-flight keys older than the processing window against the downstream system is what closes that gap, and leaving unresolved outcomes for reconciliation is the honest answer rather than pretending a wrapper that remembers successful responses is sufficient (Formation).

Agent frameworks make this worse by retrying at the LLM layer, where the model observes a failure and decides to call again, while nothing in that loop tracks whether the tool already executed (tianpan.co). That is precisely why the deadline and the reservation have to live in the orchestrator and the tool, not in the loop.

If you are building the orchestrator and would rather not assemble the retry, storage, and reconciliation pieces yourself, the team at Quanta Intelligence builds agent systems with these contracts baked in, and Quanta Inference gives every tool and model one OpenAI-compatible endpoint to call them through.