Idempotent Write Attempts: Making Escalated Writes Safe

The common implementation generates the idempotency key inside the HTTP client wrapper, right before the request goes out. That works for a single process with no pauses. It breaks the moment a human sits between the agent's decision and the write.

Trace the failure. The agent builds a refund, calls the approval tool, and waits for a human. The run is serialized and the process exits. Forty minutes later a reviewer clicks approve. The workflow resumes, rebuilds the HTTP client, generates a fresh UUID, and posts the refund. If the original dispatch never happened, nothing is wrong. If the agent had already dispatched once before discovering a policy condition that forced escalation, the service now sees two different keys and treats them as two different refunds. The row said one write; the ledger says two.

The guidance on idempotency keys is explicit about timing: keys should be created before the first request is sent, not during retries, so that every attempt at the same operation carries the same key (Zuplo's guide to idempotency keys). An approval gate is a very long retry interval. Treat it as one, and generate the key at the moment the agent first decides to write.

There is a second, subtler reason to move generation earlier. The key must be stable across process restarts, container reschedules and model re-runs. Anything derived at dispatch time depends on the runtime environment, which is the least stable thing in the system. Deriving it at intent time and persisting it makes the key a property of the decision, not of the worker that happens to execute it.

Random UUIDs are fine when one client generates the key once and keeps it in memory. They are worse when multiple components might independently reach the same decision, such as a retried LLM call that produces the same refund with a slightly different explanation string. A deterministic key built from the decision's identity collapses those variants into one.

The rule: hash only the fields that define the effect, in canonical order, excluding anything the model can phrase differently.

import hashlib, json

def attempt_key(run_id, step_index, tool_name, effect_fields):
    basis = {
        "run": run_id,
        "step": step_index,
        "tool": tool_name,
        "effect": effect_fields,   # e.g. {"invoice": "inv_8841", "amount_cents": 4200, "currency": "USD"}
    }
    canonical = json.dumps(basis, sort_keys=True, separators=(",", ":"))
    return "wa_" + hashlib.sha256(canonical.encode()).hexdigest()

Worked example. An agent handling a billing dispute decides to credit 42.00 USD against invoice inv_8841 on run r-7f3a at step 6. The effect fields are the invoice, the amount in minor units, and the currency. The reason text ("customer reported duplicate charge per ticket 5512") is excluded, because a re-run may phrase it differently. The key is the same whether the request is dispatched at 10:02 or, after a human reviews the dispute, at 11:47 the next morning.

Two cautions. First, exclude anything nondeterministic from the effect fields: timestamps, request IDs, model temperature artifacts, retry counters. Second, include the step index but not the attempt number. Including the attempt number is the classic bug, because each retry then computes a new key and the whole mechanism evaporates.

A useful decision rule for what belongs in the effect fields: if changing the field would change what a customer sees in their bank statement, their inbox, or their audit log, it belongs in the key. If it changes only how a human explains the write, it does not.

The point of putting the key in the row before approval is that the approval becomes a state transition on an existing record. The reviewer approves row wa_9c1e..., not "a refund of 42.00 against inv_8841 that will be created when you click". The distinction matters because the second phrasing invites the UI to hold the parameters in a form and submit them, which is precisely where keys get regenerated.

Approval design guidance is consistent on this: approval should pause the existing run rather than start an unrelated new interaction, long delays require durable serialized state, and the reviewer needs to see the proposed action, its external effect and its reversibility (the human-in-the-loop design guide). Elastic's implementation of this pattern persists workflow state in an index and holds a paused execution for as long as the human takes, then resumes the same workflow rather than a new one (Elastic's pause and resume walkthrough).

The review screen should show the key, truncated, with a copy affordance. Not because the human is expected to type it anywhere, but because when a support engineer later asks "did this write happen twice", the answer is a lookup, and the person reading the ledger has something to search for. The human never invents a key, never sees the full hash in the happy path, and never has to reason about deterministic hashing. They see the effect and click approve.

Two fields on the approval record are worth insisting on. The first is the hash of the payload the reviewer actually saw, so that a later modification of the row can be detected. The second is an expiry. Approval guidance notes that approval can become stale when external conditions change while the workflow is paused, so the resume path should revalidate the effect fields against current state and request a new decision if they moved (Promptessor on approval expiration). Revalidation does not touch the key: if the amount changed, that is a different effect and therefore legitimately a different key, which is the correct outcome.

The resume path has five states to handle, and each maps to one action. The rule that keeps it honest is verify before retry.

The dispatched state is where most implementations get it wrong by resending first and asking later. For a service that honors idempotency keys, resending with the same key is safe and cheap: the server looks up the key, finds the stored result, and returns it without reprocessing, which is the standard server-side algorithm (Zuplo). For a service that does not honor keys, resending is a coin flip. The paper on verified tool calls argues for querying postcondition state before retrying, and reports that this verify-before-retry approach reduces duplicate actions while keeping task success rates comparable (arXiv 2608.02645).

The third-party question deserves a blunt answer. You cannot manufacture idempotency for a remote system that does not offer it; the best you can do is send once, track what you sent, and have a reconciliation process for the ambiguous window, with manual investigation when the window closes inconclusively (the Stack Exchange discussion of third-party idempotency). What you can do locally is make the key the join column between your attempt row and the provider's records. Stripe-style APIs, and many others, echo the key back in their object metadata, so a daily reconciliation job can ask "for every row in dispatched or unknown, does the provider have an object carrying this key?" and close the loop without guessing.

A retry policy is only as good as the error it reads. Vague messages drive loops, and one documented case had an agent run hundreds of reasoning steps at high cost per minute without ever terminating, with ambiguous tool feedback listed among the root causes (the analysis of agent reasoning loops). The same dynamic applies to write attempts: a tool that returns "something went wrong, maybe try again" will be tried again, and again.

Errors should be as detailed as possible, help humans understand what went wrong, help computers decide what to do next, and be consistent across the API (Speakeasy's error design guide). In practice that means every write result should carry a machine-readable code and a message, and the resume path should branch on the code rather than on a free-text status. Postman's list of server-side error practices adds that error responses should be uniform and that logging and monitoring matter most when failures span several calls, which is exactly the position a resumed write is in (Postman on API error handling).

A workable branch table for the resumed write:

RETRY = {"rate_limited", "timeout", "service_unavailable", "connection_reset"}
VERIFY = {"unknown_outcome", "gateway_timeout"}
FATAL = {"card_declined", "insufficient_funds", "policy_violation", "invalid_request"}

def resume_action(code):
    if code in RETRY:  return "resend_same_key_with_backoff"
    if code in VERIFY: return "query_postcondition_then_decide"
    if code in FATAL:  return "close_row_and_report_to_human"
    return "escalate"

Two details make this survive contact with reality. First, cap the resends per row with a column on the row rather than an in-memory counter, because the whole point is that the process may not exist between attempts. Second, when a fatal code arrives after a verify, write the outcome into the row and surface it to the reviewer who approved it. An approval that turns out to be unexecutable is information the human needs, and it is the kind of follow-up that keeps approval fatigue from setting in.

For the presentation layer, the same principles that make error messages usable in an interface apply to what the reviewer sees on a failed write: put the message next to the thing that failed, make it visible, and make it say what to do next rather than only what went wrong (Nielsen Norman Group error-message guidelines).

You cannot prove the resume path works by reading it. The failure modes are timing-dependent, so the tests have to inject the timing.

Four scenarios cover most of the risk. Each starts with an approved row.

1. Crash before dispatch. The row is approved. Resume should produce exactly one effect at the provider. Assert the provider's object count and the key on the object.
2. Crash after dispatch, before the response is recorded. The row is dispatched. The test should simulate the provider having processed the request. Resume should not create a second object. Assert one object and that the row moves to confirmed with the stored response.
3. Approval after a long pause with changed external state. The invoice was paid by another process while the agent waited. Resume should revalidate, find the effect no longer valid, and request a new decision instead of writing.
4. Duplicate resume. Two workers pick up the same approved row concurrently. The unique constraint and a row-level lock should let exactly one dispatch. Assert one provider object and one response_body.

A test harness detail that pays for itself: make the fake provider store (idempotency_key, response) and return the stored response on a repeat, with a knob to ignore keys so you can test the verify-before-retry path against a service that is not key-aware. That single knob exercises both halves of the design.

If your agents run through a managed layer rather than raw provider calls, route every write through a single egress point so the key logic lives in one place. Quanta Inference (https://quantaintelligence.ai/inference) fronts text, image, audio and embedding models behind one OpenAI-compatible API key, which keeps the model side of an agent pipeline uniform, and the write-attempt logic described here stays in your code where the row lives.