Idempotency Keys for Agent Writes: Guards and Retries

The key must be a deterministic function of intent. If two calls represent the same intent they must hash to the same string; if they represent different intent they must not collide. A workable construction:

import hashlib, json

def idempotency_key(tenant_id, operation, target_resource, request_id):
    payload = json.dumps(
        {
            "tenant": tenant_id,          # isolation boundary
            "op": operation,              # "payout.create", "ticket.assign"
            "target": target_resource,    # "account:8842", "ticket:55120"
            "req": request_id,            # caller-supplied, stable across retries
        },
        sort_keys=True,
        separators=(",", ":"),
    )
    return hashlib.sha256(payload.encode("utf-8")).hexdigest()

The request ID is the load-bearing part. It must be generated once, before the first attempt, and reused on every retry, because keys created during a retry cannot match the original attempt as Zuplo's implementation guide notes. A UUID version 4 or another random string with enough variability to avoid collisions is the usual choice per the same guide.

What must not go in: timestamps, retry counters, random values generated per attempt, the serialized request body in raw form, or anything the model produced mid-turn. A timestamp makes every retry a new key, which is the same as having no guard. The raw body looks appealing because it captures intent, but JSON key order and field noise make two semantically identical bodies hash differently; hash a canonicalized subset instead.

A decision rule for hard cases: if two requests should produce the same side effect, they share a key; if a human operator would call them two separate actions, they must differ. When an agent retries because it got a 409, that is the same action. When an agent decides to send a second, larger payout after the first succeeded, that is a new action and needs a new request ID.

Three layers can carry the guard, and they are not substitutes.

Client. The agent runtime generates the request ID, persists it alongside the pending action in its own state, and reuses it on every retry. This is the only layer that knows a retry from a new intent, so it is the one that must generate the key per Zuplo's guide. The cheap version is a per-turn memo in the agent loop: a dictionary from action hash to request ID, so an in-loop retry is obviously the same call.

Gateway. The edge or API gateway can require the header, reject missing keys on write methods, and short-circuit exact repeats. It cannot make the final guarantee on its own, because it does not participate in your database transaction. It is a good place to enforce shape (key present, tenant ID present, body digest stored) and to return 400 when a caller reuses a key with a different body.

Datastore. This is where the guarantee actually lives. The key row and the business write must commit or roll back together, and the key column must have a unique constraint. If the constraint lives in a separate cache or table that commits first, a crash between the two writes leaves you with a key and no effect, and every retry returns a success that never happened.

How long keys are kept is a policy decision: keys must outlive the retry window, and 24 hours is the common floor, with many implementations keeping them for up to 7 days per Zuplo's guide. The same guide notes that a TTL of a few hours to a day covers most cases in this idempotency article. Expiry is a correctness boundary, not a storage optimization: a key that expires while the agent is still retrying turns a duplicate into a second write. Set the TTL from the maximum retry horizon of your slowest agent workflow, not from disk budget.

A duplicate arriving with a stored key has three possible dispositions:

1. Same key, same body digest, stored response present. Return the stored status code and body, marked as a replay. Do not touch the business tables.
2. Same key, different body digest. Return a conflict error (409 is conventional) and log the mismatch. This is the collision that signals a bug in key construction, and it must be loud.
3. Same key, first attempt still in flight. Either wait for the result and replay it, or return 409 with a retry hint. Most designs use a row-level lock or a state column (in-progress, done, failed) so the second caller blocks briefly rather than racing per Zuplo's concurrency section.

The classic race is exactly the reason you cannot use a check-then-insert pattern in application code. A read of "does this key exist" followed by an insert has a window between the two where a second caller also sees nothing. Michael Swart's experiment on concurrent insert-or-update ran 10,000 concurrent calls against the naive pattern and got primary key violations on 0.42 percent of calls, which is small but non-zero and grows with load as he documents in his mythbusting post. That is the adverse condition: low enough to pass a smoke test, high enough to double-write in production.

The schema shape that closes it:

begin;

insert into idempotency_keys
    (key, tenant_id, operation, target_resource, request_id, body_digest, state, created_at)
values
    ($1, $2, $3, $4, $5, $6, 'in_progress', now());

-- rows affected = 1 means you own the execution
insert into payouts (payout_id, tenant_id, amount_cents, ...)
values (...);

update idempotency_keys
   set state = 'done', response_status = 201, response_body = $7
 where key = $1 and tenant_id = $2;

commit;

If the first insert violates the unique constraint, the transaction aborts and your handler catches the duplicate-key error, rolls back, and reads the existing row in a separate transaction. That read returns either the stored response (replay it) or in_progress (wait or return 409). Compare the body digest before replaying; mismatch is a 409, not a replay.

Two operational details matter here. First, the unique constraint should include the tenant (unique (tenant_id, key)) so a key collision in one tenant cannot block another. Second, keep the key row in the same database and same schema as the business write, or at least in a store that participates in the same transaction. A Redis-only key store reduces duplicate work but cannot guarantee atomicity with the write, which makes it a performance layer, not a guard.

The identity of the key is only as good as the identity of the tenant. If agent tooling runs under a shared service account and forwards a tenant ID from the model's output rather than from the authenticated token, an agent can address tenant B with tenant A's request ID and produce a genuine cross-tenant key collision. The rule is that tenant ID comes from the credential, never from the model. The same reasoning applies to the target resource: an agent that computes "target ticket" from a fuzzy search can produce a key for the wrong record.

Scheduling products show how bad overlapping reservations get once two writers share time ranges: double bookings arise from calendars that cannot see each other, slots taken while a reply is pending, and schedulers that cannot read real availability, and the structural fix is a single source of truth plus provisional holds as Cal.com's write-up sets out. Agent writes have the same shape. The equivalent of a provisional hold is an in-progress key row: it reserves the logical action while the side effect is still running.

The retry loop is where idempotency and load meet. Three rules keep it sane:

• Reuse the key. Generate the request ID once, persist it, and reuse it across every attempt. A retry with a new key is a new write.
• Back off exponentially with full jitter. Delay = random(0, min(cap, base * 2^attempt)). Full jitter spreads a fleet of agents that failed at the same moment across the backoff window, so the service does not get a synchronized retry pulse.
• Retry only what is retryable. Timeouts, connection resets, 429 and 5xx are candidates. A 409 conflict or 400 validation error will fail identically on every attempt, so retrying it just burns tokens and quota.

A practical ceiling for an agent workflow is three to five attempts, then surface the failure to the orchestrator with the request ID intact. That ID is the handle a human needs to check whether the write landed. If the request was recorded as in_progress and the process died, a sweeper should mark stale in-progress rows and decide whether to replay or fail them; without a sweeper, a crash mid-write leaves the key blocked until TTL.

There is a capacity argument for treating retries as a first-class load class. A test author who watched agent-driven CI reported 168 runs with 49 failures, 11,721.8 job-minutes burned and roughly 1.7M to 2.5M tokens of agent time spent debugging the same structural problem in Christopher Meiklejohn's incident write-up. The numbers there describe a test suite, but the amplification pattern is the same one a retry loop produces against a write API: a shared, brittle contract failing in ways that cost agent time rather than surfacing a single clear error.

Idempotency bugs hide in exactly the cases a happy-path test skips. Build the suite around collisions.

The concurrency tests are the ones people get wrong. Run them in parallel workers that genuinely overlap, and do not simulate concurrency by calling the function twice in a loop. Test tooling that runs workers as independent OS processes is a decent model for real race behavior, and worker counts can be pinned so CI load is predictable as the Playwright parallelism docs describe. If several tests share a database or an external service, declare a named lock so they never run simultaneously, while everything else stays parallel per the same docs. For the concurrent-same-key case, a lock is wrong: you want the two calls to land inside the same window.

Flaky tests deserve suspicion rather than loosening. A fixture-ordering bug in a workflow engine produced intermittent failures on unrelated pull requests because discovery returned filesystem order and an assertion compared an exact ordered list; the maintainers kept the strict assertion and fixed the ordering at discovery instead of relaxing the test to a set comparison as that issue describes. The same instinct applies to idempotency tests: if the conflict test flakes, the guard is probably order-dependent.

One more habit from test hygiene: use distinct fixtures per scenario, because an agent that reuses a global test user to add an RSVP can break an unrelated spec that expected a different attendee count as Meiklejohn recounts. In idempotency tests, shared keys across test cases are the equivalent mistake: they make failures look like implementation bugs when the test suite itself reused state.

The guard is invisible when it works, so instrument it. Log four fields on every attempt: the derived key, the request ID, the disposition (created, replayed, conflict, in-progress) and the latency. A replay rate that spikes is a signal that the agent is retrying more than it should, which is usually a timeout misconfiguration upstream. A conflict rate above zero means key construction is inconsistent somewhere, and the request IDs in the log tell you which caller is at fault.

Alert on any in_progress row older than a few times the p99 write latency. That is a stuck attempt, and it is blocking a key that a caller may be retrying against. Also alert on cross-tenant key reads, which should be impossible if tenant ID is part of the constraint and comes from the credential.

For teams running many agent workflows through one API surface, routing model calls through a single gateway keeps the retry and timeout policy in one place, which is the same discipline the write path needs: Quanta Inference exposes text, image, video, audio and embedding models through one OpenAI-compatible key, so the retry budget and headers are configured once rather than per vendor see Quanta Inference. If the harder problem is deciding which writes need an approval gate at all, a short scoping call is usually faster than reading more code: Quanta Consulting offers a free 30-minute discovery session here.