Idempotency Keys for Agent Triggers: What to Store and Where

Put the key in the same database as the side effect, not in Redis, a queue attribute, or process memory. Three reasons:

1. Atomicity. The key row and the business row must commit or roll back together. If the key is in Redis and the order is in Postgres, a crash between the two writes leaves you either a lost order or a duplicate one, and no retry logic can tell which.
2. Durability across restarts. An in-memory set dies with the process, and a rolling deploy is a restart.
3. Queryability during an incident. When someone asks "did we charge twice," you want to run one SQL query, not reconcile two systems.

A cache in front is fine as an optimization. It is not the source of truth.

Providers that publish their own contract land in the same place. Stripe's API v1 saves the first request's status code and response body once endpoint execution begins, and a same-key retry returns that saved response, including a saved 500 error (Stripe). That is a server-side record of outcome keyed by the client's key, exactly the shape recommended here. Stripe also documents that the v1 guidance bounds unresolved network retries to the first day; past that, stop and reconcile instead of retrying blindly (Stripe). Note that the v2 API uses different replay semantics and a 30-day window, so read the contract for the API version you call.

Note where agent calls differ. A webhook arrives with an identifier the provider chose (X-Shopify-Webhook-Id, sg_event_id, or similar), while an agent tool call often arrives with nothing. For agent triggers, derive the key from something stable in the request (conversation ID plus step index, or a hash of the tool name and normalized arguments) and pass it explicitly, for example as an Idempotency-Key header. Never generate a fresh key inside the retry: that turns every attempt into a new action, which is the failure mode Stripe describes as a fresh key repeating the side effect (Stripe).

Two tables: one for the key and its outcome, one for whatever the business effect is. Both live in the same database and are touched in one transaction.

CREATE TABLE idempotency_keys (
    key             text PRIMARY KEY,          -- provider id or derived hash
    scope           text NOT NULL,             -- 'stripe.charge', 'agent.send_email'
    request_hash    text NOT NULL,             -- sha256 of canonical request body
    status          text NOT NULL,             -- 'in_progress' | 'completed' | 'failed'
    response_code   int,
    response_body   jsonb,
    created_at      timestamptz NOT NULL DEFAULT now(),
    updated_at      timestamptz NOT NULL DEFAULT now(),
    expires_at      timestamptz NOT NULL
);

CREATE INDEX idempotency_keys_expires_at_idx ON idempotency_keys (expires_at);

Design choices worth defending:

• text PRIMARY KEY, not a synthetic integer. The insert then doubles as the duplicate check, with no extra index to maintain.
• scope in the row. Provider event IDs are only unique within a provider. Without a scope, two systems that both emit evt_123 will collide.
• request_hash. Same key with different parameters is a client bug, not a retry. Reject it with a 422 so the mismatch surfaces early instead of silently returning a result for different inputs.
• status. in_progress lets a second delivery detect a concurrent run rather than block indefinitely.
• response_body. This is the replay payload. Store what the caller needs, not the internal trace.

The first strategy in the Hookdeck guide is exactly this shape: a unique constraint inherited from the event data, side effects performed after the insert, and the unique violation handled as a duplicate rather than an error (Hookdeck).

For the checklists most teams get wrong:

If you run several agent tools across different model providers and want one place to observe calls, Quanta Inference puts text, image, video, audio and embedding models behind a single OpenAI-compatible API key (Quanta Inference), which means one place to attach and log the key you pass on each call.

The naive pattern, check then insert, loses under concurrency. Two workers both read "no row," both run the charge. Use the unique constraint as the lock.

def handle(key: str, scope: str, payload: dict, work):
    h = sha256(canonical(payload))
    with db.transaction():
        row = db.execute(
            """INSERT INTO idempotency_keys (key, scope, request_hash, status, expires_at)
               VALUES (%s, %s, %s, 'in_progress', now() + interval '7 days')
               ON CONFLICT (key) DO NOTHING
               RETURNING key""",
            (key, scope, h),
        ).fetchone()

        if row is None:                      # we lost the race
            existing = db.select_key(key)
            if existing.request_hash != h:
                raise Mismatch422()
            if existing.status == 'completed':
                return existing.response_code, existing.response_body
            if existing.status == 'in_progress':
                raise Conflict409()          # let the sender retry later
            # status == 'failed': allow one retry from the same worker
            db.update_status(key, 'in_progress')

        result = work(payload)               # side effect inside the transaction
        db.update_key(key, 'completed', result)
        return result

ON CONFLICT DO NOTHING plus RETURNING tells you in one statement whether you own the key (Hookdeck). The alternative, catching the Postgres unique violation and then reading, works too; just make sure the check and the insert are not separated by application logic. The Hookdeck example handles SQLSTATE 23505, the Postgres code for a unique violation, and returns a success response so the sender stops retrying.

For the released-on-error pattern (delete the key so the next attempt can proceed), remember the risk: if the work succeeded but the transaction rolled back after the external call, deleting the key re-opens the duplicate. Keep the external call and the row update in the same transaction where possible, or record a pending_external state and reconcile.

The receiver's response code is part of the contract:

• 200 with the stored body. Duplicate of a completed action. The sender should stop retrying.
• 409. The first attempt is still running. The sender should retry later with the same key.
• 422. Same key, different payload. Not retriable; alert.
• 5xx. Transient. The key row should be left in failed or deleted so the next attempt can proceed.

On the sending side of your own agent calls, use exponential backoff with jitter, cap the maximum delay, and stop after a fixed number of attempts (Hookdeck). Retriable failures are 5xx, timeouts, connection resets and temporary DNS errors; 4xx responses (except 429) will not fix themselves, and 429 should honor Retry-After. Jitter matters because without it every client that failed at the same moment retries at the same moment, a thundering herd that can knock a recovering service over (Svix).

A worked example with numbers. SendGrid gives you 10 seconds to respond and retries for up to 24 hours (Hookdeck). Suppose your handler needs 18 seconds because an agent step calls a model. Options:

1. Insert the key, return 202 immediately, and process asynchronously. The fastest correct answer, and the key row already exists when the retry arrives.
2. Insert the key, do the work, and return 200 after 18 seconds. The provider times out, retries, your second delivery sees in_progress and returns 409; the sender retries later and eventually gets 200 with the stored body. Correct, but noisier.
3. Raise timeouts nowhere and hope. This is the double-charge path.

Option 1 is the default. If you need option 2 for a synchronous API, at minimum make the key insert the first thing you do, so a slow run never becomes two runs. For agent steps that call several model vendors, putting those calls behind one API removes a variable from your timing budget (Quanta Inference).

Retention is a bounded problem: keep keys longer than the longest retry window you can receive, then delete. Published windows give you numbers to work from:

The rule: retention = longest sender retry window x 2, rounded up. If you are unsure, 7 days is cheap for most tables and covers manual replays and incident forensics. If a provider documents a 30-day replay window, use 31 days for that scope, not for everything.

Cleanup is a scheduled DELETE ... WHERE expires_at < now() LIMIT 10000, looped until zero rows, run off-peak. Alternatively, partition by created_at month and drop whole partitions, which avoids the bloat and vacuum cost of row-by-row deletes. If you need the history for audits, archive to object storage before deleting, or move completed rows past expiry into a slower table.

One caution: do not set expiry so short that a legitimately slow sender's late retry misses the row. The failure mode is silent (a second charge), and the only symptom is a very confused customer.

You cannot test this by reading code. You test it by replaying.

1. Get the exact body and key. Capture a real delivery in a staging environment, including the provider's event ID or your derived key. SendGrid payloads include sg_event_id and sg_message_id per event (Hookdeck).
2. Replay it N times, concurrently and sequentially. Assert exactly one row in the business table and exactly one call to each downstream system.
3. Assert the response. First delivery 200, subsequent deliveries 200 with the same body (or 409 while in progress).
4. Simulate the timeout. Sleep inside the handler past the provider timeout, replay, assert one charge and one send.
5. Simulate the crash. Kill the process between the external call and the key update, replay, and confirm the reconciliation path catches it.
6. Assert no key is deleted on failure without a compensating record. A key deleted after a successful external call is a duplicate waiting to happen.

For counting side effects, wrap the downstream client in a recorder during tests: an email sender that increments a counter, a payment client that returns a fixed ID and increments a counter. Assert the counter is 1. Stripe's guidance points the same way: the operation is identified by its key, and a same-key retry must return the saved response instead of executing the operation again (Stripe).

If your webhook receives batched events, as SendGrid does with a JSON array per POST (Hookdeck), test at the event level: a batch of 5 events where 3 already exist should produce 2 side effects, not 5 and not 0.

Verifying the sender's signature is a separate check and does not make you idempotent. SendGrid signs with ECDSA over the timestamp plus body, using the X-Twilio-Email-Event-Webhook-Signature and X-Twilio-Email-Event-Webhook-Timestamp headers (Twilio), and verifying that signature proves origin, not uniqueness (Stack Overflow). Keep both.

For teams running many agent workflows, the same replay harness applies to tool calls: invoke the tool again with the same key and assert one effect. If you would rather not build every integration yourself, Quanta experts read your context and hand back finished deliverables, including the harness (Quanta Experts).