Idempotency Keys for Webhook Receivers: Design and Verification

The first design decision is what to key on. There are three viable sources, and the choice affects how much duplicate work you absorb versus how much state drift you risk.

Start with the provider's event ID when one exists. Asaas states plainly that when it resends an event, the id stays the same, and recommends persisting it under a unique constraint in its idempotency guide. Shopify sends X-Shopify-Webhook-Id in the request headers, and repeated requests for the same webhook carry the same identifier per Hookdeck's patterns. Stripe's payload includes a unique event identifier alongside the type and the changed object, which is enough to deduplicate on the delivery twice scenario is what the events table exists for.

Use the business object ID when the provider does not give you a stable event ID, or when the same business fact can be re-emitted as a new event (for example, an order sync where a re-created order should collapse into one local row). Shopify's orders/created webhook can key on shopify_order_id with a unique constraint in your local orders table as shown in Hookdeck's example. The risk is that a legitimate second event about the same object gets swallowed.

The composite key is the middle ground and the right default for payment flows: stripe:pi_12345:payment_intent.succeeded deduplicates retries of one event but still lets payment_intent.payment_failed and charge.refunded about the same intent both process. Write the rule down. The mistake to avoid is hashing the whole payload, because a provider that adds a field or changes serialization between attempts will make the same event look new.

Never build the key from the signature header or the delivery timestamp. Those change between attempts by design, and HMAC signatures exist to prove authenticity, not to identify events; the signature procedure, HMAC-SHA256, and replay prevention via timestamps are documented separately in the HMAC guidance on Webhooks.fyi.

There are two defensible architectures. The first is claim-and-process: insert the key into a table, let the unique constraint arbitrate, run the business logic, then mark the row complete. The second is persist-then-process: insert the event with status PENDING, return 200 immediately, and let a worker drain pending rows.

Claim-and-process is simpler and fine when the work is fast and transactional. The core is a single insert that either wins or raises a duplicate-key error:

CREATE TABLE processed_webhooks (
  provider     text        NOT NULL,
  event_id     text        NOT NULL,
  status       text        NOT NULL CHECK (status IN ('PROCESSING','DONE')),
  received_at  timestamptz NOT NULL DEFAULT now(),
  PRIMARY KEY (provider, event_id)
);

INSERT INTO processed_webhooks (provider, event_id, status)
VALUES ($1, $2, 'PROCESSING')
ON CONFLICT (provider, event_id) DO NOTHING
RETURNING event_id;

If the insert returns a row, you own the event and can proceed. If it returns nothing, another delivery already claimed it, so acknowledge with 2xx and do nothing. Asaas recommends exactly this shape, with ON CONFLICT DO NOTHING and a status column that starts at PENDING and moves to DONE in its documentation.

Three details decide whether this survives production.

First, avoid check-then-process. Two copies of the same event arriving near-simultaneously can both pass an if exists check before either writes. A unique-key insert lets the datastore arbitrate, which is the race-condition fix that Prismatic calls out.

Second, do not run side effects before the claim commits. If you send the email and then insert, a crash in between loses the dedup record and the retry sends a second email. Insert first, then act.

Third, track three states rather than two: PROCESSING, DONE, and a reclaim path for stuck rows. If a worker dies mid-flight, the row sits in PROCESSING forever and the retry gets skipped. A lease column holding a timestamp, plus a sweeper that reclaims rows older than the lease, handles the slow-worker-versus-recovery-worker collision that Prismatic describes. Set the lease longer than your slowest handler, not your median.

On failure, the cleanest option is to delete the claim row and return 500, letting both your queue and the provider retry. Hookdeck's Node example does precisely that: on a thrown error, remove the processed_webhooks row so the next attempt is not ignored in the dev.to version of its guide. Alternatively, keep the row but mark it failed and let the sweeper reclaim it; safer if you also want an audit trail.

Persist-then-process becomes worth the complexity when handlers call slow third-party APIs. GitHub terminates the connection if you do not respond within 10 seconds and queues payloads for asynchronous processing in its own guidance, recommending you acknowledge fast and process in the background per its best practices. Asaas' key warning applies either way: respond 200 only after the event is persisted, because once delivery is confirmed you must not depend on a new delivery to recover an event you failed to store in its guide.

A key with no scope is an accident waiting for a second provider. If you integrate Stripe, Shopify and a KYC vendor, three providers can each emit an event called an ID that looks similar. Include the provider (or endpoint) in the primary key, and if you run multiple accounts per provider, include the account or shop ID too. The composite key from the previous section, prefixed with the source, gives you stripe:acct_123:evt_456.

TTL is the other parameter people get wrong. The rule is simple: your deduplication window must be at least as long as the provider's retry window, plus margin for manual replays. A one-hour TTL against a three-day retry period protects only the first hour as Prismatic puts it. Provider schedules differ enough that a single global value is risky; Stripe retries for up to three days while Svix runs a fixed eight-attempt schedule, and GitHub does not auto-retry at all per Devlume's retry comparison. Pick the longest you can afford.

In practice that means 72 hours minimum for payment providers, with manual replay support extending it further for dispute and refund flows. A useful pattern is tiered retention: keep the key row itself for the full window with just the ID and status, and archive the payload separately if you need it for audit.

Storage follows from access pattern. A relational table with a primary key on (provider, event_id) is the default because the unique constraint does the arbitration in one statement. Redis with SET key NX EX 86400 is fast and works well as a first line of defense in front of a durable table, but it is a cache: if Redis is evicted you fall back to the database anyway, so do not make it the only store. Key-value stores like DynamoDB work if you accept the conditional-write pattern and give the item a TTL attribute.

Whatever you choose, make the claim and the business state change commit together when they can. In Postgres, wrapping the insert and the UPDATE orders SET status = 'paid' in one transaction means a rollback removes the claim and the retry is free to try again. When the side effect is an external API call, you cannot have that atomicity, and the fallback is absolute state changes plus a reconcile job.

Deduplication stops the same event from running twice. It does not stop a retry from doing damage when the first attempt half-completed. That is why the second half of idempotency is making the mutation itself repeat-safe: SET status = 'paid' is idempotent, balance = balance + amount is not again from Prismatic's framing.

Apply three rules.

Prefer absolute writes over deltas. Store the provider's authoritative state (status, amount_captured, refunded_amount) rather than incrementing counters, and derive aggregates from those fields. If the payload is stale, refetch from the provider's API before writing; Stripe offers thin events that contain only an ID precisely so you fetch current state as Hookdeck's scale article notes.

Order your effects so the irreversible one is last. In a payment flow, commit the database state, then call the ledger, then send the receipt. If the ledger call fails, transaction and idempotency key are recoverable; if you email first, a re-drive sends a second receipt.

For workflows with genuine multi-step side effects, add a step marker. A last_completed_step column on the event row, updated after each external call, tells a reclaimed event where to resume. That is the same idea as Asaas' PENDING to DONE status, at finer granularity in its docs.

Testing is where most teams under-invest. The minimum test is a replay: send the same delivery twice and assert both return 200 while the side effect happens once. Asaas states the expected result directly: only one record for the ID, both requests receive HTTP 200, and the business rule executes once in its validation section.

A test that consistently catches real bugs needs four cases:

1. Sequential duplicate. Post the same payload twice. Assert one order row, one charge, one email.
2. Concurrent duplicate. Fire both requests in parallel. This is the case that catches check-then-process; without a unique constraint, both usually succeed.
3. Crash mid-processing. Inject a failure after the claim but before the side effect. Assert the retry completes, rather than being skipped as already processed.
4. Stale ordering. Deliver payment_intent.payment_failed then an older payment_intent.processing about the same intent. Assert final state reflects the latest truth.

Here is a minimal harness for the first three, using the HTTP layer only so it works against any stack:

import asyncio, httpx

async def post(client, payload, headers):
    return await client.post("/webhooks/provider", json=payload, headers=headers)

async def test_duplicate_is_skipped(seed_order):
    async with httpx.AsyncClient(base_url="http://localhost:8000") as c:
        h = sign(seed_order)                 # valid HMAC over raw body
        first  = await post(c, seed_order, h)
        second = await post(c, seed_order, h)
        assert first.status_code == 200 and second.status_code == 200
        await asyncio.sleep(1)               # let the worker drain
        assert await count_charges(seed_order) == 1

async def test_concurrent_duplicate(seed_order):
    async with httpx.AsyncClient(base_url="http://localhost:8000") as c:
        h = sign(seed_order)
        r = await asyncio.gather(post(c, seed_order, h), post(c, seed_order, h))
        assert all(x.status_code == 200 for x in r)
        await asyncio.sleep(1)
        assert await count_charges(seed_order) == 1

Two practical aids. GitHub publishes a worked example with the secret It's a Secret to Everybody and payload Hello, World! that yields a known sha256= digest, so you can prove signature verification works before testing anything else per its validation docs. And when signing test payloads, HMAC over the raw body is the standard; SHA-256 is the baseline algorithm, and constant-time comparison avoids timing leaks as Hookdeck explains.

For payloads, prefer real captured events from your provider's dashboard over invented ones. Stripe, Asaas and Shopify all let you replay deliveries, so capture a production event, replay it twice into staging, and grep your logs for two executions of the business handler.

Instrument the dedup table itself. Emit a counter on every ON CONFLICT DO NOTHING hit and alert if the ratio of duplicates to new events changes by an order of magnitude in either direction. A jump suggests a provider retry storm; a drop to zero suggests duplicate detection silently broke.

Log the key with every event, and map it to your internal order or payment ID. When a customer claims a double charge, the fastest way to resolve it is a query on the event ID showing whether it was claimed once or twice. Track processing latency and dead-letter age alongside endpoint 4xx and 5xx rates, which is the observability checklist that teams running webhooks at volume converge on per Hookdeck's scale lessons.

Finally, keep the receiver's shape conventional. Verify the signature first, enqueue, return fast, process with idempotency, dead-letter on repeated failure as the Shopify reliability write-up puts it. The Standard Webhooks effort exists to make that shape portable across providers, since every sender otherwise reinvents its own conventions as the specification explains. If you are building a receiver that must talk to many providers, the same conventions are available as a packaged skill you can drop into a coding agent, which handles signature verification and dedup scaffolding so you spend your time on business rules: Quanta Skills.