Every provider promises webhooks for delivered, bounced, complained and opened events. The differences live in the fine print, and they show up at the worst possible time: a delivery event lost during a deploy, a duplicate that sends your customer two "your order shipped" emails, or a bounce you never receive because the retry window closed overnight.
Webhooks are POST requests, and most providers operate on an at-least-once guarantee, meaning duplicates are normal and your handler has to be idempotent (Hookdeck on webhook idempotency). Because POST implies side effects, the burden of making the endpoint safe falls on you, usually through a unique constraint on an event ID or a processed-events table.
Ask each provider these five questions, and write the answers into your integration design:
| Contract item | What to confirm |
|---|
| Retries | How many attempts, over what window, with what backoff |
| Idempotency | Is there a stable event ID in the payload or headers |
| Ordering | Are events per message ordered, or must you sort by timestamp |
| Signatures | Which header carries the signature, which algorithm, key rotation |
| Dead letters | Where failed events go after retries, and how you replay them |
A common pattern is exponential backoff with a capped number of attempts, often three to five, plus logging of failed requests so you can troubleshoot (2Point on retries and idempotency keys). If your provider retries for 24 hours and yours does not, a two-hour outage in your own infrastructure means permanently lost delivery data. That is a decision rule worth stating plainly: choose a retry window at least as long as your worst realistic outage, and choose a dead-letter path for everything beyond it.
Here is a minimal idempotent handler that turns a duplicate delivery into a no-op:
import psycopg
def handle_event(event_id: str, payload: dict, db: psycopg.Connection) -> bool:
try:
with db.cursor() as cur:
cur.execute(
"INSERT INTO processed_webhooks (id) VALUES (%s)", (event_id,)
)
except psycopg.errors.UniqueViolation:
db.rollback()
return True # already processed, acknowledge and move on
try:
apply_delivery_status(payload)
db.commit()
return True
except Exception:
db.rollback()
with db.cursor() as cur:
cur.execute("DELETE FROM processed_webhooks WHERE id = %s", (event_id,))
db.commit()
raise
The delete-on-failure step matters. Without it, a handler that crashes mid-processing records the event as seen and silently swallows the retry (Hookdeck).
Test all of this with failure injection during the pilot. Point the webhook at an endpoint that returns 500 for the first two attempts, then 200. Point it at an endpoint that sleeps past the provider's timeout. Send the same event twice and confirm your database shows one shipment email, not two. Verify signatures by replaying a captured request with a tampered body and confirming your code rejects it. If the provider cannot give you a sandbox with controllable failure modes, that is a finding, and you should write it down next to the price.
Antonio Evans
