Building a Webhook Retry System with a DLQ

Plain exponential backoff has a failure mode on the day it matters most. When a receiver has an outage, every delivery to that endpoint fails at roughly the same moment, backs off on the same curve, and retries in unison. The recovering receiver gets hit by a synchronized wave that can knock it over again. This is the thundering herd problem, and jitter solves it by adding a random offset to each delay so retries spread across the interval instead of arriving together (Hookdeck's outbound retry guide).

Three variants are common, and the choice is a real tradeoff:

The formulas above follow the comparison in the Hookdeck outbound guide. Whichever you choose, cap the maximum delay. Unbounded doubling produces absurd waits: with a one-second base, ten attempts means waiting over seventeen minutes between retries, and twenty attempts exceeds twelve days. A common pattern caps at one hour and continues hourly until the retry window closes.

A worked example makes the numbers concrete. Take a one-second base, a one-hour cap, full jitter, and a published schedule of eight attempts across roughly 24 hours:

• Attempt 1 fires immediately.
• Attempts 2 through 6 wait up to 1, 2, 4, 8, and 16 seconds respectively, each shortened by jitter.
• Later attempts stretch to minutes and then hours, always capped.
• The event either succeeds or is declared exhausted and moves to the DLQ.

That schedule shape mirrors the published Svix policy of eight attempts spread across about a day, moving to failure once the schedule is spent (Svix retry best practices). Providers disagree enough here that copying one is less useful than understanding the range: Stripe retries for up to three days with exponential backoff, Svix runs its fixed eight-attempt schedule, and GitHub does not automatically redeliver at all, offering manual redelivery within a three-day window (Devlume's webhook reliability analysis). The lesson is to publish your own exact schedule and to design as if your stingiest partner's behavior were the norm.

One more design decision hides in the cap: fix a total window, not just an attempt count. A slow schedule with no window can leave an event in flight for a week while the business fact it describes has long since been superseded.

Retries create the conditions for confusion on the receiver side, so your sender should make duplicates and replays cheap to detect. Three headers do most of the work.

A unique delivery identifier per attempt sequence lets the receiver deduplicate. GitHub sends X-GitHub-Delivery, and the same value is reused on redelivery so the receiver can recognize the repeat (GitHub's webhook best practices). A signature header built with HMAC-SHA256 over the raw body keeps the payload tamper-evident. A timestamp inside the signed payload lets the receiver reject stale messages, which is the core replay-prevention control, and the timestamp has to be part of the integrity check or an attacker can alter it (Webhooks.fyi on provider best practices).

HMAC is the dominant scheme, used by 65% of the webhooks studied by Webhooks.fyi, and good implementations share a few traits: they use SHA-256 or stronger, they include sensitive headers in the digest, and they provide documentation and replay tooling (Webhooks.fyi on HMAC). GitHub publishes a fixed test vector (a known secret, a known payload, and the expected signature) so implementers can check their verification code before trusting it in production (GitHub on validating deliveries).

The standard that ties this together is Standard Webhooks, an open specification with reference libraries in several languages. It defines header names, signature construction, and the replay window, and it exists precisely because every provider implementing webhooks differently makes integration worse for everyone (the Standard Webhooks spec). Kong ships a gateway plugin that validates incoming webhooks against it using HMAC-SHA256 secrets (Kong's Standard Webhooks plugin), which is a sign that the spec is landing in infrastructure rather than staying a document.

On your side as the sender, two details are easy to get wrong. First, sign the raw request body exactly as sent, not a re-serialized JSON object, or the receiver's recomputed digest will not match. Second, give the receiver a way to trigger a test and a replay so feedback stays close; the provider best-practice list calls that out directly (Webhooks.fyi).

A minimal signing routine looks like this:

import hmac, hashlib, time

def sign(body_bytes: bytes, secret: str, msg_id: str, ts: int = None) -> dict:
    ts = ts or int(time.time())
    signed = f"{msg_id}.{ts}.".encode() + body_bytes
    digest = hmac.new(secret.encode(), signed, hashlib.sha256).hexdigest()
    return {"webhook-id": msg_id, "webhook-timestamp": str(ts), "webhook-signature": digest}

The receiver recomputes the same digest from the raw body plus the two headers and rejects the request if the timestamp is outside its tolerance window.

Every retry system needs a floor. When the schedule is exhausted, the event cannot simply vanish, because the business fact it represents (a payment settled, an order shipped) still happened. The dead-letter queue is the durable record of deliveries that the primary path could not complete. It exists for three reasons: it preserves the event, it carries the diagnostic context needed to fix the cause, and it makes replay possible once the cause is fixed (Hookdeck's dead-letter queue guide).

A DLQ record that works in production carries more than the body. Include the endpoint ID, the event ID, the payload as sent, the signature headers used, the attempt count, the timestamp of each attempt, the final HTTP status or transport error, and a failure class (exhausted versus permanent). Operators triaging an incident will ask for exactly these fields, and reconstructing them from application logs at 2 a.m. is miserable.

Some messages should skip retries entirely and go straight to the DLQ, because retrying cannot help: a malformed payload, a handler bug that throws every time, or a deleted endpoint URL (Hookdeck's DLQ guide). Tagging these as permanent rather than exhausted matters, because the two classes want different runbooks.

Isolate the DLQ from the main pipeline in three ways. Cap DLQ depth so a misbehaving endpoint cannot fill the disk. Store DLQ payloads with the same encryption and retention policy as the primary store, because they contain live customer data. And give the DLQ its own alerts, because a queue nobody watches is a data loss incident waiting for an audit.

A schema sketch that has held up well:

dlq_record:
  id                 (uuid, primary key)
  endpoint_id        (which receiver)
  event_id           (stable business event id)
  payload            (raw bytes as sent)
  signature_headers  (as sent, for exact replay)
  attempts           (list of {ts, status, error, duration_ms})
  failure_class      (exhausted | permanent)
  first_seen_at / last_attempt_at
  replay_count       (default 0)
  state              (open | replaying | resolved | abandoned)

Do not delete DLQ rows when you replay them. Move them to a resolved state and link the replay record to the original. You will want that history the second time the same endpoint breaks.

Replay is not a backdoor into the queue; it is an operation with its own control surface, its own audit trail, and its own failure modes. Two shapes are worth building.

The first is single-event replay from an admin view: an operator opens a DLQ record, inspects the payload and error history, and clicks replay. The second is bulk replay filtered by time window or endpoint, which is what you actually need after an outage. Bulk replay must be rate-limited against the target endpoint, because dumping twenty thousand events at a receiver that just recovered reproduces the outage you are recovering from. Letting customers manually replay failed messages from the point an outage began turns a fixed mechanism into one teams can adapt to their own reliability needs (Svix retry best practices).

Replay must be safe to run more than once. The receiver deduplicates on the event ID, and your sender preserves the same event and message IDs on replay so that deduplication catches it. This matters because every webhook provider worth integrating operates at-least-once, and duplicates are expected, with idempotency being the receiving side's obligation (Hookdeck's idempotency guide). Prismatic makes the same point sharply: absolute state changes like SET status = 'paid' are retry-safe, while balance += amount is not, and the deduplication window must be longer than the sender's retry window or it protects only the first hour (Prismatic on idempotent processing).

A replay endpoint that fits these constraints:

@app.post("/admin/dlq/{record_id}/replay")
def replay(record_id: str, operator: str):
    rec = dlq.get(record_id)
    if rec.state not in ("open", "resolved"):
        return json_response({"error": "not replayable"}, 409)
    attempt = deliver(rec.endpoint_id, rec.payload, rec.signature_headers, rec.event_id)
    dlq.append_attempt(record_id, attempt)
    dlq.set_state(record_id, "replaying" if attempt.ok else "open")
    audit.log(operator=operator, action="dlq.replay", record=record_id, result=attempt.status)
    metrics.increment("dlq.replay.attempts", tags={"result": "ok" if attempt.ok else "fail"})
    return json_response({"status": attempt.status})

Note what the handler does not do: it does not regenerate the payload, and it does not mint a fresh event ID. A replay must be byte-identical to the original attempt or the receiver's deduplication and signature checks both become unreliable.

Silent failure is the default state of a webhook system that is not instrumented. Every request succeeds from the sender's point of view until a retry exhausts, and by then the event sits in the DLQ where nobody is looking. Observability has to cover the retry path itself, not just the HTTP endpoint. The practical corollary is the one GitHub states plainly: if your server goes down, you should redeliver the missed events once it is back up, which means you need a record of what was missed (GitHub's webhook best practices).

Track these signals, at minimum:

• Delivery success rate by status class (2xx, 4xx permanent, 4xx rate-limited, 5xx) per endpoint.
• Retry rate: retries scheduled per minute, and the ratio of retries to first attempts, which spikes first during a receiver outage.
• Retry exhaustion rate: deliveries that hit the end of the schedule, the leading indicator of DLQ growth.
• DLQ depth and, more importantly, the age of the oldest open record. A depth of 50 that is three days old is a worse signal than a depth of 5,000 that is thirty seconds old.
• Replay volume and replay success rate, because a replay that keeps failing means the root cause was never fixed.
• Acknowledgement latency at the receiver, so you can see a slow endpoint before it fails outright.

The age of the oldest DLQ record is the metric most teams omit and the one that catches silent failures. A steady DLQ depth with a growing oldest-age means events are arriving in the queue and nothing is leaving it, which is exactly the state where a partner integration quietly rots.

Alerting follows from a simple rule: alert on rates and on age, not on raw counts. A threshold on "DLQ depth above 100" fires every afternoon during a batch cycle and gets muted. A threshold on "retry exhaustion rate above 1% for ten minutes" or "oldest open DLQ record older than 24 hours" fires when something is actually wrong. Also alert when a single endpoint's success rate slips while everything else looks healthy, because that is often the first visible sign of a revoked credential or a deleted endpoint.

If you are building the monitoring layer alongside model-backed services (classifying failure reasons, summarizing alert storms, or triaging replay decisions), a single API key covering every model type keeps that glue small: Quanta Inference.

A DLQ without a runbook is just a slower way to lose data. Arguments about perfect policy can wait; the operator's sequence is short and it works.

The triage sequence:

1. Group open records by endpoint and failure class. Permanent failures on one endpoint usually mean a configuration change (revoked key, deleted URL) rather than a code bug.
2. For exhausted groups, check the receiver's status page and your own last successful delivery time to that endpoint.
3. Fix the cause, then replay a small sample of ten records and watch the success rate before replaying the rest.
4. Record the incident in the DLQ record's history, and set the state to resolved.

Pruning needs a policy too. Keep resolved records long enough to answer audit questions, then delete. Records that have been replayed repeatedly and keep failing should move to an abandoned state rather than cycling through open forever, and abandoned records of a permanent class should notify the customer or internal owner that the event will not be delivered.

One last habit: replay on a schedule, not only after incidents. Pick one low-stakes integration and run a synthetic failure (point it at a URL that returns 503 for a minute) on a regular cadence, then confirm that retries fire, the DLQ captures the event, and replay succeeds. The system you have never tested is the system that fails silently. If you would rather have this run as a standing practice than a side project, a specialist team can own it with you at Quanta Intelligence.