Using Signed Webhooks to Trigger Agent Runs Safely

The mechanics are narrow enough to state precisely. Compute HMAC-SHA256 over the raw bytes as received, not over a re-serialized object. Compare against the header value with a constant-time function. Reject anything missing a signature, with a malformed signature, or with a timestamp outside the tolerance.

The raw body detail is the one that silently breaks integrations. JSON parsers reorder nothing but they do change whitespace and number formatting, so a re-serialized body will produce a different digest than the one the sender signed. Guides on Shopify webhooks call this out explicitly and recommend timing-safe comparison reliable Shopify webhooks. A deeper write-up on HMAC secrets makes the same two points, plus a warning that plain string equality leaks timing information that lets an attacker guess a signature byte by byte HMAC secrets explained.

The timestamp is what makes the signature short-lived. Webhooks.fyi documents the pattern: the provider concatenates the timestamp with the payload, signs the result, and sends both. The listener checks the timestamp is within an acceptable window (their example uses 3 to 5 minutes), then recomputes the digest over payload plus timestamp replay prevention. Svix recommends a five-minute tolerance as the common choice: old enough to absorb clock skew, short enough that a captured request expires before it is useful webhook security best practices. Keep listener clocks synced with NTP, because a drifting clock turns into silent 403s.

Here is the shape of a verifier you can lift. It assumes you can reach the raw body, which in Express means capturing it in a middleware before the JSON parser runs.

const crypto = require('crypto');

function verify(rawBody, signatureHeader, secret, toleranceSeconds = 300) {
  const parts = Object.fromEntries(signatureHeader.split(',').map(p => p.split('=')));
  const timestamp = Number(parts.t);
  if (!Number.isFinite(timestamp)) return { ok: false, reason: 'no_timestamp' };
  if (Math.abs(Date.now() / 1000 - timestamp) > toleranceSeconds) {
    return { ok: false, reason: 'stale' };
  }
  const signed = `${parts.t}.${rawBody}`;
  const expected = crypto.createHmac('sha256', secret).update(signed).digest('hex');
  const a = Buffer.from(expected, 'hex');
  const b = Buffer.from(parts.v1 || '', 'hex');
  if (a.length !== b.length || !crypto.timingSafeEqual(a, b)) {
    return { ok: false, reason: 'bad_signature' };
  }
  return { ok: true };
}

Notice what the function does not do: it does not parse the payload, does not touch the database and does not call a model. That is the point. It returns a boolean and a reason, and the caller decides.

At-least-once delivery is the norm, not an edge case. Svix's own material states it plainly: providers retry until they receive a successful response, so a server that processes an event and then crashes before returning 200 will see it again idempotency and deduplication. Hookdeck lists the same causes: the same webhook arriving multiple times is expected, and the application must handle it webhook idempotency guide.

The naive check-then-insert has a race. Two copies arriving at two workers can both pass the "have I seen this" test before either records the event. The fix is to let the datastore arbitrate with a unique constraint and treat the conflict as a duplicate webhook idempotency. One insert, one winner:

CREATE TABLE processed_events (
  event_id   text PRIMARY KEY,
  payload_sha256 text NOT NULL,
  status     text NOT NULL DEFAULT 'queued',
  received_at timestamptz NOT NULL DEFAULT now()
);

INSERT INTO processed_events (event_id, payload_sha256)
VALUES ($1, $2)
ON CONFLICT (event_id) DO NOTHING
RETURNING event_id;

No row returned means you have seen this event. Return 200 or 202 and do nothing else. A row returned means you own the event and may enqueue it.

One subtlety worth building in: dedup on the provider event ID alone lets an attacker who learns an event ID pre-register it and block the genuine delivery. Storing the payload hash alongside the ID closes that, because a mismatch between the stored hash and the incoming body is a signal, not a duplicate. Practical guides suggest keeping the dedup record for longer than the provider's retry window; Prismatic points out that a one-hour TTL against a 72-hour retry period protects only the first hour webhook processing idempotency.

The single biggest performance and reliability change is refusing to do the work in the request. A handler that verifies, deduplicates and enqueues can finish in tens of milliseconds. A handler that starts an agent finishes in seconds to minutes, and the sender will time out and retry long before it returns.

Documented consumer timeouts vary enough to design conservatively: Svix expects responses within 15 seconds and GitHub within 10 seconds webhook reliability and retries. Neither is a budget you want an LLM loop inside. The Shopify-oriented guide recommends responding in under 500 milliseconds and moving heavy work to a worker queue reliable Shopify webhooks. Returning 202 Accepted is the honest status: the request was accepted, the work has not happened yet.

Stripe and its peers give a rough sense of how long a sender will keep trying: Stripe retries for up to three days, Svix uses a fixed eight-attempt schedule and GitHub does not auto-retry at all webhook reliability and retries. Since you cannot know which behavior you will get from a given provider, the queue is what makes the two cases equivalent, because retries land on a durable job rather than a half-finished model call.

The worker does the second half of the trust decision. It reads the job, re-checks that the payload hash matches the record, and only then assembles the agent context. Guides on payment webhooks make the same structural point for money movement: verify signature, validate the payload, handle idempotency, then process handling payment webhooks reliably. An agent run deserves the same skepticism.

The signature proves who sent the event. It says nothing about what the event is allowed to cause. That is a separate decision, and it should be a lookup, not a model judgment.

Map event type to tools with an explicit table. Anything not in the table gets no tools and no model call.

Three rules make this hold. First, the mapping lives in configuration that a code review can read, not in a prompt. Second, tool credentials are scoped to the run: a read-only token for a read-only event. Third, the verified event body is untrusted input to the model, so tool arguments get validated the same way you would validate a user form. An invoice number arrived inside a signed payload, which means the provider believed it, not that it is safe in a shell command or a SQL string.

This is where the signature boundary earns its keep. If a forged request could reach the agent, it could name an event type that unlocks tools. Because the type is read from a body you have verified, and the tool set is selected from your table rather than from the model's own reasoning, forgery and prompt injection collapse into the same failure mode: the run proceeds with only the tools your table already permitted.

An agent loop is not a single API call. It is a sequence of calls whose length depends on the model's output, which means an event you accept can turn into an unknown number of billable requests. Anyone operating agents needs a per-event ceiling.

Set three numbers per event type and enforce them in the worker, not in the prompt:

• Step ceiling. A hard cap on tool calls or model turns per run. When it is hit, the run stops and hands its state to a human or a dead-letter path.
• Token and dollar budget. A prepaid account makes this concrete: track spend per run and abort when the counter exceeds the ceiling.
• Wall clock. A maximum runtime, so a retried job cannot overlap with a stuck one.

Add a rate limit per sender and per event type at the enqueue step. A flood of valid signatures is still a flood, and the timestamp window means a burst of captured requests from the last five minutes can all pass verification at once.

A worked example shows the arithmetic. Suppose a run averages eight model turns at roughly 2,000 input and 400 output tokens per turn. A ceiling of twelve turns and a budget of three times the observed average gives you a boundary that normal runs never touch and runaway runs hit quickly. When the budget trips, the worker marks the event budget_exceeded, records the spend, and does not retry. Prismatic's guidance on failure states applies directly: track processing, completed and failed as separate states, and reclaim stuck work with a lease so a slow worker and a recovery worker do not both act webhook processing idempotency.

Building that plumbing, then wiring every provider into one client, is usually two jobs. Quanta Inference gives you text, image, video, audio and embedding models through one OpenAI-compatible API key with prepaid pricing in US dollars, which suits a worker that needs to cap spend per run Quanta Inference.

Keep these two lists next to the code. The first covers the receiving endpoint; the second covers the worker.

Endpoint checklist

• Reject if the signature header is absent, malformed or not hex or base64 of the expected length.
• Reject if |now - timestamp| exceeds the tolerance (300 seconds is common).
• Compute the HMAC over raw bytes, with the timestamp inside the signed string.
• Compare in constant time.
• Insert the event ID and payload hash with a conflict clause before any other work.
• Enqueue, then return 202 with the event ID.
• Log the outcome and reason for every request, accepted or not, with the signature omitted.

Worker checklist

• Re-verify the payload hash against the stored record.
• Select tools from the event-type table; default to none.
• Issue credentials scoped to that tool set and expire them with the run.
• Enforce step, budget and wall-clock ceilings, and write the spend to the event record.
• On failure, retry a bounded number of times with jitter, then move the job to a dead-letter queue with the error and attempt count.
• Never delete a failed event silently; alert on dead-letter depth and age.

The observability advice from the webhook reliability literature is worth taking literally here: track endpoint 4xx and 5xx rates, acknowledgement latency, worker latency and success rate, and dead-letter size and age reliable Shopify webhooks. With an agent behind the queue, add spend per event type to that list, because cost is the failure mode that does not page anyone until the invoice arrives.