An Outbox Table for Agent-Triggered Side Effects

One table, written inside the same transaction as the agent's state change. The key design decision is what goes in the key column.

CREATE TABLE agent_outbox (
  id              BIGSERIAL PRIMARY KEY,
  tenant_id       UUID        NOT NULL,
  operation       TEXT        NOT NULL,          -- e.g. 'create_invoice'
  entity_id       TEXT        NOT NULL,          -- the business object
  intent_version  INTEGER     NOT NULL DEFAULT 1,
  idempotency_key TEXT        NOT NULL,
  endpoint        TEXT        NOT NULL,
  payload         JSONB       NOT NULL,
  status          TEXT        NOT NULL DEFAULT 'pending',
  attempts        INTEGER     NOT NULL DEFAULT 0,
  next_attempt_at TIMESTAMPTZ NOT NULL DEFAULT now(),
  locked_until    TIMESTAMPTZ,
  last_error      TEXT,
  downstream_id   TEXT,
  created_at      TIMESTAMPTZ NOT NULL DEFAULT now(),
  CONSTRAINT uq_outbox_key UNIQUE (idempotency_key)
);

CREATE INDEX ix_outbox_due
  ON agent_outbox (next_attempt_at)
  WHERE status = 'pending';

Four things earn their place here. The unique constraint on idempotency_key means a duplicate insert fails loudly at commit time rather than silently creating a second intent, which is the same first-writer-wins primitive that idempotency middleware relies on (Idempotency Patterns). intent_version handles the case where a human or an agent revises the intent before it has been dispatched: bumping the version produces a new key, so the revision is a distinct operation rather than a silent overwrite. next_attempt_at drives scheduling so no external timer is needed. The partial index keeps the claim query cheap as the table fills with completed rows.

That last point matters more than it looks. Freezing the payload at insert time means the relay sends exactly the arguments that were authorized, not whatever the model produced on a later turn.

The key must be deterministic for the same intent and different for different intents. Identical arguments alone do not establish identical intent, because a caller may legitimately want two identical resources (Agent Tool Call Retries). So include something that identifies the logical step.

import hashlib, json

def derive_key(tenant_id: str, operation: str, entity_id: str,
               intent_version: int, approved_args: dict) -> str:
    canonical = json.dumps(
        {
            "tenant": tenant_id,
            "op": operation,
            "entity": entity_id,
            "v": intent_version,
            "args": approved_args,
        },
        sort_keys=True,
        separators=(",", ":"),
    )
    return f"{operation}:{hashlib.sha256(canonical.encode()).hexdigest()[:40]}"

Three properties to check before you ship this. It is stable: the same inputs always produce the same key, so a relay crash and restart reuses it. It is scoped: tenant_id in the hash prevents cross-tenant collisions if you ever share a downstream account. It is human-readable enough to debug: the operation prefix tells an on-call engineer what the row is trying to do without decoding anything.

Monday.com's API documentation makes the same point from the provider side. Keys should be unique per intended operation and deterministic for the same intent, and the docs explicitly warn against generating a new key on each retry, because deduplication then fails (monday.com idempotency). Their example key format for webhook-driven mutations is event id plus operation name, which is the same shape as the derivation above.

If the downstream takes a client-chosen resource id rather than an Idempotency-Key header, use the hash as that id. Stripe stores the key alongside the result and replays the original response for repeats, which is the behavior you are asking for even when the vendor does not expose that header (Stripe idempotent requests).

The relay is a separate process from the agent. It claims due rows, dispatches them with the stored key, and records the outcome. Claiming must be atomic, because two relay workers racing on the same row produces exactly the duplicate effect the outbox was meant to prevent.

WITH due AS (
  SELECT id FROM agent_outbox
  WHERE status = 'pending' AND next_attempt_at <= now()
  ORDER BY next_attempt_at
  LIMIT 20
  FOR UPDATE SKIP LOCKED
)
UPDATE agent_outbox o
SET status = 'in_flight',
    locked_until = now() + interval '60 seconds',
    attempts = attempts + 1
FROM due
WHERE o.id = due.id
RETURNING o.*;

SKIP LOCKED lets each worker take a disjoint batch, so no row is dispatched twice by two relays. The lease (locked_until) covers the nasty case where a worker dies mid-call: the row becomes claimable again after the lease expires, and because the key has not changed, the redispatch is safe as long as the downstream enforces it.

Once claimed, the call itself is short. Send the key in the Idempotency-Key header, treat any non-2xx as a failure, and classify before scheduling a retry. The classification table below is the one that saves the most wasted work.

The 409 row is worth pausing on. Monday.com returns a conflict with a Retry-After header when a request with the same key is already being processed, so the relay should wait the indicated time and retry rather than treat it as an error (monday.com idempotency). Hookdeck's guidance draws the same line: 5xx, timeouts, connection refused, and DNS failures are retriable; 4xx except 429 are permanent and should go straight to the dead-letter state rather than wasting attempts (Hookdeck retry best practices).

One subtlety in the timeout row: a timeout after dispatch does not tell you whether the effect committed. The payment service may have charged the card even though the caller never saw a response. That is why the redispatch must carry the same key rather than a new one, and why treating a timeout as final failure is the most dangerous bug in the whole relay (Agent Tool Call Retries, Stripe idempotent requests).

The full flow in one picture:

agent transaction:  update state + insert outbox row (status=pending)
      |
      v
relay claim:        status=pending, locked_until=now()+60s, attempts+=1
      |
      v
dispatch:           POST endpoint with Idempotency-Key = row.idempotency_key
      |                      |
      v                      v
   2xx                  5xx / 429 / timeout
 mark succeeded         retryable=true?
 store downstream_id      |          |
                       yes|         no|
                          v          v
                  backoff + set   status=dead,
                  next_attempt_at  alert + replay tool

Dead rows are not garbage. A dead-letter state preserves the event, its error context, and the metadata needed to debug and replay it once the underlying problem is fixed, instead of letting a burst of failures block the healthy rows behind them (Dead-Letter Queues for Webhook Reliability, Webhook Reliability: Retry and Dead Letter Queues). Give the relay a replay command that resets a dead row to pending with attempts = 0 and keeps the key unchanged.

Fixed-interval retries hammer a struggling endpoint and slow its recovery. Exponential backoff increases the gap between attempts, and a cap keeps the schedule finite (Svix retry best practices, Mastering Webhook Retry Logic).

delay = min(cap, base * 2 ** attempts)
jitter = random.uniform(0, delay)      # full jitter
next_attempt_at = now() + jitter

Work the numbers. With a base of 2 seconds and a cap of 1 hour, attempt 5 waits at most 32 seconds, attempt 10 at most 17 minutes, and every attempt past 11 waits at most an hour. Ten attempts with full jitter fit inside roughly 3 to 4 hours of wall-clock time, which covers the redeploy-and-recover window of most downstream services. For comparison, Svix publishes a schedule that spans 8 attempts over roughly 24 hours: immediate, 5 seconds, 5 minutes, 30 minutes, 2 hours, 5 hours, 10 hours, then 10 hours (Svix retry best practices). If your downstream is a payment or ticketing API with SLA-backed recovery measured in minutes, a 24-hour spread is overkill; if it is a partner system that goes dark for hours, it is about right. Match the window to the recovery time you can actually observe, not to a default.

Jitter is not decoration. When many rows fail at once during an outage, they all back off on the same schedule and retry at the same instant, which can knock the recovering service over again. Full jitter spreads them across the whole interval; equal jitter guarantees at least half the delay while still dispersing the wave (Hookdeck retry best practices, Webhook Retry Best Practices for Sending Webhooks). The thundering herd is a real production failure mode, not a theoretical one.

The decision rule for choosing between strategies: use full jitter when many rows share the same downstream and failures correlate, equal jitter when you need a predictable floor for latency-sensitive work, and a published fixed schedule when external teams need to reason about your timing.

None of the above helps unless the receiving service enforces the key. A local transaction does not include the provider's commit, so the only way to close the gap is to carry the operation's identity through to a downstream contract that deduplicates, or to leave the outcome for reconciliation (Agent Tool Call Retries). Where the mutation and the deduplication record share a transactional store, commit them together; otherwise a crash between changing state and recording completion leaves a hole. The order matters too. A commit-then-call relay can lose the effect if it crashes between the commit and the call, which is precisely why the outbox row exists. If you skip the outbox and call the downstream first, a failure before your commit leaves a phantom effect the database never records (CodeToByte's outbox guide).

Testing is where most teams learn their relay is wrong. A checklist that catches the real failures:

• Kill the relay process after the request is sent but before the response is read, then restart it. The redispatch must carry the same key and the downstream must report one effect, not two.
• Insert the same outbox row twice from two concurrent transactions. The unique constraint must reject the second insert, not create a duplicate.
• Return 500 for five attempts, then 200. Confirm attempts increments, next_attempt_at grows, and the row lands in succeeded with a stored downstream_id.
• Return 400. Confirm the row goes to dead on the first attempt with no further dispatches.
• Change the intent version mid-retry and confirm the relay produces a different key for the revised intent.

For a full run against a real provider, point a test relay at a sandbox endpoint that records every Idempotency-Key it sees. You want to see repeats of the same key rather than a stream of new ones. Then use the same test to confirm that a model-driven retry, where the model asks for the tool again, reuses the row instead of inserting a new one. That check is the one that catches the orchestrator wiring the key to model output rather than to the durable step.

Once this is running, the relay is a good candidate for consolidation. If your agent pipeline also calls model endpoints for summarization, classification, or embedding of the outbox payloads, Quanta Inference puts text, image, video, audio, and embedding models behind one OpenAI-compatible API key, which removes a second set of credentials and retry policies from the relay box (https://quantaintelligence.ai/inference).

The outbox plus key design also pays off outside agents. The same table handles human-triggered mutations, batch imports, and webhook-driven automations from other systems, because the identity of the operation is derived from data rather than from who pressed the button. The pattern does not care whether the caller was a person or a model; it only needs one durable row per intended effect and one stable key per row.