Invalidation Hooks for Business Data

A webhook flow has four parts: the source registers your HTTPS endpoint, it fires a POST with a payload describing what changed, your handler verifies and deduplicates, then you delete the affected cache keys and let the next read repopulate them.

Verification is the part teams skip. Public webhook endpoints accept requests from anyone, so the three main attack vectors are man-in-the-middle reads of unencrypted payloads, forged requests, and replay of captured requests (Webhook Security Vulnerabilities Guide). The standard defense is an HMAC over the raw request body using a shared secret, compared against a signature header. The guide uses Shopify's X-Shopify-Hmac-Sha256 header as the canonical example and notes that providers differ in algorithm, so your implementation must match theirs.

Three practical rules follow from that:

1. Hash the raw bytes. If your framework's JSON parser re-serializes the body before you compute the HMAC, the signature will not match, and you will be tempted to disable verification.
2. Use a constant-time comparison, not ==, on the digest.
3. Reject anything outside a short timestamp window (five minutes is a common default) and store seen event IDs for at least that window to stop replays.

The handler must also be idempotent. Webhook providers retry on non-2xx responses and sometimes deliver duplicates, so evicting the same key twice must be harmless. The correct action on receipt is usually eviction, not writing the payload into the cache: the payload may be partial, and re-reading from the source gives you one code path that is always authoritative.

import hmac, hashlib, time

def verify(raw_body: bytes, signature: str, timestamp: str, secret: bytes, seen: set) -> bool:
    # 1. freshness window
    if abs(time.time() - int(timestamp)) > 300:
        return False
    # 2. HMAC over raw bytes
    mac = hmac.new(secret, timestamp.encode() + b"." + raw_body, hashlib.sha256).hexdigest()
    if not hmac.compare_digest(mac, signature):
        return False
    # 3. replay protection by event id
    event_id = hashlib.sha256(raw_body).hexdigest()
    if event_id in seen:
        return False
    seen.add(event_id)
    return True

Where webhooks fail: providers without signing support, providers that batch events hours late, providers that silently drop deliveries after repeated failures, and any source you do not control. A webhook you cannot verify is worse than a TTL, because it looks authoritative.

Change data capture takes a different route. Instead of the application telling you about changes, you read the database's internal log: PostgreSQL's write-ahead log, MySQL's binlog, SQL Server's transaction log. Debezium captures those log entries, converts them into events, and publishes them to Kafka, and it does not query tables at all (Understanding Change Data Capture with Debezium).

That design choice is the whole value proposition. Polling tables for updated_at > last_run misses updates when timestamps overlap, duplicates rows on retries, cannot see deletes, and loads the production database. Log-based capture sees inserts, updates and deletes as a stream of events in commit order, and because Kafka holds the events, multiple consumers can each maintain their own cache, search index or analytics store from one feed (Change Data Capture with Kafka and Debezium).

The canonical cache use case is documented by Debezium itself. Hibernate's second-level cache is shared across sessions, so an entity cached there goes stale when a batch job or a DBA updates the row directly, because the ORM never sees the change. CDC-driven invalidation evicts the affected cache region on UPDATE and DELETE in near real time, and Hibernate reloads the entity from the database on the next request (Automating Cache Invalidation With Change Data Capture). The packaged Quarkus extension scans the JPA metamodel at build time, registers handlers for the matching tables, and evicts regions on update or delete (Hibernate Second-Level Cache Invalidation).

Where CDC fails: schema changes need planning, backfilling history is not trivial, and the pipeline is a system, not a script, so it needs monitoring and owned on-call (Understanding Change Data Capture with Debezium). It also only covers data whose writes reach a log you can read. If a partner's API is the source of truth, there is no log to tap.

Decide per data source, not per system. One application routinely needs both, and a single entity can need both (push for the primary path, TTL as a backstop).

The fastest way to apply this is to sort each source into one of four buckets:

• Push, high stakes: prices, stock levels, entitlements and permissions, fraud flags. Use CDC if you own the database, webhooks if a vendor provides them.
• Push, low stakes but high churn: feature flags, catalog metadata. CDC is cheap once the pipeline exists.
• Pull, slow reference data: country lists, tax codes, currency tables, plan definitions. A daily TTL is generous.
• Pull, no events available: third-party pricing feeds, partner inventories, exports from a warehouse. TTL plus a manual bust button.

Two tiebreakers are worth writing into your design doc. First, if a wrong value would cause a charge, a refund, or a permission grant, push. Second, if you cannot verify who sent an event, pull and accept the lag, because an unverified webhook turns a small staleness problem into a security problem (Webhook Security Vulnerabilities Guide).

TTL sizing is arithmetic, not intuition. Measure the change rate over a representative week, then set the TTL so the expected staleness window matches what the business tolerates.

The rule: TTL = tolerable staleness, capped by your re-fetch budget. If a product catalog record changes 3 times per hour and the business tolerates 10 minutes of staleness, a 10-minute TTL gives you an average staleness of 5 minutes and a worst case of 10, and requires at most 6 fetches per hour per hot key.

Worked example: a supplier price feed with 200,000 SKUs, a 15-minute TTL, and 4% of SKUs read in any given 15-minute window. Your origin sees about 8,000 re-fetches per window, roughly 32,000 per hour. Halve the TTL and you double that load for a 7.5-minute worst case; if the origin starts rate-limiting, the shorter TTL is not free.

Three habits make TTLs behave:

• Jitter the expiry. Add a random fraction (say up to 10%) to each TTL so a large batch of keys written at the same time does not all expire on the same second.
• Serve stale on error. If the origin is down, return the expired value with a staleness marker rather than failing the request. A TTL is a budget, not a hard contract.
• Alert on origin error rate, not on cache miss rate. Miss rate is a design property; a spike in origin errors is an incident.

For LLM-side caches, the same arithmetic applies and the economics are worth watching. Anthropic's prompt cache charges a premium on writes against a discount on reads, so a cached prefix needs at least two hits to break even, and its lifetime is measured from the start of the request that wrote or read it, not from when the response finished (Prompt caching). A four-minute streaming response can eat most of a five-minute window before your follow-up request even starts.

A real deployment shows the ceiling. One security platform serving agent workflows raised its cache hit rate from 7% to 84% and cut overall LLM costs by 59% against what the same token volume would have cost at full input rates, partly by moving a static system prompt to a one-hour TTL while keeping per-session breakpoints at five minutes (How We Cut LLM Costs by 59% With Prompt Caching). That is the pattern: long TTL where content is stable, short TTL where it churns.

Most production systems end up with two or three layers, and the layering is what makes push pipelines safe.

Push plus a long TTL backstop. Every entry written by the CDC consumer gets a TTL of, say, 24 hours. Normal operation evicts on change; the TTL catches the event you missed during a broker outage. This converts "stale forever" into "stale for a bounded window", which is the property that makes push debuggable.

Versioned keys. Store the value under a key that includes a source version or updated-at hash, and let a pointer entry move to the new key. A missed invalidation now leaves an old key that ages out naturally, and you never overwrite a value a concurrent reader is using.

Read-through with a grace period. On a miss, fetch from the origin, then write with TTL. On an expired entry, serve it while triggering a background refresh. Combined with push eviction, reads stay fast and freshness comes from events.

Webhook to queue to worker. Terminate the webhook at a thin endpoint that verifies the signature, writes the event to a queue, and returns 200 immediately. A worker does the eviction. This keeps your handler inside the provider's timeout and lets you replay the queue after a bug.

For teams running LLM features on top of this data, the cache layer is only half the problem; the other half is that every model behind it has a different API surface, streaming mode and pricing. Quanta Inference (https://quantaintelligence.ai/inference) puts text, image, video, audio and embedding models behind one OpenAI-compatible key with prepaid dollar pricing, which keeps the freshness logic in your code and out of your provider integrations.

You cannot improve staleness you do not measure. Two metrics cover most of it:

• Event lag: time from source commit timestamp to cache eviction, reported as p50 and p99. Anything above your stated freshness SLO is a bug.
• Stale-serve rate: share of reads that returned a value older than the SLO, either because an event was missed or a TTL expired under load.

Track these per source, not fleet-wide. A global average hides the one table that stopped emitting events three days ago. The same lesson appears in LLM analytics: a prompt template change that breaks prefix stability can silently halve a hit rate and only show up later as a cost spike, so alert on per-app drops rather than a blended number (Cache Hit Rate (LLM)).

A simple, durable implementation is to have every write carry a source_updated_at alongside the value, and have the reader compare it against the freshness SLO for that entity. That gives you the stale-serve rate for free, and it works whether the value arrived by webhook, CDC event or TTL refresh.

Finally, test the failure paths. Kill the Kafka broker and confirm the TTL backstop still serves reads. Replay a week of webhook deliveries and confirm your handler is idempotent. Send a request with a valid signature and a two-hour-old timestamp and confirm it is rejected. These three tests catch most of the incidents this architecture produces.