What is a database-backed webhook delivery system, and when does a monolith need one?
An outgoing webhook looks like a single HTTP POST, and the simplest implementation treats it that way: a request handler commits a change, then calls the customer's endpoint inline. This has two failure modes. If the process crashes or the endpoint times out after the commit, the event is lost. If the call happens before the commit and the transaction later rolls back, the customer is told about a change that never existed. Both are hard to detect and harder to explain to the customer.
A database-backed webhook delivery system replaces the inline call with a durable job pipeline. The business change and a record of the event are written in the same transaction, so an event exists only if its change exists. Background workers inside the same deployable read pending deliveries, send them, record each attempt, and schedule retries. The relational database you already run serves as the queue, the audit log, and the source of truth for delivery state.
This design suits teams with one deployable and a relational database, whose event volume does not yet justify running and operating a separate message broker. You get durability and observability without adding infrastructure.
Two expectations should be stated up front. Delivery is at-least-once: a receiver can see the same event more than once and must deduplicate by event ID. Ordering is not free. You either pay for it with reduced throughput or explicitly leave it out.
How does the transactional outbox pattern work for outgoing webhooks?
The core problem the outbox solves is the dual write. If you commit the order and then publish the webhook, a crash between the two loses the event. If you publish first and the transaction rolls back, a customer receives an event for something that never happened. The fix is to write the event row in the same database transaction as the business change. Either both commit or neither does.
The data model
Use two tables with different lifecycles:
- events is immutable. It holds event ID, type, payload and
created_at. The payload is serialized once, so every retry sends identical bytes under the same ID. - deliveries has one row per (event, endpoint). It holds status, attempt count,
next_attempt_at, last response code, a truncated response body and an error class such as timeout, DNS failure, TLS error or non-2xx.
The worker loop
- Claim. In a short transaction, select due deliveries with
SELECT ... FOR UPDATE SKIP LOCKEDwhere your database supports it. This lets several workers pull disjoint batches without blocking each other. Pushnext_attempt_atforward as a lease, then commit. - Call. Make the HTTP request outside any transaction. Holding row locks or a pooled connection open while a slow endpoint stalls will starve the rest of the app.
- Record. In a new transaction, write the outcome and either mark the row delivered or schedule the next attempt. If a worker dies mid-call, the lease expires and the row becomes claimable again. This is the reason delivery is at-least-once.
Keeping deliveries separate from events gives you two things. Customers get a per-endpoint delivery log showing what was sent and how their server responded. Operators get a replay tool: resetting a delivery's status and next_attempt_at resends the original event without regenerating or mutating it.
-- PostgreSQL. Immutable events: payload serialized once so every retry sends identical bytes.
CREATE TABLE events (
id uuid PRIMARY KEY,
type text NOT NULL,
payload text NOT NULL,
created_at timestamptz NOT NULL DEFAULT now()
);
-- One mutable row per (event, endpoint): this is both the delivery log and the replay handle.
CREATE TABLE deliveries (
id bigserial PRIMARY KEY,
event_id uuid NOT NULL REFERENCES events(id),
endpoint_id bigint NOT NULL,
status text NOT NULL DEFAULT 'pending', -- pending | delivered | failed
attempt_count int NOT NULL DEFAULT 0,
next_attempt_at timestamptz NOT NULL DEFAULT now(),
last_response_code int,
last_response_body text, -- truncated by the worker
error_class text, -- timeout | dns | tls | non_2xx
UNIQUE (event_id, endpoint_id)
);
CREATE INDEX deliveries_due ON deliveries (next_attempt_at) WHERE status = 'pending';
-- Claim (short transaction): lock a disjoint batch, push next_attempt_at forward as a lease.
UPDATE deliveries
SET next_attempt_at = now() + interval '2 minutes',
attempt_count = attempt_count + 1
WHERE id IN (SELECT id FROM deliveries
WHERE status = 'pending' AND next_attempt_at <= now()
ORDER BY next_attempt_at LIMIT 50
FOR UPDATE SKIP LOCKED)
RETURNING id, event_id, endpoint_id;
-- (HTTP call happens here, with no transaction or row lock held.)
-- Record outcome in a new transaction; $1 = response body, $2 = delivery id.
UPDATE deliveries
SET status = 'delivered', last_response_code = 200,
last_response_body = left($1, 1024), error_class = NULL
WHERE id = $2;
A PostgreSQL schema for immutable events and per-endpoint deliveries, plus the claim query that uses FOR UPDATE SKIP LOCKED with a lease, and the separate update that records the outcome after the HTTP call.
How should you sign webhooks with HMAC?
Every endpoint gets its own randomly generated secret. Store it encrypted at rest with an application-level key, not in plaintext next to the URL. A per-endpoint secret limits the blast radius: if one customer's secret leaks, you rotate it without touching anyone else.
Sender. At delivery time, serialize the payload once and keep those exact bytes. Take the current Unix timestamp and build the signed string as timestamp + "." + body. Compute HMAC-SHA256 over it with the endpoint's secret and hex-encode the result. Send the signature and the timestamp in separate headers, for example Webhook-Signature and Webhook-Timestamp, and send the same bytes you signed as the request body. Including the timestamp in the signed input means an attacker cannot capture a valid request and attach a fresh timestamp to it.
Receiver. Read the raw request body before any JSON parsing. Recompute the HMAC and compare it to the header value with a constant-time function such as hmac.compare_digest in Python or crypto.timingSafeEqual in Node. Then reject the request if the timestamp is more than about five minutes from the receiver's clock.
Rotation. During rotation, keep two secrets active and send one signature per secret in the header. Receivers accept a match on either, which gives them time to deploy the new secret before you retire the old one.
Trade-offs. Most "invalid signature" reports come from receivers that parse the JSON and re-serialize it. Key order and whitespace change, so the bytes no longer match. Put this in your documentation. The replay window also relies on both clocks being synced with NTP. A tight window rejects legitimate requests from servers whose clocks have drifted, and a loose window gives an attacker more time to replay a captured request.
import hashlib, hmac, json, time
TOLERANCE_SECONDS = 300 # replay window; assumes both sides sync clocks with NTP
def sign(secret: bytes, timestamp: str, body: bytes) -> str:
# Sign "timestamp.body" over the exact bytes that go on the wire
return hmac.new(secret, timestamp.encode() + b"." + body, hashlib.sha256).hexdigest()
# --- Sender ---
def build_request(payload: dict, active_secrets: list[bytes]):
body = json.dumps(payload, separators=(",", ":")).encode() # serialize once, keep the bytes
ts = str(int(time.time()))
# During rotation active_secrets holds old and new: one signature per secret
sigs = ",".join(sign(s, ts, body) for s in active_secrets)
headers = {"Content-Type": "application/json",
"Webhook-Timestamp": ts, "Webhook-Signature": sigs}
return headers, body # send exactly these body bytes
# --- Receiver ---
def verify(raw_body: bytes, headers: dict, secret: bytes) -> bool:
# raw_body must be the unparsed request bytes, never re-serialized JSON
ts = headers.get("Webhook-Timestamp", "")
if not ts.isdigit():
return False
expected = sign(secret, ts, raw_body)
candidates = headers.get("Webhook-Signature", "").split(",")
# Constant-time compare; accept a match on any signature in the header
if not any(hmac.compare_digest(expected, c) for c in candidates):
return False
# Reject stale (or far-future) timestamps to limit replays
return abs(time.time() - int(ts)) <= TOLERANCE_SECONDS
if __name__ == "__main__":
old, new = b"whsec_old_placeholder", b"whsec_new_placeholder"
headers, body = build_request({"event": "order.paid", "id": "ord_123"}, [old, new])
print(verify(body, headers, old), verify(body, headers, new)) # True True
reserialized = json.dumps(json.loads(body)).encode() # adds spaces after separators
print(verify(reserialized, headers, new)) # False: bytes changed
A Python sender that signs "timestamp.body" with HMAC-SHA256 and sends one signature per active secret during rotation, and a receiver that verifies the raw body with a constant-time comparison and a five-minute staleness check. The final lines show that re-serialized JSON fails verification.
How should webhook retries and backoff be designed?
replay a captured request. Five minutes is a common compromise.
Classifying responses
A retry only helps if the failure is transient. Timeouts and connection errors usually are. A malformed payload or a revoked credential is not, and retrying it wastes worker time.
| Response | Class | Action |
|---|---|---|
| Timeout, connection refused, TLS error | Retryable | Schedule next attempt |
| 5xx | Retryable | Schedule next attempt |
| 429 | Retryable | Honor Retry-After if present |
| Most other 4xx | Terminal | Mark failed, surface to the customer |
| 2xx | Success | Mark delivered |
Backoff
Use exponential backoff with jitter and a cap. Without jitter, every delivery that failed during a receiver outage retries at the same moment and hits the receiver again as it recovers. When Retry-After is present, treat it as a minimum delay, clamped to your cap.
Illustrative estimate (assumed inputs: 30s base, doubling, 6-hour cap, 72-hour window, jitter ignored): the first 10 retries take about 8.5 hours. Ten more at the 6-hour cap add 60 hours. That gives 21 attempts over roughly 68.5 hours. After that, mark the delivery failed and show it in the customer's dashboard with a manual redeliver option.
A long window covers weekend outages, but it keeps stale events queued and delays the failure signal. A short window surfaces problems quickly, but it drops events the receiver could still have processed.
At-least-once semantics
If a worker crashes after the POST succeeds but before the status update commits, the event is sent again. Put a stable event ID in both the payload and a header so receivers can deduplicate.
import random
from datetime import datetime, timedelta, timezone
from email.utils import parsedate_to_datetime
BASE, CAP = 30, 6 * 3600 # seconds: 30s base, 6-hour cap
WINDOW = timedelta(hours=72) # bounded retry window
def classify(status=None, network_error=False):
if network_error: # timeout, refused, TLS failure
return "retryable"
if 200 <= status < 300:
return "success"
if status >= 500 or status == 429:
return "retryable"
return "terminal" # most 4xx: retrying will not help
def parse_retry_after(value, now):
# Retry-After is either delta-seconds or an HTTP date
if value is None:
return None
if value.strip().isdigit():
return int(value)
try:
return max(0, (parsedate_to_datetime(value) - now).total_seconds())
except (TypeError, ValueError):
return None # unparseable: fall back to backoff
def next_attempt_at(first_sent_at, retry_number, retry_after_header=None):
now = datetime.now(timezone.utc)
ceiling = min(CAP, BASE * 2 ** (retry_number - 1))
delay = random.uniform(ceiling / 2, ceiling) # jitter spreads retries out
hint = parse_retry_after(retry_after_header, now)
if hint is not None:
delay = max(delay, min(hint, CAP)) # minimum delay, clamped to cap
at = now + timedelta(seconds=delay)
if at - first_sent_at > WINDOW:
return None # window exhausted: mark failed, surface it
return at
A Python sketch that classifies delivery outcomes, computes capped exponential backoff with jitter, treats Retry-After as a clamped minimum delay, and stops scheduling once the 72-hour retry window is exhausted.
Can a webhook delivery system guarantee ordering?
Not globally, at any reasonable cost. Strict global ordering means a single serial pipeline: one event in flight across all endpoints, so one slow or failing customer stalls every other customer. Most systems drop that idea immediately.
The practical version is ordering per key, usually the endpoint or the resource ID such as order:123. A worker may claim a delivery only when no earlier delivery with the same key is pending or in flight. Different keys still run in parallel.
The cost is head-of-line blocking. If an endpoint returns 500s and your retry schedule spans many hours, every later event for that key waits behind the failing one for the whole window. When the first event finally exhausts its retries, you have to choose between skipping it, which breaks the ordering promise, and halting the key until someone intervenes. Per-resource keys contain the damage better than per-endpoint keys, but they do not remove it.
| Approach | What the receiver can rely on | Cost to you | Work for the receiver |
|---|---|---|---|
| Per-key serial delivery | Events for a key arrive in order | Head-of-line blocking, more complex claim queries | Little |
| Unordered + sequence number or timestamp | Can detect stale events | Monotonic counter per resource | Store the last-seen value and discard older events |
| Thin events ("order 123 changed") | Fetches current state, so order does not matter | Extra read load on your API | One API call per event |
Unordered delivery with a per-resource sequence number is usually the best default. Thin events suit receivers that already call your API.
How do you protect the monolith from slow endpoints and SSRF?
Every outbound request is a call into infrastructure you do not control, so treat each one as potentially slow, huge or hostile.
Bound every request
Set separate, short connect and read timeouts (a few seconds each) plus an overall deadline. An endpoint that accepts the connection and then trickles bytes can otherwise hold a worker indefinitely. Read only the first few kilobytes of the response body for logging, then close the connection. You need the status code, not a 50 MB error page. Run deliveries on a dedicated worker pool or queue with its own concurrency limit. When a large customer's endpoint goes down and the backlog grows, password-reset emails and billing jobs keep moving.
Disable failing endpoints
Track consecutive failures per endpoint. After a threshold, open a circuit and pause deliveries, or auto-disable the endpoint after a longer failure window. Retrying a dead endpoint wastes worker time and delays healthy deliveries in the same pool. When you disable an endpoint, notify its owner. Give them a re-enable action and a way to replay the events that queued up while it was off, so disabling it costs them a delay rather than lost data.
Defend against SSRF
Customers supply the URLs, which makes your server a proxy into your own network. Take these steps:
- Require HTTPS.
- Resolve DNS at connect time, not just at registration, and refuse private, loopback and link-local addresses, including cloud metadata addresses. Checking only at registration lets an attacker re-point the DNS record later (DNS rebinding). Connect to the IP you validated.
- Do not follow redirects by default. A redirect can send the request to an internal host that bypasses your address checks.
import ipaddress, socket, ssl, time
from urllib.parse import urlsplit
CONNECT_TIMEOUT, READ_TIMEOUT, DEADLINE, MAX_READ = 3.0, 5.0, 10.0, 4096
def deliver(url: str, payload: bytes) -> tuple[int, bytes]:
u = urlsplit(url)
if u.scheme != "https" or not u.hostname:
raise ValueError("webhook URL must use https")
port = u.port or 443
# Resolve at send time, not registration time, so a re-pointed DNS record is caught.
infos = socket.getaddrinfo(u.hostname, port, type=socket.SOCK_STREAM)
for *_, sockaddr in infos:
ip = ipaddress.ip_address(sockaddr[0])
# is_global is False for private, loopback, link-local (incl. 169.254.169.254) etc.
if not ip.is_global or ip.is_multicast:
raise PermissionError(f"blocked address {ip} for {u.hostname}")
start = time.monotonic()
# Connect to the IP we just validated; never let the HTTP layer re-resolve the name.
raw = socket.create_connection(infos[0][4][:2], timeout=CONNECT_TIMEOUT)
ctx = ssl.create_default_context() # verifies the cert against the original hostname
with ctx.wrap_socket(raw, server_hostname=u.hostname) as s:
s.settimeout(READ_TIMEOUT) # per-recv limit against byte-trickling endpoints
path = (u.path or "/") + (f"?{u.query}" if u.query else "")
head = (f"POST {path} HTTP/1.1\r\nHost: {u.hostname}\r\n"
f"Content-Type: application/json\r\nContent-Length: {len(payload)}\r\n"
"Connection: close\r\n\r\n")
s.sendall(head.encode() + payload)
buf = b""
# Read at most MAX_READ bytes and stop at the overall deadline.
while len(buf) < MAX_READ and time.monotonic() - start < DEADLINE:
chunk = s.recv(MAX_READ - len(buf))
if not chunk:
break
buf += chunk
parts = buf.split(b" ", 2)
if len(parts) < 2 or not parts[1].isdigit():
raise ValueError("no valid HTTP status line")
# A 3xx is recorded as the result; the Location header is never followed.
return int(parts[1]), buf
A standard-library Python delivery function that requires HTTPS, resolves and validates DNS at send time, connects to the validated IP with hostname-verified TLS, enforces connect, read and overall time limits, caps how much response it reads, and treats redirects as final responses.
When should you avoid an in-app webhook system, and what are the alternatives?
The in-app approach earns its place because it costs almost nothing to start. It reuses the database, transactions and deploy pipeline you already run, so there is no new infrastructure to operate. Replay and the delivery log also fall out of it as product features, since every attempt is already a queryable row.
The costs grow quietly:
- Outbox inserts and status updates add write load on the primary.
- Workers poll constantly, even when nothing is pending.
- Concurrent workers contend for row locks.
- A high-churn table bloats and needs regular vacuum or index maintenance.
- Workers killed mid-attempt during deploys resend events that had already been delivered.
- Webhook throughput scales with the app, so a customer's flood of events competes with your request traffic.
Track these so you can see the limits coming: backlog depth, age of the oldest pending event, success rate per endpoint, retry volume, and a retention policy that keeps the table bounded.
| Concern | In-app (database-backed) | Dedicated queue or delivery service |
|---|---|---|
| New infrastructure | None | Broker or service to run and monitor |
| Atomic with business writes | Yes, same transaction | Needs an outbox relay |
| Load isolation | Shares the primary | Scales independently |
| Replay and audit log | Already queryable | Must be built or retained separately |
| Failure blast radius | Can slow the app | Contained to the delivery tier |
Consider migrating when any of these persist:
- Oldest pending age keeps rising despite adding workers.
- Outbox queries show up among your top database load.
- Vacuum cannot keep up with the table's churn.
- One noisy endpoint's retries degrade delivery for everyone else.
When you move, keep the outbox table as the handoff point and relay from it into the queue, so delivery stays atomic with your business writes.
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to webhook delivery system, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Changing a shared variable is one edit and one review instead of editing every service's
.envby hand.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.