Webhook Delivery Design: Signatures, Retries and Replay Protection

What webhook delivery design is and why it is at-least-once

A webhook is an HTTP request a sender makes to a URL the receiver registered, announcing that something happened. Webhook delivery design is the contract around that request. It covers what the payload contains, how the receiver proves it came from the sender, what response counts as "accepted," and what the sender does when it does not get one. Both sides have to agree on these rules, because neither can fix delivery problems alone.

The contract is at-least-once because the sender cannot always tell whether a delivery succeeded. Suppose the receiver commits the event to its database, then the connection drops before the 200 response arrives. Or the receiver takes longer than the sender's timeout. From the sender's side, these look the same as a request that never landed. The only safe choice is to retry, so the receiver sees the same event twice. Retries also break ordering. Event B can succeed while event A is still being retried, so A arrives after B. Exactly-once delivery would require the two systems to share transactional state, which a plain HTTP call does not provide. Receivers must therefore handle duplicates and out-of-order events by design.

This matters to backend engineers writing handlers and to platform engineers running delivery infrastructure. Webhooks are a good fit for pushing events to third parties or loosely coupled services that cannot hold a persistent connection or poll efficiently.

Copy  SENDER                                      RECEIVER
 +---------------------+                     +---------------------+
 | event source        |                     | webhook handler     |
 | delivery queue      |                     | event store (DB)    |
 +----------+----------+                     +----------+----------+
            |                                           |
            | 1. POST event A (signed, id=A)            |
            |------------------------------------------>|--> commit A
            |                                           |
            |      200 OK lost / sender timeout         |
            |  x<---------------------------------------|
            |                                           |
            | 2. POST event B (id=B)                    |
            |------------------------------------------>|--> commit B
            |<------------------------------------------|
            |      200 OK                               |
            |                                           |
            | 3. RETRY event A (same id=A)              |
            |------------------------------------------>|--> A again:
            |<------------------------------------------|    duplicate,
            |      200 OK                               |    arrives
            |                                           |    after B
            v                                           v
 +------------------------------------------------------------------+
 | Sender cannot tell "never arrived" from "processed, reply lost"  |
 | -> it must retry -> receiver sees duplicates and reordering      |
 +------------------------------------------------------------------+

A lost or late 200 response forces the sender to retry, so the receiver commits event A twice and receives it after event B, which is why webhook delivery is at-least-once and unordered.

How webhook signature verification works

The sender computes an HMAC, typically HMAC-SHA256, over a string that combines a timestamp and the exact request body bytes, for example timestamp + "." + body. It keys the HMAC with a secret shared with the receiver and sends both the timestamp and the hex or base64 digest in request headers. Including the timestamp in the signed material is what makes replay windows enforceable later, because an attacker cannot change the timestamp without invalidating the signature.

The receiver must recompute the HMAC over the raw bytes it received, before any JSON parsing. Parsing and re-serializing changes key order, whitespace, number formatting and Unicode escaping, so the receiver ends up computing a different digest. Many frameworks parse bodies automatically. You need to capture the raw buffer explicitly, usually with a route-specific body handler. Proxies and API gateways that decompress, re-encode or normalize payloads cause the same mismatch. Charset conversion between hops does too.

Compare digests with a constant-time function such as hmac.compare_digest in Python or crypto.timingSafeEqual in Node.js. A naive == returns at the first differing byte. An attacker who can measure response times can then guess a valid signature one byte at a time.

Copyimport hashlib
import hmac
import json
import time

WEBHOOK_SECRET = b"whsec_example_placeholder_not_a_real_secret"
TOLERANCE_SECONDS = 300  # replay window, enforced because the timestamp is signed


def verify_webhook(raw_body: bytes, timestamp_header: str, signature_header: str) -> bool:
    # Accept only plain ASCII digits so the bytes we sign match what the sender signed.
    if not (timestamp_header.isascii() and timestamp_header.isdigit()):
        return False
    if abs(time.time() - int(timestamp_header)) > TOLERANCE_SECONDS:
        return False

    # Rebuild the signed string from the exact bytes received: timestamp + "." + raw body.
    # raw_body must be the unparsed request buffer, never re-serialized JSON.
    signed_payload = timestamp_header.encode("ascii") + b"." + raw_body
    expected = hmac.new(WEBHOOK_SECRET, signed_payload, hashlib.sha256).hexdigest()

    # Constant-time comparison: it does not return early at the first differing byte.
    return hmac.compare_digest(expected.encode("ascii"), signature_header.strip().encode("utf-8"))


def handle_request(raw_body: bytes, headers: dict) -> dict:
    if not verify_webhook(raw_body, headers.get("X-Webhook-Timestamp", ""),
                          headers.get("X-Webhook-Signature", "")):
        raise PermissionError("invalid webhook signature")
    # Parse JSON only after the signature over the raw bytes has been checked.
    return json.loads(raw_body)


if __name__ == "__main__":
    body = b'{"event":"invoice.paid","id":"evt_123"}'
    ts = str(int(time.time()))
    sig = hmac.new(WEBHOOK_SECRET, ts.encode() + b"." + body, hashlib.sha256).hexdigest()
    print(handle_request(body, {"X-Webhook-Timestamp": ts, "X-Webhook-Signature": sig}))

A receiver that checks the timestamp window, recomputes HMAC-SHA256 over the timestamp and raw body bytes, compares digests with hmac.compare_digest, and parses JSON only after verification succeeds.

Replay attack protection with timestamps and event IDs

A valid signature proves the payload came from the sender and was not modified. It says nothing about when it was sent. Anyone who captures a delivery, whether through a logging proxy, a leaked request dump, or a compromised intermediary, can resend it byte for byte, and it will verify every time.

Timestamp plus event ID

The first control is a timestamp included in the signed content. If the timestamp sits outside the signature, an attacker can simply update it. The receiver rejects any request whose timestamp falls outside a tolerance window around its own clock. This limits replays to a short window but does not prevent them inside it.

The second control is a unique event ID that the receiver stores after processing. A repeat ID is acknowledged and discarded. The ID store must keep entries for at least the tolerance window. Otherwise a replay could arrive after its ID was evicted but while its timestamp is still accepted.

Choosing the window

• Short windows reject legitimate requests when clocks drift or when queued deliveries arrive late.
• Long windows give attackers more time to replay and increase the size of the ID store.

Retries add a further requirement. Senders usually re-sign each attempt with a fresh timestamp, so every retry passes the window check. Only deduplication catches them. For that reason, ID retention must cover the sender's maximum retry horizon, not just the window.

Illustrative estimate (assumed numbers, not a benchmark): Store size = event rate × retention seconds × bytes per entry = 50/s × 259,200 s (72 h) × 100 B ≈ 1.3 GB

Copyimport hashlib, hmac, time

SECRET = b"whsec_example_placeholder"   # placeholder, never a real secret
TOLERANCE_S = 300                       # accept timestamps within +/- 5 minutes
RETRY_HORIZON_S = 72 * 3600             # sender's maximum retry span (assumed)
RETENTION_S = max(TOLERANCE_S, RETRY_HORIZON_S)

seen = {}  # event_id -> expiry; use a shared, persistent store in production

def verify(body: bytes, timestamp: str, event_id: str, signature: str) -> str:
    # Sign timestamp and ID together with the body so neither can be altered.
    signed = f"{timestamp}.{event_id}.".encode() + body
    expected = hmac.new(SECRET, signed, hashlib.sha256).hexdigest()
    if not hmac.compare_digest(expected, signature):
        return "reject: bad signature"

    # Bound how old (or how far in the future) a request may be.
    now = time.time()
    if abs(now - int(timestamp)) > TOLERANCE_S:
        return "reject: timestamp outside window"

    # Evict expired IDs, then dedupe. Retries re-signed with fresh timestamps
    # pass the window check, so this is the only control that catches them.
    for eid, exp in list(seen.items()):
        if exp < now:
            del seen[eid]
    if event_id in seen:
        return "ack: duplicate, discarded"
    seen[event_id] = now + RETENTION_S
    return "ok: process event"

A receiver that verifies an HMAC over the timestamp, event ID and body, enforces a tolerance window, and deduplicates event IDs for the longer of the window and the sender's retry horizon.

Implementing an idempotent, fast-acknowledging receiver

The receiver's HTTP handler should do as little as possible: verify the signature, check the timestamp, write the raw payload to a durable queue or table, and return 2xx. All business logic runs later in a worker. This keeps response times well under the sender's timeout. If processing runs inline, a slow downstream call causes timeouts, the sender retries, and you receive duplicates while you are already under load.

Never acknowledge before the enqueue is durable

A 2xx tells the sender to stop retrying. If you acknowledge first and then lose the event to a crash, a failed write, or an in-memory queue that dies with the process, the event is gone, and nothing will redeliver it. Return 2xx only after the queue or database has confirmed the write. If the write fails, return 5xx so the sender's retries can recover it.

Deduplicate with a unique constraint

Store each event ID in a table with a unique constraint. Insert it in the same database transaction as the side effect, such as an order status change. Both then commit or roll back together. If you check for the ID in a separate step before writing, two concurrent deliveries can both pass the check. When the insert hits the constraint, the event has already been handled, so return 2xx. A 4xx or 5xx would only trigger more retries.

Don't trust delivery order

Retries and parallel delivery mean events can arrive out of order. Store a resource version or source timestamp with each record, and skip any update older than the one you already hold.

Copy-- PostgreSQL. Run by the async worker, not the HTTP handler.
-- The handler has already verified the signature and durably stored the raw event.

CREATE TABLE processed_events (
  event_id     text PRIMARY KEY,  -- unique constraint is the dedup mechanism
  processed_at timestamptz NOT NULL DEFAULT now()
);

CREATE TABLE orders (
  id      text PRIMARY KEY,
  status  text NOT NULL,
  version bigint NOT NULL  -- resource version from the sender's payload
);

-- Per event: record the ID and apply the side effect in ONE transaction,
-- so both commit or both roll back.
BEGIN;

WITH claimed AS (
  -- A concurrent duplicate blocks here until the first transaction finishes,
  -- then inserts nothing. No separate check-then-insert race.
  INSERT INTO processed_events (event_id)
  VALUES ('evt_01HZX3K9Q2')
  ON CONFLICT (event_id) DO NOTHING
  RETURNING event_id
)
UPDATE orders
SET    status = 'paid', version = 7
WHERE  id = 'ord_4821'
  AND  version < 7                         -- discard stale, out-of-order updates
  AND  EXISTS (SELECT 1 FROM claimed);     -- skip if this event was already handled

COMMIT;

-- Zero rows updated means a duplicate or a stale event. Both are success:
-- mark the queue item done. Never retry them or report them as errors.

A worker transaction in PostgreSQL. It claims the event ID through a unique constraint and applies the order update in the same statement, and a version check makes duplicate and out-of-order events no-ops.

Sender-side retries, dead-letter handling and secret rotation

Retry schedule and classification

Classify each delivery attempt before scheduling the next one. Timeouts, connection errors, 5xx responses and 429 are retryable because the receiver may recover. Most 4xx responses are not: a 400 or 404 will fail again with the same payload. Honour a Retry-After header on 429 when the receiver sends one.

Use exponential backoff with jitter and a hard cap on attempts. Illustrative estimate, assumed inputs: a 1-minute base delay, doubling each attempt, capped at 60 minutes, with 8 retries. That gives 1, 2, 4, 8, 16, 32, 60 and 60 minutes, roughly 3 hours in total. Full jitter picks a random delay between zero and each computed value. This spreads out events that failed together during the same outage, so they don't all retry at the same moment.

Dead letters

After the final attempt, write the event to a dead-letter store with its last status code and error. Expose manual and API-driven redelivery so receivers can replay events after they fix the problem. Redelivery is safe because receivers deduplicate by event ID.

Secret rotation

Allow at least two active secrets per endpoint. During the overlap window, sign each request with both secrets and send both signatures. The receiver accepts either one, moves to the new secret, and then the old secret is revoked. No deliveries fail during the switch.

Payload design, operations and when to avoid webhooks

Payload shape determines how far a receiver can trust the request body. A thin event carries only the event type and resource ID, and the receiver fetches current state from your API. A fat event carries the full resource, so the receiver needs no callback. Because retries and out-of-order arrival are normal, however, a fat payload may describe a state that has already been superseded.

On the operations side, keep per-endpoint delivery logs that record each attempt's timestamp, response status, latency and truncated response body, and offer manual redelivery. Customers can then debug their own failures instead of opening tickets. If an endpoint fails persistently over a sustained period, auto-disable it and notify its owner. Otherwise it consumes retry capacity indefinitely and its errors drown out signals from healthy endpoints.

Customer-supplied URLs are an SSRF vector. Resolve DNS before connecting and reject loopback, private and link-local addresses, including cloud metadata endpoints. Connect to the IP you already validated so that DNS rebinding cannot redirect the request, and don't follow redirects blindly.

When to avoid webhooks

Webhooks are a poor fit in four cases: when consumers need strict ordering, when they need consistently low latency, when receivers cannot expose a public endpoint, and when the traffic is a high-volume internal stream. Alternatives include polling with cursors, message brokers or event streams for internal traffic, and long-lived connections such as WebSockets or server-sent events.

Impact on development cost and engineering time

If you do keep webhooks, budget for the infrastructure that makes them reliable. Most of it is ordinary, but it is not free. The receiver needs a dedup store keyed by event ID, with a retention window at least as long as the sender's retry horizon. The sender needs a durable queue, a retry scheduler with backoff, a dead-letter store, per-attempt delivery logs, a redelivery API or console, and key management that supports two active secrets during rotation. Each piece has a build cost plus ongoing storage and on-call ownership.

These pieces cost less when you design them together. Some examples:

• Event IDs do double duty. A stable event ID that is included in the signed payload is the dedup key, the replay check and the lookup key in delivery logs.
• Duplicates become harmless. Signed timestamps bound replay windows, and idempotent handlers turn retries into no-ops instead of double charges or duplicate emails.
• Receivers can recover on their own. Delivery logs plus self-serve redelivery let a receiver fix a missed event without filing a ticket.

Illustrative estimate (assumed numbers, not a benchmark or measured result)

Monthly hours saved = (incidents × hours per incident × reduction) + (tickets × hours per ticket × reduction)

All inputs below are assumed for illustration only. They are not measurements from any real team; substitute your own figures.

• Assumed incident load (illustrative): 4 duplicate or missed-event incidents a month, assumed to take 6 hours each, with an assumed 75% reduction
• Assumed ticket load (illustrative): 40 "resend this event" tickets a month, assumed to take 0.5 hours each, with an assumed 90% reduction

With these assumed inputs, the illustrative result is (4 × 6 × 0.75) + (40 × 0.5 × 0.9) = 18 + 18 = 36 hours a month. Compare a figure calculated from your own inputs against your one-time build estimate and ongoing maintenance cost to find the payback period.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to webhook delivery design, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.