Idempotency in Distributed Systems: Making Retries Safe

What idempotency means and why retries make it necessary

An operation is idempotent when applying it once or many times with the same input leaves the system in the same state and gives the caller an equivalent response. "Equivalent" matters. A repeated DELETE may return 404 instead of 204, which the HTTP specification allows because the specification judges idempotency by the effect on server state, not by the response bytes. Clients that branch on status codes often need more than that, so idempotency-key designs usually replay the original response.

Idempotency is different from safety. A safe operation is read-only and changes nothing. Every safe method is idempotent, but an idempotent method is not necessarily safe.

Retries cannot be avoided. When a request times out, the caller cannot tell whether the request never arrived, the work failed, or the work succeeded and only the reply was lost. The only way to make progress is to send the request again. That makes delivery at-least-once, and idempotent processing turns at-least-once delivery into an effectively-once outcome.

Brokers and stream processors that advertise exactly-once semantics usually guarantee it only inside their own boundary. When a consumer charges a card or sends an email, that external side effect falls outside the guarantee.

Any team that owns operations with side effects needs this, as long as their callers, SDKs, proxies or queues retry.

How idempotency keys work end to end

The lifecycle starts on the client. Before the first attempt, the caller generates a key, typically a random UUID, once per logical operation: "pay invoice 4411," not "this HTTP request." The client stores that key with the pending operation and sends the same value on every retry, including retries after a process restart. If the key is regenerated per attempt, the server sees each retry as new work and deduplication does nothing.

On the server, the first step is a lookup for a dedup record matching the key. If none exists, the server inserts one marked in progress, executes the work, and then records the final status along with the serialized response: status code, headers that matter, and body. When a retry arrives, the server finds the completed record and replays the stored response byte for byte without executing anything. The client cannot tell the replay from the original, which is the point.

Scope and fingerprinting

Keys need a namespace. Scope them per tenant, and usually per endpoint, so that two customers who happen to generate the same value don't collide, and so that a key used for a refund can't match a stored charge.

The record also stores a fingerprint, such as a hash of the normalized request payload. If a key comes back with a different fingerprint, the server rejects it as a client error instead of replaying a response that belongs to other parameters. Normalization is where this goes wrong. Serialize JSON with sorted keys so that reordered fields hash the same. Exclude client-generated timestamps and request IDs from the hash, because they change on every retry and would make every legitimate retry look like a conflict.

Copy+---------------------+          +-----------------------------------+
| Client              |          | Server                            |
| key = UUID, once per|  POST    | scope = tenant + endpoint + key   |
| logical operation;  |--------->| fp = hash(normalized payload)     |
| persisted and reused| same key | - JSON keys sorted                |
| on every retry      |          | - timestamps, request IDs omitted |
+---------------------+          +-----------------------------------+
           ^                                       |
           |                                       v
           |            +-------------------------------------------+
           |            | Dedup store: look up record by scoped key |
           |            +-------------------------------------------+
           |                    |               |               |
           |                no record       fp match       fp differs
           |                    v               v               v
           |             +-------------+ +-------------+ +-------------+
           |             | insert      | | replay      | | reject      |
           |             | IN_PROGRESS | | stored      | | (client     |
           |             | execute work| | response    | |  error):    |
           |             | store status| | byte for    | | same key,   |
           |             | headers,body| | byte        | | new params  |
           |             +-------------+ +-------------+ +-------------+
           |                    |               |               |
           +--------------------+---------------+---------------+
                    response: original, replay, or error

An idempotency key is created once per logical operation and reused on every retry; the server looks it up within its tenant and endpoint scope and executes, replays, or rejects based on the payload fingerprint.

Implementing a transactional dedup record

Store the dedup record in the same database as the business data, and write it in the same transaction. Put a unique constraint on (scope, key). The scope is the tenant or API client, so two customers who happen to pick the same key never collide. Each row stores a status (in_progress or completed), the request fingerprint, the serialized response and a lease expiry.

When a request arrives, insert the row as in_progress. If the insert succeeds, this request owns the key. It performs the business write, stores the response, marks the row completed and commits. The order row and the dedup row then become visible together or not at all. If the insert fails on the unique constraint, read the existing row:

• Different fingerprint: the client reused the key for a different request, so reject it.
• completed: replay the stored response.
• in_progress: a concurrent duplicate is still running. Return a retryable conflict, or wait on a row lock until the first request commits.

Keys get stuck in_progress when the claim has to commit before the work finishes. That happens when the handler calls something outside the database, and then the process crashes. A lease expiry lets a later request take over the key. A naive timeout is risky, though: the original worker may only be paused or waiting on a slow downstream call, and taking over its key means the operation runs twice. To prevent that, increment a version number on every takeover, and make the final completed update succeed only if the version still matches. Run a reconciliation job that checks the downstream system's actual state before it re-executes anything.

Keeping keys in a separate cache breaks this guarantee. The cache and the database can't commit atomically. A crash between the two writes leaves either a recorded key with no business write, so retries are blocked or get a false success, or a business write with no key, so the next retry duplicates it. Cache eviction can also silently drop keys while clients are still retrying.

Copy-- PostgreSQL. The key table lives in the same database as orders.
CREATE TABLE idempotency_keys (
  scope       text        NOT NULL,  -- tenant or API client id
  key         text        NOT NULL,  -- client-supplied idempotency key
  status      text        NOT NULL CHECK (status IN ('in_progress', 'completed')),
  fingerprint bytea       NOT NULL,  -- hash of method, path and canonical body
  response    jsonb,                 -- stored so duplicates can be replayed
  version     integer     NOT NULL DEFAULT 1,  -- incremented on every takeover
  lease_until timestamptz NOT NULL,
  PRIMARY KEY (scope, key)           -- the unique constraint that serializes duplicates
);

BEGIN;
-- Claim the key. A concurrent insert of the same (scope, key) waits until the
-- first transaction ends, then gets no row back.
INSERT INTO idempotency_keys (scope, key, status, fingerprint, lease_until)
VALUES ($1, $2, 'in_progress', $3, now() + interval '30 seconds')
ON CONFLICT (scope, key) DO NOTHING
RETURNING version;
-- No row returned: SELECT status, fingerprint, response for the key, then
-- reject (fingerprint differs), replay (completed) or return 409 (in_progress).

INSERT INTO orders (id, tenant_id, amount_cents) VALUES ($4, $1, $5);

-- Finish only if this worker still owns the key. 0 rows updated means another
-- worker took over after the lease expired, so ROLLBACK.
UPDATE idempotency_keys
SET status = 'completed', response = $6
WHERE scope = $1 AND key = $2 AND version = $7;
COMMIT;  -- the order row and the completed key become visible together

-- Takeover, only needed when the claim was committed before external work:
-- bump the version so a paused original worker's final UPDATE matches 0 rows.
UPDATE idempotency_keys
SET version = version + 1, lease_until = now() + interval '30 seconds'
WHERE scope = $1 AND key = $2
  AND status = 'in_progress' AND lease_until < now()
RETURNING version;

A PostgreSQL dedup table keyed on (scope, key): the claim, the business write and the version-checked completion all run in one transaction, and a version-bumping takeover handles keys left stuck in progress.

Idempotent message consumers and side effects beyond your database

Message queues raise the same problem from the other side. Most brokers deliver at-least-once: if a consumer processes a message but crashes before acknowledging it, the broker redelivers it. The consumer has to treat every message as a possible duplicate.

Deduplicate on the message ID, in the same transaction

Give every message a stable ID assigned by the producer, not by the broker on delivery. Record that ID in a processed-messages table inside the same database transaction as the consumer's state change. A redelivered message then fails the uniqueness check and is acknowledged without being applied again. If you store the ID separately, for example in a cache or in a second transaction, you recreate the gap described above: a crash between the two writes leaves the state and the dedup record out of sync.

The producer side has a matching hazard. A service that writes to its database and then publishes an event can fail between the two steps, which loses the event or, after a retry, publishes it twice. The transactional outbox closes this gap. Write the event to an outbox table in the same transaction as the business change, and let a separate relay publish rows from that table. The relay can still publish the same event more than once, so consumers still need to deduplicate. What the outbox guarantees is that the event and the database write cannot diverge.

Side effects you don't control

Emails, payments and third-party APIs sit outside your transaction. If a provider accepts an idempotency key, derive one from your own key and pass it downstream. The provider then handles deduplication for that call.

If the provider has no such support, record your intent before making the call, such as "charge pending for order X." Mark the record complete once the call succeeds. A reconciliation job looks for records that stayed pending and checks the provider's actual state before deciding whether to retry. Without this, a timeout leaves you with no reliable way to know whether the side effect happened.

Copyimport json
import sqlite3

conn = sqlite3.connect("orders.db")
conn.executescript("""
CREATE TABLE IF NOT EXISTS processed_messages (message_id TEXT PRIMARY KEY);
CREATE TABLE IF NOT EXISTS orders (order_id TEXT PRIMARY KEY, status TEXT);
CREATE TABLE IF NOT EXISTS outbox (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    event_id TEXT UNIQUE,
    payload TEXT,
    published INTEGER DEFAULT 0
);
""")

def handle(message):
    # message["id"] is set by the producer, so a redelivery carries the same ID.
    with conn:  # One transaction. Commits on success, rolls back on exception.
        cur = conn.execute(
            "INSERT OR IGNORE INTO processed_messages (message_id) VALUES (?)",
            (message["id"],))
        if cur.rowcount == 0:
            return  # Duplicate. Already applied, so the caller only acknowledges.

        conn.execute(
            "UPDATE orders SET status = 'paid' WHERE order_id = ?",
            (message["order_id"],))

        # The outbox row commits atomically with the state change.
        # A separate relay publishes unpublished rows, and may do so more than once,
        # so downstream consumers dedupe on event_id the same way.
        conn.execute(
            "INSERT INTO outbox (event_id, payload) VALUES (?, ?)",
            (f"order-paid-{message['order_id']}",
             json.dumps({"type": "OrderPaid", "order_id": message["order_id"]})))
    # Acknowledge to the broker only after this function returns, i.e. after commit.

handle({"id": "msg-7f3a", "order_id": "order-1042"})
handle({"id": "msg-7f3a", "order_id": "order-1042"})  # Redelivery: no second update or event

A consumer that records the producer-assigned message ID, applies the state change and writes an outbox event in one database transaction, so a redelivered message is detected and skipped.

Alternatives: natural idempotency without keys

Not every operation needs a key store. Often you can make the operation itself safe to repeat, so a duplicate request finds nothing left to do.

Conditional writes. Add a version column and make each update conditional: "set status to paid and version to 8 where id is 42 and version is 7." The first attempt matches the row and changes it. A retry matches nothing, because the version has already moved on. The caller treats "zero rows affected" as "already applied" and reads back the current state. Compare-and-set works the same way in key-value stores that support it.

Upserts on a business identifier. If a record has a natural unique identity, such as an order number, an invoice ID from an upstream system, or a (tenant, external reference) pair, put a unique constraint on it. Then insert-or-update against that constraint. A duplicate collapses into the existing row, and the uniqueness check comes from the database instead of your application code.

Absolute state assignments. "Set balance to 110" or "mark order shipped" gives the same result however many times it runs. "Add 10" does not. Changing a relative operation into an absolute one removes the duplication risk. The cost is that you now need to know the target state, which usually requires a conditional write to avoid overwriting a concurrent change.

Use the natural options wherever the domain allows them. Keep idempotency keys for creates that have no stable identity and for operations whose original response must be returned exactly as it was the first time.

Copy-- PostgreSQL. Three ways to make a write safe to repeat without a key store.

CREATE TABLE orders (
  id           bigint PRIMARY KEY,
  tenant_id    bigint NOT NULL,
  external_ref text   NOT NULL,       -- ID from the upstream system
  status       text   NOT NULL,
  total_cents  bigint NOT NULL,
  version      int    NOT NULL DEFAULT 1,
  UNIQUE (tenant_id, external_ref)    -- the business identity
);

-- 1. Conditional write (compare-and-set on a version column).
-- The first attempt matches and bumps the version. A retry matches 0 rows,
-- so the caller reads the row back and treats it as already applied.
UPDATE orders
SET    status = 'paid', version = 8
WHERE  id = 42 AND version = 7;

-- 2. Upsert on the business identifier.
-- A duplicate create turns into an update of the existing row
-- instead of adding a second order.
INSERT INTO orders (id, tenant_id, external_ref, status, total_cents)
VALUES (1001, 7, 'INV-2026-0042', 'pending', 2500)
ON CONFLICT (tenant_id, external_ref)
DO UPDATE SET total_cents = EXCLUDED.total_cents;

-- 3. Absolute assignment instead of a relative change.
-- Unsafe to retry: each run adds again.
--   UPDATE orders SET total_cents = total_cents + 1000 WHERE id = 42;
-- Safe to retry: gives the same result every time. The version check
-- stops it from overwriting a change made by someone else in between.
UPDATE orders
SET    total_cents = 3500, version = version + 1
WHERE  id = 42 AND version = 8;

PostgreSQL statements that make retries harmless without an idempotency key: a version-checked update, an upsert on a unique business identifier, and an absolute assignment that replaces a relative increment.

Retry policy and testing that pair with idempotency

Idempotency on the server only pays off if clients retry with discipline. A sound policy has four rules:

• Bound the attempts. Cap retries per request and set an overall deadline. Unbounded retries turn a brief outage into a sustained overload.
• Back off exponentially, with jitter. Doubling the delay gives a struggling dependency room to recover. Randomizing it stops thousands of clients from retrying in lockstep, which would recreate the spike that caused the failure.
• Retry only errors that may be transient. Timeouts, connection resets, and throttling or unavailable responses are candidates. Validation failures and authorization errors will fail the same way every time, so retrying them only adds load.
• Never retry a non-idempotent call that lacks a key. A timeout tells you nothing about whether the write happened. Without a key, a retry can apply it twice.

Clients also have to handle keys correctly. Generate the key once per logical operation, persist it with the pending request, and send the same key on every retry. Two mistakes are common, and neither produces an error. Minting a fresh key per attempt turns every retry into a new operation, which brings back duplicates. Reusing one key across distinct operations makes the server return a stale response for work it never performed.

Tests should target the failure that matters most: the side effect commits, but the response never arrives. Inject a fault at exactly that point, then retry. Replay identical requests concurrently and in sequence. Redeliver messages, including out of order. In each case, assert that database state, emitted events, and external calls match a single execution, and that every replay returns the original response.

When to avoid it, and the impact on cost and engineering time

Not every endpoint needs a key store. Read-only operations such as GETs, searches, and reports change nothing, so a retry costs only compute. Operations that are already naturally idempotent, such as an absolute PUT or an upsert on a business identifier, get the guarantee from their own semantics. Low-stakes side effects, such as a duplicate analytics event or a cache warm, can tolerate an occasional repeat. Adding keys to any of these buys little and adds a component that can fail.

What it costs

Every key needs durable storage. How long you keep it is a business decision, not just a tuning knob, because a retry that arrives after the key expires runs again. Match retention to the longest realistic retry window, including clients that come back online after hours or days.

Each request also adds a lookup and a write to the hot path. Under concurrent traffic the unique index can become a point of contention. Measure latency and lock behaviour in your own environment rather than assuming the overhead is negligible.

Replayed responses are snapshots. A client that retries may receive a status that has since changed.

What it saves

For operations where duplicates matter, the trade is usually favourable. Fewer duplicate charges reach production, so there are fewer incidents and less manual reconciliation work. Teams can also enable retries across service boundaries without negotiating, endpoint by endpoint, whether a repeat is safe.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to idempotency distributed systems, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Changing a route's URL or method automatically notifies the services that consume it — no hunting through repos for callers.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• Consumers learn about a breaking change at review time rather than from production errors.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.