What are idempotency keys and why do POST requests need them?

An idempotency key is a unique value, usually a client-generated UUID sent in a request header, that identifies one logical operation. The server stores the key with the outcome of the first request. When the same key arrives again, the server returns that stored outcome and does not perform the operation a second time.

POST needs this because HTTP semantics do not treat it as idempotent. GET, PUT and DELETE are defined so that repeating them has the same effect as sending them once. POST is defined as "process this," and each call usually creates something new: a charge, an order, a shipment. Two identical POSTs mean two charges.

Duplicates rarely come from careless clients. They come from normal infrastructure behavior:

  • Timeouts. The client gives up after 10 seconds, but the server committed the payment at second 11. The client cannot tell whether the request failed or only the response was lost, so it retries.
  • Load balancer and proxy retries. Some proxies resend a request to another upstream when a connection resets. The first upstream may already have done the work.
  • Client SDK retries. Many HTTP libraries retry with backoff on network errors or 5xx responses, sometimes without the calling code knowing.
  • At-least-once message delivery. Queue consumers that call your API will see the same message again after a crash or a missed acknowledgment.

If your service creates payments, orders, bookings or any side effect that must not happen twice, you cannot prevent these retries. You can only make them safe to repeat.

How do idempotency keys work end to end?

The contract has two halves. The client generates a unique key, typically a random UUID, once per logical operation: one checkout or one transfer, not one HTTP attempt. It sends the key in a request header, conventionally Idempotency-Key, and reuses the same value on every retry of that operation. A client that mints a new key per attempt defeats the mechanism entirely.

The server scopes keys to the caller, usually the tenant or API credential, so two customers who happen to generate the same value cannot collide or read each other's responses. Keys are retained for a fixed window, often 24 hours to a few days. The window must outlast the client's longest realistic retry schedule. Beyond that point a reused key is treated as new, which keeps the table bounded.

A request then moves through four steps:

  1. Claim. Insert a row for (scope, key) along with a fingerprint of the request, such as a hash of the method, path and body. If the row already exists, skip to replay.
  2. Side effect. Perform the business write, such as creating the order.
  3. Commit. Store the response status and body on the key row.
  4. Replay. On retry, return the stored response unchanged.

The design depends on one rule. The key, the fingerprint, the business write and the stored response must commit in a single Postgres transaction. If they commit separately, a crash between them leaves either an order with no recorded key or a key with no order, and the next retry produces exactly the duplicate you were trying to prevent.

+-----------------+    POST /orders              +-------------------------+
| Client          |    Idempotency-Key: K1       | API server              |
| key K1 = UUID   |----------------------------->| scope = tenant/cred     |
| per operation,  |                              | fp = hash(method,       |
| reused on retry |<-----------------------------|       path, body)       |
+-----------------+    stored status + body      +------------+------------+
                                                              |
                                                              v
+------------------------------------------------------------------------+
| Postgres: ONE transaction per request                                  |
|                                                                        |
|  1. CLAIM    INSERT key row (scope, K1, fp)                            |
|   |                                                                    |
|   +-- row exists (retry)? ------------------> 4. REPLAY                |
|   |                                           return stored            |
|   v                                           response as-is           |
|  2. SIDE EFFECT  INSERT order row                                      |
|   |                                                                    |
|   v                                                                    |
|  3. COMMIT   UPDATE key row SET status, body                           |
|   |                                                                    |
|   v                                                                    |
|  TX COMMIT: key, fp, order, response land together or not at all       |
+------------------------------------------------------------------------+
  Key rows expire after the retention window (e.g. 24h to a few days);
  a key reused after that is treated as a new operation.

The client reuses one key per logical operation, and the server claims the scoped key, performs the business write and stores the response in a single Postgres transaction, so a retry either replays the stored response or finds nothing was committed.

How do you implement idempotency keys in Postgres?

The table needs six things beyond the key itself: scope (typically the tenant or API client, so two customers can't collide on the same key), request_fingerprint, status (in_progress or completed), locked_at, response_code and response_body, and created_at for expiry. Put the primary key on (scope, key). That constraint is the whole concurrency story: a check-then-insert lets two requests both see "no row" and both proceed, while a unique index cannot be raced.

The claim is a single INSERT ... ON CONFLICT (scope, key) DO NOTHING RETURNING .... If a row comes back, this request owns the key. If nothing comes back, someone else got there first, and the handler moves to the retry path. When a concurrent insert of the same key is still uncommitted, Postgres makes the second insert wait for that transaction to finish rather than guessing.

Inside the same transaction, perform the business write, serialize the response you are about to return, and update the key row to completed with that code and body. Commit once. locked_at matters when the claim must commit separately, which is covered below, because it lets you detect claims abandoned by a crashed worker.

The fingerprint is a hash of method, path, and a canonicalized body. It catches clients that reuse a key for a different request. Naive hashing produces false mismatches:

  • Key order: sort JSON object keys before hashing.
  • Defaulted fields: an omitted currency and an explicit "USD" differ unless you hash after applying defaults.
  • Client timestamps: these change on every retry, so exclude them.
CREATE TABLE idempotency_keys (
  scope               text        NOT NULL,  -- tenant or API client
  key                 text        NOT NULL,  -- client-supplied Idempotency-Key
  request_fingerprint text        NOT NULL,  -- sha256 of method + path + canonical body
  status              text        NOT NULL DEFAULT 'in_progress'
                      CHECK (status IN ('in_progress', 'completed')),
  locked_at           timestamptz,           -- detects abandoned claims
  response_code       integer,
  response_body       jsonb,
  created_at          timestamptz NOT NULL DEFAULT now(),  -- used for expiry
  PRIMARY KEY (scope, key)                   -- settles concurrent duplicates
);

BEGIN;

-- Atomic claim. One row back = this request owns the key.
-- Zero rows = key already exists: ROLLBACK and take the retry path
-- (compare fingerprints, replay the stored response or report in-progress).
INSERT INTO idempotency_keys (scope, key, request_fingerprint, locked_at)
VALUES ('tenant_42', '5f1c9a7e-2b44-4d0e-9c1a-8e3f6b2d7a10',
        'a3f9...e21c', now())
ON CONFLICT (scope, key) DO NOTHING
RETURNING key;

-- Business write in the same transaction.
INSERT INTO payments (id, tenant_id, amount_cents, currency)
VALUES ('pay_123', 'tenant_42', 5000, 'USD');

-- Store the exact response the client will receive, then commit once.
UPDATE idempotency_keys
SET status        = 'completed',
    response_code = 201,
    response_body = '{"id": "pay_123", "amount_cents": 5000, "currency": "USD"}',
    locked_at     = NULL
WHERE scope = 'tenant_42'
  AND key   = '5f1c9a7e-2b44-4d0e-9c1a-8e3f6b2d7a10';

COMMIT;

The idempotency_keys table with a (scope, key) primary key, an INSERT ... ON CONFLICT DO NOTHING claim, and the business write plus completion update with the serialized response committed in one transaction.

How should a retry be handled when the key already exists?

When the insert hits the unique constraint, read the existing row and branch on its state and fingerprint. Check the fingerprint first, so a mismatched request is never served someone else's response.

  • Same key, different fingerprint: reject with 422. The client reused a key for a different request. That is a client bug, and retrying won't fix it.
  • Completed: replay the stored status code and body exactly, even if the original was a 4xx. The client should not be able to tell a replay from the first response.
  • In progress: another worker holds the key. Either return a conflict or wait briefly for it to finish.
Strategy Client experience Under a retry storm
Return 409 with Retry-After Client must retry Each duplicate is a cheap read and exits fast, so connections stay free
Block until done or timeout Often a single round trip Every duplicate holds a connection and a worker, which can exhaust the pool

Prefer the conflict response by default. If you block, cap the wait at a few hundred milliseconds and then fall back to the conflict.

Workers can crash and leave rows stuck in progress. Treat locked_at as a lease: a later request may take over the key with an UPDATE only where locked_at is older than the lease interval. Reclaiming is only safe if the original attempt either rolled back completely or performed side effects that are themselves idempotent. An expired lease shows the worker was slow. It does not show the worker is dead. A live but stalled worker can wake up and charge the card while the new owner charges it again.

-- Run after INSERT ... ON CONFLICT (idempotency_key) DO NOTHING inserted 0 rows.
-- $1 = idempotency key, $2 = request fingerprint (e.g. hash of method, path, body).

-- 1. Decide what to do with the existing row. Fingerprint is checked first,
--    so a mismatched request never sees another request's stored response.
SELECT
  CASE
    WHEN fingerprint <> $2 THEN 'reject_422'      -- key reused for a different request
    WHEN status = 'completed' THEN 'replay'       -- return stored code and body as-is
    WHEN locked_at < now() - interval '30 seconds'
      THEN 'try_reclaim'                          -- lease expired; owner may be dead or just slow
    ELSE 'conflict_409'                           -- live owner; send 409 with Retry-After
  END AS action,
  response_code,
  response_body
FROM idempotency_keys
WHERE idempotency_key = $1;

-- 2. Only when action = 'try_reclaim': take over the lease atomically.
--    The WHERE clause repeats every condition, so if two retries race,
--    only one UPDATE matches the row. The loser gets 0 rows and returns 409.
UPDATE idempotency_keys
SET locked_at = now()
WHERE idempotency_key = $1
  AND fingerprint = $2
  AND status = 'in_progress'
  AND locked_at < now() - interval '30 seconds'
RETURNING idempotency_key;

-- Caveat: winning this UPDATE proves only that the lease expired, not that
-- the old worker stopped. Re-run the work only if the original attempt rolled
-- back fully or its side effects are idempotent (for example, the payment
-- call passes its own idempotency key downstream).

Postgres queries that branch on an existing idempotency key's fingerprint, status, and lease age, then reclaim an expired in-progress key with a conditional UPDATE so that only one of several concurrent retries can take it over.

What about side effects outside Postgres?

That is why the key row alone cannot make a request safe. Postgres can roll back its own writes, but it cannot un-send an email or reverse a captured payment.

Pass the key downstream

Many payment providers accept an idempotency key of their own. Derive it deterministically from yours, for example your key plus a step name, so every attempt presents the same value to the provider, including an attempt from a stalled worker. The provider then deduplicates the charge for you. Generating a fresh random key per attempt defeats this.

Use an outbox for messages

Email services and message brokers often will not deduplicate for you. Do not publish from inside the request. Insert a row into an outbox table in the same transaction that updates the key row, and let a separate relay publish it and mark it sent. Delivery is at-least-once, so consumers must deduplicate on a message ID.

Record phases on the key row

For multi-step work, store a phase such as started, payment_captured, order_created, or notified. Each step commits its result and advances the phase, and a retry resumes from the last recorded phase.

The simpler-looking alternative is one transaction wrapped around the slow external call. That holds a connection and row locks for the whole network round trip, and a pool sized for millisecond transactions drains quickly when a provider slows down. Phases avoid this but add states. Every transition needs a recovery path and a test that simulates a crash between steps.

-- Step 1 (after the provider confirms a charge sent with idempotency key 'ord-7f3a:capture'):
-- advance the phase and enqueue the receipt in ONE transaction, with no network call inside it.
-- The CTE only inserts the outbox row if this worker actually won the phase transition.
WITH advanced AS (
  UPDATE idempotency_keys
  SET phase = 'payment_captured',
      provider_charge_id = 'ch_placeholder_123',
      updated_at = now()
  WHERE idempotency_key = 'ord-7f3a'
    AND phase = 'started'          -- guards against a stalled worker replaying this step
  RETURNING idempotency_key
)
INSERT INTO outbox (message_id, topic, payload)
SELECT idempotency_key || ':receipt',  -- deterministic ID so consumers can deduplicate
       'email.receipt',
       jsonb_build_object('order_key', idempotency_key)
FROM advanced;
-- Zero rows affected means another attempt already advanced the phase: re-read the row and resume from it.

-- Step 2 (relay process, runs separately): claim unsent messages without blocking other relays.
BEGIN;
SELECT id, message_id, topic, payload
FROM outbox
WHERE sent_at IS NULL
ORDER BY id
LIMIT 10
FOR UPDATE SKIP LOCKED;
-- Publish each claimed message to the broker or email service here.
-- A crash before COMMIT releases the locks and the batch is retried: delivery is at-least-once.
UPDATE outbox SET sent_at = now()
WHERE id = ANY (ARRAY[101, 102, 103]);  -- the ids claimed above
COMMIT;

A phase transition guarded on the current phase commits atomically with an outbox insert. A separate relay then claims and publishes outbox rows using FOR UPDATE SKIP LOCKED, which gives at-least-once delivery keyed by a deterministic message ID.

When should you avoid idempotency keys, and what are the alternatives?

Keys only work if the client creates one per logical operation and reuses it on every attempt. A client that generates a fresh key inside its retry loop gets no protection, because each attempt looks like a new request. Keys also cannot catch logical duplicates: two users, or two browser tabs, submitting the same order produce two valid keys. If the operation is already idempotent, such as a PUT to a client-chosen ID, a key only adds a table to maintain.

Alternative Catches Atomic with the write? Adoption cost
Domain unique constraint (e.g. on order_reference) Retries and logical duplicates sharing the reference Yes, enforced in the same transaction Low, but needs a meaningful business identifier
Gateway or middleware idempotency Retries with a reused key No, the response is cached outside the database transaction Lowest, with no service changes
Client-supplied resource IDs Retries, because a repeat insert hits the primary key Yes Moderate, since clients need collision-safe ID generation

A gateway is attractive because one component covers every service. It cannot see your commit, though. A crash between the database write and the response cache leaves the gateway unaware that the operation succeeded, so the retry runs again.

On storage, saving the full response body replays exactly what the client saw, but it costs space and may retain sensitive fields. Storing only the resource ID is cheaper, but re-rendering returns the current state, which may differ from the original response.

The retention window is a hard limit. A retry that arrives after its key expires executes again. Size the window to the longest retry horizon you expect from clients, including queued and offline clients.

What does adopting idempotency keys cost, and where does it save engineering time?

The cost is mostly up front. The idempotency table lives in each service's own database so the key insert and the business write commit in one transaction. That means every adopting service needs schema and migration work rather than a shared library dropped in. Request fingerprinting needs a canonical form, with sorted fields and normalized numbers and whitespace, or legitimate retries get rejected as mismatches. The in-progress, completed and mismatch states each need tests, including concurrent duplicates. Expired rows need cleanup, either through batched deletes against a created_at index or through time-partitioned tables where you drop whole partitions. Dropping partitions avoids long-running deletes and table bloat.

Illustrative estimate (assumed numbers, not a benchmark). Storage ≈ keyed requests per day × retention days × bytes per row. With an assumed 2,000,000 POSTs/day, 7-day retention and 1 KB per row (key, fingerprint, stored response, index overhead), that is 2,000,000 × 7 × 1 KB ≈ 14 GB.

The savings come from duplicate charges and orders. Without keys, those get found and reversed by hand, often long after the fact. With keys, retries become safe to enable by default in clients, gateways and job runners, so transient failures stop turning into support tickets.

Clients carry part of the contract:

  • Generate the key before the first attempt, not after a failure.
  • Persist the key across process restarts for operations that matter, such as payments.
  • Retry with exponential backoff and jitter.
  • Never mint a new key on retry, because that turns a retry into a new request.
  • Document what happens after the key expires, so callers know a late retry will execute again.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to idempotency keys api, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • The dependency graph answers "what does this touch?" in seconds instead of a manual audit.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • You can size a change's blast radius before merging, avoiding surprise cross-team firefighting.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.