Idempotency Keys in Microservices: Making Retries Safe End to End

Why retries create duplicate requests

An operation is idempotent when performing it more than once has the same effect as performing it once. Setting a user's email address is naturally idempotent. Charging a card, sending an email, creating an order or incrementing a counter is not. An idempotency key is a unique value the caller attaches to a request so the server can recognize a repeat of that same logical operation and return the original result instead of executing it again.

The need comes from a basic limit of networks. When a call times out, the caller learns nothing about what happened on the server. The request may never have arrived. It may have failed halfway. Or it may have committed successfully, with only the response lost on the way back. From the caller's side these three cases look identical, so the reasonable move is to retry.

Retries come from many layers at once. A mobile client retries on a flaky connection. An API gateway retries an upstream that returned an error. A service mesh sidecar retries according to its own policy. A queue consumer that crashes before acknowledging a message receives that message again. Each layer acts correctly by its own logic, and the result can still be a customer charged twice.

If your services create, charge, send or increment, this applies to you. The rest of this guide argues that a dedup cache in front of one endpoint is not enough. Safe retries depend on keys that are stored atomically with the work and carried across every service, queue and external provider the request touches.

Who should use idempotency keys

Idempotency keys are for teams whose services perform non-idempotent side effects behind a network boundary where something might retry. In practice that includes:

• Teams running payment, billing, or order flows. A duplicate charge or a duplicate order is a customer-visible failure that needs manual reconciliation to fix.
• Owners of public or partner-facing APIs. You don't control how third-party clients retry. Some will retry aggressively on timeouts, and some will retry with the same request after a long delay. Accepting a key gives those clients a safe way to retry, and it gives you a defined behavior when they do.
• Services that call external providers. Examples include payment processors, email and SMS gateways, and shipping APIs. Once a side effect leaves your system, a database rollback cannot undo it.
• Event-driven systems with at-least-once delivery. Any consumer that can receive the same message twice needs a deduplication record, and a derived key is the natural thing to record.
• Platform teams building shared request-handling libraries. The pattern pays off when it is implemented once and applied consistently. A per-endpoint reinvention does not deliver the same benefit.

If none of your write paths create, charge, send or increment, or every one of them is already protected by a business-level unique constraint, you may not need a key store at all. The next sections cover how to tell.

How idempotency keys work end to end

At a high level, the mechanism has five parts. Each one is covered in detail later in this guide.

1. The client mints one key per logical operation. It generates a random identifier, typically a UUIDv4, when the user action begins. It sends that identifier with the request, commonly in a request header, and reuses the same value on every retry of that operation.
2. The server claims the key atomically. Before doing any work, the server inserts a row keyed on (tenant_id, key) with status in_progress and a fingerprint of the request. A unique constraint guarantees that only one concurrent request can win the claim.
3. The winner executes and records the result. The server performs the business change and stores the response status and body on the key row. Where possible, it does this in the same database transaction as the business change, so the claim, the effect and the recorded result commit or roll back together.
4. Retries are answered from the record. A later request with the same key and the same fingerprint gets the stored response without re-executing. A request whose original attempt is still running gets a 409 and should retry later. A request that reuses the key with a different payload is rejected as a client error.
5. The key travels downstream. Calls to other services, messages published through an outbox, and requests to external providers carry keys derived deterministically from the original key. A retried upstream request therefore produces the same downstream keys at every hop, and each consumer deduplicates against its own record.

Keys are kept for a defined retention period (TTL) that must be longer than every retry window in the path. After expiry, a retry is treated as a new request.

When you need a key and when you don't

Not every endpoint needs that machinery. The question is whether executing the request twice leaves the system in a different state than executing it once.

Treat the "yes" rows with care. A PUT that also emits an event or calls a provider is idempotent in stored state but not in effects, so the side effect needs its own guard.

Where the business already defines uniqueness, use a uniqueness constraint instead of a key. Examples include one subscription per account per plan, one invoice per order, or one refund per return ID. A unique index on that natural identifier rejects duplicates from any client, any retry path and any time window, and it needs no expiring key store.

Keys have a blind spot. They identify a request, not an intent. If a user double-clicks across two tabs, or a client generates a fresh key on each attempt, the server sees two legitimate requests. To prevent that, generate the key once per user action, for example when the form renders. If that isn't enough, back the key with a business-level constraint.

When to avoid idempotency keys

A key store adds a write, an index, a retention policy and a cleanup job to every path it guards. Skip it, or use something simpler, in these cases:

• Reads. GET and other read-only requests have no side effects to deduplicate. Retrying them is already safe.
• Naturally idempotent writes with no external effects. A full-replacement PUT or a set-to-value update that only touches your own database is safe to repeat. A key adds cost without changing the outcome.
• Operations with a real business identity. If "one invoice per order" or "one refund per return" already defines uniqueness, a unique constraint on that identifier is stronger than a key. It has no TTL, and it catches duplicates that arrive with different keys.
• Creates where the client can choose the ID. If clients can generate the resource ID and send a PUT to that ID, the create becomes naturally idempotent. The resource ID then does the work a separate key would do.
• Effects where duplicates are harmless or already handled downstream. Debug logging, or metrics pipelines that tolerate or deduplicate repeated events, rarely justify a key store. The cost of an occasional duplicate is lower than the cost of the machinery.
• Batch jobs that are rerun from the start with upsert semantics. If a rerun overwrites rather than appends, the job is idempotent by construction.

Also be careful about relying on a key where you cannot store it transactionally with the effect. Examples include a key kept only in a cache, or an effect that happens in a system that cannot join your transaction and accepts no key of its own. In those setups the key reduces duplicates but does not prevent them. Design the rest of the system with that in mind, for example with reconciliation, rather than treating the key as a guarantee.

How the server-side state machine works

Minting and scoping the key

The client generates a UUIDv4 once per logical operation and sends the same value on every retry. Don't derive it from a timestamp: two requests in the same millisecond collide, and a retry issued a second later gets a different key, which defeats the purpose. Scope keys per tenant or caller, with a unique constraint on (tenant_id, key) rather than on key alone. That way, one customer's buggy client can't collide with another's, and a guessed key can't replay someone else's response.

Claim, execute, record

The server first inserts the key with status in_progress. The unique constraint makes this insert the lock: only one request can win it, and the database enforces that even across many API instances. The winner runs the operation, then writes the final status code and response body to the same row. Store the response as it was sent, not a recomputation, so replays are byte-for-byte consistent even if downstream state has changed since.

The server also stores a fingerprint of the request, such as a hash of the method, path, and canonicalized body. Clients sometimes reuse a key for a different payload, usually because of a bug. Silently replaying the old response would hide that bug.

Implementing the claim and record in one transaction

Start with a table whose primary key is (tenant_id, idempotency_key). Add these columns: request_hash, status (in_progress or completed), response_code, response_body, locked_at, and created_at. The unique constraint does the deduplication, not application logic. Two concurrent inserts cannot both succeed. Scoping by tenant stops two clients that happen to generate the same key from colliding.

Handler flow

The handler opens a transaction and attempts an insert-or-ignore of the claim row. If the insert succeeds, the handler performs the business write, updates the row to completed with the serialized response, and commits. If the insert conflicts, the handler reads the existing row, compares request_hash, and responds according to the table above.

The claim and the business change commit atomically. A crash therefore rolls back both, so you never have a record without an effect or an effect without a record.

Lock timeouts and stuck records

A concurrent duplicate blocks on the unique index until the first transaction finishes. Set a short lock timeout and return 409 with Retry-After, so callers retry instead of tying up connections.

Stuck in_progress rows only appear when the claim must commit on its own. This happens, for example, before calling an external provider that cannot join your transaction. Treat locked_at as a lease. A retry may reclaim the row with a conditional update that matches only rows where locked_at is older than the lease.

Replaying errors

Store and replay deterministic failures, such as validation 4xx responses. For transient 5xx failures, the rolled-back transaction left nothing behind, so allow re-execution.

Copy-- PostgreSQL. The primary key does the deduplication, scoped per tenant.
CREATE TABLE idempotency_keys (
  tenant_id       text        NOT NULL,
  idempotency_key text        NOT NULL,
  request_hash    text        NOT NULL,
  status          text        NOT NULL CHECK (status IN ('in_progress', 'completed')),
  response_code   int,
  response_body   jsonb,
  locked_at       timestamptz NOT NULL DEFAULT now(),
  created_at      timestamptz NOT NULL DEFAULT now(),
  PRIMARY KEY (tenant_id, idempotency_key)
);

BEGIN;
-- A concurrent duplicate waits on the first transaction. Fail fast and map to 409 + Retry-After.
SET LOCAL lock_timeout = '2s';

-- Claim. Zero rows returned means a conflict: SELECT the existing row and compare request_hash.
INSERT INTO idempotency_keys (tenant_id, idempotency_key, request_hash, status)
VALUES ('tenant-42', 'b7f3c1e0-order-create', 'sha256:9a1f...', 'in_progress')
ON CONFLICT (tenant_id, idempotency_key) DO NOTHING
RETURNING idempotency_key;

-- Business write in the same transaction, so claim and effect commit or roll back together.
INSERT INTO orders (tenant_id, order_id, amount_cents) VALUES ('tenant-42', 'ord-1001', 2500);

UPDATE idempotency_keys
SET status = 'completed', response_code = 201, response_body = '{"order_id": "ord-1001"}'
WHERE tenant_id = 'tenant-42' AND idempotency_key = 'b7f3c1e0-order-create';
COMMIT;

-- Only for claims committed on their own (e.g. before an external call): reclaim an expired lease.
-- One row updated means this retry now owns the key. Zero rows means the lease is still held.
UPDATE idempotency_keys
SET locked_at = now()
WHERE tenant_id = 'tenant-42' AND idempotency_key = 'b7f3c1e0-order-create'
  AND status = 'in_progress' AND request_hash = 'sha256:9a1f...'
  AND locked_at < now() - interval '30 seconds';

A PostgreSQL schema keyed on (tenant_id, idempotency_key), a transaction that commits the claim, the business write and the stored response together under a short lock timeout, and a conditional update that reclaims a stuck in-progress row after its lease expires.

Propagating keys across services, queues, and external providers

Derive downstream keys, don't mint them

When a service handling an idempotent request calls another service, it needs a key for that call too. Generate it deterministically from the parent key plus a fixed step identifier, such as parentKey + ':reserve-inventory'. A retried upstream request then produces exactly the same downstream keys, so each dependency recognizes the repeat and returns its stored result. This only holds if every hop derives its keys. One service that generates a random UUID per attempt breaks the chain for everything below it. Renaming a step identifier between deployments has the same effect: a retry that spans the deploy looks new, and the side effect runs twice. Treat step identifiers like a wire format and change them only on purpose.

External providers

Many payment and messaging providers accept an idempotency key on write requests. Pass the derived key through and don't create a separate one for the provider. Check how long the provider keeps keys. If your retry window is longer than its retention period, a late retry can still create a duplicate.

Queues and the outbox

Publishing a message and committing a database write can't be made atomic directly. Write the message to an outbox table in the same transaction as the business change. A relay then publishes from committed rows and retries until the broker acknowledges. This delivers at least once, so consumers must deduplicate. Each consumer records the message ID, or a business ID, in a processed-messages table in the same transaction as its own writes. A duplicate delivery then fails the uniqueness check and the consumer acknowledges it without reapplying the change.

Copy-- PostgreSQL. Dedup table: the primary key is what rejects redelivered messages.
CREATE TABLE processed_messages (
  message_key  text PRIMARY KEY,
  processed_at timestamptz NOT NULL DEFAULT now()
);

-- Producer (order service): the business row and the outbox row commit together.
-- A relay later publishes committed outbox rows and retries until the broker acks.
BEGIN;
INSERT INTO orders (id, status, total_cents)
VALUES ('ord_7f3a', 'placed', 4999);

-- Downstream key = parent key + fixed step id, never a fresh UUID,
-- so a retried upstream request yields the identical key.
INSERT INTO outbox (idempotency_key, topic, payload)
VALUES ('req_9c21e' || ':reserve-inventory', 'inventory.reserve',
        '{"order_id":"ord_7f3a","sku":"SKU-1042","qty":2}');
COMMIT;

-- Consumer (inventory service): claim the key and apply the change in one transaction.
-- On redelivery the INSERT hits the conflict, returns no row, and the UPDATE matches nothing.
BEGIN;
WITH claimed AS (
  INSERT INTO processed_messages (message_key)
  VALUES ('req_9c21e:reserve-inventory')
  ON CONFLICT (message_key) DO NOTHING
  RETURNING message_key
)
UPDATE inventory
SET reserved = reserved + 2
WHERE sku = 'SKU-1042'
  AND EXISTS (SELECT 1 FROM claimed);
COMMIT;
-- After COMMIT, acknowledge the message whether or not a row was updated.

The producer writes an outbox row carrying a derived key (parent key plus ':reserve-inventory') in the same transaction as its business change. The consumer records that key in a processed-messages table in the same transaction as its own update, so a redelivered message changes nothing.

Alternatives and where they fall short

That pattern works because the dedup record and the side effect commit together. Most alternatives drop that property, which is where duplicates get through.

A Redis cache sits outside the transaction that performs the write. You can mark a key and then crash before committing, or commit and then lose the key. Either way, the cache and the database disagree.

Unique constraints are the strongest alternative when a real business key exists. They stop the second insert, but the caller sees a constraint error, not the original result.

Version checks prevent lost updates. They do not identify a retry, so the client cannot tell "already applied" from "someone else changed it."

Turning off retries only moves the duplicate problem to whoever retries next.

Impact on cost and engineering time

The pattern has a runtime cost. Every side-effecting request pays for at least one extra write and a unique-index lookup on the key. That work usually runs inside the same transaction as the business change, so it adds latency and contention to your busiest write paths.

The key table is append-heavy and grows continuously. Its live size is roughly write rate × retention:

Illustrative estimate (assumed numbers, not a benchmark): live rows ≈ side-effecting requests per second × TTL in seconds. If you assume 50 requests/sec and a 24-hour TTL, that is 50 × 86,400 = 4,320,000 rows. At that size you need an index on (scope, key), plus either a cleanup job or time-based partitions you can drop whole, and alerting when cleanup falls behind.

What you store matters. Caching full response bodies can copy card data or personal information into a table with weaker access controls, and large payloads inflate storage. A safer and cheaper approach is to store the status code and a resource reference, then re-read the resource on replay. The trade-off is that a replay may show newer state than the original response did.

Choose a TTL longer than every retry window in the path, including client backoff, gateway retries, queue redelivery, and provider retries. Expiry is a hard boundary: a retry that arrives after it runs again. Document the TTL in the API contract.

The engineering savings come from applying the pattern consistently. One shared library and table design replace ad hoc deduplication on each endpoint. Teams also spend less time on duplicate-charge incidents and manual reconciliation.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to idempotency keys microservices, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Changing a route's URL or method automatically notifies the services that consume it — no hunting through repos for callers.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• Consumers learn about a breaking change at review time rather than from production errors.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.