What is the dual write problem and why can't you just publish after commit?

A dual write happens when one request changes state in two systems that share no transaction, typically an UPDATE in Postgres and a send to a Kafka topic. Each write can succeed or fail independently, so every ordering leaves a window where the two systems disagree.

Ordering Failure point Result
Commit, then publish Process crashes after commit, before send Order is saved, no event. Downstream services never learn about it.
Publish, then commit Commit fails or rolls back after send Consumers act on an order that does not exist.
Either Publish times out Unknown outcome. The broker may have the message or may not.

The timeout case is the one that defeats defensive code. A try/catch only sees the error your process observed, not what the broker actually did. Retrying after a timeout can produce a duplicate. Not retrying can lose the event. Retries also cannot survive a crash: if the process dies between the two writes, the code that would have retried is gone, and nothing durable records that a publish was still owed.

Distributed transactions are the textbook answer, but not a practical one here. Kafka's transactions are atomic only across Kafka partitions and offsets. Kafka does not take part in XA two-phase commit with Postgres. Even where 2PC is available, it couples the availability of both systems and leaves locks held whenever the coordinator fails.

The outbox removes the second write from the request path. The service writes the business row and an event row in one Postgres transaction. The commit then becomes the single source of truth for both state and events, and publishing becomes a separate, retryable process that reads from data that has already been committed.

How does the transactional outbox with Debezium work?

The guarantee comes first because it shapes everything downstream: at-least-once delivery, ordered per aggregate key, with no global ordering across aggregates. Consumers will see duplicates and must handle them. They can rely on the events for one order arriving in sequence, but not on the relative order of events for two different orders.

The write path

Inside a single local transaction, the application updates its business tables and inserts an outbox row describing the event: aggregate type, aggregate ID, event type and payload. Either both commit or neither does. There is no broker call in the request path, so a Kafka outage cannot fail or half-complete a business operation.

The read path

Debezium runs as a source connector on Kafka Connect. It does not poll the outbox table. Instead it uses Postgres logical decoding over a replication slot to stream committed changes from the write-ahead log. Uncommitted or rolled-back inserts never appear, and the WAL's commit order is preserved.

The outbox event router, a single message transform (SMT), reshapes each outbox insert into a clean Kafka record. It typically routes by aggregate type to a topic, sets the aggregate ID as the record key, and uses the payload as the value. Keying by aggregate ID sends all events for one aggregate to the same partition, which is where per-key ordering comes from.

Duplicates come from offsets. Kafka Connect commits the connector's WAL position periodically, so after a crash or restart Debezium re-reads from the last committed position and re-emits some records it had already published.

What gets published

Unlike raw table CDC, these events reflect only what the application chose to write to the outbox. If a code path forgets the outbox insert, or someone fixes data with direct SQL against business tables, nothing is published.

+--------------------------- Application ----------------------------+
|  BEGIN                                                             |
|    UPDATE orders ...  (business tables)                            |
|    INSERT INTO outbox (aggregatetype, aggregateid, type, payload)  |
|  COMMIT  -- both rows commit, or neither does                      |
+---------------------------------+----------------------------------+
                                  |
                                  v
+---------------------------- Postgres ------------------------------+
|  write-ahead log: committed changes only, in commit order          |
|          |                                                         |
|          v  logical decoding over a replication slot (no polling)  |
+----------+---------------------------------------------------------+
           |
           v
+----------------------- Kafka Connect worker -----------------------+
|  Debezium Postgres source connector                                |
|          |                                                         |
|          v                                                         |
|  Outbox event router SMT                                           |
|    topic <- aggregate type, key <- aggregate ID, value <- payload  |
|  WAL offsets committed periodically: restart re-emits some records |
+----------+---------------------------------------------------------+
           |
           v
+------------------------------ Kafka -------------------------------+
|  topic per aggregate type; same key -> same partition              |
|  at-least-once, ordered per aggregate key, no global order         |
+--------------------------------------------------------------------+

Business change and outbox row commit in one transaction; Debezium streams the committed insert from the WAL and the outbox event router SMT turns it into a keyed Kafka record.

How should you design the outbox table?

Because the outbox is the only path events take to consumers, its schema is a contract. A minimal design has six columns:

  • id (UUID, primary key): a stable event identity that consumers use for deduplication. Generate it in the application so retries reuse it.
  • aggregatetype: the entity kind, such as order. The Debezium outbox event router uses it to choose the destination topic.
  • aggregateid: the entity's identifier, which becomes the Kafka message key.
  • type: the event name, such as OrderPlaced.
  • payload: JSONB if you want to inspect events with SQL, or bytea for Avro or Protobuf.
  • Optional headers: a trace or correlation id, mapped to Kafka headers.

The key matters because Kafka hashes the key to pick a partition, and ordering is guaranteed only within a partition. With aggregateid as the key, every event for one order lands in the same partition. Those events appear in the order the WAL recorded their commits, so consumers never see OrderShipped before OrderPlaced.

The application writes the business row and the outbox row in one transaction: BEGIN, update orders, insert into outbox, then COMMIT. Either both rows exist or neither does.

The outbox table should stay small. You can delete the row in the same transaction right after inserting it, or remove old rows on a schedule. Deleting does not lose the event, because Debezium reads the insert from the WAL. Configure the router to drop the resulting delete events so that consumers never receive them.

This design has a cost. Every business write generates extra WAL and adds commit latency, and both grow with payload size. Keep payloads lean.

-- Outbox table: column names match the Debezium outbox event router defaults
CREATE TABLE outbox (
  id             uuid PRIMARY KEY,       -- event identity, used by consumers to deduplicate
  aggregatetype  text NOT NULL,          -- entity kind, e.g. 'order'; the router uses it to pick the topic
  aggregateid    text NOT NULL,          -- entity id; becomes the Kafka message key
  type           text NOT NULL,          -- event name, e.g. 'OrderPlaced'
  payload        jsonb NOT NULL,         -- use bytea instead for Avro or Protobuf
  correlation_id text                    -- optional; can be mapped to a Kafka header
);

-- Application side: business row and outbox row commit or roll back together
BEGIN;

UPDATE orders
   SET status = 'PLACED'
 WHERE id = 'ord-10042';

-- The id is generated by the application, so a retried request reuses the same value
INSERT INTO outbox (id, aggregatetype, aggregateid, type, payload, correlation_id)
VALUES (
  '3f6c2a1e-8b4d-4c2f-9e1a-7d5b0c9a4e21',
  'order',
  'ord-10042',
  'OrderPlaced',
  '{"orderId": "ord-10042", "total": 129.90, "currency": "EUR"}',
  'req-7f3a9c'
);

-- Optional: delete immediately to keep the table empty.
-- Debezium has already got the INSERT from the WAL; the router should drop this DELETE.
DELETE FROM outbox
 WHERE id = '3f6c2a1e-8b4d-4c2f-9e1a-7d5b0c9a4e21';

COMMIT;

An outbox table using Debezium's default column names, and one transaction that updates the business row, inserts the outbox event, and deletes it before commit. The WAL still records the insert, so the event is not lost.

How do you set up CDC on Postgres and the Debezium connector on Kafka Connect?

Postgres has to write enough detail into the WAL for logical decoding, so set wal_level = logical. The change requires a server restart, so plan it as a maintenance step. Give the connector a dedicated role with only what it needs: CREATE ROLE debezium WITH LOGIN REPLICATION PASSWORD '...'; and GRANT SELECT ON outbox TO debezium;. As the table owner, create the publication yourself with CREATE PUBLICATION outbox_pub FOR TABLE outbox;. Then configure the connector not to auto-create one. This keeps the connector away from ownership privileges, and logical decoding never ships changes from other tables.

When the connector first connects, it creates a replication slot. The slot records the last WAL position the connector has confirmed, and Postgres keeps all WAL after that point. If the connector restarts, it resumes from the slot exactly where it stopped, so it does not skip committed events.

The outbox event router is a single message transform. It replaces the raw CDC envelope (before/after images and source metadata) with a clean message. By default, aggregatetype selects the topic, aggregateid becomes the message key (which preserves per-aggregate ordering within a partition), and payload becomes the value. You can map extra columns such as id or type into headers.

An illustrative, version-dependent config looks like this: plugin.name=pgoutput, publication.name=outbox_pub, slot.name=outbox_slot, table.include.list=public.outbox, and transforms.outbox.type=io.debezium.transforms.outbox.EventRouter. Property names and accepted values change between releases, so check the Debezium docs for your version.

On first startup, choose a snapshot mode that streams only new changes. A full snapshot would re-publish any outbox rows that already exist.

# Illustrative Debezium Postgres connector config (Kafka Connect .properties style).
# VERSION-DEPENDENT: property names and accepted values change between releases.
# Check the Debezium docs for your version before using any of these keys.
name=outbox-connector
connector.class=io.debezium.connector.postgresql.PostgresConnector

# Connect as the dedicated, minimally privileged replication role
database.hostname=db.internal.example
database.port=5432
database.user=debezium
database.password=CHANGE_ME_PLACEHOLDER
database.dbname=orders
topic.prefix=orders

# Use the built-in pgoutput plugin and the publication the table owner created
plugin.name=pgoutput
publication.name=outbox_pub
# Do not let the connector create or alter publications itself
publication.autocreate.mode=disabled
# Replication slot that records the connector's confirmed WAL position
slot.name=outbox_slot
table.include.list=public.outbox

# First startup: stream only new changes, do not re-publish existing outbox rows
# (older releases used a different value for this mode)
snapshot.mode=no_data

# Outbox event router: replace the CDC envelope with a clean event
transforms=outbox
transforms.outbox.type=io.debezium.transforms.outbox.EventRouter
# Topic is derived from aggregatetype (defaults shown explicitly)
transforms.outbox.route.by.field=aggregatetype
# aggregateid becomes the message key, preserving per-aggregate ordering
transforms.outbox.table.field.event.key=aggregateid
transforms.outbox.table.field.event.payload=payload
# Copy the event type column into a message header named eventType
transforms.outbox.table.fields.additional.placement=type:header:eventType

An illustrative, version-dependent Debezium Postgres connector config that streams only the outbox table through a pre-created publication and slot, skips the initial data snapshot, and uses the outbox event router to set topic, key, value and headers.

How do you make consumers idempotent with the inbox pattern?

Debezium delivers at least once, so every consumer will eventually see the same event twice. Duplicates come from three places. The connector can restart before it flushes its source offset. A consumer group can rebalance before a partition's offsets are committed. More generally, offset commits lag behind delivery. Kafka transactions on the producer side do not fix this, because the duplicate is created before the record reaches the broker or after the consumer reads it.

The inbox table

Give each consumer a processed_events table with event_id as its primary key. In the same database transaction that applies the business change, run INSERT INTO processed_events (event_id, processed_at) VALUES ($1, now()) ON CONFLICT (event_id) DO NOTHING. If the insert affects zero rows, the event was already handled, so skip the side effects and commit. Commit the Kafka offset only after the database commit succeeds. If the process crashes between the two, the redelivered event hits the conflict and is dropped. Prune old rows on a schedule that is longer than any plausible redelivery window.

Ordering limits

Kafka orders events only within a partition. If you change the partition count or the key scheme, events for a key that is already in flight can land on two partitions and be consumed out of order. Drain the topic first, or migrate to a new topic.

Contract and tracing

The payload is a public API. Keep it separate from your internal tables, version your event types, and make only additive changes. Carry a trace id in the headers so you can follow one event through the application, the WAL, the connector, the SMT, Kafka and the consumer.

-- Inbox table: one row per event this consumer has applied.
CREATE TABLE processed_events (
  event_id     uuid        PRIMARY KEY,
  processed_at timestamptz NOT NULL DEFAULT now()
);
CREATE INDEX processed_events_processed_at_idx
  ON processed_events (processed_at);

-- Per event: claim the id and apply the side effect in ONE transaction.
-- $1 = event id from the outbox payload or headers
-- $2 = order id, $3 = new status
BEGIN;

INSERT INTO processed_events (event_id, processed_at)
VALUES ($1, now())
ON CONFLICT (event_id) DO NOTHING;
-- The application checks the affected row count:
--   1 row  -> first delivery, apply the business change below
--   0 rows -> duplicate, skip the business change and go to COMMIT

UPDATE orders
SET    status = $3, updated_at = now()
WHERE  id = $2;

COMMIT;
-- Commit the Kafka offset only after COMMIT succeeds. A crash in between
-- causes a redelivery, and the redelivered event hits ON CONFLICT.

-- The same guard as one statement: the UPDATE runs only if the claim
-- inserted a row.
WITH claimed AS (
  INSERT INTO processed_events (event_id) VALUES ($1)
  ON CONFLICT (event_id) DO NOTHING
  RETURNING event_id
)
UPDATE orders SET status = $3, updated_at = now()
WHERE  id = $2 AND EXISTS (SELECT 1 FROM claimed);

-- Scheduled pruning. Keep rows longer than any plausible redelivery window.
DELETE FROM processed_events
WHERE  processed_at < now() - interval '30 days';

A Postgres inbox table that claims each event id in the same transaction as the consumer's business update, so redelivered events hit ON CONFLICT and are skipped, plus a one-statement form of the same check and a scheduled prune.

What can go wrong when operating Debezium on Postgres?

Replication slot lag

The most dangerous failure fills the primary's disk without touching the application. A logical replication slot makes Postgres retain every WAL segment from the slot's restart_lsn onward until the consumer confirms it. If the connector stops, or falls behind a busy database, WAL accumulates until the disk fills and the primary stops accepting writes. Monitor pg_replication_slots and compute retained WAL as pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn) per slot. Alert on that byte count, on active = false, and separately on the connector task leaving the RUNNING state in the Kafka Connect REST API. A failed task can leave the slot looking healthy while WAL grows.

Snapshots

On first startup, the default snapshot behavior reads the captured tables before streaming. If offsets are lost, for example because the offsets topic was deleted or the connector was renamed, the connector may snapshot again. For an outbox table, a snapshot re-emits whatever rows remain. The inbox pattern absorbs those duplicates, but only if old outbox rows are actually being cleaned up.

Failover

Depending on your Postgres version and HA tooling, logical slots may exist only on the old primary. After a failover, the connector then finds no slot. Recreating the slot starts from the current position, so you either accept a gap or resnapshot. Run a real failover in staging before production.

Recovery

Dropping a slot frees retained WAL immediately. It is the right call when the disk is about to fill. The cost is that any outbox rows written after the slot's confirmed position are never streamed. Before restarting the connector, republish those rows from the outbox table by their timestamps, or trigger a snapshot.

When should you use the Debezium outbox, and what are the alternatives?

The Debezium outbox fits teams that already run Kafka and Kafka Connect, and teams where several services must agree on state changes at a volume where polling gets expensive. Avoid it if volume is low, nobody on the team has operated Kafka Connect, or only one service publishes. In those cases a polling relay or an in-process publisher that drains the outbox is enough.

Approach Consistency Latency Operational load Typical failure
Debezium outbox At-least-once, atomic with the write Low Connect workers, offsets, replication slot WAL growth while the connector is down
Polling relay At-least-once Poll interval One scheduled job Polling load; gaps when IDs commit out of order
Raw table CDC At-least-once Low Same as Debezium Internal schema becomes a public contract
Distributed transactions Atomic Higher, blocking Coordinator; Kafka has no XA support In-doubt transactions holding locks
Publish after commit None Lowest None Events lost on crash

The development cost is real. You take on Connect workers, offset storage, and logical replication to run. Debugging also gets harder, because each event passes through the WAL, the slot, the connector, and a transform before it reaches a topic.

In return, you stop writing retry loops and reconciliation jobs. You also stop building a separate relay in every service. Outage recovery becomes predictable: the WAL is the backlog, and the connector replays it from its last confirmed position.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to transactional outbox debezium, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.