What is the dual write problem and why can't retries fix it?
A service that handles "place order" usually has to do two things: insert the order row and tell the rest of the system with an OrderPlaced event on Kafka. When those are two separate calls to two separate systems, there is no moment at which both succeed or both fail together. That gap is the dual write problem.
The two failure orderings
Commit, then publish. The database transaction commits, and the process crashes, gets killed by a deploy, or times out talking to the broker before the event is sent. The order exists, but no downstream service ever hears about it. Billing never charges and shipping never ships.
Publish, then commit. The event goes out first, and then the transaction rolls back because of a constraint violation, a deadlock, or a lost connection. Consumers now act on an order that does not exist.
Why retries don't close the gap
Retrying the publish after a commit helps only while the process is alive to retry. If the retry state lives in memory, a crash erases it. If you persist it, you have written a second record that must stay consistent with the first, which recreates the same problem. Retries also cause duplicates when a send succeeded but its acknowledgment was lost.
Why not two-phase commit?
Kafka does not act as a participant in XA-style distributed transactions. Its transactions cover only writes to Kafka itself. Coordinating a database and Kafka atomically would need a transaction manager that both systems support, and most stacks lack one. You would also take on blocking and in-doubt recovery. So the practical fix is to stop writing to two systems at once, which is what the outbox pattern does.
How does it work? The transactional outbox with Debezium
The service writes only to its own database. In one local transaction, it updates the business tables and inserts a row into an outbox table that describes the event. Either both commit or neither does, so an event exists if and only if the state change committed. Nothing talks to the broker inside the request path.
Publishing becomes a separate, asynchronous job, and Debezium handles it through change data capture. It does not poll the outbox table with SELECT queries, which add load and can miss or reorder rows. Instead, it reads the database's transaction log: logical decoding through a replication slot on PostgreSQL, or the binlog on MySQL. The log already records committed changes in commit order, so Debezium sees outbox inserts exactly as the database committed them.
The end-to-end flow:
- The service commits the business change and the outbox row together.
- The database appends the commit to its transaction log.
- Debezium, running as a source connector on Kafka Connect, reads the outbox insert from the log.
- Kafka Connect writes the event to a Kafka topic.
- Consumers read from the topic and update their own state.
Kafka Connect stores the connector's log position as offsets. In distributed mode, these live in an internal Kafka topic. After a crash or redeploy, the connector resumes from the last committed offset. Events published after that offset but before the crash are sent again. Delivery is therefore at-least-once, and consumers see changes with eventual consistency, not inside the producer's transaction.
+-----------------------------------------------+
| Service: one local transaction |
| UPDATE business tables |
| INSERT INTO outbox (event row) |
+-----------------------+-----------------------+
| 1. COMMIT: both rows or neither
v
+-----------------------------------------------+
| Database: [business tables] [outbox] |
| transaction log: PostgreSQL WAL via logical |
| decoding + replication slot, or MySQL binlog |
+-----------------------+-----------------------+
| 2. read log in commit order (no polling)
v
+-----------------------------------------------+ +-------------------+
| Kafka Connect (distributed mode) | | offsets topic |
| Debezium source connector | -->| (internal Kafka |
| restart: resume from last committed offset | <--| topic): log pos |
+-----------------------+-----------------------+ +-------------------+
| 3. publish event
v
+-----------------------------------------------+
| Kafka topic |
+-----------------------+-----------------------+
| 4. at-least-once, eventually consistent
v
+-----------------------------------------------+
| Consumers: read topic, update own state |
+-----------------------------------------------+
The service commits the business change and the outbox row in one local transaction. Debezium on Kafka Connect reads that commit from the transaction log, stores its log position in an internal offsets topic, and publishes the event to Kafka for consumers with at-least-once delivery.
How do you design the outbox table and write to it?
The outbox is an ordinary table in the service's own database. It does not need to be generic, but a small set of columns covers most needs:
id(UUID): a unique event identifier. Consumers use it later for deduplication, so generate it once, at write time.aggregatetype: the kind of entity, such asOrderorCustomer. The event router typically uses it to choose the destination topic.aggregateid: the identifier of the specific entity instance.type: the event name, such asOrderPlaced.payload: the event body, usually JSON or JSONB.timestamp: when the event was created.
The aggregate id matters most. It becomes the Kafka message key. Kafka assigns every message with the same key to the same partition, and ordering holds only within a partition. That means all events for order 42 arrive in commit order. Events for order 42 and order 43 can land on different partitions, so their relative order is not defined. If a consumer needs ordering across two entities, either model them as one aggregate or design the consumer so it does not depend on that order.
The write itself has to happen in the same local transaction as the business change. For example, the service updates the order's status and inserts the OrderPlaced row, then commits. Either both changes become durable or neither does. There is no second system to coordinate.
Treat payload as a public API. Other teams will deserialize it long after you have forgotten the original shape. Make only additive changes, put a version in the event type or payload, or enforce compatibility rules through a schema registry.
-- PostgreSQL. Outbox table that sits beside the business tables.
CREATE TABLE outbox (
id UUID PRIMARY KEY, -- event id, generated once; consumers dedupe on it
aggregatetype VARCHAR(255) NOT NULL, -- e.g. 'Order'; the router picks the topic from it
aggregateid VARCHAR(255) NOT NULL, -- becomes the Kafka message key
type VARCHAR(255) NOT NULL, -- event name, e.g. 'OrderPlaced'
payload JSONB NOT NULL, -- public contract: additive changes only
timestamp TIMESTAMPTZ NOT NULL DEFAULT now() -- when the event was created
);
-- One local transaction: the state change and the event commit together or not at all.
BEGIN;
UPDATE orders
SET status = 'PLACED'
WHERE id = 42;
INSERT INTO outbox (id, aggregatetype, aggregateid, type, payload)
VALUES (
gen_random_uuid(), -- built in from PostgreSQL 13; use the pgcrypto extension on older versions
'Order',
'42', -- same key for every event about order 42, so they share a partition
'OrderPlaced',
'{"schemaVersion": 1, "orderId": 42, "customerId": 7, "total": "59.90"}'::jsonb
);
COMMIT;
A PostgreSQL outbox table, plus one transaction that updates an order's status and inserts the matching OrderPlaced event keyed by the order id.
How do you configure Debezium and the outbox event router on Kafka Connect?
Prepare the database
Debezium reads the database's change log, not the tables, so the log must contain full row data.
- PostgreSQL: set
wal_level = logicaland leave headroom inmax_replication_slotsandmax_wal_senders. Changingwal_levelrequires a restart. Create a dedicated role with theREPLICATIONandLOGINattributes, plusSELECTon the outbox table. The built-inpgoutputplugin avoids installing a server extension. - MySQL: enable the binlog with
binlog_format = ROWandbinlog_row_image = FULL. Grant the connector userSELECT,RELOAD,SHOW DATABASES,REPLICATION SLAVE, andREPLICATION CLIENT. Statement-based binlogs record SQL, not row values, so Debezium cannot reconstruct events from them.
Configure the connector
The connector needs connection details (database.hostname, database.port, database.user, database.password, database.dbname for PostgreSQL) and a table.include.list that names only the outbox table. Capturing business tables as well would publish your internal schema, which the outbox exists to prevent.
Next, add the outbox event router SMT. Set transforms=outbox and transforms.outbox.type=io.debezium.transforms.outbox.EventRouter. By default it does three things:
- Routes by aggregate type. It reads the
aggregatetypecolumn and writes tooutbox.event.<aggregatetype>. Useroute.topic.replacementto change the topic pattern. - Keys messages by aggregate id. It sets the Kafka key from
aggregateid. All events for one aggregate land in the same partition, so consumers see them in order. - Emits only the payload. It replaces Debezium's before/after envelope with the
payloadcolumn.
Column mappings such as table.field.event.key and table.field.event.payload let you rename columns. Property names have changed between Debezium releases, so check each one against the documentation for your version.
name=orders-outbox-connector
connector.class=io.debezium.connector.postgresql.PostgresConnector
database.hostname=orders-db.internal
database.port=5432
database.user=debezium_outbox
database.password=change-me-placeholder
database.dbname=orders
topic.prefix=orders
plugin.name=pgoutput
table.include.list=public.outbox
transforms=outbox
transforms.outbox.type=io.debezium.transforms.outbox.EventRouter
transforms.outbox.route.by.field=aggregatetype
transforms.outbox.route.topic.replacement=outbox.event.${routedByValue}
transforms.outbox.table.field.event.key=aggregateid
transforms.outbox.table.field.event.payload=payload
A Debezium PostgreSQL connector config that captures only the outbox table and uses the EventRouter SMT to route rows to per-aggregate topics, key messages by aggregate id, and emit only the payload.
How do you build idempotent consumers for at-least-once delivery?
Debezium's guarantee is at-least-once. Duplicates come from the following sources:
- The connector commits source offsets periodically, not after every record. If it crashes, it resumes from the last committed offset and re-emits everything after it.
- A Kafka Connect rebalance can stop a task before its offsets are flushed. The task's replacement then re-emits the same records.
- A re-snapshot re-reads every row still in the outbox table and publishes each one again.
- Your own consumer can apply an event and then fail before committing its Kafka offset.
Consumers therefore need to tolerate duplicates.
The processed-events pattern
Use the outbox row's id as the event identity, and carry it in a header or in the payload. Give the consumer a processed_events table with a unique constraint on that id. In one local database transaction, insert the id and apply the side effect, then commit. A duplicate insert violates the constraint, the transaction rolls back, and the consumer skips the event and commits its Kafka offset. Both writes must be in the same transaction. If they are not, the consumer has its own dual write problem.
Ordering and versions
The event router keys messages by aggregate id. As a result, events for one aggregate land in one partition and arrive in order. Events for different aggregates have no relative ordering. If an OrderPlaced event can arrive before the CustomerCreated event it references, the consumer must handle the missing reference, either by retrying later or by creating a placeholder.
Within an aggregate, include a monotonically increasing version in the payload. Store the last applied version per aggregate, and ignore any event whose version is not newer. The id check catches exact duplicates. The version check catches stale replays of older state that arrive after newer state has already been applied.
-- PostgreSQL. Consumer-side tables, in the consumer's own database.
CREATE TABLE processed_events (
event_id uuid PRIMARY KEY, -- outbox row id, read from header/payload
processed_at timestamptz NOT NULL DEFAULT now()
);
CREATE TABLE aggregate_versions (
aggregate_id text PRIMARY KEY,
last_version bigint NOT NULL
);
-- Executed by the consumer for each event. The ids and values are placeholders.
BEGIN;
-- 1. Dedup guard. A replayed event fails here with unique_violation (SQLSTATE 23505).
-- The consumer then rolls back, skips the event and commits its Kafka offset.
INSERT INTO processed_events (event_id)
VALUES ('3f6c2a9e-8b1d-4c57-9a0e-1d2b7e4f5a10');
-- 2. Version guard. The row is written only if this event is newer than the last applied version.
-- If the reported row count is 0, the event is stale. The consumer skips step 3 and still
-- commits, so the id is recorded as processed.
INSERT INTO aggregate_versions (aggregate_id, last_version)
VALUES ('order-1042', 7)
ON CONFLICT (aggregate_id) DO UPDATE
SET last_version = EXCLUDED.last_version
WHERE aggregate_versions.last_version < EXCLUDED.last_version;
-- 3. Side effect, in the same transaction. All three writes commit together or not at all.
UPDATE order_projection
SET status = 'SHIPPED'
WHERE order_id = 'order-1042';
COMMIT;
-- Commit the Kafka offset only after this COMMIT succeeds.
A consumer transaction that records the event id in a dedup table, applies a per-aggregate version check and performs the side effect atomically, so replayed or stale events are skipped.
How do you keep the outbox small and operate the connector in production?
Because the connector reads the write-ahead log or binlog rather than the table, the outbox does not need to retain rows after commit. That leaves two cleanup strategies.
| Strategy | How it works | Benefit | Cost |
|---|---|---|---|
| Delete in same transaction | Insert the outbox row, then delete it before commit | Table stays near empty; no cleanup job | No queryable history; debugging relies on Kafka |
| Scheduled cleanup | A job deletes rows older than a retention window | Rows available for audit and replay investigation | Table growth, index bloat, one more job to own |
The immediate delete works because the log still records the insert. The event router is built around inserts, so the delete does not become a business event.
Operational checklist
- Connector and task status. A connector can report running while a task has failed. Alert on task state and on consumer lag for the outbox topics.
- Log retention. A stalled connector makes PostgreSQL keep WAL for its replication slot, or lets MySQL purge binlogs it has not read yet. Alert on slot or binlog growth before the disk fills.
- Rehearsals. Test connector restarts, database failover, major version upgrades and re-snapshots in staging. Each can replay events or lose your log position, so confirm that your idempotent consumers handle the duplicates and that you can detect gaps.
Kafka Connect and Debezium are production systems. Someone must deploy, upgrade, secure and monitor them, and that work belongs in a team's on-call scope. The return is that services no longer need their own ad hoc publish retries and reconciliation jobs. One shared, well-understood pipeline replaces many of those per-service workarounds.
How can it save engineering time?
The time savings come from code and incidents you no longer have, not from faster publishing.
- No per-service publishing machinery. Without the outbox, each service that emits events tends to grow its own retry loop, failure table, and reconciliation job that compares database state with what was published. With the outbox and Debezium, a service's only publishing responsibility is one extra
INSERTinside a transaction it already runs. Retries, offset tracking, and broker connectivity live in one shared connector configuration. - No broker client in the request path. Request handlers do not need Kafka producer configuration, timeout tuning, or error handling for broker outages. A Kafka outage delays event delivery but does not fail user requests, so it stops being a reason to page the service owner.
- Fewer "lost event" investigations. Dual-write bugs show up as rare, hard-to-reproduce inconsistencies between services. Each one typically means reading logs across several systems and writing a one-off backfill script. With the outbox, an event exists if and only if its transaction committed, so this class of investigation largely disappears. The remaining failure mode, connector lag, is visible in metrics.
- Simpler tests. Whether the right event is produced becomes a database assertion: after the operation, check the outbox row. Integration tests for the business logic do not need a running broker.
- A repeatable pattern. Once one service uses the outbox, onboarding another is mostly a new table, a migration, and a connector config entry, rather than a new design discussion about delivery guarantees.
- Replay and backfill from one place. Because events flow through Kafka topics with retained offsets, a new consumer can read history from the topic instead of asking the producing team for a custom export.
As an illustrative estimate with assumed inputs: suppose an organization has an assumed 8 event-producing services, and building and maintaining a hand-rolled retry and reconciliation mechanism costs an assumed 2 engineer-weeks per service up front plus an assumed 1 engineer-day per service per quarter for fixes and investigations. That totals 16 engineer-weeks initially and about 8 engineer-days per quarter. Against that, set the assumed cost of running the shared pipeline: initial Kafka Connect and Debezium setup, upgrades, and on-call ownership. The outbox approach saves time only when the per-service costs you avoid exceed that shared operating cost. With few services or no existing Kafka Connect deployment, it may not.
When should you use transactional outbox Debezium, and what are the alternatives?
That pipeline pays off when a team already runs Kafka and needs several services to publish events reliably from a relational database. It is the wrong choice for low-volume systems, where a scheduled poller is simpler. It also does not fit teams that cannot own a connector on call, or managed databases that restrict access to the replication log.
| Approach | Consistency | Latency | DB load | Operational burden | Typical failure mode |
|---|---|---|---|---|---|
| Outbox + Debezium | Atomic with business write | Low, log-driven | Log reading only | Kafka Connect, replication slots, offsets | Connector lag or a stalled slot that retains log on the database |
| Polling publisher | Atomic with business write | Bounded by poll interval | Repeated queries on the outbox | Low, runs in-process | Duplicates after a crash, lock contention between pollers |
| Naive dual write | None | Lowest | None | Lowest | Silent loss or phantom events |
| Two-phase commit | Atomic across participants | Higher, blocking | Locks held during prepare | High | Blocked in-doubt transactions; Kafka has no XA support |
| Listen-to-yourself / event sourcing | Eventual; log is the source of truth | Low | Depends on projections | Redesign of the data model | Stale reads before the service consumes its own event |
Decision guide
- Small scale, no Kafka Connect: start with a polling publisher. It gives the same atomicity guarantee.
- Already running Kafka Connect, need the same pattern across many services: use Debezium. A single pipeline replaces a separate poller in every service.
- The event stream is already your system of record: event sourcing removes the outbox entirely.
- Naive dual writes: never acceptable.
- Two-phase commit: rarely practical for these pipelines, because Kafka does not support XA and so cannot take part in a distributed transaction with the database.
When should it be avoided?
Avoid the outbox with Debezium in these situations:
- You do not run Kafka or Kafka Connect and have no other reason to. Standing up a broker cluster, a Connect cluster, and their monitoring to publish a handful of events adds more operational surface than it removes. A polling publisher against the outbox table gives the same atomicity with components you already have.
- Your database does not expose its change log. Some managed database offerings restrict or do not support logical replication or row-based binlog access for external clients. Without that access, Debezium cannot read the log, and a polling publisher is the fallback.
- No team can own the connector. A stalled replication slot keeps WAL on the primary until the disk fills, and a connector that loses its log position may need a re-snapshot. If nobody is on call for that, the pipeline becomes a new source of outages.
- Consumers need the result synchronously. The outbox is eventually consistent. If the caller must know, inside the same request, that another service has acted on the change, an event is the wrong tool. Use a synchronous call, or redesign the workflow so it does not require that guarantee.
- The event does not need to be reliable. Telemetry, analytics pings, or cache-warming hints where occasional loss is acceptable do not justify writing an outbox row in every transaction. Publish them directly.
- Correctness depends on global ordering across aggregates. The pattern orders events per aggregate key only. If consumers need a single total order across all entities, partitioning by aggregate id will not provide it, and you need a different design, such as a single partition with its throughput limits, or modeling the related entities as one aggregate.
- The database is already the bottleneck. Every event adds a row insert, plus a delete or later cleanup, to the business transaction, along with extra log volume and log-reading load from the connector. On a database with no write headroom, measure that cost before adopting the pattern.
- Everything lives in one process. In a monolith where the "consumers" are modules in the same codebase and the same database, in-process handlers within the same transaction are simpler and strictly consistent.
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to transactional outbox debezium, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Changing a shared variable is one edit and one review instead of editing every service's
.envby hand.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.