What is Debezium Postgres CDC and how does logical replication make it work?
Change data capture (CDC) turns committed database writes into an ordered stream of events. You don't query tables for differences. A CDC tool reads the database's own record of what changed. For Postgres, that record is the write-ahead log (WAL), and Debezium consumes it through logical replication.
By default, the WAL contains physical page changes, which are useful only to an identical server. Setting wal_level=logical makes Postgres write enough extra information to reconstruct row-level changes. A logical decoding plugin then turns WAL entries into inserts, updates, and deletes. Debezium supports pgoutput, the plugin built into Postgres, so you don't need to install a server extension.
Two server objects control the stream:
- A publication defines which tables are included.
- A replication slot tracks the last position, or LSN, that the consumer has confirmed. Postgres keeps every WAL segment after that position until the consumer acknowledges it.
The data path works like this:
- A transaction commits.
- Logical decoding emits its changes in commit order through the slot. Uncommitted or rolled-back work never appears.
- The Debezium connector, running as a Kafka Connect task, converts each row change into an event.
- The connector writes each event to a topic for its table. Topic names default to the pattern
<prefix>.<schema>.<table>. - After Kafka acknowledges the write, Debezium records the LSN in Connect's offsets and confirms it back to the slot.
This makes Debezium a replica in every sense that matters operationally. If the connector stops, the slot pins WAL on the primary, and disk usage grows until the connector catches up or someone drops the slot.
+--------------------------- Postgres primary ---------------------------+
| (1) txn COMMIT ---> WAL (wal_level=logical: row-level detail) |
| | |
| v |
| publication ------> logical decoding: pgoutput |
| (table list) | (2) committed changes only, commit order |
| v |
| replication slot (confirmed LSN; pins WAL after it) |
| | ^ |
+-----------------------|-------------------------------|----------------+
| change events in commit order | (5) confirm LSN
v |
+-------------------- Kafka Connect worker -------------|----------------+
| (3) Debezium Postgres connector task | |
| row change --> event --> Kafka producer | |
| | +------+ |
| (5) on ack: LSN to offsets | ^ |
+----------------------------------------|-------|-----------------------+
v | ack
+-------------------- Kafka ---------------------|-----------------------+
| (4) one topic per table: <prefix>.<schema>.<table> |
| app.public.orders app.public.customers ... |
+------------------------------------------------------------------------+
Committed transactions flow from the WAL through pgoutput and the replication slot to the Debezium connector in Kafka Connect, which writes per-table topics and, only after Kafka acknowledges each write, stores the LSN in Connect offsets and confirms it back to the slot.
Why and when should you use change data capture instead of dual writes or polling?
The reason to accept that operational coupling is that the alternatives each give up something harder to recover. The classic failure is the dual write. A service commits to Postgres and then publishes to Kafka. If the process crashes between the two, or the publish fails after the commit, downstream systems diverge from the source of truth, and nothing records that it happened. Change data capture removes the second write. The WAL entry is the event, so an event exists if and only if the transaction committed.
Polling avoids dual writes but adds different problems. A query like "rows where updated_at > last seen" misses hard deletes and collapses intermediate updates. It can also skip rows whose transactions commit out of timestamp order. It adds query load on every interval, too. Application-level publishing and LISTEN/NOTIFY are closer to real time. However, NOTIFY is not durable, so a disconnected listener simply misses messages.
| Approach | Consistency with DB | Latency | Source load | Operational cost |
|---|---|---|---|---|
| Dual write | Can diverge on crash | Low | Low | Low until it breaks |
| Timestamp polling | Misses deletes, reorderings | Poll interval | Repeated queries | Low |
| App-level publish | Same as dual write unless paired with an outbox | Low | Low | Code in every service |
LISTEN/NOTIFY |
Lost if listener is down | Low | Low | Low, but no replay |
| Trigger audit table | Transactional | Depends on shipper | Extra writes per change | Triggers plus a shipper |
| Log-based CDC | Transactional, ordered per row | Low | WAL reading | Kafka Connect, slot monitoring |
CDC fits teams whose services already own a Postgres database and need reliable event publication without rewriting every write path.
How do you prepare Postgres and run the Debezium connector on Kafka Connect?
Prepare Postgres
- Enable logical decoding. Set
wal_level = logicalin the server configuration and restart Postgres. This setting cannot be changed with a reload, so schedule the restart. While you're there, check thatmax_replication_slotsandmax_wal_sendersleave room for the connector. - Create a dedicated role. Give it
LOGINandREPLICATION, plusSELECTon every captured table.REPLICATIONlets it open a replication slot and stream WAL.SELECTis needed for the initial snapshot. Keeping this role separate from application roles makes it easy to audit and revoke. - Decide who owns the publication. Debezium can create the publication itself, but that requires more privileges. A publication for all tables needs superuser, and a publication for specific tables requires ownership of those tables. In most production setups, a DBA creates the publication for the exact tables you need. Debezium is then configured not to auto-create one (
publication.autocreate.mode=disabled).
Register the connector
Post a config to Kafka Connect with these properties:
- Connection details:
database.hostname,database.port,database.user,database.password,database.dbname plugin.name=pgoutput: uses Postgres's built-in decoder, so no server extension is required.slot.name: use one unique slot per connector. Two connectors sharing a slot will conflict.publication.name: the publication you created.table.include.list: fully qualified names such aspublic.orders.topic.prefix: a logical name for this source database.
Each table's events go to a topic named prefix.schema.table, so public.orders with prefix shop lands in shop.public.orders. Choose the prefix carefully. Changing it later moves events to new topics.
To confirm the connector is attached, query pg_replication_slots. Your slot should appear with plugin = 'pgoutput' and active = true.
# 1. Enable logical decoding (as superuser). Takes effect only after a restart.
psql -h db.example.internal -U postgres -d shop -c "ALTER SYSTEM SET wal_level = logical;"
# ...restart Postgres, then confirm: SHOW wal_level; -- should return 'logical'
# 2. Dedicated CDC role, and 3. a DBA-owned publication for the exact tables
psql -h db.example.internal -U postgres -d shop <<'SQL'
CREATE ROLE debezium WITH LOGIN REPLICATION PASSWORD 'change-me';
GRANT USAGE ON SCHEMA public TO debezium;
GRANT SELECT ON public.orders, public.outbox TO debezium; -- needed for the snapshot
CREATE PUBLICATION dbz_shop_pub FOR TABLE public.orders, public.outbox;
SQL
# 4. Register the connector with Kafka Connect's REST API
curl -s -X POST -H 'Content-Type: application/json' \
-d '{
"name": "shop-postgres-cdc",
"config": {
"connector.class": "io.debezium.connector.postgresql.PostgresConnector",
"database.hostname": "db.example.internal",
"database.port": "5432",
"database.user": "debezium",
"database.password": "change-me",
"database.dbname": "shop",
"plugin.name": "pgoutput",
"slot.name": "dbz_shop_slot",
"publication.name": "dbz_shop_pub",
"publication.autocreate.mode": "disabled",
"table.include.list": "public.orders,public.outbox",
"topic.prefix": "shop"
}
}'
# Events land in topics shop.public.orders and shop.public.outbox
# 5. Verify the connector created its slot and is streaming from it
psql -h db.example.internal -U postgres -d shop -c \
"SELECT slot_name, plugin, slot_type, active FROM pg_replication_slots WHERE slot_name = 'dbz_shop_slot';"
End-to-end setup: enable logical WAL, create a replication role and DBA-owned publication, register a pgoutput-based Debezium connector with topic prefix 'shop', and confirm its slot in pg_replication_slots.
How do you read a Debezium change event, and what happens during snapshots and restarts?
Each change event carries an envelope with four fields that matter: before, after, op, and source. The source block records the database, schema, table, transaction ID, commit timestamp, and the LSN. The LSN gives you a total order within the server and a value you can compare when deduplicating.
op |
Meaning | before |
after |
|---|---|---|---|
c |
Insert | null | new row |
u |
Update | old row (depends on replica identity) | new row |
d |
Delete | old row or key only | null |
r |
Snapshot read | null | current row |
With the default replica identity, Postgres writes only the primary key of the old row to the WAL. On deletes, before holds just the key. On updates, before is usually null. ALTER TABLE ... REPLICA IDENTITY FULL makes Postgres log the entire old row. Consumers can then diff values or audit deletes. The cost is real: every update and delete writes the full prior tuple, which inflates WAL volume and replication bandwidth. Enable it only on tables where consumers need the old values.
On first start, the default initial mode reads each table inside a consistent transaction and emits r events. It then switches to streaming from the slot position captured at the start of the snapshot, so no change falls into the gap. After each event batch, Kafka Connect commits the last processed LSN to its offsets topic. The connector then reports that position back to Postgres, which advances the slot's confirmed_flush_lsn and lets Postgres discard older WAL. On restart, streaming resumes from the stored offset. Events after the last committed offset may be delivered again.
For very large tables, a blocking initial snapshot can run for hours while the slot holds WAL. Use no_data, or never on older releases, to start streaming without one. Then backfill with incremental snapshots triggered through a signaling table. These read in chunks and interleave with live changes.
-- 1. Check the current replica identity of the captured tables.
-- relreplident: d = default (primary key), f = full, i = index, n = nothing
SELECT c.relname, c.relreplident
FROM pg_class c
JOIN pg_namespace n ON n.oid = c.relnamespace
WHERE n.nspname = 'public'
AND c.relname IN ('orders', 'customers');
-- 2. Log the full old row only where consumers need old values.
-- After this, every UPDATE and DELETE on orders writes the full prior tuple to WAL.
ALTER TABLE public.orders REPLICA IDENTITY FULL;
-- To revert to key-only before images:
-- ALTER TABLE public.orders REPLICA IDENTITY DEFAULT;
-- 3. Inspect the slot during a long snapshot or after a restart.
-- confirmed_flush_lsn advances only after the connector reports
-- the offsets that Kafka Connect has committed.
-- retained_wal shows how much WAL the slot is holding back.
SELECT slot_name,
plugin,
active,
restart_lsn,
confirmed_flush_lsn,
pg_size_pretty(
pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)
) AS retained_wal
FROM pg_replication_slots
WHERE slot_type = 'logical';
Checks and sets REPLICA IDENTITY on a captured table, then shows how much WAL a Debezium logical slot is holding and how far its confirmed_flush_lsn has advanced.
How do you implement the transactional outbox pattern with Debezium?
Capturing a service's tables directly turns its internal schema into a public API. If you rename a column, split a table or change a type, every downstream consumer breaks. Those consumers also see row-level churn instead of business facts. The transactional outbox fixes this. The service writes explicit events to a dedicated table, and Debezium publishes only that table.
Write the event in the same transaction
In one database transaction, the service updates its business row, such as inserting an order, and inserts a row into outbox. That row needs five columns:
id: a UUID, which consumers use for deduplicationaggregatetype: for example,Orderaggregateid: the order's IDtype: for example,OrderCreatedpayload: a JSONB document shaped for consumers, not copied from the table
Both writes commit or roll back together. This removes the dual-write gap without a distributed transaction.
Configure the connector
Set table.include.list to the outbox table only. Then add the outbox event router SMT, io.debezium.transforms.outbox.EventRouter. With its defaults, it routes each event to a topic derived from aggregatetype, such as outbox.event.Order. It sets the Kafka message key to aggregateid and uses payload as the message value. Because the key is the aggregate ID, all events for one order land in the same partition and stay in order.
Cleanup and cost
Debezium reads from the WAL, not from the table. You can therefore delete outbox rows right after inserting them, even in the same transaction. The router ignores the resulting delete events. Alternatively, run a periodic purge job. Either way, the outbox table stays small.
The trade-off is development cost. Every state change needs an extra insert and deliberately designed event payloads. You also have to version those payloads as a contract.
-- Outbox table: the only table Debezium captures (table.include.list=public.outbox)
CREATE TABLE outbox (
id uuid PRIMARY KEY, -- event ID; consumers deduplicate on it
aggregatetype varchar(255) NOT NULL, -- EventRouter routes on this, e.g. outbox.event.Order
aggregateid varchar(255) NOT NULL, -- becomes the Kafka message key
type varchar(255) NOT NULL, -- business event name
payload jsonb NOT NULL -- consumer-facing contract, not a row copy
);
-- One transaction: the business write and the event commit or roll back together
BEGIN;
INSERT INTO orders (id, customer_id, total_cents, status)
VALUES ('ord-1001', 'cust-42', 4999, 'CREATED');
INSERT INTO outbox (id, aggregatetype, aggregateid, type, payload)
VALUES (
gen_random_uuid(), -- built in since PostgreSQL 13; otherwise generate the UUID in the app
'Order',
'ord-1001',
'OrderCreated',
jsonb_build_object(
'orderId', 'ord-1001',
'customerId', 'cust-42',
'totalCents', 4999
)
);
-- Optional immediate cleanup: the INSERT is already in the WAL, so Debezium
-- still emits the event; the EventRouter skips the resulting delete event.
DELETE FROM outbox WHERE aggregateid = 'ord-1001' AND type = 'OrderCreated';
COMMIT;
An outbox table schema and a single transaction that writes an order row and its OrderCreated event, then deletes the outbox row. Debezium still captures the event from the WAL.
How should consumers handle duplicates, ordering, and tombstones?
Debezium delivers at least once. The connector commits source offsets periodically, so after a crash or restart it re-emits changes since the last committed position. On the consumer side, a group rebalance hands partitions to another instance, which resumes from the last committed Kafka offset and reprocesses anything in flight. Design every consumer to expect duplicates.
Idempotency keyed on event ID
Give each event a stable ID. The outbox row's primary key works well. In the consumer's own database transaction, insert that ID into a processed_events table with a unique constraint and apply the business change. If the insert conflicts, the event was already handled, so skip it. Because both writes commit or roll back together, a crash between them cannot leave the change applied without a record of it. Commit the Kafka offset only after the database transaction succeeds.
Ordering
Kafka orders messages only within a partition. Key every event by aggregate ID, such as an order ID, so all changes to one entity land on the same partition and are processed in sequence. Do not assume any ordering across different aggregates.
Deletes and tombstones
A row delete produces two messages. The first is a delete event with op set to d and a null after. The second is a tombstone with the same key and a null value, which lets log compaction remove the key. Act on the delete event. Treat a null-value tombstone as a no-op rather than letting deserialization throw.
Schema evolution
Register schemas in a schema registry and enforce backward compatibility. A rename looks like a dropped field plus an added one, so add the new column first and migrate consumers before you remove the old one. New fields need defaults, and consumers should ignore fields they do not recognize.
When should you avoid Debezium, and how do you operate it safely in production?
Debezium is not free to run, and some teams are better served without it.
| Situation | Better fit | Why |
|---|---|---|
| Small team with no one to own Kafka Connect | Outbox table plus a polling worker | Connectors, offsets and replication slots all need ongoing attention |
| Low event volume, latency of seconds is fine | Polling outbox | A query on an indexed created_at column is simpler to debug than a replication stream |
| Managed Postgres where logical slots are lost on failover | Polling, or confirm slot failover support first | After a promotion the connector has no slot to resume from and silently misses changes |
If you do run it, the replication slot is the main risk. Postgres keeps WAL until the slot's restart_lsn advances. If the connector stops, WAL accumulates until the disk fills and the primary goes down. Query pg_replication_slots on a schedule and compute retained WAL with pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn). Alert on that value and on active = false well before free disk space becomes critical, so the alert fires while there is still time to act.
On quiet databases, enable Debezium heartbeats. Without them, a busy database elsewhere on the server can generate WAL while the captured tables see no writes, so the slot never advances.
Keep a written runbook for two procedures: dropping a stalled slot, and re-snapshotting after a failover. Both are destructive, and the middle of an incident is a bad time to work them out.
These costs are paid up front. In return, you remove dual-write bugs and the custom sync jobs that tend to collect around every service boundary.
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to debezium postgres kafka, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.