What is backfilling a microservice database from events, and when is the event log enough?
Backfilling a microservice database from events means replaying historical events through a dedicated, idempotent consumer to build a new service's state, then stopping at a recorded cutover point (a set of per-partition offsets) where live consumption takes over. Resetting an existing consumer group to the earliest offset and letting it run is a different thing. A reset gives you no boundary to verify against. It also gives you no way to tell whether a write came from history or from live traffic.
Teams use a backfill when a new service needs history it never consumed, such as order states, balances or entitlements. This is common for backend and platform engineers who are splitting a monolith or rebuilding a service with a new data model.
The deciding test
The log can act as the source of truth only if every live entity's current state can be rebuilt from what the broker still holds and what your consumers can still decode.
| Log property | Suitable for backfill | Risky for backfill |
|---|---|---|
| Time-retained topic | Retention outlives every entity you need | Old entities' creation events already deleted |
| Compacted topic | You need latest state per key | You need intermediate history or audit trail |
| Tombstones | Replay finishes within the tombstone retention window | A slow replay reads an old value, and the tombstone is removed before the consumer reaches it, so the delete is missed |
| Schema history | Every historical schema version is still retrievable | Early versions lost, so old payloads are undecodable |
| Event payload | Full state per event, so gaps are survivable | Deltas only, so every event since creation must exist and be applied in order |
If any row lands in the risky column for data you need, the log alone is not enough.
Who should use an event-based backfill?
This approach suits teams that consume an event stream they cannot or should not bypass. Typical users include:
- Backend engineers building a new service that needs state another service has been publishing for a long time, such as a fulfilment service that needs every open order.
- Platform and data engineers splitting a monolith, where the event stream is the agreed contract between the old system and the new service, and direct access to the monolith's tables is off limits or about to disappear.
- Teams maintaining read models or CQRS projections that need to rebuild a projection after changing its data model or fixing a bug in the projection logic.
- Teams moving a service to a new datastore, where replaying events into the new store is easier to verify than copying tables and translating their schema.
It is a good fit when the team can run a long job against the broker, has write access to its own target database, and can coordinate with the owning service on verification. It is a poor fit for a team that needs a one-time copy of a small table it could simply export, or for any case listed under "When to avoid this approach" at the end of this article.
How does an event-based backfill work?
At a high level, a safe backfill runs these steps:
- Check the log. Confirm with the deciding test above that retention, compaction, schema history and payload shape let you rebuild every entity you need. If they do not, plan a snapshot for the part the log cannot cover.
- Choose a starting point. Either start from the earliest retained offset, or bulk-load a consistent snapshot and record the log position P it corresponds to.
- Run a dedicated consumer group. The backfill reads under its own
group.id, so the production group's offsets and lag stay untouched. - Normalize old events. Each event goes through an upcaster chain that converts historical schema versions into the current model.
- Write idempotently. Each event becomes a conditional upsert keyed by entity ID and guarded by a per-entity version, so duplicates and stale events are no-ops.
- Suppress side effects. A backfill mode stops emails, webhooks, billing and outgoing events while history is replayed.
- Catch up and record the cutover point. When backfill lag is near zero, record the per-partition offsets where history ends and live consumption takes over.
- Verify and cut over. Compare counts, checksums and sampled rows against the owning service, run shadow reads, and then move reads behind a flag.
The sections below cover each step in detail.
What are the alternatives to full event replay, and how does snapshot-plus-replay work?
| Approach | Works well when | Main trade-off |
|---|---|---|
| Full replay from the log | Retention covers the entity's whole history | Slow, and every historical schema must still parse |
| Snapshot-plus-replay | Log retention is shorter than entity lifetime | Snapshot must be tied to an exact log position |
| Direct DB copy or dual-write | You control both services' storage | Couples you to another team's schema; dual-writes drift on partial failure |
| Owning service's API | Low volume, well-paged endpoints | Loads production, and the API rarely exposes a consistent point in time |
Snapshot-plus-replay bulk-loads current state, then uses the log only for changes made after that state was captured:
owning service ──► consistent snapshot ──► backfill job ──┐
│ │ ▼
│ log position marker (P) target database
▼ │ ▲
event log ─────────────┴─ replay from P ──► live consumer ┘
The snapshot comes from a database and the events come from a log, and the two rarely share a transaction. If the service publishes after it commits, no single offset matches "the state at snapshot time." If P is too late, events written between the snapshot read and P are never applied, and nothing reports the gap. If P is too early, events already reflected in the snapshot get applied again. A full-state upsert absorbs that. A delta such as "add 5 to balance" corrupts the row.
The usual fix is to pick a P that is safely early and make every write compare a per-entity version, so replayed events become no-ops. Another option is an outbox table read inside the same transaction as the snapshot, which gives you the exact position.
+-----------------------------+
| Owning service |
| (database + publisher) |
+-----------------------------+
| |
| consistent read | publish after commit
v v
+------------+ +-------------------------------------------+
| Snapshot | | Event log |
| + marker P |--->| ... e3 e4 [P] e5 e6 e7 ... |
+------------+ +-------------------------------------------+
| |
v | replay from P onward
+------------+ v
| Backfill | +-----------------------------+
| job | | Live consumer |
+------------+ | apply if event.version > |
| | row.version, else no-op |
| bulk upsert +-----------------------------+
| |
v v
+----------------------------------------------------+
| Target database |
| one row per entity id, with stored version |
+----------------------------------------------------+
P too early: events already in the snapshot are applied again
P too late: events between snapshot read and P are silently lost
Snapshot-plus-replay: the backfill job bulk-loads a consistent snapshot tagged with log position P, and the live consumer replays from P using per-entity version checks so that an early P causes only no-ops while a late P leaves gaps that nothing reports.
How do you make consumers idempotent and order-tolerant during event replay?
Replay will deliver some events more than once. Retries after a consumer crash, overlapping offset ranges between the backfill and the live consumer, and a snapshot boundary chosen to be early all produce duplicates. The consumer has to treat every event as possibly already applied.
Upsert by entity ID, guarded by version
Write each entity with an upsert keyed by its ID, never a blind insert or an increment. Store the last applied event version alongside the row. This can be a producer-assigned sequence number or, for a single-partition entity, the Kafka offset. The write applies only if the incoming version is greater than the stored one. A duplicate or stale event then matches zero rows and becomes a no-op. That is why a replay can safely overlap live traffic. The conditional upsert shown below does this in a single statement, so there is no gap between read and write where two consumers could race.
This has costs. Every row carries extra version columns. Every write becomes a compare-and-set, which adds index lookups and causes more contention on hot entities. Consumer code must also separate "skipped as stale" from "failed", or real errors can be hidden as no-ops.
Ordering is per partition, not global
Kafka orders events only within a partition. If one read model combines several keys or topics, such as orders joined with customer records, replay can interleave them in a sequence that never happened in production. Per-entity versions protect each entity's own row. They do not protect derived state that spans entities. Design that state to converge regardless of arrival order, or recompute it after the backfill completes.
-- Read model: one row per order, plus the version of the last event applied to it.
CREATE TABLE IF NOT EXISTS orders_read (
order_id text PRIMARY KEY,
status text NOT NULL,
total_cents bigint NOT NULL,
last_version bigint NOT NULL, -- producer sequence number or per-entity offset
updated_at timestamptz NOT NULL DEFAULT now()
);
-- Conditional upsert (PostgreSQL). The consumer runs this once per event.
-- $1 = order_id, $2 = status, $3 = total_cents, $4 = event version
INSERT INTO orders_read AS cur (order_id, status, total_cents, last_version)
VALUES ($1, $2, $3, $4)
ON CONFLICT (order_id) DO UPDATE
SET status = EXCLUDED.status,
total_cents = EXCLUDED.total_cents,
last_version = EXCLUDED.last_version,
updated_at = now()
-- Apply only strictly newer versions. A duplicate or stale event
-- matches no row and leaves the stored state unchanged.
WHERE cur.last_version < EXCLUDED.last_version
-- RETURNING yields a row only when the insert or update was applied.
-- No returned row with no error means "skipped as stale". The consumer
-- logs or counts that case and commits the offset. A raised error is a
-- real failure and must not be committed as if it were a no-op.
RETURNING cur.order_id, cur.last_version;
A PostgreSQL upsert keyed by entity ID that applies an event only if its version is newer than the stored one, so a consumer can tell a stale duplicate (no row returned) apart from a real failure (an error).
How should you run a Kafka backfill without disrupting live consumers?
Run the replay under its own consumer group, either inside the new service or as a dedicated backfill job, and leave the production group's offsets alone. Rewinding the production group with --reset-offsets looks simpler, but it couples two jobs that fail differently. If the service crashes halfway through a rewind, live traffic stalls behind years of history. On restart, the store holds a mix of fully rebuilt entities and stale ones, and nothing records which is which. A separate group lets you throttle, pause, or restart the replay while live consumption continues normally.
Give the backfill group a distinct group.id and set auto.offset.reset=earliest so it starts at the beginning of retained history. Set enable.auto.commit=false and commit only after a batch is durably written. Raise max.poll.records to suit large batch writes. Check both groups with kafka-consumer-groups.sh --bootstrap-server <host:port> --describe --group <group>. The LAG column shows how far the backfill has to go and confirms that live lag is not growing.
Isolate the load on the target database as well:
- Batch writes. Write many rows per transaction instead of one at a time.
- Defer secondary indexes. Where your database allows it, build them after the bulk load.
- Use an explicit backfill mode. This mode suppresses outgoing events, notifications, webhooks, and billing actions. Replaying a historical "order paid" event must not charge a customer again.
The backfill mode is a code path that rarely runs. Give it dedicated tests asserting that no side effects escape, because production traffic will never exercise it.
#!/usr/bin/env bash
set -euo pipefail
BOOTSTRAP="kafka-1.internal:9092"
LIVE_GROUP="orders-service"
BACKFILL_GROUP="orders-service-backfill-2026-11"
cat > backfill-consumer.properties <<'EOF'
bootstrap.servers=kafka-1.internal:9092
group.id=orders-service-backfill-2026-11
client.id=orders-backfill
auto.offset.reset=earliest
enable.auto.commit=false
max.poll.records=2000
max.poll.interval.ms=600000
isolation.level=read_committed
EOF
kafka-consumer-groups.sh --bootstrap-server "$BOOTSTRAP" --list
kafka-consumer-groups.sh --bootstrap-server "$BOOTSTRAP" \
--describe --group "$BACKFILL_GROUP"
kafka-consumer-groups.sh --bootstrap-server "$BOOTSTRAP" \
--describe --group "$LIVE_GROUP"
This example creates a consumer config for a dedicated backfill group with manual commits and large polls, then uses kafka-consumer-groups.sh to compare backfill lag with live-group lag.
How do you handle schema evolution across years of events?
A topic that has existed for years rarely holds one event shape. Fields get renamed, amounts move from floats to integer cents, and a single AddressChanged event gets split into ShippingAddressChanged and BillingAddressChanged. The backfill consumer should not carry a separate handler for every historical variant. Instead, put a chain of upcasters in front of the handler. Each upcaster converts version n into version n+1. Events are migrated step by step until they reach the current model, and only that model reaches your projection logic.
The chain is easier to maintain than one large converter. Each step is small, can be tested on its own, and stays unchanged when a new version is added. The cost is that upcasting happens on every read, which matters during a backfill of hundreds of millions of records. Measure it before you assume it is negligible.
Test the replay against a sample of the oldest data before you run it in full. The earliest events are the ones written by code nobody remembers. They are most likely to contain nulls where you expect values, undocumented enum members, or timestamps in a different zone. A failure at hour six of a full run is far more expensive than one found on a ten-thousand-event sample.
The deeper risk is semantic. When old schemas are poorly documented, writing an upcaster means deciding what a field meant. For example, you may have to decide whether a missing currency field implied the account's default currency. Each of those decisions encodes a business rule that was never written down. Record each one next to the upcaster and get it reviewed by someone who owns the domain.
Expect some rebuilt records to disagree with the owning service's current view. This is a policy question, not a bug to patch silently. Decide in advance whether the owning service is authoritative, whether the event log is, or whether mismatches go to a person for review.
from decimal import Decimal, ROUND_HALF_UP
CURRENT_VERSION = 3
def v1_to_v2(event):
# v1 called the field "amt". v2 renamed it to "amount".
data = dict(event["data"])
data["amount"] = data.pop("amt")
return {**event, "version": 2, "data": data}
def v2_to_v3(event):
# v3 stores integer cents instead of floats. str() avoids float artifacts.
data = dict(event["data"])
cents = Decimal(str(data.pop("amount"))) * 100
data["amount_cents"] = int(cents.quantize(Decimal("1"), rounding=ROUND_HALF_UP))
# BUSINESS RULE (unwritten until now, needs domain-owner review):
# pre-v3 events had no currency, so assume the account's default currency.
data.setdefault("currency", event["account_default_currency"])
return {**event, "version": 3, "data": data}
UPCASTERS = {1: v1_to_v2, 2: v2_to_v3}
def upcast(event):
# Events written before versioning existed carry no version field.
version = event.get("version", 1)
while version < CURRENT_VERSION:
step = UPCASTERS.get(version)
if step is None:
# Fail loudly. Do not let an unknown shape reach the projection.
raise ValueError(f"no upcaster for version {version}: {event.get('id')}")
event = step(event)
version = event["version"]
if version != CURRENT_VERSION:
raise ValueError(f"event newer than consumer: version {version}")
return event # only the current model reaches projection logic
A versioned upcaster chain migrates each event one version at a time until it reaches the current model, fails on unknown versions, and records the inferred-currency business rule next to the step that applies it.
How long will the backfill take, and how does it affect development cost?
Duration is set by the slowest stage in the pipeline, so compute each stage's ceiling before choosing parallelism.
Illustrative estimate (assumed numbers, not a benchmark). Assume 600 million events across 24 partitions. One consumer handles 5,000 events/s per partition, and the target database sustains 40,000 writes/s.
- Consumer ceiling = partitions × per-partition throughput = 24 × 5,000 = 120,000/s
- Effective rate = min(consumer ceiling, database write rate) = 40,000/s
- Useful parallelism = database rate ÷ per-partition throughput = 8 consumers
- Duration = events ÷ effective rate
The database is the bottleneck. Running all 24 partitions in parallel would only add lock contention and retries, so run 8.
| Mode | Effective rate | Duration | Live events to catch up (at an assumed 500/s) |
|---|---|---|---|
| Unthrottled | 40,000/s | ~4.2 h | ~7.5M |
| Throttled to 50% | 20,000/s | ~8.3 h | ~15M |
Throttling protects brokers and other tenants on the cluster, but it doubles the run time. A longer run means more live writes land during it, so the reconciliation window grows and the catch-up phase gets longer.
Most of the development cost comes before the long run:
- Idempotency keys and a version check on every write.
- Backfill-mode flags that suppress side effects such as emails.
- Several test runs against a sample-sized dataset.
- Days of dual-run time before cutover.
That investment pays off on reruns. A restartable job resumes from committed offsets after a failure. An ad hoc script, or a full rewind from offset zero, starts over each time. A reusable verification job, such as count and checksum comparison per key range, lets every rerun be checked the same way at no extra cost.
How can an event-based backfill save engineering time?
The upfront work described above is real, but a well-built backfill removes several recurring sources of engineering effort:
- Less cross-team coordination. The new service reads the event stream it already has access to. It does not need credentials for another team's database, a custom export endpoint, or a dump-and-transform schedule negotiated with the owning team.
- No dual-write cleanup. Dual-writes drift when one write succeeds and the other fails, and someone has to find and repair those rows by hand. A replay through idempotent upserts converges on its own when it is rerun.
- Failures cost a restart, not a rewrite. Because offsets are committed only after durable writes and every write is version-guarded, a crash or a bad deploy means restarting the job. Nobody has to work out which rows were half-applied.
- Projection bugs become cheap to fix. When a bug in projection logic corrupts the read model, the fix is to correct the code and replay, instead of writing a one-off repair script for each bad field.
- Schema changes reuse the same pipeline. When the read model changes, you add an upcaster step or adjust the projection and run the same job into a new table. The consumer group setup, side-effect suppression, and verification job are already built and tested.
- Verification is automated once. The count, checksum and sample comparison written for the first cutover keeps running as scheduled reconciliation, which replaces manual spot checks after each release.
Whether this saves time overall depends on how often you will rebuild. A service that will be backfilled once, from a small table it could export directly, may be better served by a one-time copy. A read model that will be rebuilt every time its data model or projection logic changes recovers the setup cost on the second or third run, because each rerun reuses tooling instead of starting a new migration project.
How do you cut over and verify the data migration, and when should you avoid this approach?
Cutover sequence
Start the cutover only when consumer lag has stopped shrinking and holds near zero under normal production traffic. Lag that is still falling means the new database is still catching up. Then switch reads in stages, behind a flag you can turn off without a deploy. For a shadow period, keep serving responses from the owning service while also querying the new one and logging mismatches. Shadow reads expose problems that bulk checks miss, such as stale caches, timezone handling, and default values the projection fills in differently.
Verification against the owner
Compare three things, from cheapest to most precise:
- Entity counts per type and per key range
- Checksums over normalized fields in the same ranges
- Sampled field-level comparisons to locate rows that differ
Counts and checksums tell you that two ranges differ. The sample tells you which rows and fields. The example below picks random IDs from the owner, fetches the same rows from both stores, and reports per-field differences. It skips rows whose last update falls within the current lag window, because those rows may legitimately not have arrived yet.
Reconciliation after cutover
Keep the verification job running on a schedule after cutover. Have it alert on drift and re-project the affected keys. This catches dropped events and bugs in new consumer code.
When to avoid this approach
Avoid an event-based backfill in these cases:
- Retention or compaction has removed early history.
- Events carry only deltas, and the events that set initial state are gone.
- Side effects such as emails or webhooks cannot be reliably suppressed during replay.
- A direct snapshot from the owning service is simpler and gives you everything you need.
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to backfill microservice database, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Changing a shared variable is one edit and one review instead of editing every service's
.envby hand.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.