Eventual Consistency in Practice: Guarantees Across Services

What eventual consistency actually guarantees, and what it does not

Eventual consistency promises one thing: if writes stop, every replica will converge on the same value. It says nothing about when that happens or what a reader sees in the meantime. "Eventually" has no fixed upper bound. Under normal load it might mean milliseconds. During a consumer outage or a backlog it might mean hours.

Engineers usually reason with a ladder of stronger models. Each one rules out a specific anomaly that users can see:

Splitting a system into services does not give you any of these guarantees by default. When one service reads another's data from a projection or cache, you start at plain eventual. Anything stronger is a choice you make and then build, for example with session tokens, version checks, or routing a user's reads to the primary. You also need to make the guarantee visible, so people can tell which one a given read actually has.

Most consistency bugs are not caused by eventual consistency on its own. They come from code that quietly assumes a stronger guarantee than the one the system actually provides.

Why teams accept it: avoiding distributed transactions and the dual-write trap

Strong consistency across service boundaries usually means a distributed transaction such as two-phase commit (2PC), and most teams avoid it for concrete reasons. 2PC couples services at runtime: every participant must be up and responsive for any commit to complete, so one slow inventory database stalls checkout. During a network partition, participants that have voted "prepared" must hold their locks until the coordinator returns, which trades availability for consistency on every write. Those held locks also cause contention under load. In practice, support is the deciding constraint, since most message brokers and many managed datastores cannot act as XA participants at all.

The obvious workaround is to commit to the database and then publish an event. This creates the dual-write trap, because the two operations are not atomic and neither ordering is safe:

• Lost events. The commit succeeds and the process crashes, or the broker is briefly unreachable, before publishing. The order exists, but no downstream service ever learns about it, and nothing records that anything went wrong.
• Phantom events. You publish first and the commit then fails on a constraint violation or rollback. Consumers act on an order that does not exist.

Retries do not close either gap. They only make the gap narrower.

This guide is for backend and platform engineers who own service boundaries and decide how data crosses them. Eventual consistency is the right default when each service owns its data, cross-service reads can tolerate seconds of staleness, and invariants that must hold atomically can be kept inside a single service's database.

Copyimport json
import sqlite3

def create_order_commit_then_publish(db: sqlite3.Connection, broker, order: dict) -> None:
    # Ordering A: commit first, then publish.
    with db:  # commits on success, rolls back on exception
        db.execute(
            "INSERT INTO orders (id, customer_id, total_cents) VALUES (?, ?, ?)",
            (order["id"], order["customer_id"], order["total_cents"]),
        )
    # The row is now durable. If the process crashes here, or publish()
    # raises because the broker is unreachable, the event is lost:
    # the order exists, no consumer hears about it, and nothing records
    # that an event was owed.
    broker.publish("orders.created", json.dumps(order))


def create_order_publish_then_commit(db: sqlite3.Connection, broker, order: dict) -> None:
    # Ordering B: publish first, then commit.
    broker.publish("orders.created", json.dumps(order))
    # Consumers may already be acting on the event. If this INSERT fails
    # (e.g. a duplicate id violates the primary key) the transaction rolls
    # back and the event describes an order that never existed: a phantom.
    with db:
        db.execute(
            "INSERT INTO orders (id, customer_id, total_cents) VALUES (?, ?, ?)",
            (order["id"], order["customer_id"], order["total_cents"]),
        )

# Wrapping either call in retries only narrows the window between the two
# writes; the database and the broker still do not share one transaction,
# so neither ordering is atomic.

Both orderings of the naive dual write in Python, with comments marking where a crash or failed commit produces a lost event or a phantom event.

How reliable data synchronisation works: transactional outbox and change data capture

The outbox pattern removes the dual write by turning it into a single local write. Inside the transaction that updates business state, the service also inserts a row into an outbox table with columns such as id, aggregate_id, event_type, payload, created_at and published_at. Both rows commit or neither does, so the database is the only place a decision is made. An event can no longer exist for a change that rolled back, or go missing for one that committed.

A separate relay then moves rows to the broker. A simple relay works in a loop. It selects unpublished rows ordered by id, publishes each one, and marks it published. Because the relay can crash after publishing but before marking the row, delivery is at-least-once. Consumers must deduplicate on the event id.

Change data capture (CDC) skips the polling query. A connector tails the database's replication log, which already records every committed change in commit order, and publishes from there. It can stream the outbox table itself or the business tables directly. Streaming business tables exposes internal schema to every consumer.

Either way, you add a component to deploy, monitor and alert on.

Copy-- PostgreSQL. Outbox table: one row per event, written with the state change.
CREATE TABLE outbox (
  id           BIGSERIAL PRIMARY KEY,         -- relay orders by this; consumers dedupe on it
  aggregate_id UUID        NOT NULL,          -- entity the event is about
  event_type   TEXT        NOT NULL,          -- e.g. 'OrderPaid'
  payload      JSONB       NOT NULL,          -- the explicit event contract
  created_at   TIMESTAMPTZ NOT NULL DEFAULT now(),
  published_at TIMESTAMPTZ                    -- NULL until the relay has sent it
);
-- Small partial index so the relay's poll stays cheap as published rows pile up.
CREATE INDEX outbox_unpublished ON outbox (id) WHERE published_at IS NULL;

-- Service write: state change and event commit or roll back together.
BEGIN;
UPDATE orders SET status = 'paid'
 WHERE id = '6f1c2a4e-0000-4000-8000-000000000001';
INSERT INTO outbox (aggregate_id, event_type, payload)
VALUES ('6f1c2a4e-0000-4000-8000-000000000001', 'OrderPaid',
        jsonb_build_object('order_id', '6f1c2a4e-0000-4000-8000-000000000001',
                           'amount_cents', 4200));
COMMIT;

-- Relay: one iteration of its polling loop (repeat every poll interval).
BEGIN;
-- SKIP LOCKED lets several relay instances poll without blocking each other.
-- Sequence ids are assigned at insert, not commit, so order is approximate.
SELECT id, aggregate_id, event_type, payload
  FROM outbox
 WHERE published_at IS NULL
 ORDER BY id
 LIMIT 100
 FOR UPDATE SKIP LOCKED;
-- ...relay publishes each row to the broker and waits for acks...
-- A crash before this UPDATE commits means a resend: at-least-once delivery.
UPDATE outbox SET published_at = now()
 WHERE id = ANY($1);  -- $1: array of ids the broker acknowledged
COMMIT;

A PostgreSQL outbox table, a transaction that writes the business change and its event together, and one iteration of a polling relay whose crash window between publish and mark produces at-least-once delivery.

Implementing consumers and read-your-writes correctly

At-least-once delivery means every consumer will see duplicates eventually. Idempotency is a correctness requirement.

The version check also handles reordering. Brokers order messages only within a partition, which in practice means per key. Events for different keys interleave freely. If you add partitions or change the partition key, a key can move to a new partition while its older events are still queued on the old one. The consumer can then receive version 5 before version 4. Discard stale versions or park them for inspection. Never apply them blindly.

Read-your-writes needs deliberate design:

• Pin reads. Route a user's reads to the primary or the writing service for a fixed window after a write. This is simple, but the window is a guess.
• Version token. The write returns a version. Readers wait, with a timeout, until the projection reaches that version, then fall back to the source.
• Optimistic UI. The client shows the expected result immediately and reconciles when the real data arrives.

Session-held pins and tokens do not follow a user to another device. If cross-device consistency matters, store the latest version against the user on the server.

Copy-- PostgreSQL. Dedup table: one row per message the consumer has applied.
CREATE TABLE processed_messages (
  message_id   uuid PRIMARY KEY,
  processed_at timestamptz NOT NULL DEFAULT now()
);
CREATE INDEX ON processed_messages (processed_at);  -- supports the retention delete

-- Read-side projection. version is copied from the owning service's row version.
CREATE TABLE order_view (
  order_id bigint PRIMARY KEY,
  status   text   NOT NULL,
  version  bigint NOT NULL
);

-- Handling one event: {message_id, order_id: 4182, status: 'shipped', version: 5}
BEGIN;

-- A redelivered message violates the primary key and aborts the whole transaction,
-- so the side effect below never runs twice. The consumer treats that error as "already done" and acks.
INSERT INTO processed_messages (message_id)
VALUES ('b3e1c2a4-5d6f-4a7b-8c9d-0e1f2a3b4c5d');

-- Absolute state, not an increment. The WHERE clause drops stale versions,
-- so version 4 arriving after version 5 (e.g. after repartitioning) updates nothing.
INSERT INTO order_view (order_id, status, version)
VALUES (4182, 'shipped', 5)
ON CONFLICT (order_id) DO UPDATE
  SET status  = EXCLUDED.status,
      version = EXCLUDED.version
  WHERE order_view.version < EXCLUDED.version;

COMMIT;

-- Scheduled retention job. Keep the window longer than the longest redelivery or replay window,
-- otherwise an old message replayed after its row is deleted would be applied again.
DELETE FROM processed_messages
WHERE processed_at < now() - interval '14 days';

A PostgreSQL consumer transaction that records the message ID together with a version-guarded upsert, so duplicate and out-of-order events are rejected, plus a retention job for the dedup table.

Multi-step workflows with sagas: orchestration, choreography and compensation

When a business process spans services, such as reserving stock, charging a card and booking a courier, a two-phase commit would hold locks across systems that fail independently. A saga replaces it with a sequence of local transactions. Each step commits on its own, and each step has a compensating action that semantically undoes it if a later step fails.

There are two ways to coordinate the steps:

An orchestrator is a persisted state machine, for example: STARTED → STOCK_RESERVED → PAYMENT_CAPTURED → SHIPPED → COMPLETED. On failure it moves into COMPENSATING and works backwards through the steps that succeeded. If compensation cannot finish, it ends in NEEDS_ATTENTION.

Compensation is not a rollback. The customer saw the charge on their statement, and the confirmation email is already in their inbox. A refund and an apology email are new business events, and product owners must decide what they say. Compensations can fail too, for example when a refund is declined or a downstream service is unavailable. Make them idempotent and retry them with backoff. After a bounded number of attempts, escalate to a human queue with enough context to resolve the case by hand.

Copy// Orchestrator as a persisted state machine; each transition is saved before acting.
type State = 'STARTED' | 'STOCK_RESERVED' | 'PAYMENT_CAPTURED' | 'SHIPPED' | 'COMPLETED' | 'COMPENSATING' | 'COMPENSATED' | 'NEEDS_ATTENTION';

interface Step { done: State; run: (id: string) => Promise<void>; compensate: (id: string) => Promise<void>; }

// Compensations are business actions (release stock, refund, notify), not rollbacks.
const steps: Step[] = [
  { done: 'STOCK_RESERVED',   run: reserveStock,   compensate: releaseStock },
  { done: 'PAYMENT_CAPTURED', run: capturePayment, compensate: refundAndEmailApology },
  { done: 'SHIPPED',          run: bookCourier,    compensate: cancelCourier },
];

async function runSaga(sagaId: string): Promise<void> {
  const completed: Step[] = [];
  for (const step of steps) {
    try {
      await step.run(sagaId);               // local transaction in one service
      await saveState(sagaId, step.done);   // persist progress so a restart can resume
      completed.push(step);
    } catch {
      await saveState(sagaId, 'COMPENSATING');
      // Undo only the steps that succeeded, newest first.
      for (const done of completed.reverse()) {
        const ok = await retryWithBackoff(() => done.compensate(sagaId), 5);
        if (!ok) {
          await saveState(sagaId, 'NEEDS_ATTENTION');
          await escalateToHumanQueue(sagaId, { failedCompensation: done.done });
          return;
        }
      }
      await saveState(sagaId, 'COMPENSATED');
      return;
    }
  }
  await saveState(sagaId, 'COMPLETED');
}

// Compensations must be idempotent: a retry after a timeout must not refund twice.
async function retryWithBackoff(fn: () => Promise<void>, maxAttempts: number): Promise<boolean> {
  for (let attempt = 0; attempt < maxAttempts; attempt++) {
    try { await fn(); return true; }
    catch { await new Promise(r => setTimeout(r, 2 ** attempt * 1000)); }
  }
  return false;
}

A minimal saga orchestrator that saves each state change, compensates completed steps newest first using idempotent retries with exponential backoff, and moves the saga to NEEDS_ATTENTION for manual handling when a compensation keeps failing.

Making staleness observable: lag, event age and reconciliation

Eventual consistency is only acceptable if you can say how long "eventual" is. Measure three things:

• Replication lag: how far a replica or change-data-capture reader is behind the source log.
• Consumer lag: how many messages, or how many seconds of messages, a consumer group has not processed yet.
• End-to-end event age: what users actually experience. Stamp each event at the source with its commit time. When the consumer finishes writing the derived copy, record how much time has passed since that stamp. Hosts' clocks drift apart, so treat small values as rough and watch the trend.

Set an event-age target for each data flow and alert on that, not on raw message counts. A backlog of a thousand cheap updates can matter less than ten delayed payment events.

Some messages fail on every retry. After a bounded number of attempts, move the message to a dead-letter queue along with its error and payload, so one malformed event cannot block a whole partition. Alert on dead-letter queue depth as well.

Lag metrics can't catch silent loss, such as a dropped event or a buggy projection. Reconciliation jobs can. They compare the source of truth with each derived copy, usually by checksumming key ranges or checking rows changed in a recent window. Full comparisons cost a lot of compute and database load, so run them against replicas, off-peak, and in increments. Decide ownership in advance: the owning service's record wins, and the job repairs the copy.

This saves time when something looks wrong. "Search shows stale orders" becomes "order events reaching the search index are eleven minutes old because consumer X is stuck behind a dead-lettered message." That report names the pipeline, the cause and the owner.

Choosing a guarantee per data item: when to avoid eventual consistency, alternatives, and cost

Observability tells you how stale data is. The harder decision is which data may be stale at all. Make that decision per data item, based on whether a wrong read can break an invariant.

Avoid eventual consistency when a stale read can cause an irreversible action, such as a payment or a shipment. In those cases, use one of these alternatives:

• Keep one owning service that enforces the invariant.
• Have other services make synchronous calls to the owner.
• Merge services that share an invariant, because a boundary drawn through an invariant is in the wrong place.
• Use distributed transactions only where your infrastructure supports them reliably.

Strong consistency has a cost. Every check on a hot invariant, such as a popular SKU's stock count, goes through one owner. That owner caps throughput, and when it is down, every dependent write is down too. Shard the invariant (per SKU, per account) so contention stays local.

The machinery for eventual consistency is also expensive: outbox tables, CDC connectors, dedup storage, saga and compensation logic, and lag monitoring. Weigh that against the incidents it prevents.

Illustrative estimate (assumed numbers, not a benchmark):

Break-even when build + yearly upkeep < incidents avoided per year × hours per incident.

Assume 300 build hours, 100 upkeep hours, and 8 avoided incidents at 40 hours each. That gives 400 < 320, so this case does not pay back in year one. It does in year two, when only the 100 upkeep hours recur.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to eventual consistency, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• The dependency graph answers "what does this touch?" in seconds instead of a manual audit.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• You can size a change's blast radius before merging, avoiding surprise cross-team firefighting.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.