Event-Driven Architecture: A Practical Guide for Backend Engineers

What event-driven architecture is: commands, domain events and state transfer

Event-driven architecture is a decision about who owns state and how quickly other services must agree with it. The broker and the wire format come after that. When a service publishes an event instead of calling another service, it accepts that consumers will see the change later and will act on it independently. If your business rules cannot tolerate that delay, no messaging technology will fix the design.

Three kinds of message travel through these systems, and they carry different obligations:

Hidden coupling appears when these types get mixed up. An "event" named SendInvoiceEmail is a command in disguise. The publisher now expects one particular consumer to act, but nothing enforces that expectation. A thin notification looks decoupled, yet every consumer still makes a synchronous call back to the publisher, so a burst of events turns into a burst of API load. The opposite failure is a fat event that exposes internal columns, which freezes the publisher's data model.

This style suits backend and platform teams whose services are owned and deployed independently, and whose domains can accept eventual consistency between those services.

How a message broker changes the interaction model

In a synchronous design, the caller sends a request and blocks until the callee responds. Both services must be up at the same moment, and the caller has to know who handles the work. A message broker removes both requirements. The publisher writes an event to a topic or queue and moves on. Consumers read it when they are ready, whether that is milliseconds or hours later. This is temporal decoupling: a consumer that is down for a deploy catches up from the broker instead of causing errors upstream.

The model shifts from "call and wait" to "publish and react." As microservices.io notes, an operation can be triggered by events published by other applications and services, not only by client requests [1]. Fan-out follows from this. One OrderPlaced event can drive billing, fulfillment and analytics, and the order service does not need to know any of those consumers exist.

Why teams adopt it

Ownership becomes cleaner. A team can add a new consumer without coordinating a change to the producer. Each consumer scales on its own backlog and deploys on its own schedule.

The ordering caveat

Brokers usually guarantee order only within a single partition or queue, not across a whole topic. Choose partition keys on purpose. Keying by aggregate ID, such as the order ID, keeps every event for one entity in sequence. Even with that, retries, replays and redeliveries mean consumers must still handle events that arrive late or out of order. Common techniques are comparing version numbers on incoming events or rejecting state transitions that are no longer valid.

Copy          +-------------+-------------+
          |  Order Service (producer) |
          | publishes, does not wait  |
          +-------------+-------------+
                        |
                        | OrderPlaced, key = order ID
                        v
+--------------------------------------------------------------+
| Broker topic: orders                                         |
|                                                              |
| partition 0: [A-v1] -> [A-v2] -> [A-v3]   (order A, in seq)  |
| partition 1: [B-v1] -> [B-v2]             (order B, in seq)  |
| partition 2: [C-v1]                       (order C)          |
|                                                              |
| ordering guaranteed within a partition, not across topic     |
| events wait in broker; a down consumer catches up later      |
+------+--------------------+--------------------+-------------+
       |                    |                    |
       | fan-out            | fan-out            | fan-out
       v                    v                    v
+-----------------+  +-----------------+  +-----------------+
| Billing         |  | Fulfillment     |  | Analytics       |
| consumer        |  | consumer        |  | consumer        |
| scales and      |  | scales and      |  | added with no   |
| deploys alone   |  | deploys alone   |  | producer change |
+-----------------+  +-----------------+  +-----------------+
       |                    |                    |
       v                    v                    v
 consumers compare event versions; stale or invalid transitions rejected

An order service publishes OrderPlaced events keyed by order ID without waiting for a response, the broker preserves sequence only within each partition, and billing, fulfillment and analytics each consume the same events independently while guarding against late or out-of-order delivery.

Implementing reliable publishing and consumption: outbox and idempotent consumers

Those defenses only help if the event was published at all, which is a separate problem. A service that commits an order row and then calls the broker has performed a dual write. If the process crashes between the two steps, the order exists but no event was ever sent. If the broker accepts the message and the database transaction then rolls back, consumers act on an order that never existed. Retries do not solve this, because no single step can see both systems.

Transactional outbox

The fix is to make the event part of the same local transaction as the state change. Add an outbox table with columns such as id (UUID primary key), aggregate_id, event_type, payload (JSON), created_at, and a nullable published_at. Insert the outbox row inside the same transaction as the business update, so the change and the event are committed together or not at all. A separate relay then delivers outbox rows to the broker. A polling relay selects unpublished rows in created_at order and marks them published after the broker acknowledges them. A change data capture relay tails the database's write-ahead log instead. Both can crash after sending a message and before recording that it was sent, so both can publish duplicates.

Idempotent consumers

Treat at-least-once delivery as the normal case. In the same transaction that applies its side effects, the consumer runs INSERT INTO processed_messages (message_id, consumer) VALUES (...) ON CONFLICT DO NOTHING. If no row is inserted, the message has already been handled, and the consumer skips it. Upserts keyed on business identifiers add a second line of defense. With an outbox on the producer and deduplication on the consumer, you get exactly-once effects end to end. No broker setting provides that on its own.

Copy-- Outbox table: events are written in the same transaction as the state change.
CREATE TABLE outbox (
  id           uuid        PRIMARY KEY DEFAULT gen_random_uuid(),
  aggregate_id text        NOT NULL,
  event_type   text        NOT NULL,
  payload      jsonb       NOT NULL,
  created_at   timestamptz NOT NULL DEFAULT now(),
  published_at timestamptz            -- NULL until the relay confirms broker ack
);
-- Lets a polling relay find unpublished rows in created_at order cheaply.
CREATE INDEX outbox_unpublished ON outbox (created_at) WHERE published_at IS NULL;

-- Producer: the business change and its event commit together or not at all.
BEGIN;
UPDATE orders SET status = 'confirmed' WHERE id = 'ord-1042';
INSERT INTO outbox (aggregate_id, event_type, payload)
VALUES ('ord-1042', 'OrderConfirmed', '{"orderId": "ord-1042", "total": 129.90}');
COMMIT;

-- Consumer dedupe table: one row per (message, consumer) pair.
CREATE TABLE processed_messages (
  message_id   uuid        NOT NULL,
  consumer     text        NOT NULL,
  processed_at timestamptz NOT NULL DEFAULT now(),
  PRIMARY KEY (message_id, consumer)
);

-- Consumer: the dedupe insert and the side effects share one transaction.
BEGIN;
INSERT INTO processed_messages (message_id, consumer)
VALUES ('3f6c1a2e-8b4d-4e9a-9c1f-2d7e5b8a0c41', 'shipping-service')
ON CONFLICT DO NOTHING
RETURNING message_id;
-- Zero rows returned: already handled, so the app rolls back and acks the message.
-- One row returned: apply side effects; the business-key upsert is a second guard.
INSERT INTO shipments (order_id, status) VALUES ('ord-1042', 'pending')
ON CONFLICT (order_id) DO NOTHING;
COMMIT;

PostgreSQL example of an outbox table written in the same transaction as the order update, and a consumer that records the message ID in processed_messages with ON CONFLICT DO NOTHING inside the same transaction as its side effects, so a duplicate delivery changes nothing.

Eventual consistency, read models and sagas

Once data crosses a broker, some reads will lag behind writes. Decide which ones matter with the product owner, not in a code review. A user's account balance after a transfer, or stock levels at checkout, usually need a strongly consistent read from the owning service. A dashboard or search index can trail by seconds. Where lag is visible, model it: return 202 Accepted, show the order as "pending confirmation", and update it when the confirming event arrives. A visible pending state is more honest than a spinner that hides a race.

Read models are local projections of other services' events. Because they are derived data, treat them as disposable. Version the projection code, and keep enough event history, or a snapshot plus history, to rebuild into a fresh table and swap it in. If you can't rebuild a projection, it has quietly become a second source of truth.

Multi-step processes such as reserve stock, charge card and ship become sagas. Compensations are new forward actions, not rollbacks. A refund does not erase a capture, and a confirmation email that has been sent stays sent. Order your steps so the irreversible ones run last.

Once a flow has more than a handful of steps or needs timeouts, an orchestrator is easier to reason about than a web of subscriptions.

Event schemas as public contracts

Once an event crosses a team boundary, its schema is no longer an implementation detail. Other teams have written code against the payload's shape. You usually cannot see all of those consumers, and you cannot deploy them at the same moment as your producer.

Give every event one owner

Each event type should belong to one team: the team whose service emits it and whose domain it describes. That team approves changes, publishes the schema somewhere consumers can find it, and decides when old versions are retired. Without a single owner, schemas get changed by whoever needs a field next.

Make additive changes and read tolerantly

Adding an optional field is safe. Renaming, removing or retyping a field is not, and neither is changing what a field means. Suppose a producer renames customerId to buyerId. Every consumer that reads customerId will either fail to deserialize the event or quietly get a null. The failure shows up in a service owned by a team that changed nothing. It can also arrive hours later, when a backlog drains or someone replays retained events.

Additive changes only stay safe if consumers are tolerant readers. A consumer should ignore fields it does not recognise. It should not use strict validation that rejects them.

In the example below, version 1 of OrderPlaced carries orderId, customerId and totalCents. Version 2 adds an optional salesChannel. Consumers built for version 1 skip the new field and keep working.

For changes that must break the contract, publish a new event version alongside the old one. Move consumers across, and retire the old version only after nobody reads it.

Copy// OrderPlaced v1: the contract existing consumers were built against.
interface OrderPlacedV1 {
  orderId: string;
  customerId: string;
  totalCents: number;
}

// v2 only adds an optional field. Nothing is renamed, removed or retyped.
interface OrderPlacedV2 extends OrderPlacedV1 {
  salesChannel?: "web" | "mobile" | "store";
}

// A v2 event as the upgraded producer now emits it.
const payload: OrderPlacedV2 = {
  orderId: "ord_7f3a91",
  customerId: "cus_20481",
  totalCents: 4599,
  salesChannel: "mobile",
};
const raw = JSON.stringify({ schemaVersion: 2, ...payload });

// A consumer written against v1. It checks only the fields it needs and
// ignores everything else, so the new salesChannel field cannot break it.
function handleOrderPlaced(json: string): void {
  const data = JSON.parse(json) as Record<string, unknown>;
  const { orderId, customerId, totalCents } = data;
  if (typeof orderId !== "string" || typeof customerId !== "string" || typeof totalCents !== "number") {
    // Missing or retyped v1 fields are a real contract break, so fail loudly.
    throw new Error("OrderPlaced is missing a required v1 field");
  }
  const event: OrderPlacedV1 = { orderId, customerId, totalCents };
  console.log(`charge ${event.customerId} ${event.totalCents} cents for ${event.orderId}`);
}

handleOrderPlaced(raw); // works unchanged on v2 events

Version 2 of OrderPlaced adds an optional salesChannel field. A tolerant v1 consumer validates only the fields it uses, so it processes v2 events unchanged and still fails loudly if a required field is removed or retyped.

Operating an event-driven system in production

Once events are flowing, most of the work shifts from design to operations. The failure modes are quieter than in synchronous systems: nothing returns a 500 to a user, so problems pile up unseen unless you instrument for them.

Retries, poison messages and dead-letter queues

Separate transient failures from permanent ones. A timeout calling a downstream database deserves a retry with exponential backoff and jitter, which keeps a recovering dependency from being hit by every consumer at the same moment. A message that fails deserialization or validation will fail on every attempt. That is a poison message, and retrying it without limit can stall a partition or burn CPU in a tight loop. Cap the number of attempts, then move the message to a dead-letter queue along with the error and attempt count. A DLQ only helps if someone owns it. That means alerting on its depth and having a documented way to inspect, fix and replay messages.

Consumer lag and tracing

Consumer lag is the gap between the newest event and the last one a consumer processed. It is the most direct health signal you have. Rising lag means consumers are falling behind, and every read model built on them is going stale.

Put a correlation ID and trace context in the event headers. Each consumer should forward them on anything it publishes. Without them, you cannot reconstruct a chain of asynchronous hops after an incident.

The broker is infrastructure

The broker needs capacity planning for throughput and storage. It also needs retention settings long enough to cover replay and recovery. Partition counts are hard to change later and set a ceiling on consumer parallelism. Finally, it needs its own on-call rotation, because when the broker fails, everything that depends on it fails too.

When to use events, when to avoid them, and the impact on cost and engineering time

Treat events as a cost you accept to get team autonomy. The microservices.io forces are a good test: simple interactions and "prefer ACID over BASE" both push against distributing an operation. You need a concrete reason to override them.

Avoid events when

• The team is small. No release dependencies across teams means no coordination for events to remove.
• The domain is tightly coupled. Invariants that span aggregates belong in one transaction.
• Users need strongly consistent reads. Read-your-writes is easy to promise and hard to deliver with eventual consistency.

The honest cost

Every consumer needs idempotency work. Stale reads create support tickets. Reconciliation jobs have to catch drift. Debugging means correlating logs across services.

Where events pay back

The publisher ships without waiting on downstream teams. New consumers subscribe without any change to the publisher. A failing consumer falls behind instead of taking the publisher down with it.

Illustrative estimate (assumed numbers, not a benchmark)

• Extra cost = consumers × (idempotency days + reconciliation days). With 4 × (3 + 2), that is 20 engineer-days.
• Monthly saving = avoided coordinated releases × hours each. With 2 × 6, that is 12 hours.

Under these assumptions, the 20 days (160 hours) are recovered in roughly 13 months, before counting support load. Plug in your own numbers. If you cannot name the coordinated releases events would remove, stay with the monolith.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to event-driven architecture, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• The dependency graph answers "what does this touch?" in seconds instead of a manual audit.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• You can size a change's blast radius before merging, avoiding surprise cross-team firefighting.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.