RabbitMQ vs Kafka vs NATS: Choosing a Broker for Microservices

Three different mental models: routed queues, a replayable log, and subject-based messaging

RabbitMQ, Kafka and NATS are often compared as if they were interchangeable pipes. They aren't. Each one starts from a different idea of what a message is and how long it should live.

RabbitMQ is a router in front of queues. Producers publish to an exchange, not to a queue. The exchange type (direct, topic, fanout or headers) and its bindings decide which queues receive a copy. A consumer takes a message from a queue and acknowledges it, and the broker then deletes it. A message is a unit of work that exists until someone handles it. That makes RabbitMQ a natural fit for task distribution, flexible per-message routing and workloads where each job should be processed once and then removed.

Kafka is a partitioned, append-only log. A topic is split into partitions. Records are appended in order and kept according to a retention policy based on time or size. Consumers don't remove anything. Each consumer group tracks its own offset per partition, so a new service can read from the beginning, and an existing one can rewind after a bug. Teams use Kafka when events are a durable record that several systems need to read independently.

NATS is subject-based messaging. Publishers send to subjects such as orders.created, and subscribers match them, including with wildcards. Core NATS is at-most-once: if no subscriber is listening, the message is gone. JetStream adds streams that capture subjects and store messages, with consumers that can acknowledge and replay. Teams choose NATS for low-latency service-to-service traffic and add persistence only where they need it.

These differences in delivery and retention decide how your system behaves when a consumer crashes, lags behind or is deployed next month. That matters more than any throughput headline.

Copy RabbitMQ: exchange routes to queues; message deleted once acknowledged
 +----------+    +-----------+  binding  +---------+    +------------+
 | Producer |--->| Exchange  |---------->| Queue A |--->| Consumer 1 |
 +----------+    | direct,   |           +---------+    +------------+
                 | topic,    |  binding  +---------+    +------------+
                 | fanout,   |---------->| Queue B |--->| Consumer 2 |
                 | headers   |           +---------+    +------------+
                 +-----------+

 Kafka: partitioned append-only log; read by offset, not deleted on read
 +----------+  append  +--- topic: orders (time/size retention) ---+
 | Producer |--------->| P0 | 0 | 1 | 2 | 3 | 4 | 5 |             |
 +----------+          | P1 | 0 | 1 | 2 | 3 |                     |
                       +----------^---------------^----------------+
                                  |               |
                                  |               +- group B: offset 5
                                  +- group A: offset 1 (can rewind)

 NATS: subject-based pub/sub; core NATS is at-most-once, no storage
 +-----------+  orders.created  +--------+  orders.*   +--------------+
 | Publisher |----------------->|  NATS  |------------>| Subscriber A |
 +-----------+                  | server |  orders.>   +--------------+
                                |        |------------>| Subscriber B |
                                +---+----+             +--------------+
                                    |  JetStream (optional persistence)
                                    v
                          +----------------------------+
                          | stream: stores orders.*    |
                          | consumers ack and replay   |
                          +----------------------------+

RabbitMQ routes each message through an exchange to queues and deletes it after acknowledgment, Kafka appends records to a retained partitioned log that each consumer group reads at its own offset, and NATS delivers by subject at most once unless JetStream stores the subject in a replayable stream.

Retention, replay and delivery guarantees

A message's lifetime, and whether you can get it back, differs sharply between the three.

Kafka keeps records whether or not anyone has read them. Resetting a consumer group's offset lets you reprocess a week of events after fixing a bug or bootstrap a new service from history. RabbitMQ classic queues are work buffers: an acknowledged message is deleted, so reprocessing requires an upstream source. Streams add append-only, replayable semantics where you need them. Core NATS delivers only to subscribers connected at that moment. Without JetStream, a crashed consumer loses those messages.

In practice, all three run at-least-once. A consumer that commits its work and crashes before acknowledging will see the message again. Consumers therefore have to be idempotent. Kafka's transactions give exactly-once processing only for read-process-write cycles that stay inside Kafka. A database row or an HTTP call to a payment API falls outside that boundary.

A common idempotent pattern works like this. In one database transaction, insert the message ID into a processed_messages table that has a unique constraint, then apply the business write and commit. If the insert conflicts, the message is a duplicate, so skip the write. Acknowledge only after the commit succeeds.

Copy# Idempotent consumer: dedupe on message ID inside the same DB transaction
# as the business write. Works with any at-least-once broker.
#
# Schema (run once):
#   CREATE TABLE processed_messages (
#       message_id   TEXT PRIMARY KEY,
#       processed_at TIMESTAMPTZ NOT NULL DEFAULT now()
#   );
import psycopg2

conn = psycopg2.connect("dbname=orders user=app password=change-me host=db.internal")

def handle(message_id: str, order_id: str, amount_cents: int, ack) -> None:
    # 'with conn' commits on success and rolls back on any exception,
    # so the dedupe record and the business write succeed or fail together.
    with conn:
        with conn.cursor() as cur:
            cur.execute(
                "INSERT INTO processed_messages (message_id) VALUES (%s) "
                "ON CONFLICT (message_id) DO NOTHING",
                (message_id,),
            )
            if cur.rowcount == 0:
                # Already processed: a redelivery after a crash or a lost ack.
                # Skip the write and fall through to acknowledge.
                pass
            else:
                cur.execute(
                    "UPDATE orders SET paid_cents = paid_cents + %s WHERE id = %s",
                    (amount_cents, order_id),
                )
    # Acknowledge only after the commit. If we crash before this line,
    # the broker redelivers and the INSERT above detects the duplicate.
    ack()

A broker-agnostic Python consumer that records the message ID and applies the business write in one PostgreSQL transaction, skips duplicates using a unique constraint, and acknowledges only after the commit.

Ordering, scaling and backpressure under load

Every broker gives up strict ordering when you add consumers, so ordering has to be designed in from the start. Kafka orders messages only within a partition, and a consumer group can't use more active members than there are partitions. Competing consumers on one RabbitMQ queue process messages in parallel, so message 2 can finish before message 1. NATS queue groups spread messages across members with no ordering between them.

Ordering usually breaks without any error:

• Horizontal scaling: adding consumers to fix throughput removes the single reader that kept messages in sequence.
• Partition-key changes: if you switch a Kafka key from customer_id to order_id, events for the same customer land on different partitions.
• Redelivery: a nacked or timed-out message comes back after later messages have already been processed.
• Dead-letter flows: a replayed message from the DLQ arrives long after its successors.

Slow consumers fail in a different way on each broker:

RabbitMQ pushes backpressure onto producers. Kafka absorbs backpressure as retained lag. Core NATS protects the server by dropping the slow client. Only JetStream pull consumers let each consumer decide how fast it reads.

Routing and request/reply: implementing common patterns in each broker

RabbitMQ: routing lives in the exchange

Producers publish to an exchange, and bindings decide which queues receive each message. A direct exchange matches the routing key exactly, so billing.invoice reaches only queues bound with that key. A topic exchange matches dot-separated keys against patterns: * matches one word and # matches zero or more, so a binding of orders.*.created receives orders.eu.created. A fanout exchange ignores the key and copies each message to every bound queue. A headers exchange matches on message headers, using x-match set to all or any. Because routing is declared in the broker, consumers can change what they receive without any change to producers.

For request/reply, the client sets reply_to to a queue it consumes from and attaches a correlation_id. The server publishes its response to that queue, and the client uses the ID to match the response to the request. The amq.rabbitmq.reply-to pseudo-queue lets clients skip declaring a queue per caller.

NATS: routing lives in the subject

Subscribers use wildcards. * matches one token, and > matches one or more trailing tokens. Request/reply is built in: request() creates a unique inbox subject, publishes with it as the reply address and waits for one response. Queue groups spread requests across multiple responders.

Kafka: routing lives in your code

Kafka has no broker-side routing. Producers choose the topic and key, and consumers filter on the client side or use a stream processor that splits one topic into several. RPC over Kafka is awkward for three reasons:

• Replies must be correlated across partitions.
• Consumer group rebalances can stall responders.
• Short-lived replies end up stored in a durable, retained log.

Each of these works against what the log is designed for.

Copyimport time, uuid
import pika  # RabbitMQ client

conn = pika.BlockingConnection(pika.ConnectionParameters("localhost"))
ch = conn.channel()

# Topic routing: this queue receives order-created events from any region.
ch.exchange_declare(exchange="orders", exchange_type="topic", durable=True)
ch.queue_declare(queue="audit.order-created", durable=True)
ch.queue_bind(exchange="orders", queue="audit.order-created",
              routing_key="orders.*.created")
ch.basic_publish(exchange="orders", routing_key="orders.eu.created",
                 body=b'{"order_id": "A-1001"}')

# Request/reply: exclusive, server-named queue that only this caller reads.
reply_queue = ch.queue_declare(queue="", exclusive=True).method.queue
corr_id = str(uuid.uuid4())
response = None

def on_reply(channel, method, props, body):
    global response
    if props.correlation_id == corr_id:  # drop stale or unrelated replies
        response = body

ch.basic_consume(queue=reply_queue, on_message_callback=on_reply, auto_ack=True)

# Default exchange ("") routes by queue name to the responder's queue.
ch.basic_publish(
    exchange="",
    routing_key="rpc.pricing",
    properties=pika.BasicProperties(reply_to=reply_queue, correlation_id=corr_id),
    body=b'{"sku": "SKU-42"}',
)

# Wait for the matching reply, but never block forever.
deadline = time.monotonic() + 5
while response is None and time.monotonic() < deadline:
    conn.process_data_events(time_limit=0.5)
print(response or "timed out")
conn.close()

A Python pika client binds a queue to a RabbitMQ topic exchange with the pattern orders..created, then makes an RPC call using an exclusive reply_to queue and a correlation_id to match the response to the request.*

Operational footprint and impact on cost and engineering time

Kafka asks for the most up-front planning. Brokers coordinate through a controller quorum, and you must size disk, retention and replication factor before go-live. Partition count needs the most care. Producers map keys to partitions by hashing, so adding partitions later sends existing keys to new partitions, and per-key ordering breaks during the transition. Teams usually over-provision partitions to avoid that migration, which adds broker overhead.

RabbitMQ clusters are smaller, but quorum queues replicate through Raft, so every write waits for a majority of nodes to acknowledge it. RabbitMQ is built for short queues. When large backlogs build up, it pages messages to disk and raises memory alarms, and throughput falls exactly when consumers are already behind.

NATS ships as a single binary with a simple full-mesh cluster model. JetStream adds the decisions NATS otherwise skips: storage type, replica count, retention limits and stream placement.

Ecosystem maturity changes how much you build. Kafka has an established connector and stream-processing ecosystem, so you can often configure an integration instead of writing it. With NATS, you will more often write and maintain the integration yourself.

Illustrative estimate (assumed numbers, not a benchmark)

Build days = integrations × days per integration

• Assumed: 4 integrations.
• Off-the-shelf connector, assumed 1 day each: 4 × 1 = 4 days.
• Custom code, assumed 5 days each: 4 × 5 = 20 days.

The 16-day gap is time a better-fitting broker gives back to the team.

Decision matrix: which broker fits which workload

The table maps each workload to a default. It is a starting point for evaluation, not a verdict.

Two factors outweigh the table more often than engineers admit. The first is team familiarity: a broker your on-call rotation already understands fails in ways they can diagnose. The second is an existing managed offering on your cloud provider, which removes most of the operational work counted in the previous section.

Lock-in is the counterweight. If you pick a queue for order events and later need to replay six months of history for a new service, RabbitMQ has nothing to replay because consumed messages are gone. Ask early whether any consumer could plausibly need history. If one could, start with a log, or with NATS JetStream streams.

Finally, don't choose on published benchmarks. Their message sizes, acknowledgement modes, fsync behaviour and replication factors rarely match yours. Load-test with your own payloads, durability settings and replication factor. Measure tail latency during a broker restart as well as steady-state throughput.

When to avoid each broker and when running two is justified

Kafka is a poor fit for a plain task queue. If you need work items dispatched to a pool of workers, retried individually and discarded once done, Kafka makes you build that yourself. Its unit of parallelism is the partition, so one slow message blocks everything behind it, and per-message retries or delays require extra topics and consumer logic. RabbitMQ or a NATS JetStream work queue gives you acknowledgements, redelivery and dead-lettering per message.

RabbitMQ should not be your long-term event store, even with streams. Streams add replay, but the tooling around compaction, schema management and reprocessing years of history is far more mature in Kafka. If services need to rebuild state from the full history of a domain, start with a log designed for that.

Core NATS delivers at most once. A subscriber that is down or slow loses messages, and nothing on the server keeps a copy. If losing a message is unacceptable and you are not adopting JetStream, use something else.

Running two brokers makes sense when each handles a clearly different job. A common example is Kafka as the durable event backbone, with RabbitMQ or NATS handling task dispatch and request/reply. The split holds up when teams can say which broker a new flow belongs on without debating it. A second broker hurts when it exists because one team preferred it, or when the same events end up bridged into both. Then you pay for two on-call runbooks, two sets of monitoring and two failure models, and you get no new capability in return.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to rabbitmq vs kafka vs nats, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• The dependency graph answers "what does this touch?" in seconds instead of a manual audit.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• You can size a change's blast radius before merging, avoiding surprise cross-team firefighting.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.