Kafka Consumer Lag: Diagnosing and Fixing Slow Consumers

What Kafka consumer lag actually measures

Kafka consumer lag is a per-partition number: the partition's log end offset minus the offset the consumer group has committed for that partition. If partition 7 has a log end offset of 1,050,000 and the group's last committed offset is 1,048,000, that partition is 2,000 messages behind. Committed offset is the key detail. Lag reflects what the group has durably acknowledged, not what a consumer has fetched into memory. A consumer that processes quickly but commits rarely will show inflated lag.

Why the group total misleads

Most dashboards sum lag across every partition. That sum hides where the problem is. A total of 50,000 could mean 500 messages behind on each of 100 partitions, which is healthy churn. It could also mean 50,000 behind on one partition, which points to a stuck consumer, a hot key, or a poison message. The per-partition breakdown tells you which consumer instance to investigate. The total does not.

Count versus time

Message count does not tell you whether users are affected. A lag of 10,000 messages is trivial on a topic that receives 50,000 per second and severe on one that receives ten per minute. Time-based lag, meaning the age of the oldest unprocessed message, maps directly to what users experience: "order confirmations are four minutes late." That is the number to put in your SLOs.

A symptom, not a diagnosis

Lag tells you consumption is falling behind production. It does not tell you why. Slow processing, rebalance churn, partition skew, and a traffic spike all produce rising lag. Engineers who own consumer groups should treat lag as the alarm that starts an investigation, not as the explanation.

Copy+-----------+  produce  +---------------------------------------------+
| Producers |---------->| Partition 7 log                             |
+-----------+           |                                             |
                        |  committed offset          log end offset   |
                        |  1,048,000                 1,050,000        |
                        |  |<------ lag = 2,000 messages ----->|      |
                        |                                             |
                        |  time lag = age of oldest unprocessed msg   |
                        |  (e.g. 4 min late -> use this in SLOs)      |
                        +---------------------------------------------+
                                  ^ commit (durable ack)
                                  |
                        +-----------------------------+
                        | Consumer group              |
                        | fetched-in-memory offsets   |
                        | do NOT count; rare commits  |
                        | inflate lag                 |
                        +-----------------------------+

Same group total = 50,000 -- the sum hides WHERE the lag is:
+----------------------------------+  +----------------------------------+
| A: 100 partitions x 500 each     |  | B: p99 = 50,000, rest = 0        |
|    -> healthy churn              |  |    -> stuck consumer, hot key,   |
|                                  |  |       or poison message          |
+----------------------------------+  +----------------------------------+

[ lag rising ] --> symptom only; investigate: slow processing,
                   rebalance churn, partition skew, traffic spike

Lag is computed per partition as log end offset minus the group's committed offset; the group total can hide a single stuck partition, time-based lag is what users feel, and rising lag starts an investigation rather than explaining one.

How lag tracking works

Two pieces of state determine lag, and they live in different places.

The end of the log. Each partition leader tracks the offset that the next produced record will receive. Consumers can only read records up to the high watermark, which is the point up to which records have been replicated to the in-sync replicas. Tools that report "log end offset" for lag purposes ask the broker for the latest readable offset, so lag is measured against what a consumer could actually fetch. For read_committed consumers, the readable limit is the last stable offset, which stops before any open transaction.

The group's committed position. When a consumer commits, it sends the offset of the next record it intends to process to the group coordinator, a broker chosen for that group. The coordinator persists commits in the internal __consumer_offsets topic. After a restart or rebalance, whichever consumer owns a partition resumes from that committed offset.

Lag is the difference between the two, computed per partition. Nothing in Kafka stores lag as a value. Every tool derives it by reading both numbers at a point in time. kafka-consumer-groups.sh fetches committed offsets from the coordinator and latest offsets from the partition leaders. Consumer client metrics such as records-lag use the consumer's in-memory fetch position instead of the committed offset. That is why the two views disagree, and why a consumer that commits rarely looks further behind in the CLI than in its own metrics.

Lag changes whenever either side moves. Producers advance the end of the log continuously, while the committed offset advances only when a consumer commits. A rising number therefore means one of two things: commits have slowed or stopped, or production has sped up beyond what the group can absorb. The rest of this article shows how to tell those apart.

Who should use this approach, and when

Who. The techniques here are for engineers who own or operate Kafka consumer groups. That includes service teams whose applications consume topics, platform and SRE teams who run shared clusters and write alerting, and on-call engineers responding to a "consumers are behind" page. The material assumes you can read consumer configuration, run the standard Kafka CLI tools against the cluster, and change consumer code or deployment settings. If a managed connector or stream-processing framework runs your consumers, the same concepts apply. The knobs you can turn may be exposed differently, though.

When. Use this diagnosis workflow in these situations:

• Lag alerts fire, or users report stale data. Examples include late notifications, dashboards behind real time, or downstream systems missing recent events.
• Before scaling a consumer group. Adding replicas helps only in specific cases, and the sizing check at the end of this article tells you whether it can help at all.
• After changing a deployment. A new handler version, a slower downstream dependency, a change in partition count, or a change to poll and session settings can each alter lag behaviour.
• When designing alerts and SLOs for a new consumer. Decide up front whether you will alert on time-based lag or on sustained growth, and export per-partition metrics from the start.
• When duplicates appear downstream. Repeated processing often goes together with rebalance churn, which also shows up as lag.

A small, steady, non-zero lag that drains on its own is normal and does not call for this process. Kafka consumers always trail producers by at least the records in flight.

Reading lag shapes to find the cause

Before scaling anything, plot lag over time and look at its shape. Each common cause leaves a distinct pattern, because each one affects the gap between the log-end offset and the committed offset differently.

These shapes tell you whether more consumers will help. Only linear growth reflects a throughput deficit that extra consumers can fix, and only until the consumer count matches the number of Kafka partitions. A sawtooth is often harmless. Recurring spikes call for rebalance or memory tuning. Skew and poison messages call for fixes in code or partitioning.

The recovery slope after a spike shows your headroom. If lag drains slowly, the group is running close to capacity even when it looks healthy.

Per-partition trends matter most on topics with many partitions. There, aggregate lag can stay flat while one partition falls hours behind. A committed offset that does not advance at all, while its neighbours keep moving, almost always points to a poison message rather than slowness.

Lag monitoring with stable tooling and sensible alerts

The first place to confirm that pattern is kafka-consumer-groups.sh --describe --group <name>, which prints one row per partition. CURRENT-OFFSET is the group's last committed offset, LOG-END-OFFSET is the offset the next produced record will receive, and LAG is the difference between them. CONSUMER-ID, HOST and CLIENT-ID show which group member owns the partition. Read the rows together. If high lag clusters on partitions owned by one consumer, suspect that instance. If one partition lags regardless of owner, suspect its data. A dash in the consumer columns means nobody owns the partition, which usually indicates a rebalance in progress or too few members. Because this view is built from committed offsets, a consumer that commits infrequently will appear further behind than it really is.

The CLI only gives you a snapshot, so you need a time series as well. Consumer clients expose fetch-manager metrics such as records-lag-max and per-partition records-lag. These are computed from the consumer's fetch position, so they read lower than commit-based lag. Export lag per partition into your metrics system, not as a group total, so that skew stays visible.

Avoid alerting on raw message counts. A lag of 50,000 records is a few seconds on a busy topic and hours on a quiet one. Alert on lag age, the time since the oldest unconsumed record was produced, or on lag growing over a sustained window. Lag age needs record timestamps, and it is unreliable on idle partitions. When nothing new arrives, there is no fresh timestamp to compare against, so an idle partition can look stale. Pair lag age with a lag-greater-than-zero condition so idle partitions don't fire alerts.

Copykafka-consumer-groups.sh --bootstrap-server kafka-1.example.internal:9092 \
  --describe --group orders-processor

kafka-consumer-groups.sh --bootstrap-server kafka-1.example.internal:9092 \
  --describe --group orders-processor 2>/dev/null |
  awk '$1 == "orders-processor" {
         owner = ($7 == "-") ? "UNASSIGNED" : $7
         if ($6 == "-") { nocommit[owner]++; next }
         lag[owner] += $6; parts[owner]++
       }
       END {
         for (o in lag) printf "%-40s partitions=%d total_lag=%d\n", o, parts[o], lag[o]
         for (o in nocommit) printf "%-40s partitions_without_commit=%d\n", o, nocommit[o]
       }'

Running kafka-consumer-groups.sh --describe, reading per-partition offsets, lag and ownership from illustrative output, and summing lag per consumer to show whether lag clusters on one instance or on unowned partitions.

Fixing processing-bound consumers and the partition ceiling

If lag grows steadily while fetches return full batches, the consumer is processing-bound. Instrument the handler before tuning anything: time each stage of a record (deserialization, downstream HTTP calls, database writes) and export per-stage latency histograms. Usually one blocking call per record dominates, and that call determines your maximum throughput per partition.

Reduce per-record cost

Batch writes. One multi-row insert or bulk upsert per poll costs far less than one round trip per record. Keep the poll loop light: if handling a polled batch takes longer than max.poll.interval.ms, the consumer is evicted from the group, and processing stalls during the resulting rebalance.

When ordering only matters per key, you can process records within a partition with bounded concurrency, for example a fixed pool of workers keyed by record key. The cost is in commits. A partition's committed offset can only advance to just past the lowest offset not yet completed. If one slow record is stuck, every record after it that already finished still gets reprocessed after a crash. More concurrency means a wider reprocessing window, so handlers must be idempotent.

The partition ceiling

A group can have at most one active consumer per partition. Any consumers beyond that number sit idle. Adding partitions raises this ceiling, but it changes how keys map to partitions. Records with an existing key start landing on a different partition, which breaks ordering for that key across the transition. More partitions also increase broker overhead and lengthen rebalances.

Scale out only after per-record cost is under control and the consumer count already equals the partition count.

Copy// Assumes enable.auto.commit=false, the consumer is subscribed, and conn.setAutoCommit(false).
while (running) {
    ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(500));
    if (records.isEmpty()) continue;

    // Stage 1: deserialization only. No network or DB calls per record.
    long t0 = System.nanoTime();
    List<Order> orders = new ArrayList<>(records.count());
    for (ConsumerRecord<String, String> r : records) {
        orders.add(Order.parse(r.value()));
    }
    long t1 = System.nanoTime();

    // Stage 2: one batched write per poll instead of one round trip per record.
    // The conflict clause makes replays harmless (upsert syntax varies by database).
    try (PreparedStatement ps = conn.prepareStatement(
            "INSERT INTO orders (id, amount_cents) VALUES (?, ?) ON CONFLICT (id) DO NOTHING")) {
        for (Order o : orders) {
            ps.setString(1, o.id());
            ps.setLong(2, o.amountCents());
            ps.addBatch();
        }
        ps.executeBatch();
    }
    conn.commit();
    long t2 = System.nanoTime();

    // Export per-stage latency so you can see which stage dominates.
    // These are histograms from your metrics library.
    deserializeNanos.record(t1 - t0);
    dbWriteNanos.record(t2 - t1);

    // Advance offsets only after the batch is durable. A crash before this line
    // replays the batch, and the idempotent insert absorbs the replay.
    consumer.commitSync();
}

A poll loop that times deserialization and database writes separately, writes each polled batch with a single JDBC batch insert, and commits offsets only after the write is durable.

Consumer group rebalance churn, skew and poison messages

When processing between two poll() calls exceeds max.poll.interval.ms, the consumer leaves the group even though its heartbeat thread is still running. Its partitions are reassigned, and the new owner resumes from the last committed offset. The slow batch is therefore processed again, often by a consumer that is just as slow. Lag grows in a sawtooth pattern, and duplicate writes show up downstream.

Size the two settings together. The worst-case per-record time multiplied by max.poll.records should stay well under max.poll.interval.ms. Lowering max.poll.records is usually safer than raising the interval, because a long interval also means a genuinely stuck consumer goes undetected for that long.

Two features reduce the cost of rebalances:

• Cooperative incremental rebalancing revokes only the partitions that actually move, so the rest of the group keeps consuming.
• Static membership (group.instance.id) lets a restarting pod rejoin without triggering a rebalance. The trade-off is that a consumer that really has died keeps its partitions, unconsumed, until session.timeout.ms expires.

Skew

If one partition's lag climbs while the others stay flat, suspect a hot key. You have three options:

• Choose a better key. This is cleanest, but it may break ordering assumptions.
• Salt the key across N partitions. This gives up per-key ordering and requires merging downstream.
• Route known hot keys to a dedicated topic with its own consumers.

Poison messages

A record that always fails will block its partition indefinitely. Retry a bounded number of times, then publish the record to a dead-letter topic. A dead-letter topic only helps if three things are in place: a named owner, an alert on its inflow, and a tested replay path. Without them, it becomes a place where records go missing.

Copymax.poll.records=100

max.poll.interval.ms=120000

enable.auto.commit=false

partition.assignment.strategy=org.apache.kafka.clients.consumer.CooperativeStickyAssignor

group.instance.id=orders-consumer-0

session.timeout.ms=30000
heartbeat.interval.ms=10000

Consumer configuration that sizes max.poll.records against max.poll.interval.ms using an illustrative per-record cost and enables cooperative rebalancing and static membership, with comments explaining the trade-off between faster restarts and slower detection of dead consumers.

Fetch and commit tuning, and alternatives to adding consumers

Fetch settings trade latency for throughput

fetch.min.bytes tells the broker to hold a fetch response until that much data is available. fetch.max.wait.ms caps how long it holds. Raising both means fewer, larger requests and less per-request overhead, but each record waits longer before delivery. Raising max.partition.fetch.bytes produces bigger batches per partition. The cost is more memory and more processing per poll, so you are more likely to exceed max.poll.interval.ms and trigger a rebalance.

Commits decide how much you redo

Auto-commit is cheap, but after a crash it can replay everything since the last interval. Manual synchronous commits after each batch shrink that window, but they block the poll loop. Asynchronous commits avoid the blocking, though a failed commit is not retried in order. A common pattern is to commit asynchronously in the loop and synchronously on shutdown or partition revocation. Every option except transactions gives at-least-once delivery, so handlers must be idempotent. Exactly-once transactions remove duplicates, but they add coordination, and read_committed consumers wait for commit markers.

Be careful with lag-based autoscaling. Each scale event triggers a rebalance that pauses consumption. The pause raises lag, which can trigger another scale-up. Use cooldowns, scale on sustained lag trends rather than single samples, and cap the replica count at the partition count.

A diagnosis runbook and its impact on cost and engineering time

Before changing any configuration, work through the checks in order. Each step rules out a class of cause, so the fix you choose addresses what is actually slowing the consumer.

Runbook

1. Confirm lag is real and growing. Compare lag across several sampling intervals. A sawtooth that drains on its own does not need a fix.
2. Check the per-partition distribution. If a few partitions hold most of the lag, the problem is skew or a stuck record, not total capacity.
3. Check rebalance frequency in consumer logs and group metrics. Frequent rebalances point to session or poll timeouts.
4. Measure per-record processing time, including downstream calls.
5. Compare producer rate against consumer throughput. If producers consistently outpace consumers, you need more capacity.

Sizing

Illustrative estimate (assumed numbers, not a benchmark): Assume an input rate of 6,000 records/s, 4 ms of processing per record per consumer, and a 70% target utilization (30% headroom).

consumers = rate × time ÷ utilization = 6,000 × 0.004 ÷ 0.7 ≈ 34.3, so 35 consumers.

If the topic has 24 partitions, consumers 25 through 35 would have no partitions and sit idle. Within that limit, the time per record must drop to 24 × 0.7 ÷ 6,000 = 2.8 ms or less.

This calculation is the reason to diagnose before scaling. It tells you whether adding replicas can help at all. When it cannot, you avoid paying for idle consumers. You may also avoid a repartitioning project that a processing fix would have made unnecessary. And a fix that addresses the actual cause does not have to be repeated during the next traffic peak.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to kafka consumer lag, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.