Monitor Kafka Consumer Lag

The basic way to monitor Kafka Consumer Lag is to use the Kafka command line tools and see the lag in the console. We can use the kafka-consumer-groups.sh script provided with Kafka and run a lag command similar to this one:

$ bin/kafka-consumer-groups.sh - bootstrap-server localhost:8080 - describe - group console-consumer-12342

The result would be the lag for the provided consumer group. Here is a very simple example that uses the console consumer:

GROUP                 TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG            CONSUMER-ID                                                           HOST            CLIENT-ID
console-consumer-3452 blog_kr    0          -               26              -              consumer-console-consumer-3452-1-3dr56342f-a045-2e4d-e13a-5434b5e446ed /127.0.0.1      consumer-console-consumer-3452-1

The thing you are most interested in is the current offset. If the offset is positive, that means that there is a lag. In most cases, if your Kafka Producer is actively producing messages and the Kafka Consumers are actively consuming, you will have a small lag here. This is expected. The problems start when the lag is significant or is constantly growing. That means that the data is not processed fast enough.

Using the console tools is possible and a viable solution if we have access to Kafka brokers, we are online when the issues are happening, and we know that the issues are happening. But without a proper monitoring…