Causes & Fix for Consumer Lag in Kafka

What Causes Kafka Consumer Lag?

There are many things that can be causing the Kafka Consumer Lag, including:

• big jump in traffic resulting in producing way more Kafka messages
• poorly written code
• various software bugs and issues resulting in slow processing
• issues with the pipeline elements
• uneven load in Kafka partitions

Of course, the above list is just an example, but let’s look at one of the mentioned causes a bit deeper, the one about issues with the pipeline elements. Let’s imagine a very simplified architecture that looks like this:

We have the data sent to the Data Receiver that works as the Kafka Producer and sends the data to our Kafka Broker. Next, the Data Processor reads the messages from the Kafka Broker. The data process is in fact, the Kafka Consumer which, in addition to reading the data, also enriches it and finally writes it to the Data Store.

What can cause the Kafka Consumer Lag in this scenario? From a technical point of view, the cause can be in almost every part of this architecture, but to be very strict, the Consumer Lag is only present when the consumer cannot keep up. So in our case, if the Data Processor cannot read, process…