Introduction to Consumer Lag Metrics in Kafka

Introduction

The world lives by processing the data. Humans process the data — each sound we hear, each picture we see — everything is data for our brain. The same goes for modern applications and algorithms — the data is the fuel that allows them to function and provide useful features.

Even though such thinking is not new, what is new in recent years is the requirement of near-real-time processing of large quantities of events processed by our systems. That’s why the technological stack grew from simple applications to whole processing pipelines. And the more complex the systems become, the less visibility into its working we have — at least when not using proper tools.

One system that allows us to process large amounts of data is Apache Kafka — an open-source, distributed event streaming platform designed to stream massive amounts of data. However, as with everything, we need to monitor it to ensure that everything works well and is healthy. One of the most crucial metrics for Kafka and the systems using it is consumer lag. In this blog post, we will learn how to monitor it.

What Is Consumer Lag in Kafka

Kafka Consumer Lag indicates how much lag there is between Kafka producers and consumers.