Previously, in Service integrations help article and Kafka dashboard blog, we presented a built-in Kafka dashboard with a collection of graphs for monitoring the health of your Kafka system. In this help article we show additions to the original set namely, Consumer group graphs, which are intended to provide insight into the behavior of the consumers. Before we show the details, let’s get some terminology out of the way.
What is a topic in a Kafka
A topic can be thought of as a unique channel over which a discussion can take place. Let's say there's a channel called "soccer". The channel typically has producers who send messages to the channel and consumers who consume messages from the channel. So you for example can read what others are saying about soccer (in this case you are acting like a consumer), and further, you yourself can post messages to the soccer channel (in this case you are acting like a producer).
What are topic partitions in Kafka
The storage of messages destined for a topic can be spread across one or more partitions. For example, let's say there are 100 messages for a topic and the topic is setup to have 5 partitions. In this case, for simplicity, each partition will be assigned 20 messages each.
What are consumer groups in Kafka
In a Kafka system, multiple consumers can be used to consume messages from a Kafka topic. Multiple consumers are generally used instead of one to improve the rate of consumption of messages from a topic and therefore improve performance. It is common practice to have a set of consumers to belong to a so-called consumer group, identified by a group id, to consume messages from a topic whose messages are spread across multiple partitions. It is also important to a certain class of applications to not have a message consumed by more than one consumer. Kafka can guarantee this behavior.
The notion of multiple consumers belonging to a consumer group and how they interact with available topic partitions to consume messages is better explained with an example. Let's look at the earlier example with 100 messages and 5 partitions. Let's say we have configured 5 consumers in the consumer group for the topic. In this scenario, each consumer will be assigned to one-and-only-one partition and hence will consume the messages in that partition, that is, 20 messages each. If the consumer group instead had 3 consumers, then consumers 1 thru 3 will be assigned partitions A thru C and subsequently consumers 1 and 2 will also be assigned partitions D and E. Note that in cases where there are more consumers in a group than available partitions, the remaining consumers in the group will be idle until perhaps another member of the groups has terminated for some unforeseen reason. Further it should be noted that, a consumer will not be allowed to read from a partition not assigned to it.
Benefits of the Consumer group telemetry
Built-in consumer group graph - Consumer group lag
The graph shows how far behind the consumers in a group are, called consumer (group) lag, in consuming messages in the topic. It is our hope that the lag graph provides useful insight into the behavior of consumers allowing you to take necessary actions if needed. Significant and persistent lag for very long durations may indicate the system not behaving according to plan and hence would make sense to take appropriate followup actions to alleviate it.
Other available consumer group telemetry
Offsets, what are they - A partition is divided in to cells with a monotonically increasing number called the offset. Each cell holds a message in it. Aiven Kafka also provides offset telemetry.
Q. Is the consumer group lag graph enabled by default?
Yes, it is.