This article expands on the service integrations article by introducing consumer group graphs and some of the related key terminology. Consumer group graphs provide insights into the behavior of the Apache Kafka consumers.
Here are some of the relevant concepts:
Think of an Apache Kafka topic as a unique channel where a discussion can take place, with producers that send messages to the channel and consumers that read messages from the channel. For example, in a topic called
soccer, you can read what others are saying about soccer, which would be the role of a consumer, or you can post messages about soccer, which would be the role of a producer.
You can spread the storage of messages for an Apache Kafka topic across one or more topic partitions. For example, in a topic that has 100 messages and is set up to have 5 partitions, 20 messages would be assigned to each partition.
A Kafka system can use multiple consumers to consume messages from a topic. Compared to using a single consumer, this improves the rate of consumption of messages from a topic and therefore improves performance. It is common practice to have a set of consumers belong to a consumer group, identified by a group ID, to consume messages from a topic where the messages are spread across multiple partitions. For certain classes of application, it is also important that a message is only consumed by one consumer, and Kafka can guarantee this behavior.
Following on from the example of a topic with 100 messages and 5 partitions, that topic could be configured with 5 consumers in its consumer group. Each consumer is assigned to a single partition, and therefore only consumes the messages in that partition, amounting to 20 messages each. If the consumer group instead had 3 consumers, each consumer would be assigned one of partitions A, B, and C, and the first two consumers would be each assigned partitions D and E.
Note that if there are more consumers in a group than there are available partitions, the remaining consumers in the group are idle until another member of the group has terminated for some unforeseen reason. In addition, a consumer cannot read from a partition that is not assigned to it.
Consumer group telemetry
Built-in consumer group graph - Consumer group lag
This graph is enabled by default and shows how far behind the consumers in a group are in consuming messages in the topic, called consumer lag (or consumer group lag). This graph provides useful insight into the behavior of consumers, allowing you to take necessary actions if needed. Significant and persistent lag for long durations may indicate that the system is not behaving according to plan, which would require investigation and follow-up actions to resolve the issue.
Other available consumer group telemetry - Offsets
A partition is divided into cells with an incremental number called the offset. Each cell holds a message in it. Aiven for Apache Kafka also provides offset telemetry.
Learn how Aiven simplifies working with Apache Kafka: