Estimated reading time: 4 minutes
Describing Kafka Consumer Groups gives you valuable insights into the state of the consumers in the group and helps you identify any lags or issues. Regularly monitoring the state of consumers is essential to ensure you take proactive action if there are unhealthy consumers.
Table of contents
TL;DR – The command to describe Kafka consumers
We can understand if you want to skip all the explanations and jump straight to the command to describe the consumers. So here’s the command:
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-group Now, let’s get to the details of the describe command and the different options we can pass to the command.
Introduction
Kafka Consumers are applications that subscribe to topics and read data from their partitions. Each consumer may subscribe to one or more partitions. However, if there are more consumers in a consumer group than the number of partitions in the topic, the excess number of consumers may sit idle.
When managing a Kafka cluster, one common task is to check the status of a consumer group or the hosts subscribed to a topic. Despite many management software with great UIs for managing Kafka clusters, a Kafka Administrator must know how to describe Kafka consumer groups with the CLI.
Read Also:
What is a Kafka Consumer Group?
A Kafka consumer group is a group of consumers who read and consume data on a topic. Each consumer in the group may subscribe to a unique subset of partitions in that topic, allowing the number of consumers to scale horizontally.
Kafka automatically assigns partitions to consumers within a consumer group to ensure that only one consumer reads the data per partition at any time. Whenever a new consumer joins the consumer group, Kafka rebalances the consumer group and reassigns the partitions. Similarly, when a consumer leaves the group or crashes, Kafka reassigns the partitions to the remaining consumers.
Using the Describe Kafka Consumer command, you can see the number of consumers in the group and the partitions to which they are subscribed.
Read also: Reset Kafka Offsets For a Partition: Quick and Easy
Describing Kafka Consumers
Kafka provides the ConsumerGroupCommand tool, which allows you to list, describe, or delete consumer groups. This tool also shows the position of all consumers in the consumer groups. Let’s now see how we can use this tool.
The basic Kafka Consumer Group describe command would look like this:
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-group Output example:
The above command describes the my-group consumer group.
Listing consumer groups
It’s often the case that you may not know all the consumer groups in your cluster. You can list all the CGs with the --list command.
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --list Output Example:
my-group
sd-group1
sd-group2
my-consumer-group
demo-groupDescribe the Kafka Consumer to check it’s offset positions
When you run the describe command, the default option is to list the consumer position of all the consumers in that group. Alternatively, you can also run the command with the --offsets option, which will also provide you with the same output.
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-group --offsets Example output:
Check what partitions are assigned to the members in a consumer group
You can use the --members --verbose option to list the partitions assigned to the members in the consumer group.
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-group --members --verbose Example output:
