How to check the number of messages in a Kafka topic?

@jackie 

To check the number of messages in a Kafka topic, you can use the kafka-run-class.sh script that comes with Kafka to run the kafka.tools.GetOffsetShell tool. This tool allows you to see the current offset for each partition in a topic, as well as the log end offset (LEO) for each partition. The difference between the current offset and the LEO is the number of messages in that partition.

Here's an example of how to use the GetOffsetShell tool to see the number of messages in a topic:

1. First, navigate to the directory where Kafka is installed on your machine. For example:

1

cd /path/to/kafka

1. Then, use the kafka-run-class.sh script to run the GetOffsetShell tool, specifying the topic name and the name of the Kafka server to connect to:

1

./bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic my-topic

This will show you the current offset and LEO for each partition in the my-topic topic. To see the number of messages in each partition, subtract the current offset from the LEO.

Note: You can also use the kafka-consumer-groups.sh tool to get the number of messages in a topic. This tool allows you to see the number of messages that have been consumed by a particular consumer group, as well as the number of messages that have not yet been consumed. To use this tool, run the following command:

1

./bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group my-consumer-group --topic my-topic --describe

This will show you the current offset and LEO for each partition in the my-topic topic, as well as the offset and lag for the my-consumer-group consumer group. The lag is the difference between the current offset and the LEO, so you can use this to see the number of messages that have not yet been consumed by the consumer group.