Estimated reading time: 3 minutes
Apache Kafka is a popular distributed event streaming platform. It is open-source software that provides a unified, high-throughput, low-latency platform for handling real-time data feeds. One of the core features of Apache Kafka is that the topics are partitioned and spread across the brokers. This design helps distribute the load across multiple brokers, enhances parallelism, and improves overall performance. Typically, you would define the number of partitions a topic is split into when creating the topic. However, there are times when you might want to increase the number of partitions for a topic. This article discusses the command to increase the number of partitions for a topic.
Table of contents
TL;DR – Command to increase the partitions
If you are running Apache Kafka v2.2+, use the bootstrap server names in the command. Otherwise, for versions older than 2.2, pass the zookeeper quorum in the command.
Examples:
On Apache Kafka v2.2+
kafka-topics.sh --bootstrap-server localhost:9092 --alter --topic my_topic --partitions 5 On older versions of Apache Kafka
kafka-topics.sh --zookeeper localhost:2181 --alter --topic my_topic --partitions 5 Now, let’s get into more detail about increasing the partitions on a Kafka Topic.
A few words
Increasing the number of partitions on a Kafka topic is a handy task that allows you to scale your topic and improve parallelism. However, there are some things to keep in mind.
Impact on key-based ordering
Remember that adding new partitions changes the hashing and impacts where new data goes. So, if your applications depend on key-based ordering, your consumers may need to be updated.
You can’t reduce the number of partitions.
Be extremely careful when you increase the number of partitions because there is no way to revert or reduce the number of partitions. This limitation is because the “log” or the data on the topics are immutable. Additionally, Kafka Consumers track the offsets of their subscribed partitions. Moving around data would disrupt the Kafka consumers and can cause data integrity issues and even data loss. So, it’s impossible to re-shuffle or move around data between partitions, and Kafka provides no way of doing this. If you need to reduce partitions, you would need to consider re-creating the topic.
Consider networking overheads
A higher number of partitions means more network overhead as inter-broker traffic will increase, especially during replication and rebalancing processes.
Pre-requisites
The rest of this post assumes you have the following pre-requisites met:
- You have Apache Kafka up and running
- You have the Apache Kafka Command Line tools available on your system
- You have proper access to the Apache Kafka brokers
- You already have the Apache Kafka Topic for which you want to increase the partitions created on your cluster. If you need to create the topic, read our post on how to create Kafka Topics.
Increasing the partitions for a Kafka Topic
Now, let’s look at the steps to increase the number of partitions for a Kafka Topic.
List and identify your topic
To list all the topics in the cluster, use the --list command on the kafka-topics.sh tool. Identify the topic for which you want to increase the partitions.
Example:
kafka-topics.sh --bootstrap-server <broker-address> --list Describe the topic and identify the current count of partitions
Run the describe command on the topic to identify the current partition count.
Example:
kafka-topics.sh --bootstrap-server <broker-address> --describe --topic <topic-name> Increase the partitions
Run the -alter command to increase the number of partitions for your Kafka topic. Ensure that the new partition count is greater than the existing count.
kafka-topics.sh --bootstrap-server localhost:9092 --alter --topic my_topic --partitions 5 Note: The above command is for Apache Kafka v2.2 and above. Use the --zookeeper option instead of the --bootstrap-servers option if you are using an older version.
Verify the new partitions
Run the describe command again to verify the changes are affected.
kafka-topics.sh --bootstrap-server <broker-address> --describe --topic <topic-name> Before you go
If you’re interested, you can read more of our other articles.
