Apache Kafka: Creating Kafka topics

Creating Kafka Topics: Examples and Syntax

Introduction

In Apache Kafka, topics are the fundamental abstraction that serves as a category or a bucket in which we store data. Producers publish data to the Kafka topic, while the consumers subscribe to the topic and read the data. In this blog, we look at creating Kafka topics, with practical examples. We’ll also look at some advanced configurations that you can pass to the create command when you are creating your topic.

Creating Kafka Topics: The basics

Let’s look at a simple Kafka Topic create command. For this blog, we assume you are already inside your Kafka’s installation directory or you have set your environment $PATH variable to include your Kafka bin directory. Therefore, the commands we explain here assume you are running the commands from the Kafka Installation Directory.

The basic topic create command would look something like this:

$ bin/kafka-topics.sh --create --topic <topic_name> --bootstrap-server <localhost:9092>

This command will create a topic on the Kafka cluster with the default number of partitions and replication factor configured in the Kafka broker’s properties file.

Here’s an example of how the command would look like:

$ bin/kafka-topics.sh -create -topic socketdaddy-demo --bootstrap-server 5c1f1412bd9f.socketdaddy.io:9092

Output:

Created topic socketdaddy-demo.

To confirm whether the topic was created use the --list command option:

$ bin/kafka-topics.sh --bootstrap-server 5c1f1412bd9f.socketdaddy.io:9092 --list

Advanced configurations and options for creating Kafka topics

When creating a Kafka topic you can pass additional arguments to the command to tailor it according to your requirements. Here’s a list of all the arguments you can pass with the create command. We’ll explore some examples after this section.

  • Replication Factor (--replication-factor) [integer]: Specify the number of replicas of each partition you’d like to have for this topic. This should not exceed more than the number of brokers available. Replication ensures data durability and fault tolerance. The default value is 1.
  • Partition Count (--partitions) [integer]: Specify the number of partitions you want to have for the topic. While partitions enable parallel processing and scalability, you must pay attention to the resource utilization and zookeeper overhead. The default value is 1
  • Retention Time (--retention-ms) [long_integer]: This kafka topic configuration controls the maximum time in milliseconds a message is retained in the topic. After this time, the message becomes eligible for deletion.
  • Cleanup Policy (--cleanup-policy) [“delete” / “compact”]: Defines the criteria for deleting old log segments. Options include ‘delete’ and ‘compact’.
  • Minimum In-Sync Replicas (--min-insync-replicas): [integer] Specifies the minimum number of in-sync replicas required for a produce request to be considered successful.
  • Segment Size (--segment-bytes) [long_integer]: Sets the size of each log segment. Smaller segments may lead to faster log compaction.
  • Unclean Leader Election (--unclean-leader-election) [boolean]: Determines whether a replica that is not in sync with the leader can become the leader during an election. We recommended keeping this value false.
  • Maximum Message Bytes (--max-message-bytes) [long_integer]: Defines the maximum size of a message that can be published on the topic.

Examples

Here are some examples of creating Kafka topics with advanced configurations.

Specify replication factor and number of partitions:

Let’s create a topic with 10 partitions and 3 replicas for each partition.

$ bin/kafka-topics.sh -create -topic socketdaddy-demo --partitions 10 --replication-factor 3 --bootstrap-server 5c1f1412bd9f.socketdaddy.io:9092

Specify the retention time:

In this example, we are creating a topic with a 24-hour retention period. Notice that we are passing this value in milliseconds.

$ bin/kafka-topics.sh -create -topic socketdaddy-demo --partitions 10 --replication-factor 3 -retention-ms 86400000 --bootstrap-server 5c1f1412bd9f.socketdaddy.io:9092

Disable Unclean Leader Election:

While this setting is disabled by default in most Kafka installations, if you need to specifically disable the unclean leader election, you can pass the --unclean-leader-election argument and set it to false.

$ bin/kafka-topics.sh -create -topic socketdaddy-demo --partitions 10 --replication-factor 3 --unclean-leader-election false --bootstrap-server 5c1f1412bd9f.socketdaddy.io:9092

Maximum Message Bytes:

To control the maximum size of a message that can be published to a Kafka topic, we can use the --max-message-bytes option with the create command.

So, the example below creates a topic that can receive messages of up to a maximum of 10 MB in size.

$ bin/kafka-topics.sh -create -topic socketdaddy-demo --partitions 10 --replication-factor 3 --max-message-bytes 10485760 --bootstrap-server 5c1f1412bd9f.socketdaddy.io:9092

References and further reading

Leave a Reply

Your email address will not be published. Required fields are marked *