Apache Kafka, a distributed event-streaming platform, organizes data into topics. It is important to understand how to manage and monitor these topics. Listing topics in Kafka is a common task for checking existing configurations, debugging issues, or ensuring that producers and consumers are correctly interacting with the cluster.
This documentation provides detailed instructions on listing topics in Kafka using the Kafka command-line interface (CLI) and programmatic approaches. It also explains how to filter results, troubleshoot common issues, and implement best practices for managing Kafka topics.
TL;DR
- Use the
kafka-topics.shCLI to list topics in a Kafka cluster:
kafka-topics.sh --list --bootstrap-server <BROKER_LIST> - Filter topics programmatically using Kafka client libraries in Java or Python.
- Use
kafka-configs.shor REST APIs to fetch detailed topic configurations.
Listing Topics Using the Kafka CLI
Kafka provides the kafka-topics.sh command-line tool for managing topics. This tool is located in the bin directory of your Kafka installation and supports various operations, including listing topics.
Basic Command to List Topics
To list all topics in the Kafka cluster, use the --list option with kafka-topics.sh:
kafka-topics.sh --list --bootstrap-server <BROKER_LIST> --bootstrap-server: Specifies the Kafka brokers to connect to (e.g.,localhost:9092)
Example:
kafka-topics.sh --list --bootstrap-server localhost:9092 Output:
topic1
topic2
__consumer_offsetsThis command lists all topics, including internal topics such as __consumer_offsets.
Filtering Topic Names
Using Shell Utilities
To filter topics by name, pipe the output of kafka-topics.sh to grep:
kafka-topics.sh --list --bootstrap-server localhost:9092 | grep 'user' Example Output:
user-activity
user-logs Fetching Detailed Topic Information
Listing topics only provides the names. To fetch detailed configurations, use the kafka-configs.sh tool.
Example: Fetching Topic Configuration
kafka-configs.sh --bootstrap-server <BROKER_LIST> --entity-type topics --entity-name <TOPIC_NAME> --describe Example Output:
Configs for topic 'user-activity' are:
cleanup.policy=compact
retention.ms=604800000 Programmatically Listing Kafka Topics
For dynamic use cases, you can programmatically list Kafka topics using client libraries. Below are examples in Java and Python.
1. Using the Kafka Java Client
The Kafka Java client library provides a straightforward way to list topics.
Code Example:
import org.apache.kafka.clients.admin.AdminClient;
import org.apache.kafka.clients.admin.AdminClientConfig;
import java.util.Properties;
public class KafkaListTopics {
public static void main(String[] args) throws Exception {
Properties props = new Properties();
props.put(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
try (AdminClient adminClient = AdminClient.create(props)) {
adminClient.listTopics().names().get().forEach(System.out::println);
}
}
}2. List Using the Python confluent-kafka Library
Python users can leverage the confluent-kafka library to list topics.
Code Example:
from confluent_kafka.admin import AdminClient
admin = AdminClient({'bootstrap.servers': 'localhost:9092'})
topics = admin.list_topics().topics
for topic in topics:
print(topic)Common Issues and Troubleshooting
1. No Topics Listed
- Cause: Misconfigured
--bootstrap-server. - Solution: Ensure the broker address is correct and reachable.
2. Authentication Errors
- Cause: SASL/SSL authentication is required but not configured.
- Solution: Pass the
--command-configoption with a properties file:
kafka-topics.sh --list --bootstrap-server <BROKER_LIST> --command-config /path/to/client.properties 3. Lagging Internal Topics
- Internal topics like
__consumer_offsetsare automatically created and managed by Kafka. Use the--exclude-internalflag if your tool or custom script supports it.
Reference Links
- Apache Kafka Documentation: Managing Topics
- Kafka GitHub Repository
- Python Client for Kafka
- Java Kafka Client API
