Estimated reading time: 2 minutes
Kafka Broker logs are invaluable for troubleshooting issues in the Kafka cluster. They provide insights into broker operations, message flows, and errors. This post explains changing the directory location where the broker process writes its logs.
Note: This post explains the steps to change the location of the Kafka Broker Process logs. Do not confuse this configuration with the topic data directory configuration, also called “logs” in Apache Kafka.
Updating the Kafka Server logs config property
Follow these steps to update the Kafka Server Logs directory in the server.properties file:
Step 1: Stop the broker
Stop the Kafka broker to prevent it from writing new data while you make configuration changes.
bin/kafka-server-stop.shStep 2: Update the server.properties File
Locate and open the server.properties file in your Kafka configuration directory. This file contains all the configuration settings for the Kafka broker.
cd kafka/config
vim server.propertiesStep 3: Change the Kafka Broker Logs directory
Find the kafka.log4j.dir property in the file. This property defines where Kafka Broker process log files are written. Example:
kafka.log4j.dir=/var/log/kafkaStep 3: Restart the Kafka Broker and verify
After making the changes to the configuration file, restart the broker and verify that the new Kafka broker logs are written in the new directory.
bin/kafka-server-start.sh config/server.propertiesTo verify the changes, you can tail the log file:
tail -f /var/log/kafka/server.logNote: Implement this change across all the brokers that require the process logs directory to be changed.
Before you go
It is advisable not to store the logs on the root mount point when changing the default Kafka Broker Log directory location. If the root mount point fills up, it can lead to issues on the server. Also, be careful when updating the configuration. A similar property, log.dirs defines where the Topic data is stored. Double-check before committing any changes.
