Reset Kafka Offset on a Partition

Reset Kafka Offsets For a Partition: Quick and Easy

In this article, we explain how to reset a Kafka consumer group’s offsets for a partition. By the end of this step-by-step guide, you should be able to identify the current position of your consumers on the topic and reset them using the kafka-consumer-groups command.

Introduction: Reset Kafka Offsets For a Partition

In Apache Kafka, consumer groups are logical groupings of consumers. These consumers work together to consume and process messages from one or more topics. Only one consumer consumes each message in a Kafka topic within a consumer group. This provides a scalable and fault-tolerant way to process data.

Kafka keeps track of the consumer’s position on a partition with an identifier called the offset. Offsets track the consumer’s progress. Therefore, Kafka ensures that the messages are not processed multiple times. On the other hand, this also means that a particular consumer in a group can be lagging only on a particular partition. In such a case, you may want to reset its offset only for that partition.

In the following sections, we explain how to identify the current position of the consumers for each partition and how to reset them.

Table of contents

Step 1: Identify the current position of the Kafka Offset for each partition.

The first step to reset Kafka Offsets for a partition is to identify the current position of the consumers. You can use this using the kafka-consumer-groups describe command. Apache Kafka v 2.4.0 or higher supports listing all consumer groups and their lag on the CLI:

$ bin/kafka-consumer-groups.sh --bootstrap-server 5c1f1412bd9f.socketdaddy.io:9092 --all-groups -describe

Similarly, you can run the command with the –-group option to list the lag of a specific consumer:

$ bin/kafka-consumer-groups.sh --bootstrap-server 5c1f1412bd9f.socketdaddy.io:9092 --group myConsumerGroup --describe

Example output:

Step 2: Resetting the offsets with the CLI

IMP: It is important to note that you can only reset the offsets for a consumer group only if the consumer is STOPPED. In other words, you can’t reset the offsets when there is an active consumer.

We can reset the offsets once we’ve identified the consumer group. The kafka-consumer-groups command can be used to achieve this. The syntax is:

$ bin/kafka-consumer-groups.sh --bootstrap-server 5c1f1412bd9f.socketdaddy.io:9092 --group myConsumerGroup --reset-offsets --to-earliest --topic <topic_name>:<partition_number> --execute

The command offers multiple options:

Specifying a topic and the partition number to reset the offsets

The below example resets the offset to the earliest offset for partition number 1:

$ bin/kafka-consumer-groups.sh --bootstrap-server 5c1f1412bd9f.socketdaddy.io:9092 --group myConsumerGroup --reset-offsets --to-earliest --topic socketdaddy-demo:1 --execute

Shift by number of offsets

Use --shift-by to move the offset forward or backwards by a number. The command takes a +ve or a -ve integer value as input. So, to move the offset forward by 100 positions, pass --shift-by 100. Similarly, to move it 100 positions backwards, use --shift-by -100.

For example, to move the offsets backwards by 100 positions for partition number 5:

$ bin/kafka-consumer-groups.sh --bootstrap-server 5c1f1412bd9f.socketdaddy.io:9092 --group myConsumerGroup --reset-offsets --shift-by -100 --topic socketdaddy-demo:5 --execute

Shift to a specific date and time

Use --to-datetime to move the offset to a specific date and time. Pass the date and time in YYYY-MM-DDTHH:mm:SS.sss format. For example, to move the offsets to Nov 11th, 2023 at 23:00 for partition number 3:

$ bin/kafka-consumer-groups.sh --bootstrap-server 5c1f1412bd9f.socketdaddy.io:9092 --group myConsumerGroup --reset-offsets --to-datetime 2023-11-11T23:00:00Z --topic socketdaddy-demo:3 --execute

Shift to the earliest offset

Use --to-earliest to move the offset position to the oldest data available on the Kafka Topic. For example, to reset the offsets to the earliest for partition 0:

$ bin/kafka-consumer-groups.sh --bootstrap-server 5c1f1412bd9f.socketdaddy.io:9092 --group myConsumerGroup --reset-offsets --to-earliest --topic socketdaddy-demo:0 --execute

Shift to the latest offset

Use –to-latest to move the offset position to the latest data available on the Kafka Topic. For example, to reset the offsets to the earliest for partition 2:

$ bin/kafka-consumer-groups.sh --bootstrap-server 5c1f1412bd9f.socketdaddy.io:9092 --group myConsumerGroup --reset-offsets --to-latest --topic socketdaddy-demo:2 --execute

Before you go

In this blog, we explained how to describe the consumer group, identify the lag and reset the offsets for a partition. Undoubtedly, knowing how to manage your consumer group’s offsets is crucial to building resilient and fault-tolerant systems. Here are some additional reading materials for you to enhance your Kafka Knowledge:

Reset Kafka Offsets: A step-by-step guide

Creating Kafka Topics: Examples and Syntax

How to reset consumer offsets in Apache Kafka

How to Reset Kafka Consumer Group Offset

Leave a Reply

Your email address will not be published. Required fields are marked *