Reset Kafka Offsets For a Partition: Quick and Easy

In this article, we explain how to reset a Kafka consumer group’s offsets for a partition. By the end of this step-by-step guide, you should be able to identify the current position of your consumers on the topic and reset them using the kafka-consumer-groups command.

Introduction: Reset Kafka Offsets For a Partition

In Apache Kafka, consumer groups are logical groupings of consumers. These consumers work together to consume and process messages from one or more topics. Only one consumer consumes each message in a Kafka topic within a consumer group, providing a scalable and fault-tolerant way to process data.

Kafka keeps track of the consumer’s position on a partition with an identifier called the offset. Offsets track the consumer’s progress, so Kafka ensures that messages are not processed multiple times. However, this also means that a particular consumer in a group can be lagging only on a particular partition. In such a case, you may want to reset its offset only for that partition.

In the following sections, we explain how to identify the current position of the consumers for each partition and how to reset them.

Step 1: Identify the current position of the Kafka Offset for each partition.

The first step in resetting Kafka Offsets for a partition is to identify the current position of the consumers. You can use this using the kafka-consumer-groups describe command. Apache Kafka v 2.4.0 or higher supports listing all consumer groups and their lag on the CLI:

$ bin/kafka-consumer-groups.sh --bootstrap-server 5c1f1412bd9f.socketdaddy.io:9092 --all-groups -describe 

Similarly, you can run the command with the –-group option to list the lag of a specific consumer:

$ bin/kafka-consumer-groups.sh --bootstrap-server 5c1f1412bd9f.socketdaddy.io:9092 --group myConsumerGroup --describe 

Example output:

Step 2: Resetting the offsets with the CLI

IMP: It is important to note that you can only reset the offsets for a consumer group only if the consumer is STOPPED. In other words, you can’t reset the offsets when there is an active consumer.

We can reset the offsets once we’ve identified the consumer group. The kafka-consumer-groups command can be used to achieve this. The syntax is:

$ bin/kafka-consumer-groups.sh --bootstrap-server 5c1f1412bd9f.socketdaddy.io:9092 \
  --group myConsumerGroup --reset-offsets --to-earliest \
  --topic <topic_name>:<partition_number> --execute

The command offers multiple options:

Specifying a topic and the partition number to reset the offsets

The below example resets the offset to the earliest offset for partition number 1:

$ bin/kafka-consumer-groups.sh --bootstrap-server 5c1f1412bd9f.socketdaddy.io:9092 \
  --group myConsumerGroup \
  --reset-offsets --to-earliest --topic socketdaddy-demo:1 --execute 

Shift by number of offsets

Use --shift-by to move the offset forward or backwards by a number. The command takes a +ve or a -ve integer value as input. So, to move the offset forward by 100 positions, pass --shift-by 100. Similarly, to move it 100 positions backwards, use --shift-by -100.

For example, to move the offsets backwards by 100 positions for partition number 5:

$ bin/kafka-consumer-groups.sh --bootstrap-server 5c1f1412bd9f.socketdaddy.io:9092 --group myConsumerGroup \
  --reset-offsets --shift-by -100 --topic socketdaddy-demo:5 \
  --execute

Shift to a specific date and time

Use --to-datetime to move the offset to a specific date and time. Pass the date and time in YYYY-MM-DDTHH:mm:SS.sss format. For example, to move the offsets to Nov 11th, 2023, at 23:00 for partition number 3:

$ bin/kafka-consumer-groups.sh --bootstrap-server 5c1f1412bd9f.socketdaddy.io:9092 \
  --group myConsumerGroup --reset-offsets --to-datetime 2023-11-11T23:00:00Z \
  --topic socketdaddy-demo:3 --execute

Shift to the earliest offset

Use --to-earliest to move the offset position to the oldest data available on the Kafka Topic. For example, to reset the offsets to the earliest for partition 0:

$ bin/kafka-consumer-groups.sh --bootstrap-server 5c1f1412bd9f.socketdaddy.io:9092 \
  --group myConsumerGroup --reset-offsets --to-earliest \
  --topic socketdaddy-demo:0 --execute

Shift to the latest offset

Use –to-latest to move the offset position to the latest data available on the Kafka Topic. For example, to reset the offsets to the earliest for partition 2:

$ bin/kafka-consumer-groups.sh --bootstrap-server 5c1f1412bd9f.socketdaddy.io:9092 \
  --group myConsumerGroup --reset-offsets --to-latest \
  --topic socketdaddy-demo:2 --execute

Before you go

In this blog, we explained how to describe the consumer group, identify the lag and reset the offsets for a partition. Managing your consumer group’s offsets is important for building resilient, fault-tolerant systems. Here are some additional reading materials for you to enhance your Kafka Knowledge:

Describe Kafka Consumer Groups – Command and Steps

Understanding and Managing Failed Partitions in Kafka

Reset Kafka Offsets: A step-by-step guide

Creating Kafka Topics: Examples and Syntax

How to reset consumer offsets in Apache Kafka

How to Reset Kafka Consumer Group Offset