Reset Kafka Consumer Group Offsets

Reset Kafka Offsets: A step-by-step guide

In this article, we explain how to reset Kafka offsets for a consumer group. By the end of this step-by-step guide, you should be able to identify the current position of your consumers on the topic and reset them using the kafka-consumer-groups command.

Introduction

In Apache Kafka, consumer groups are logical groupings of consumers. These consumers work together to consume and process messages from one or more topics. Only one consumer consumes each message in a Kafka topic within a consumer group. This provides a scalable and fault-tolerant way to process data.

Kafka keeps track of the consumer’s position on a partition with an identifier called the offset. Offsets track the consumer’s progress. Therefore, Kafka ensures that the messages are not processed multiple times.

In the following sections, we explain how to identify the current position of the consumers and how to reset them.

Table of contents

Step 1: Identify the current position of the Kafka Offset.

The first step to reset Kafka Offsets is to identify the current position of the consumers. You can use this using the kafka-consumer-groups describe command. Apache Kafka v 2.4.0 or higher supports listing all consumer groups and their lag on the CLI:

$ bin/kafka-consumer-groups.sh --bootstrap-server 5c1f1412bd9f.socketdaddy.io:9092 --all-groups -describe

Similarly, you can run the command with the -group option to list the lag of a specific consumer:

$ bin/kafka-consumer-groups.sh --bootstrap-server 5c1f1412bd9f.socketdaddy.io:9092 --group myConsumerGroup --describe

Example output:

Step 2: Resetting the offsets with the CLI

IMP: It is important to note that you can only reset the offsets for a consumer group only if the consumer is STOPPED. In other words, you can’t reset the offsets when there is an active consumer.

We can reset the offsets once we’ve identified the consumer group. The kafka-consumer-groups command can be used to achieve this. The syntax is:

$ bin/kafka-consumer-groups.sh --bootstrap-server 5c1f1412bd9f.socketdaddy.io:9092 --group myConsumerGroup --reset-offsets --to-earliest --all-topics --execute

The command offers multiple options:

Specifying a topic to reset the offsets

The above example uses the –all-topics option, which resets the consumer offset for all topics the consumer has subscribed to. Instead, pass the –topic command to specify a topic name. Example:

$ bin/kafka-consumer-groups.sh --bootstrap-server 5c1f1412bd9f.socketdaddy.io:9092 --group myConsumerGroup --reset-offsets --to-earliest --topic socketdaddy-demo --execute

Shift by number of offsets

Use --shift-by to move the offset forward or backwards by a number. The command takes a +ve or a -ve integer value as input. So, to move the offset forward by 100 positions, pass --shift-by 100. Similarly, to move it 100 positions backwards, use --shift-by -100.

For example, to move the offsets backward by 100 positions:

$ bin/kafka-consumer-groups.sh --bootstrap-server 5c1f1412bd9f.socketdaddy.io:9092 --group myConsumerGroup --reset-offsets --shift-by -100 --topic socketdaddy-demo --execute

Shift to a specific date and time

Use --to-datetime to move the offset to a specific date and time. Pass the date and time in YYYY-MM-DDTHH:mm:SS.sss format. For example, to move the offsets to Nov 11th, 2023 at 23:00:

$ bin/kafka-consumer-groups.sh --bootstrap-server 5c1f1412bd9f.socketdaddy.io:9092 --group myConsumerGroup --reset-offsets --to-datetime 2023-11-11T23:00:00Z --topic socketdaddy-demo --execute

Shift to the earliest offset

Use --to-earliest to move the offset position to the oldest data available on the Kafka Topic. For example:

$ bin/kafka-consumer-groups.sh --bootstrap-server 5c1f1412bd9f.socketdaddy.io:9092 --group myConsumerGroup --reset-offsets --to-earliest --topic socketdaddy-demo --execute

Shift to the latest offset

Use –to-latest to move the offset position to the latest data available on the Kafka Topic. For example:

$ bin/kafka-consumer-groups.sh --bootstrap-server 5c1f1412bd9f.socketdaddy.io:9092 --group myConsumerGroup --reset-offsets --to-latest --topic socketdaddy-demo --execute

Before you go

In this blog, we explained how to describe the consumer group, identify the lag and reset the offsets. Undoubtedly, knowing how to manage your consumer group’s offsets is crucial to building resilient and fault-tolerant systems. Here are some additional reading materials for you to enhance your Kafka Knowledge:

Reset Kafka Offsets For a Partition: Quick and Easy

Creating Kafka Topics: Examples and Syntax

How to reset consumer offsets in Apache Kafka

How to Reset Kafka Consumer Group Offset

Leave a Reply

Your email address will not be published. Required fields are marked *