Reset Kafka Consumer Group Offsets

In this article, we explain how to reset Kafka offsets for a consumer group. By the end of this step-by-step guide, you should be able to identify the current position of your consumers on the topic and reset them using the kafka-consumer-groups command.

Introduction

In Apache Kafka, consumer groups are logical groupings of consumers. These consumers work together to consume and process messages from one or more topics. Only one consumer consumes each message in a Kafka topic within a consumer group, providing a scalable and fault-tolerant way to process data.

Kafka keeps track of the consumer’s position on a partition with an identifier called the offset. Offsets track the consumer’s progress. Therefore, Kafka ensures that the messages are not processed multiple times.

In the following sections, we explain how to identify the consumers’ current positions and how to reset them.

Table of contents

Step 1: Identify the current position of the Kafka Offset.

The first step to reset Kafka Offsets is to identify the current position of the consumers. You can use this using the kafka-consumer-groups describe command. Apache Kafka v 2.4.0 or higher supports listing all consumer groups and their lag on the CLI:

Similarly, you can run the command with the -group option to list the lag of a specific consumer:

Example output:

Step 2: Resetting the offsets with the CLI

IMP: It is important to note that you can only reset the offsets for a consumer group only if the consumer is STOPPED. In other words, you can’t reset the offsets when there is an active consumer.

We can reset the offsets once we’ve identified the consumer group. The kafka-consumer-groups command can be used to achieve this. The syntax is:

The command offers multiple options:

Specifying a topic to reset the offsets

The above example uses the –all-topics option, which resets the consumer offset for all topics the consumer has subscribed to. Instead, pass the –topic command to specify a topic name. Example:

Shift by number of offsets

Use --shift-by to move the offset forward or backwards by a number. The command takes a +ve or a -ve integer value as input. So, to move the offset forward by 100 positions, pass --shift-by 100. Similarly, to move it 100 positions backwards, use --shift-by -100.

For example, to move the offsets backward by 100 positions:

Shift to a specific date and time

Use --to-datetime to move the offset to a specific date and time. Pass the date and time in YYYY-MM-DDTHH:mm:SS.sss format. For example, to move the offsets to Nov 11th, 2023 at 23:00:

Shift to the earliest offset

Use --to-earliest to move the offset position to the oldest data available on the Kafka Topic. For example:

Shift to the latest offset

Use –to-latest to move the offset position to the latest data available on the Kafka Topic. For example:

Before you go

In this blog, we explained how to describe the consumer group, identify the lag and reset the offsets. Managing your consumer group’s offsets is critical to building resilient, fault-tolerant systems. Here are some additional reading materials for you to enhance your Kafka Knowledge:

Describe Kafka Consumer Groups – Command and Steps

Understanding and Managing Failed Partitions in Kafka

Reset Kafka Offsets For a Partition: Quick and Easy

Creating Kafka Topics: Examples and Syntax

How to reset consumer offsets in Apache Kafka

How to Reset Kafka Consumer Group Offset

3 Comments

  1. It is the best time to make some plans for the
    longer term and it’s time to be happy. I’ve learn this put up and
    if I may just I wish to counsel you few fascinating things or advice.
    Maybe you can write next articles relating to this article.
    I wish to learn more issues about it!

  2. Hey would you mind stating which blog platform you’re using?
    I’m going to start my own blog soon but I’m having a tough
    time choosing between BlogEngine/Wordpress/B2evolution and Drupal.
    The reason I ask is because your layout seems different then most blogs and I’m looking for something unique.

    P.S Apologies for getting off-topic but I had to ask!

  3. et voluptatibus aspernatur consequuntur quia minima eaque id eligendi inventore aliquid ut ut eum ut voluptates praesentium et perferendis voluptas. nemo ab aut quasi est non quis cum perferendis reiciendis distinctio ut enim debitis rerum libero. quisquam vel quaerat ut unde ratione officia officiis. ea ut ut harum laudantium optio rerum quia consequatur. dignissimos qui deleniti vitae dolorem molestiae facere ab esse quos assumenda quas esse ea modi dolore itaque eius.

Leave a Reply

Your email address will not be published. Required fields are marked *