Skip to content

Instantly share code, notes, and snippets.

@ernestas-poskus

ernestas-poskus/how_to_reset_kafka_consumer_group_offset.md

Forked from marwei/how_to_reset_kafka_consumer_group_offset.md
Created February 24, 2020 09:22
Show Gist options
  • Save ernestas-poskus/b4dabb65a66c5b0f973c7e7904af3011 to your computer and use it in GitHub Desktop.
Save ernestas-poskus/b4dabb65a66c5b0f973c7e7904af3011 to your computer and use it in GitHub Desktop.
Download ZIP

Revisions

  1. marwei created this gist Nov 9, 2017.
    32 changes: 32 additions & 0 deletions how_to_reset_kafka_consumer_group_offset.md
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,32 @@
    Kafka 0.11.0.0 (Confluent 3.3.0) added support to manipulate offsets for a consumer group via cli `kafka-consumer-groups` command.

    1. List the topics to which the group is subscribed
    ```bash
    kafka-consumer-groups --bootstrap-server <kafkahost:port> --group <group_id> --describe
    ```
    Note the values under "CURRENT-OFFSET" and "LOG-END-OFFSET". "CURRENT-OFFSET" is the offset where this consumer group is currently at in each of the partitions.

    2. Reset the consumer offset for a topic (preview)
    ```bash
    kafka-consumer-groups --bootstrap-server <kafkahost:port> --group <group_id> --topic <topic_name> --reset-offsets --to-earliest
    ```
    This will print the expected result of the reset, but not actually run it.

    3. Reset the consumer offset for a topic (execute)
    ```bash
    kafka-consumer-groups --bootstrap-server <kafkahost:port> --group <group_id> --topic <topic_name> --reset-offsets --to-earliest --execute
    ```
    This will execute the reset and reset the consumer group offset for the specified topic back to 0.

    4. Repeat 1 to check if the reset is successful

    ### Note
    * The consumer group must have no running instance when performing the reset. Otherwise the reset will be rejected.
    * There are many other resetting options, run `kafka-consumer-groups` for details
    * --shift-by <positive_or_negative_integer>
    * --to-current
    * --to-latest
    * --to-offset <offset_integer>
    * --to-datetime <datetime_string>
    * --by-duration <duration_string>
    * The command also provides an option to reset offsets for all topics the consumer group subscribes to: `--all-topics`