Skip to content

Instantly share code, notes, and snippets.

@cemo

cemo/kafka-cheat-sheet.md

Forked from filipefigcorreia/kafka-cheat-sheet.md
Created February 19, 2018 19:14
Show Gist options
  • Save cemo/0e2b0a77ce9474f8b9c9514812b8ae3f to your computer and use it in GitHub Desktop.
Save cemo/0e2b0a77ce9474f8b9c9514812b8ae3f to your computer and use it in GitHub Desktop.
Download ZIP

Revisions

  1. @filipefigcorreia filipefigcorreia revised this gist Jun 14, 2016. 1 changed file with 6 additions and 1 deletion.
    7 changes: 6 additions & 1 deletion kafka-cheat-sheet.md
    View file
    Original file line number Diff line number Diff line change
    @@ -22,7 +22,7 @@
    `bin/kafka-console-consumer.sh --new-consumer --bootstrap-server localhost:9092 --topic mytopic --from-beginning`

    ## Get the consumer offsets for a topic
    `bin/kafka-consumer-offset-checker.sh --zookeeper=localhost:2181 --topic=playlists.playlist-change --group=users_content_api`
    `bin/kafka-consumer-offset-checker.sh --zookeeper=localhost:2181 --topic=mytopic --group=my_consumer_group`

    ### Read from __consumer_offsets

    @@ -41,6 +41,11 @@ Add the following property to `config/consumer.properties`:
    ### View the details of a consumer group
    `bin/kafka-consumer-groups.sh --zookeeper localhost:2181 --describe --group <group name>`

    ## kafkacat

    ### Getting the last five message of a topic
    `kafkacat -C -b localhost:9092 -t mytopic -p 0 -o -5 -e`

    ## Zookeeper

    ### Starting the Zookeeper Shell
  2. @filipefigcorreia filipefigcorreia revised this gist May 24, 2016. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion kafka-cheat-sheet.md
    View file
    Original file line number Diff line number Diff line change
    @@ -19,7 +19,7 @@
    `bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic mytopic --time -1`

    ### Consume messages with the console consumer
    `bin/kafka-console-consumer.sh --new-consumer --bootstrap-server localhost:9092 --topic mytopic --from_begining`
    `bin/kafka-console-consumer.sh --new-consumer --bootstrap-server localhost:9092 --topic mytopic --from-beginning`

    ## Get the consumer offsets for a topic
    `bin/kafka-consumer-offset-checker.sh --zookeeper=localhost:2181 --topic=playlists.playlist-change --group=users_content_api`
  3. @filipefigcorreia filipefigcorreia revised this gist May 24, 2016. 1 changed file with 3 additions and 1 deletion.
    4 changes: 3 additions & 1 deletion kafka-cheat-sheet.md
    View file
    Original file line number Diff line number Diff line change
    @@ -18,8 +18,10 @@
    ### Get the latest offset still in a topic
    `bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic mytopic --time -1`

    ## Get the consumer offsets for a topic
    ### Consume messages with the console consumer
    `bin/kafka-console-consumer.sh --new-consumer --bootstrap-server localhost:9092 --topic mytopic --from_begining`

    ## Get the consumer offsets for a topic
    `bin/kafka-consumer-offset-checker.sh --zookeeper=localhost:2181 --topic=playlists.playlist-change --group=users_content_api`

    ### Read from __consumer_offsets
  4. @filipefigcorreia filipefigcorreia revised this gist May 18, 2016. 1 changed file with 4 additions and 0 deletions.
    4 changes: 4 additions & 0 deletions kafka-cheat-sheet.md
    View file
    Original file line number Diff line number Diff line change
    @@ -18,6 +18,10 @@
    ### Get the latest offset still in a topic
    `bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic mytopic --time -1`

    ## Get the consumer offsets for a topic

    `bin/kafka-consumer-offset-checker.sh --zookeeper=localhost:2181 --topic=playlists.playlist-change --group=users_content_api`

    ### Read from __consumer_offsets

    Add the following property to `config/consumer.properties`:
  5. @filipefigcorreia filipefigcorreia revised this gist May 18, 2016. 1 changed file with 7 additions and 0 deletions.
    7 changes: 7 additions & 0 deletions kafka-cheat-sheet.md
    View file
    Original file line number Diff line number Diff line change
    @@ -18,6 +18,13 @@
    ### Get the latest offset still in a topic
    `bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic mytopic --time -1`

    ### Read from __consumer_offsets

    Add the following property to `config/consumer.properties`:
    `exclude.internal.topics=false`

    `bin/kafka-console-consumer.sh --consumer.config config/consumer.properties --from-beginning --topic __consumer_offsets --zookeeper localhost:2181 --formatter "kafka.coordinator.GroupMetadataManager\$OffsetsMessageFormatter"`

    ## Kafka Consumer Groups

    ### List the consumer groups known to Kafka
  6. @filipefigcorreia filipefigcorreia revised this gist May 10, 2016. 1 changed file with 3 additions and 1 deletion.
    4 changes: 3 additions & 1 deletion kafka-cheat-sheet.md
    View file
    Original file line number Diff line number Diff line change
    @@ -28,6 +28,8 @@
    ### View the details of a consumer group
    `bin/kafka-consumer-groups.sh --zookeeper localhost:2181 --describe --group <group name>`

    ## Starting the Zookeeper Shell
    ## Zookeeper

    ### Starting the Zookeeper Shell

    `bin/zookeeper-shell.sh localhost:2181`
  7. @filipefigcorreia filipefigcorreia revised this gist May 10, 2016. 1 changed file with 7 additions and 3 deletions.
    10 changes: 7 additions & 3 deletions kafka-cheat-sheet.md
    View file
    Original file line number Diff line number Diff line change
    @@ -1,4 +1,4 @@
    ## Topics
    ## Kafka Topics
    ### List existing topics
    `bin/kafka-topics.sh --zookeeper localhost:2181 --list`

    @@ -18,12 +18,16 @@
    ### Get the latest offset still in a topic
    `bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic mytopic --time -1`

    ## Consumer Groups
    ## Kafka Consumer Groups

    ### List the consumer groups known to Kafka
    `bin/kafka-consumer-groups.sh --zookeeper localhost:2181 --list` (old api)

    `bin/kafka-consumer-groups.sh --new-consumer --bootstrap-server localhost:9092 --list` (new api)

    ### View the details of a consumer group
    `bin/kafka-consumer-groups.sh --zookeeper localhost:2181 --describe --group <group name>`
    `bin/kafka-consumer-groups.sh --zookeeper localhost:2181 --describe --group <group name>`

    ## Starting the Zookeeper Shell

    `bin/zookeeper-shell.sh localhost:2181`
  8. @filipefigcorreia filipefigcorreia revised this gist May 10, 2016. 1 changed file with 2 additions and 0 deletions.
    2 changes: 2 additions & 0 deletions kafka-cheat-sheet.md
    View file
    Original file line number Diff line number Diff line change
    @@ -18,6 +18,8 @@
    ### Get the latest offset still in a topic
    `bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic mytopic --time -1`

    ## Consumer Groups

    ### List the consumer groups known to Kafka
    `bin/kafka-consumer-groups.sh --zookeeper localhost:2181 --list` (old api)

  9. @filipefigcorreia filipefigcorreia revised this gist May 10, 2016. 1 changed file with 1 addition and 0 deletions.
    1 change: 1 addition & 0 deletions kafka-cheat-sheet.md
    View file
    Original file line number Diff line number Diff line change
    @@ -1,3 +1,4 @@
    ## Topics
    ### List existing topics
    `bin/kafka-topics.sh --zookeeper localhost:2181 --list`

  10. @filipefigcorreia filipefigcorreia revised this gist May 10, 2016. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion kafka-cheat-sheet.md
    View file
    Original file line number Diff line number Diff line change
    @@ -23,4 +23,4 @@
    `bin/kafka-consumer-groups.sh --new-consumer --bootstrap-server localhost:9092 --list` (new api)

    ### View the details of a consumer group
    `bin/kafka-consumer-groups.sh --zookeeper localhost:2181 --group <group name> --describe`
    `bin/kafka-consumer-groups.sh --zookeeper localhost:2181 --describe --group <group name>`
  11. @filipefigcorreia filipefigcorreia revised this gist May 10, 2016. 1 changed file with 2 additions and 2 deletions.
    4 changes: 2 additions & 2 deletions kafka-cheat-sheet.md
    View file
    Original file line number Diff line number Diff line change
    @@ -11,10 +11,10 @@
    ### Delete a topic
    `bin/kafka-topics.sh --zookeeper localhost:2181 --delete --topic mytopic`

    ### Get the earliest offset of a topic
    ### Get the earliest offset still in a topic
    `bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic mytopic --time -2`

    ### Get the latest offset of a topic
    ### Get the latest offset still in a topic
    `bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic mytopic --time -1`

    ### List the consumer groups known to Kafka
  12. @filipefigcorreia filipefigcorreia revised this gist May 10, 2016. 1 changed file with 1 addition and 0 deletions.
    1 change: 1 addition & 0 deletions kafka-cheat-sheet.md
    View file
    Original file line number Diff line number Diff line change
    @@ -19,6 +19,7 @@

    ### List the consumer groups known to Kafka
    `bin/kafka-consumer-groups.sh --zookeeper localhost:2181 --list` (old api)

    `bin/kafka-consumer-groups.sh --new-consumer --bootstrap-server localhost:9092 --list` (new api)

    ### View the details of a consumer group
  13. @filipefigcorreia filipefigcorreia revised this gist May 10, 2016. 1 changed file with 6 additions and 0 deletions.
    6 changes: 6 additions & 0 deletions kafka-cheat-sheet.md
    View file
    Original file line number Diff line number Diff line change
    @@ -17,3 +17,9 @@
    ### Get the latest offset of a topic
    `bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic mytopic --time -1`

    ### List the consumer groups known to Kafka
    `bin/kafka-consumer-groups.sh --zookeeper localhost:2181 --list` (old api)
    `bin/kafka-consumer-groups.sh --new-consumer --bootstrap-server localhost:9092 --list` (new api)

    ### View the details of a consumer group
    `bin/kafka-consumer-groups.sh --zookeeper localhost:2181 --group <group name> --describe`
  14. @filipefigcorreia filipefigcorreia revised this gist Apr 11, 2016. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion kafka-cheat-sheet.md
    View file
    Original file line number Diff line number Diff line change
    @@ -1,5 +1,5 @@
    ### List existing topics
    `bin/kafka-topics.sh --list`
    `bin/kafka-topics.sh --zookeeper localhost:2181 --list`

    ### Purge a topic
    `bin/kafka-topics.sh --zookeeper localhost:2181 --alter --topic mytopic --config retention.ms=1000`
  15. @filipefigcorreia filipefigcorreia revised this gist Apr 11, 2016. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion kafka-cheat-sheet.md
    View file
    Original file line number Diff line number Diff line change
    @@ -9,7 +9,7 @@
    `bin/kafka-topics.sh --zookeeper localhost:2181 --alter --topic mytopic --delete-config retention.ms`

    ### Delete a topic
    `bin/kafka-topics.sh --delete --zookeeper localhost:2181 --topic mytopic`
    `bin/kafka-topics.sh --zookeeper localhost:2181 --delete --topic mytopic`

    ### Get the earliest offset of a topic
    `bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic mytopic --time -2`
  16. @filipefigcorreia filipefigcorreia revised this gist Apr 11, 2016. 1 changed file with 5 additions and 2 deletions.
    7 changes: 5 additions & 2 deletions kafka-cheat-sheet.md
    View file
    Original file line number Diff line number Diff line change
    @@ -7,10 +7,13 @@
    ... wait a minute ...

    `bin/kafka-topics.sh --zookeeper localhost:2181 --alter --topic mytopic --delete-config retention.ms`

    ### Delete a topic
    `bin/kafka-topics.sh --delete --zookeeper localhost:2181 --topic mytopic`

    ### Get the earliest offset of a topic
    `bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list kafkasrv:9092 --topic mytopic --time -2`
    `bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic mytopic --time -2`

    ### Get the latest offset of a topic
    `bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list kafkasrv:9092 --topic mytopic --time -1`
    `bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic mytopic --time -1`

  17. @filipefigcorreia filipefigcorreia revised this gist Apr 11, 2016. 1 changed file with 2 additions and 4 deletions.
    6 changes: 2 additions & 4 deletions kafka-cheat-sheet.md
    View file
    Original file line number Diff line number Diff line change
    @@ -1,9 +1,7 @@
    ### Listing existing topics
    ### List existing topics
    `bin/kafka-topics.sh --list`



    ### Purging a topic
    ### Purge a topic
    `bin/kafka-topics.sh --zookeeper localhost:2181 --alter --topic mytopic --config retention.ms=1000`

    ... wait a minute ...
  18. @filipefigcorreia filipefigcorreia revised this gist Apr 11, 2016. 1 changed file with 2 additions and 2 deletions.
    4 changes: 2 additions & 2 deletions kafka-cheat-sheet.md
    View file
    Original file line number Diff line number Diff line change
    @@ -4,11 +4,11 @@


    ### Purging a topic
    `bin/kafka-topics.sh --zookeeper localhost:2181 --alter --topic MyTopic --config retention.ms=1000`
    `bin/kafka-topics.sh --zookeeper localhost:2181 --alter --topic mytopic --config retention.ms=1000`

    ... wait a minute ...

    `bin/kafka-topics.sh --zookeeper localhost:2181 --alter --topic MyTopic --delete-config retention.ms`
    `bin/kafka-topics.sh --zookeeper localhost:2181 --alter --topic mytopic --delete-config retention.ms`

    ### Get the earliest offset of a topic
    `bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list kafkasrv:9092 --topic mytopic --time -2`
  19. @filipefigcorreia filipefigcorreia revised this gist Apr 11, 2016. 1 changed file with 2 additions and 0 deletions.
    2 changes: 2 additions & 0 deletions kafka-cheat-sheet.md
    View file
    Original file line number Diff line number Diff line change
    @@ -1,3 +1,5 @@
    ### Listing existing topics
    `bin/kafka-topics.sh --list`



  20. @filipefigcorreia filipefigcorreia revised this gist Apr 11, 2016. 1 changed file with 2 additions and 2 deletions.
    4 changes: 2 additions & 2 deletions kafka-cheat-sheet.md
    View file
    Original file line number Diff line number Diff line change
    @@ -8,9 +8,9 @@

    `bin/kafka-topics.sh --zookeeper localhost:2181 --alter --topic MyTopic --delete-config retention.ms`

    ## Get earliest offset
    ### Get the earliest offset of a topic
    `bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list kafkasrv:9092 --topic mytopic --time -2`

    ## Get latest offset
    ### Get the latest offset of a topic
    `bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list kafkasrv:9092 --topic mytopic --time -1`

  21. @filipefigcorreia filipefigcorreia revised this gist Apr 11, 2016. 1 changed file with 3 additions and 1 deletion.
    4 changes: 3 additions & 1 deletion kafka-cheat-sheet.md
    View file
    Original file line number Diff line number Diff line change
    @@ -3,7 +3,9 @@

    ### Purging a topic
    `bin/kafka-topics.sh --zookeeper localhost:2181 --alter --topic MyTopic --config retention.ms=1000`
    ... wait a minute ...

    ... wait a minute ...

    `bin/kafka-topics.sh --zookeeper localhost:2181 --alter --topic MyTopic --delete-config retention.ms`

    ## Get earliest offset
  22. @filipefigcorreia filipefigcorreia revised this gist Apr 11, 2016. 1 changed file with 7 additions and 0 deletions.
    7 changes: 7 additions & 0 deletions kafka-cheat-sheet.md
    View file
    Original file line number Diff line number Diff line change
    @@ -1,4 +1,11 @@



    ### Purging a topic
    `bin/kafka-topics.sh --zookeeper localhost:2181 --alter --topic MyTopic --config retention.ms=1000`
    ... wait a minute ...
    `bin/kafka-topics.sh --zookeeper localhost:2181 --alter --topic MyTopic --delete-config retention.ms`

    ## Get earliest offset
    `bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list kafkasrv:9092 --topic mytopic --time -2`

  23. @filipefigcorreia filipefigcorreia revised this gist Apr 11, 2016. 1 changed file with 2 additions and 2 deletions.
    4 changes: 2 additions & 2 deletions kafka-cheat-sheet.md
    View file
    Original file line number Diff line number Diff line change
    @@ -1,7 +1,7 @@

    Get earliest offset
    ## Get earliest offset
    `bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list kafkasrv:9092 --topic mytopic --time -2`

    Get latest offset
    ## Get latest offset
    `bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list kafkasrv:9092 --topic mytopic --time -1`

  24. @filipefigcorreia filipefigcorreia renamed this gist Apr 11, 2016. 1 changed file with 0 additions and 0 deletions.
    0 gistfile1.txt → kafka-cheat-sheet.md
    View file
    File renamed without changes.
  25. @filipefigcorreia filipefigcorreia created this gist Apr 11, 2016.
    7 changes: 7 additions & 0 deletions gistfile1.txt
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,7 @@

    Get earliest offset
    `bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list kafkasrv:9092 --topic mytopic --time -2`

    Get latest offset
    `bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list kafkasrv:9092 --topic mytopic --time -1`