Skip to content

Instantly share code, notes, and snippets.

@ursuad

ursuad/kafka-cheat-sheet.md

Last active June 17, 2025 10:00
Show Gist options
  • Save ursuad/e5b8542024a15e4db601f34906b30bb5 to your computer and use it in GitHub Desktop.
Save ursuad/e5b8542024a15e4db601f34906b30bb5 to your computer and use it in GitHub Desktop.
Download ZIP

Revisions

  1. ursuad revised this gist Jul 27, 2016. 1 changed file with 2 additions and 0 deletions.
    2 changes: 2 additions & 0 deletions kafka-cheat-sheet.md
    View file
    Original file line number Diff line number Diff line change
    @@ -13,8 +13,10 @@

    ### Delete a topic
    `bin/kafka-topics.sh --zookeeper localhost:2181 --delete --topic mytopic`

    ### Get number of messages in a topic ???
    `bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic mytopic --time -1 --offsets 1 | awk -F ":" '{sum += $3} END {print sum}'`

    ### Get the earliest offset still in a topic
    `bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic mytopic --time -2`

  2. ursuad revised this gist Jul 27, 2016. 1 changed file with 2 additions and 1 deletion.
    3 changes: 2 additions & 1 deletion kafka-cheat-sheet.md
    View file
    Original file line number Diff line number Diff line change
    @@ -13,7 +13,8 @@

    ### Delete a topic
    `bin/kafka-topics.sh --zookeeper localhost:2181 --delete --topic mytopic`

    ### Get number of messages in a topic ???
    `bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic mytopic --time -1 --offsets 1 | awk -F ":" '{sum += $3} END {print sum}'`
    ### Get the earliest offset still in a topic
    `bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic mytopic --time -2`

  3. ursuad revised this gist Jul 27, 2016. 1 changed file with 2 additions and 0 deletions.
    2 changes: 2 additions & 0 deletions kafka-cheat-sheet.md
    View file
    Original file line number Diff line number Diff line change
    @@ -2,6 +2,8 @@
    ### List existing topics
    `bin/kafka-topics.sh --zookeeper localhost:2181 --list`

    ### Describe a topic
    `bin/kafka-topics.sh --zookeeper localhost:2181 --describe --topic mytopic `
    ### Purge a topic
    `bin/kafka-topics.sh --zookeeper localhost:2181 --alter --topic mytopic --config retention.ms=1000`

  4. ursuad created this gist Jul 27, 2016.
    53 changes: 53 additions & 0 deletions kafka-cheat-sheet.md
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,53 @@
    ## Kafka Topics
    ### List existing topics
    `bin/kafka-topics.sh --zookeeper localhost:2181 --list`

    ### Purge a topic
    `bin/kafka-topics.sh --zookeeper localhost:2181 --alter --topic mytopic --config retention.ms=1000`

    ... wait a minute ...

    `bin/kafka-topics.sh --zookeeper localhost:2181 --alter --topic mytopic --delete-config retention.ms`

    ### Delete a topic
    `bin/kafka-topics.sh --zookeeper localhost:2181 --delete --topic mytopic`

    ### Get the earliest offset still in a topic
    `bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic mytopic --time -2`

    ### Get the latest offset still in a topic
    `bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic mytopic --time -1`

    ### Consume messages with the console consumer
    `bin/kafka-console-consumer.sh --new-consumer --bootstrap-server localhost:9092 --topic mytopic --from-beginning`

    ## Get the consumer offsets for a topic
    `bin/kafka-consumer-offset-checker.sh --zookeeper=localhost:2181 --topic=mytopic --group=my_consumer_group`

    ### Read from __consumer_offsets

    Add the following property to `config/consumer.properties`:
    `exclude.internal.topics=false`

    `bin/kafka-console-consumer.sh --consumer.config config/consumer.properties --from-beginning --topic __consumer_offsets --zookeeper localhost:2181 --formatter "kafka.coordinator.GroupMetadataManager\$OffsetsMessageFormatter"`

    ## Kafka Consumer Groups

    ### List the consumer groups known to Kafka
    `bin/kafka-consumer-groups.sh --zookeeper localhost:2181 --list` (old api)

    `bin/kafka-consumer-groups.sh --new-consumer --bootstrap-server localhost:9092 --list` (new api)

    ### View the details of a consumer group
    `bin/kafka-consumer-groups.sh --zookeeper localhost:2181 --describe --group <group name>`

    ## kafkacat

    ### Getting the last five message of a topic
    `kafkacat -C -b localhost:9092 -t mytopic -p 0 -o -5 -e`

    ## Zookeeper

    ### Starting the Zookeeper Shell

    `bin/zookeeper-shell.sh localhost:2181`