Skip to content

Instantly share code, notes, and snippets.

@chivasto

chivasto/changing kafka retention period during runtime

Created October 9, 2018 03:55
Show Gist options
  • Save chivasto/37c920fb19ab7e7081407c87dba43f62 to your computer and use it in GitHub Desktop.
Save chivasto/37c920fb19ab7e7081407c87dba43f62 to your computer and use it in GitHub Desktop.
Download ZIP

Revisions

  1. chivasto created this gist Oct 9, 2018.
    17 changes: 17 additions & 0 deletions changing kafka retention period during runtime
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,17 @@
    log.retention.hours is a property of a broker which is used as a default value when a topic is created. When you change configurations of currently running topic using kafka-topics.sh, you should specify a topic-level property.

    A topic-level property for log retention time is retention.ms.

    Property: retention.ms
    Default: 7 days
    Server Default Property: log.retention.minutes
    Description: This configuration controls the maximum time we will retain a log before we will discard old log segments to free up space if we are using the "delete" retention policy. This represents an SLA on how soon consumers must read their data.
    So the correct command is

    $ bin/kafka-topics.sh --zookeeper zk.yoursite.com --alter --topic as-access --config retention.ms=86400000
    You can check whether the configuration is properly applied with the following command.

    $ bin/kafka-topics.sh --describe --zookeeper zk.yoursite.com --topic as-access
    Then you will see something like below.

    Topic:as-access PartitionCount:3 ReplicationFactor:3 Configs:retention.ms=86400000