Skip to content

Instantly share code, notes, and snippets.

@dobeerman
Forked from dkurzaj/README.md
Created October 9, 2022 09:48
Show Gist options
  • Save dobeerman/7ba3c9814a8fb14431d4a547dfe7fac4 to your computer and use it in GitHub Desktop.
Save dobeerman/7ba3c9814a8fb14431d4a547dfe7fac4 to your computer and use it in GitHub Desktop.
Docker compose Kafka, Zookeeper and Kafka manager

Docker compose Kafka, Zookeeper and Kafka manager

Install

  • Create the environment variable that contains our host name (IP address) :
$ export DOCKER_HOST=$(ip addr | grep 'state UP' -A2 | tail -n1 | awk '{print $2}' | cut -f1  -d'/')
  • Create the folders :
$ mkdir -p zookeeper/{data,logs}

Run

  • Execute the docker compose :
$ docker-compose -f docker-compose.yml up -d
  • Access Kafka Manager : http://localhost:9000/.

  • Create a new cluster.

  • Name it as you like (Localtest for example).

  • IP : zookeeper:2181

  • Kafka version (corresponding here to the Kafka version inside the image : wurstmeister/kafka:1.0.0). But as I write this Gist, the latest Kafka version available in Kafka Manager is 0.11.0.0 so I select this one, but it's sufficiently compatible with the 1.0.0 version of Kafka according to this topic: yahoo/CMAK#451

  • Tick Enable JMX Polling in order to see the metrics of the topics

  • Tick Poll consumer information to know the consumer of a topic (it may not work)

  • Tick Enable Active OffsetCache to see the offsets

And then create it.

✨ This is done ! ✨

Uninstall

  • Stop the containers and remove them :
$ docker-compose -f docker-compose.yml stop && docker-compose -f docker-compose.yml rm -vf

Tips

  • Remove content of Docker volumes (we need to do this trick to remove hidden and non-hidden files) :
$ rm -rf zookeeper/{data,logs} && mkdir -p zookeeper/{data,logs}

Gist inspired by this one intending to be an updated version of it : https://gist.github.com/17twenty/d619f922ab209f750630824f7c6836e3

version: '3.1'
services:
zookeeper:
container_name: zookeeper
image: zookeeper:3.4
restart: on-failure
volumes:
- "./zookeeper/data:/data"
- "./zookeeper/logs:/datalog"
ports:
- "2181:2181"
network_mode: "host"
kafka:
container_name: kafka
image: wurstmeister/kafka:1.0.0
restart: on-failure
depends_on:
- zookeeper
volumes:
- /var/run/docker.sock:/var/run/docker.sock
environment:
- KAFKA_ZOOKEEPER_CONNECT=${DOCKER_HOST}:2181
- KAFKA_ADVERTISED_HOST_NAME=${DOCKER_HOST}
- JMX_PORT=9093
- KAFKA_ADVERTISED_PORT=9092
- KAFKA_DELETE_TOPIC_ENABLE=true
- KAFKA_LOG_RETENTION_HOURS=1
- KAFKA_MESSAGE_MAX_BYTES=10000000
- KAFKA_REPLICA_FETCH_MAX_BYTES=10000000
- KAFKA_GROUP_MAX_SESSION_TIMEOUT_MS=60000
- KAFKA_NUM_PARTITIONS=2
- KAFKA_DELETE_RETENTION_MS=1000
ports:
- "9092:9092"
- "9093:9093"
network_mode: "host"
kafka-manager:
container_name: kafka-manager
image: hlebalbau/kafka-manager:1.3.3.16
restart: on-failure
depends_on:
- kafka
- zookeeper
command: -Dconfig.file=/kafka-manager/conf/application.conf -Dapplication.home=/kafkamanager
environment:
- ZK_HOSTS=${DOCKER_HOST}
- APPLICATION_SECRET=letmein
ports:
- "9000:9000"
network_mode: "host"
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment