Skip to content

Instantly share code, notes, and snippets.

@abacaphiliac
Last active January 21, 2024 12:10
Show Gist options
  • Save abacaphiliac/f0553548f9c577214d16290c2e751071 to your computer and use it in GitHub Desktop.
Save abacaphiliac/f0553548f9c577214d16290c2e751071 to your computer and use it in GitHub Desktop.

Revisions

  1. abacaphiliac revised this gist Jan 17, 2017. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion run-kafka-container.md
    Original file line number Diff line number Diff line change
    @@ -46,5 +46,5 @@ This command will start an unnamed instance of `spotify/kafka` linked to the `ka
    docker run -it --rm --link kafka spotify/kafka /opt/kafka_2.11-0.10.1.0/bin/kafka-console-consumer.sh --bootstrap-server kafka:9092 --topic test --from-beginning
    ```

    # Send some messages
    # Send messages
    Send some newline-delimited messages in the Producer terminal window. The messages appear in the Consumer terminal window.
  2. abacaphiliac revised this gist Jan 17, 2017. 1 changed file with 4 additions and 1 deletion.
    5 changes: 4 additions & 1 deletion run-kafka-container.md
    Original file line number Diff line number Diff line change
    @@ -44,4 +44,7 @@ docker run -it --rm --link kafka spotify/kafka /opt/kafka_2.11-0.10.1.0/bin/kafk
    This command will start an unnamed instance of `spotify/kafka` linked to the `kafka` service, start a consumer, display existing messages from the `test` topic, and wait for new messages until you quit (which destroys the container):
    ```
    docker run -it --rm --link kafka spotify/kafka /opt/kafka_2.11-0.10.1.0/bin/kafka-console-consumer.sh --bootstrap-server kafka:9092 --topic test --from-beginning
    ```
    ```

    # Send some messages
    Send some newline-delimited messages in the Producer terminal window. The messages appear in the Consumer terminal window.
  3. abacaphiliac revised this gist Jan 17, 2017. 1 changed file with 2 additions and 2 deletions.
    4 changes: 2 additions & 2 deletions run-kafka-container.md
    Original file line number Diff line number Diff line change
    @@ -34,13 +34,13 @@ output:
    test
    ```

    # Start a producer (in another container)
    # Start a producer (in a new terminal window)
    This command will run an unnamed instance of `spotify/kafka` linked to the `kafka` service, start a producer, and wait for newline-delimited input until you quit (which destroys the container):
    ```
    docker run -it --rm --link kafka spotify/kafka /opt/kafka_2.11-0.10.1.0/bin/kafka-console-producer.sh --broker-list kafka:9092 --topic test
    ```

    # Start a consumer (in another container)
    # Start a consumer (in a new terminal window)
    This command will start an unnamed instance of `spotify/kafka` linked to the `kafka` service, start a consumer, display existing messages from the `test` topic, and wait for new messages until you quit (which destroys the container):
    ```
    docker run -it --rm --link kafka spotify/kafka /opt/kafka_2.11-0.10.1.0/bin/kafka-console-consumer.sh --bootstrap-server kafka:9092 --topic test --from-beginning
  4. abacaphiliac revised this gist Jan 17, 2017. 1 changed file with 5 additions and 4 deletions.
    9 changes: 5 additions & 4 deletions run-kafka-container.md
    Original file line number Diff line number Diff line change
    @@ -7,11 +7,12 @@ docker run -d -p 2181:2181 -p 9092:9092 --env ADVERTISED_HOST=kafka --env ADVERT

    [Why Spotify?](https://github.com/spotify/docker-kafka#why)

    `ADVERTISTED_HOST` was set to `kafka`, which will allow other containers to be able to run Producers and/or Consumers.
    Setting it to `localhost`, `127.0.0.1`, or `0.0.0.0` will work great only if Producers and Consumers are started within the `kafka` container itself, or if you are using DockerForMac (like me) and you want to run Producers and Consumers from OSX.
    `ADVERTISTED_HOST` was set to `kafka`, which will allow other containers to be able to run Producers and Consumers.

    Setting `ADVERTISED_HOST` to `localhost`, `127.0.0.1`, or `0.0.0.0` will work great only if Producers and Consumers are started within the `kafka` container itself, or if you are using DockerForMac (like me) and you want to run Producers and Consumers from OSX.
    These are far less interesting use cases though, so we'll start Producers and Consumers from other containers.
    In order for the `kafka` service to be reachable from another container, we need to use an IP address or hostname. Since
    we do not know the IP address before the container is started, we have to choose a hostname, and I chose `kafka` in this example.

    We need to use an IP address or hostname in order for the `kafka` service to be reachable from another container. IP address is not known before the container is started, so we have to choose a hostname, and I chose `kafka` in this example.

    # Create a topic
    ```
  5. abacaphiliac revised this gist Jan 17, 2017. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion run-kafka-container.md
    Original file line number Diff line number Diff line change
    @@ -8,7 +8,7 @@ docker run -d -p 2181:2181 -p 9092:9092 --env ADVERTISED_HOST=kafka --env ADVERT
    [Why Spotify?](https://github.com/spotify/docker-kafka#why)

    `ADVERTISTED_HOST` was set to `kafka`, which will allow other containers to be able to run Producers and/or Consumers.
    Setting it to `localhost`, `127.0.0.1`, or `0.0.0.0` will work great if Producers and Consumers are started within the `kafka` container itself, or if you are using DockerForMac (like me) and you want to run Producers and Consumers from OSX.
    Setting it to `localhost`, `127.0.0.1`, or `0.0.0.0` will work great only if Producers and Consumers are started within the `kafka` container itself, or if you are using DockerForMac (like me) and you want to run Producers and Consumers from OSX.
    These are far less interesting use cases though, so we'll start Producers and Consumers from other containers.
    In order for the `kafka` service to be reachable from another container, we need to use an IP address or hostname. Since
    we do not know the IP address before the container is started, we have to choose a hostname, and I chose `kafka` in this example.
  6. abacaphiliac revised this gist Jan 17, 2017. 1 changed file with 2 additions and 0 deletions.
    2 changes: 2 additions & 0 deletions run-kafka-container.md
    Original file line number Diff line number Diff line change
    @@ -10,6 +10,8 @@ docker run -d -p 2181:2181 -p 9092:9092 --env ADVERTISED_HOST=kafka --env ADVERT
    `ADVERTISTED_HOST` was set to `kafka`, which will allow other containers to be able to run Producers and/or Consumers.
    Setting it to `localhost`, `127.0.0.1`, or `0.0.0.0` will work great if Producers and Consumers are started within the `kafka` container itself, or if you are using DockerForMac (like me) and you want to run Producers and Consumers from OSX.
    These are far less interesting use cases though, so we'll start Producers and Consumers from other containers.
    In order for the `kafka` service to be reachable from another container, we need to use an IP address or hostname. Since
    we do not know the IP address before the container is started, we have to choose a hostname, and I chose `kafka` in this example.

    # Create a topic
    ```
  7. abacaphiliac revised this gist Jan 17, 2017. 1 changed file with 2 additions and 2 deletions.
    4 changes: 2 additions & 2 deletions run-kafka-container.md
    Original file line number Diff line number Diff line change
    @@ -5,6 +5,8 @@ docker pull spotify/kafka
    docker run -d -p 2181:2181 -p 9092:9092 --env ADVERTISED_HOST=kafka --env ADVERTISED_PORT=9092 --name kafka spotify/kafka
    ```

    [Why Spotify?](https://github.com/spotify/docker-kafka#why)

    `ADVERTISTED_HOST` was set to `kafka`, which will allow other containers to be able to run Producers and/or Consumers.
    Setting it to `localhost`, `127.0.0.1`, or `0.0.0.0` will work great if Producers and Consumers are started within the `kafka` container itself, or if you are using DockerForMac (like me) and you want to run Producers and Consumers from OSX.
    These are far less interesting use cases though, so we'll start Producers and Consumers from other containers.
    @@ -29,8 +31,6 @@ output:
    test
    ```

    [Why Spotify?](https://github.com/spotify/docker-kafka#why)

    # Start a producer (in another container)
    This command will run an unnamed instance of `spotify/kafka` linked to the `kafka` service, start a producer, and wait for newline-delimited input until you quit (which destroys the container):
    ```
  8. abacaphiliac created this gist Jan 17, 2017.
    44 changes: 44 additions & 0 deletions run-kafka-container.md
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,44 @@
    # Start Kafka service
    The following commands will start a container with Kafka and Zookeeper running on mapped ports 2181 (Zookeeper) and 9092 (Kafka).
    ```
    docker pull spotify/kafka
    docker run -d -p 2181:2181 -p 9092:9092 --env ADVERTISED_HOST=kafka --env ADVERTISED_PORT=9092 --name kafka spotify/kafka
    ```

    `ADVERTISTED_HOST` was set to `kafka`, which will allow other containers to be able to run Producers and/or Consumers.
    Setting it to `localhost`, `127.0.0.1`, or `0.0.0.0` will work great if Producers and Consumers are started within the `kafka` container itself, or if you are using DockerForMac (like me) and you want to run Producers and Consumers from OSX.
    These are far less interesting use cases though, so we'll start Producers and Consumers from other containers.

    # Create a topic
    ```
    docker exec kafka /opt/kafka_2.11-0.10.1.0/bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test
    ```

    output:
    ```
    Created topic "test".
    ```

    # List topics
    ```
    docker exec kafka /opt/kafka_2.11-0.10.1.0/bin/kafka-topics.sh --list --zookeeper localhost:2181
    ```

    output:
    ```
    test
    ```

    [Why Spotify?](https://github.com/spotify/docker-kafka#why)

    # Start a producer (in another container)
    This command will run an unnamed instance of `spotify/kafka` linked to the `kafka` service, start a producer, and wait for newline-delimited input until you quit (which destroys the container):
    ```
    docker run -it --rm --link kafka spotify/kafka /opt/kafka_2.11-0.10.1.0/bin/kafka-console-producer.sh --broker-list kafka:9092 --topic test
    ```

    # Start a consumer (in another container)
    This command will start an unnamed instance of `spotify/kafka` linked to the `kafka` service, start a consumer, display existing messages from the `test` topic, and wait for new messages until you quit (which destroys the container):
    ```
    docker run -it --rm --link kafka spotify/kafka /opt/kafka_2.11-0.10.1.0/bin/kafka-console-consumer.sh --bootstrap-server kafka:9092 --topic test --from-beginning
    ```