Skip to content

Instantly share code, notes, and snippets.

@piyushgarg-dev
Last active November 1, 2025 17:49
Show Gist options
  • Save piyushgarg-dev/32cadf6420c452b66a9a6d977ade0b01 to your computer and use it in GitHub Desktop.
Save piyushgarg-dev/32cadf6420c452b66a9a6d977ade0b01 to your computer and use it in GitHub Desktop.

Revisions

  1. Piyush Garg revised this gist Jul 29, 2023. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion README.md
    Original file line number Diff line number Diff line change
    @@ -3,7 +3,7 @@ Video Link: [Apache Kafka Crash Course | What is Kafka?](https://youtu.be/ZJJHm_
    ## Prerequisite
    - Knowledge
    - Node.JS Intermediate level
    - Experince with designing distributed systems
    - Experience with designing distributed systems
    - Tools
    - Node.js: [Download Node.JS](https://nodejs.org/en)
    - Docker: [Download Docker](https://www.docker.com)
  2. Piyush Garg revised this gist Jul 29, 2023. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion README.md
    Original file line number Diff line number Diff line change
    @@ -30,7 +30,7 @@ const { Kafka } = require("kafkajs");

    exports.kafka = new Kafka({
    clientId: "my-app",
    brokers: ["192.168.29.35:9092"],
    brokers: ["<PRIVATE_IP>:9092"],
    });

    ```
  3. Piyush Garg revised this gist Jul 29, 2023. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion README.md
    Original file line number Diff line number Diff line change
    @@ -1,5 +1,5 @@
    # Kafka

    Video Link: [Apache Kafka Crash Course | What is Kafka?](https://youtu.be/ZJJHm_bd9Zo)
    ## Prerequisite
    - Knowledge
    - Node.JS Intermediate level
  4. Piyush Garg revised this gist Jul 29, 2023. 1 changed file with 13 additions and 0 deletions.
    13 changes: 13 additions & 0 deletions README.md
    Original file line number Diff line number Diff line change
    @@ -122,4 +122,17 @@ async function init() {
    }

    init();
    ```
    ## Running Locally
    - Run Multiple Consumers
    ```bash
    node consumer.js <GROUP_NAME>
    ```
    - Create Producer
    ```bash
    node producer.js
    ```
    ```bash
    > tony south
    > tony north
    ```
  5. Piyush Garg revised this gist Jul 29, 2023. 1 changed file with 101 additions and 0 deletions.
    101 changes: 101 additions & 0 deletions README.md
    Original file line number Diff line number Diff line change
    @@ -21,4 +21,105 @@ docker run -p 9092:9092 \
    -e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://<PRIVATE_IP>:9092 \
    -e KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1 \
    confluentinc/cp-kafka
    ```

    ## Code
    `client.js`
    ```js
    const { Kafka } = require("kafkajs");

    exports.kafka = new Kafka({
    clientId: "my-app",
    brokers: ["192.168.29.35:9092"],
    });

    ```
    `admin.js`
    ```js
    const { kafka } = require("./client");

    async function init() {
    const admin = kafka.admin();
    console.log("Admin connecting...");
    admin.connect();
    console.log("Adming Connection Success...");

    console.log("Creating Topic [rider-updates]");
    await admin.createTopics({
    topics: [
    {
    topic: "rider-updates",
    numPartitions: 2,
    },
    ],
    });
    console.log("Topic Created Success [rider-updates]");

    console.log("Disconnecting Admin..");
    await admin.disconnect();
    }

    init();
    ```
    `producer.js`
    ```js
    const { kafka } = require("./client");
    const readline = require("readline");

    const rl = readline.createInterface({
    input: process.stdin,
    output: process.stdout,
    });

    async function init() {
    const producer = kafka.producer();

    console.log("Connecting Producer");
    await producer.connect();
    console.log("Producer Connected Successfully");

    rl.setPrompt("> ");
    rl.prompt();

    rl.on("line", async function (line) {
    const [riderName, location] = line.split(" ");
    await producer.send({
    topic: "rider-updates",
    messages: [
    {
    partition: location.toLowerCase() === "north" ? 0 : 1,
    key: "location-update",
    value: JSON.stringify({ name: riderName, location }),
    },
    ],
    });
    }).on("close", async () => {
    await producer.disconnect();
    });
    }

    init();
    ```
    `consumer.js`
    ```js
    const { kafka } = require("./client");
    const group = process.argv[2];

    async function init() {
    const consumer = kafka.consumer({ groupId: group });
    await consumer.connect();

    await consumer.subscribe({ topics: ["rider-updates"], fromBeginning: true });

    await consumer.run({
    eachMessage: async ({ topic, partition, message, heartbeat, pause }) => {
    console.log(
    `${group}: [${topic}]: PART:${partition}:`,
    message.value.toString()
    );
    },
    });
    }

    init();
    ```
  6. Piyush Garg revised this gist Jul 29, 2023. 1 changed file with 2 additions and 2 deletions.
    4 changes: 2 additions & 2 deletions README.md
    Original file line number Diff line number Diff line change
    @@ -19,6 +19,6 @@ docker run -p 2181:2181 zookeeper
    docker run -p 9092:9092 \
    -e KAFKA_ZOOKEEPER_CONNECT=<PRIVATE_IP>:2181 \
    -e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://<PRIVATE_IP>:9092 \
    -e KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1
    confluentic/cp-kafka
    -e KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1 \
    confluentinc/cp-kafka
    ```
  7. Piyush Garg revised this gist Jul 29, 2023. 1 changed file with 9 additions and 1 deletion.
    10 changes: 9 additions & 1 deletion README.md
    Original file line number Diff line number Diff line change
    @@ -10,7 +10,15 @@
    - VsCode: [Download VSCode](https://code.visualstudio.com)

    ## Commands
    - Start Zookeper Container and expose PORT `2181`
    - Start Zookeper Container and expose PORT `2181`.
    ```bash
    docker run -p 2181:2181 zookeeper
    ```
    - Start Kafka Container, expose PORT `9092` and setup ENV variables.
    ```bash
    docker run -p 9092:9092 \
    -e KAFKA_ZOOKEEPER_CONNECT=<PRIVATE_IP>:2181 \
    -e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://<PRIVATE_IP>:9092 \
    -e KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1
    confluentic/cp-kafka
    ```
  8. Piyush Garg revised this gist Jul 29, 2023. 1 changed file with 7 additions and 1 deletion.
    8 changes: 7 additions & 1 deletion README.md
    Original file line number Diff line number Diff line change
    @@ -7,4 +7,10 @@
    - Tools
    - Node.js: [Download Node.JS](https://nodejs.org/en)
    - Docker: [Download Docker](https://www.docker.com)
    - VsCode: [Download VSCode](https://code.visualstudio.com)
    - VsCode: [Download VSCode](https://code.visualstudio.com)

    ## Commands
    - Start Zookeper Container and expose PORT `2181`
    ```bash
    docker run -p 2181:2181 zookeeper
    ```
  9. Piyush Garg created this gist Jul 29, 2023.
    10 changes: 10 additions & 0 deletions README.md
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,10 @@
    # Kafka

    ## Prerequisite
    - Knowledge
    - Node.JS Intermediate level
    - Experince with designing distributed systems
    - Tools
    - Node.js: [Download Node.JS](https://nodejs.org/en)
    - Docker: [Download Docker](https://www.docker.com)
    - VsCode: [Download VSCode](https://code.visualstudio.com)