Last active
November 1, 2025 17:49
-
Star
(203)
You must be signed in to star a gist -
Fork
(150)
You must be signed in to fork a gist
-
-
Save piyushgarg-dev/32cadf6420c452b66a9a6d977ade0b01 to your computer and use it in GitHub Desktop.
Revisions
-
Piyush Garg revised this gist
Jul 29, 2023 . 1 changed file with 1 addition and 1 deletion.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -3,7 +3,7 @@ Video Link: [Apache Kafka Crash Course | What is Kafka?](https://youtu.be/ZJJHm_ ## Prerequisite - Knowledge - Node.JS Intermediate level - Experience with designing distributed systems - Tools - Node.js: [Download Node.JS](https://nodejs.org/en) - Docker: [Download Docker](https://www.docker.com) -
Piyush Garg revised this gist
Jul 29, 2023 . 1 changed file with 1 addition and 1 deletion.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -30,7 +30,7 @@ const { Kafka } = require("kafkajs"); exports.kafka = new Kafka({ clientId: "my-app", brokers: ["<PRIVATE_IP>:9092"], }); ``` -
Piyush Garg revised this gist
Jul 29, 2023 . 1 changed file with 1 addition and 1 deletion.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -1,5 +1,5 @@ # Kafka Video Link: [Apache Kafka Crash Course | What is Kafka?](https://youtu.be/ZJJHm_bd9Zo) ## Prerequisite - Knowledge - Node.JS Intermediate level -
Piyush Garg revised this gist
Jul 29, 2023 . 1 changed file with 13 additions and 0 deletions.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -122,4 +122,17 @@ async function init() { } init(); ``` ## Running Locally - Run Multiple Consumers ```bash node consumer.js <GROUP_NAME> ``` - Create Producer ```bash node producer.js ``` ```bash > tony south > tony north ``` -
Piyush Garg revised this gist
Jul 29, 2023 . 1 changed file with 101 additions and 0 deletions.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -21,4 +21,105 @@ docker run -p 9092:9092 \ -e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://<PRIVATE_IP>:9092 \ -e KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1 \ confluentinc/cp-kafka ``` ## Code `client.js` ```js const { Kafka } = require("kafkajs"); exports.kafka = new Kafka({ clientId: "my-app", brokers: ["192.168.29.35:9092"], }); ``` `admin.js` ```js const { kafka } = require("./client"); async function init() { const admin = kafka.admin(); console.log("Admin connecting..."); admin.connect(); console.log("Adming Connection Success..."); console.log("Creating Topic [rider-updates]"); await admin.createTopics({ topics: [ { topic: "rider-updates", numPartitions: 2, }, ], }); console.log("Topic Created Success [rider-updates]"); console.log("Disconnecting Admin.."); await admin.disconnect(); } init(); ``` `producer.js` ```js const { kafka } = require("./client"); const readline = require("readline"); const rl = readline.createInterface({ input: process.stdin, output: process.stdout, }); async function init() { const producer = kafka.producer(); console.log("Connecting Producer"); await producer.connect(); console.log("Producer Connected Successfully"); rl.setPrompt("> "); rl.prompt(); rl.on("line", async function (line) { const [riderName, location] = line.split(" "); await producer.send({ topic: "rider-updates", messages: [ { partition: location.toLowerCase() === "north" ? 0 : 1, key: "location-update", value: JSON.stringify({ name: riderName, location }), }, ], }); }).on("close", async () => { await producer.disconnect(); }); } init(); ``` `consumer.js` ```js const { kafka } = require("./client"); const group = process.argv[2]; async function init() { const consumer = kafka.consumer({ groupId: group }); await consumer.connect(); await consumer.subscribe({ topics: ["rider-updates"], fromBeginning: true }); await consumer.run({ eachMessage: async ({ topic, partition, message, heartbeat, pause }) => { console.log( `${group}: [${topic}]: PART:${partition}:`, message.value.toString() ); }, }); } init(); ``` -
Piyush Garg revised this gist
Jul 29, 2023 . 1 changed file with 2 additions and 2 deletions.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -19,6 +19,6 @@ docker run -p 2181:2181 zookeeper docker run -p 9092:9092 \ -e KAFKA_ZOOKEEPER_CONNECT=<PRIVATE_IP>:2181 \ -e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://<PRIVATE_IP>:9092 \ -e KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1 \ confluentinc/cp-kafka ``` -
Piyush Garg revised this gist
Jul 29, 2023 . 1 changed file with 9 additions and 1 deletion.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -10,7 +10,15 @@ - VsCode: [Download VSCode](https://code.visualstudio.com) ## Commands - Start Zookeper Container and expose PORT `2181`. ```bash docker run -p 2181:2181 zookeeper ``` - Start Kafka Container, expose PORT `9092` and setup ENV variables. ```bash docker run -p 9092:9092 \ -e KAFKA_ZOOKEEPER_CONNECT=<PRIVATE_IP>:2181 \ -e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://<PRIVATE_IP>:9092 \ -e KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1 confluentic/cp-kafka ``` -
Piyush Garg revised this gist
Jul 29, 2023 . 1 changed file with 7 additions and 1 deletion.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -7,4 +7,10 @@ - Tools - Node.js: [Download Node.JS](https://nodejs.org/en) - Docker: [Download Docker](https://www.docker.com) - VsCode: [Download VSCode](https://code.visualstudio.com) ## Commands - Start Zookeper Container and expose PORT `2181` ```bash docker run -p 2181:2181 zookeeper ``` -
Piyush Garg created this gist
Jul 29, 2023 .There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -0,0 +1,10 @@ # Kafka ## Prerequisite - Knowledge - Node.JS Intermediate level - Experince with designing distributed systems - Tools - Node.js: [Download Node.JS](https://nodejs.org/en) - Docker: [Download Docker](https://www.docker.com) - VsCode: [Download VSCode](https://code.visualstudio.com)