Skip to content

Instantly share code, notes, and snippets.

@ram-pi

ram-pi/Kafka-console-consumer-with-kerberos.md

Forked from kumar-de/Kafka-console-consumer-with-kerberos.md
Created September 8, 2022 06:43
Show Gist options
  • Save ram-pi/f72480d9be1175205ee5d7aa9dcedf2d to your computer and use it in GitHub Desktop.
Save ram-pi/f72480d9be1175205ee5d7aa9dcedf2d to your computer and use it in GitHub Desktop.
Download ZIP

Revisions

  1. Achintya Kumar revised this gist Jan 27, 2021. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion Kafka-console-consumer-with-kerberos.md
    View file
    Original file line number Diff line number Diff line change
    @@ -49,7 +49,7 @@ kafka-console-consumer --topic <topic-name> --from-beginning --bootstrap-server
    kafka-console-producer --broker-list <anybroker>:9093 --topic topic-kumar --producer.config consumer.properties < event1.json
    ```

    ### 7. Execute the Kafka-console-producer script:
    ### 7. Execute the Kafka-topic script to describe a given topic:
    ```
    kafka-topics --describe --topic topic-kumar --zookeeper <zookeeperHost>:2181
    ```
  2. Achintya Kumar revised this gist Jan 27, 2021. 1 changed file with 10 additions and 1 deletion.
    11 changes: 10 additions & 1 deletion Kafka-console-consumer-with-kerberos.md
    View file
    Original file line number Diff line number Diff line change
    @@ -44,8 +44,17 @@ kafka-topics.sh --list --zookeeper localhost:2181
    kafka-console-consumer --topic <topic-name> --from-beginning --bootstrap-server <anybroker>:9092 --consumer.config ./consumer.properties
    ```

    ### 6. Execute the Kafka-console-producer script:
    ```
    kafka-console-producer --broker-list <anybroker>:9093 --topic topic-kumar --producer.config consumer.properties < event1.json
    ```

    ### 7. Execute the Kafka-console-producer script:
    ```
    kafka-topics --describe --topic topic-kumar --zookeeper <zookeeperHost>:2181
    ```

    ### 6. Delete topic (provided delete.topic.enable is set)
    ### 8. Delete topic (provided delete.topic.enable is set)
    ```
    ./bin/kafka-topics.sh --zookeeper localhost:2181 --delete --topic remove-me
    ```
  3. Achintya Kumar revised this gist Sep 7, 2020. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion Kafka-console-consumer-with-kerberos.md
    View file
    Original file line number Diff line number Diff line change
    @@ -36,7 +36,7 @@ export KAFKA_OPTS="-Djava.security.auth.login.config=./jaas.conf"

    ### 4. List topics
    ```
    kafka-topics.sh --list --zookeeper localhost:2181 --consumer.config ./consumer.properties
    kafka-topics.sh --list --zookeeper localhost:2181
    ```

    ### 5. Execute the Kafka-console-consumer script:
  4. Achintya Kumar revised this gist Sep 7, 2020. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion Kafka-console-consumer-with-kerberos.md
    View file
    Original file line number Diff line number Diff line change
    @@ -36,7 +36,7 @@ export KAFKA_OPTS="-Djava.security.auth.login.config=./jaas.conf"

    ### 4. List topics
    ```
    kafka-topics.sh --list --zookeeper localhost:2181
    kafka-topics.sh --list --zookeeper localhost:2181 --consumer.config ./consumer.properties
    ```

    ### 5. Execute the Kafka-console-consumer script:
  5. Achintya Kumar revised this gist Sep 7, 2020. 1 changed file with 6 additions and 2 deletions.
    8 changes: 6 additions & 2 deletions Kafka-console-consumer-with-kerberos.md
    View file
    Original file line number Diff line number Diff line change
    @@ -34,14 +34,18 @@ sasl.kerberos.service.name=kafka
    export KAFKA_OPTS="-Djava.security.auth.login.config=./jaas.conf"
    ```

    ### 4. List topics
    ```
    kafka-topics.sh --list --zookeeper localhost:2181
    ```

    ### 4. Execute the Kafka-console-consumer script:
    ### 5. Execute the Kafka-console-consumer script:
    ```
    kafka-console-consumer --topic <topic-name> --from-beginning --bootstrap-server <anybroker>:9092 --consumer.config ./consumer.properties
    ```


    ### 5. Delete topic (provided delete.topic.enable is set)
    ### 6. Delete topic (provided delete.topic.enable is set)
    ```
    ./bin/kafka-topics.sh --zookeeper localhost:2181 --delete --topic remove-me
    ```
  6. Achintya Kumar revised this gist Jun 10, 2020. 1 changed file with 3 additions and 1 deletion.
    4 changes: 3 additions & 1 deletion Kafka-console-consumer-with-kerberos.md
    View file
    Original file line number Diff line number Diff line change
    @@ -36,7 +36,9 @@ export KAFKA_OPTS="-Djava.security.auth.login.config=./jaas.conf"


    ### 4. Execute the Kafka-console-consumer script:
    ```kafka-console-consumer --topic <topic-name> --from-beginning --bootstrap-server <anybroker>:9092 --consumer.config ./consumer.properties```
    ```
    kafka-console-consumer --topic <topic-name> --from-beginning --bootstrap-server <anybroker>:9092 --consumer.config ./consumer.properties
    ```


    ### 5. Delete topic (provided delete.topic.enable is set)
  7. Achintya Kumar revised this gist Jun 10, 2020. 1 changed file with 3 additions and 10 deletions.
    13 changes: 3 additions & 10 deletions Kafka-console-consumer-with-kerberos.md
    View file
    Original file line number Diff line number Diff line change
    @@ -22,30 +22,23 @@ Client {
    };
    ```




    ### 2. Create a properties file (say "consumer.properties") with the following contents:
    ```
    security.protocol=SASL_PLAINTEXT
    sasl.kerberos.service.name=kafka
    ```




    ### 3. Then at the terminal run the following command:
    ```export KAFKA_OPTS="-Djava.security.auth.login.config=./jaas.conf"```



    ```
    export KAFKA_OPTS="-Djava.security.auth.login.config=./jaas.conf"
    ```


    ### 4. Execute the Kafka-console-consumer script:
    ```kafka-console-consumer --topic <topic-name> --from-beginning --bootstrap-server <anybroker>:9092 --consumer.config ./consumer.properties```



    ### 5. Delete topic (provided delete.topic.enable is set)
    ```
    ./bin/kafka-topics.sh --zookeeper localhost:2181 --delete --topic remove-me
  8. Achintya Kumar revised this gist Jun 10, 2020. 1 changed file with 3 additions and 2 deletions.
    5 changes: 3 additions & 2 deletions Kafka-console-consumer-with-kerberos.md
    View file
    Original file line number Diff line number Diff line change
    @@ -47,10 +47,11 @@ sasl.kerberos.service.name=kafka


    ### 5. Delete topic (provided delete.topic.enable is set)
    ```./bin/kafka-topics.sh --zookeeper localhost:2181 --delete --topic remove-me```
    ```
    ./bin/kafka-topics.sh --zookeeper localhost:2181 --delete --topic remove-me
    ```
    If the above doesn't work, try the following:
    ```
    # Delete topic folder from Kafka broker machine.
    # Login to zookeeper and -
    hbase zkcli
    rmr /brokers/topics/<topic_name>
  9. Achintya Kumar revised this gist Jun 10, 2020. 1 changed file with 9 additions and 1 deletion.
    10 changes: 9 additions & 1 deletion Kafka-console-consumer-with-kerberos.md
    View file
    Original file line number Diff line number Diff line change
    @@ -47,4 +47,12 @@ sasl.kerberos.service.name=kafka


    ### 5. Delete topic (provided delete.topic.enable is set)
    ```./bin/kafka-topics.sh --zookeeper localhost:2181 --delete --topic remove-me```
    ```./bin/kafka-topics.sh --zookeeper localhost:2181 --delete --topic remove-me```
    If the above doesn't work, try the following:
    ```
    # Delete topic folder from Kafka broker machine.
    # Login to zookeeper and -
    hbase zkcli
    rmr /brokers/topics/<topic_name>
    rmr /admin/delete_topics/<topic_name>
    ```
  10. Achintya Kumar revised this gist Aug 20, 2019. 1 changed file with 1 addition and 0 deletions.
    1 change: 1 addition & 0 deletions Kafka-console-consumer-with-kerberos.md
    View file
    Original file line number Diff line number Diff line change
    @@ -1,3 +1,4 @@
    ## Kafka console consumer with Kerberos
    ### 1. Create a jaas.conf file with the following contents:
    ```
    KafkaClient {
  11. Achintya Kumar revised this gist Jul 24, 2019. 1 changed file with 2 additions and 6 deletions.
    8 changes: 2 additions & 6 deletions Kafka-console-consumer-with-kerberos.md
    View file
    Original file line number Diff line number Diff line change
    @@ -34,18 +34,14 @@ sasl.kerberos.service.name=kafka


    ### 3. Then at the terminal run the following command:
    ```
    export KAFKA_OPTS="-Djava.security.auth.login.config=./jaas.conf"
    ```
    ```export KAFKA_OPTS="-Djava.security.auth.login.config=./jaas.conf"```





    ### 4. Execute the Kafka-console-consumer script:
    ```
    kafka-console-consumer --topic <topic-name> --from-beginning --bootstrap-server <anybroker>:9092 --consumer.config ./consumer.properties
    ```
    ```kafka-console-consumer --topic <topic-name> --from-beginning --bootstrap-server <anybroker>:9092 --consumer.config ./consumer.properties```



  12. Achintya Kumar revised this gist Jul 24, 2019. 1 changed file with 6 additions and 1 deletion.
    7 changes: 6 additions & 1 deletion Kafka-console-consumer-with-kerberos.md
    View file
    Original file line number Diff line number Diff line change
    @@ -45,4 +45,9 @@ export KAFKA_OPTS="-Djava.security.auth.login.config=./jaas.conf"
    ### 4. Execute the Kafka-console-consumer script:
    ```
    kafka-console-consumer --topic <topic-name> --from-beginning --bootstrap-server <anybroker>:9092 --consumer.config ./consumer.properties
    ```
    ```



    ### 5. Delete topic (provided delete.topic.enable is set)
    ```./bin/kafka-topics.sh --zookeeper localhost:2181 --delete --topic remove-me```
  13. Achintya Kumar revised this gist Jun 21, 2019. 1 changed file with 17 additions and 4 deletions.
    21 changes: 17 additions & 4 deletions Kafka-console-consumer-with-kerberos.md
    View file
    Original file line number Diff line number Diff line change
    @@ -1,10 +1,23 @@
    ### 1. Create a jaas.conf file with the following contents:
    ```
    KafkaClient {
    com.sun.security.auth.module.Krb5LoginModule required
    useKeyTab=true
    keyTab="<path-to-the-keytab-file>"
    principal="<kafka-principal>";
    com.sun.security.auth.module.Krb5LoginModule required
    useKeyTab=true
    keyTab="keytabFile.keytab"
    storeKey=true
    useTicketCache=false
    serviceName="kafka"
    debug=true
    principal="[email protected]";
    };
    Client {
    com.sun.security.auth.module.Krb5LoginModule required
    useKeyTab=true
    keyTab="keytabFile.keytab"
    storeKey=true
    useTicketCache=false
    debug=true
    principal="[email protected]";
    };
    ```

  14. Achintya Kumar revised this gist Jun 20, 2019. 1 changed file with 2 additions and 2 deletions.
    4 changes: 2 additions & 2 deletions Kafka-console-consumer-with-kerberos.md
    View file
    Original file line number Diff line number Diff line change
    @@ -22,7 +22,7 @@ sasl.kerberos.service.name=kafka

    ### 3. Then at the terminal run the following command:
    ```
    export KAFKA_OPTS="-Djava.security.auth.login.config=<path-to-jaas.conf>"
    export KAFKA_OPTS="-Djava.security.auth.login.config=./jaas.conf"
    ```


    @@ -31,5 +31,5 @@ export KAFKA_OPTS="-Djava.security.auth.login.config=<path-to-jaas.conf>"

    ### 4. Execute the Kafka-console-consumer script:
    ```
    kafka-console-consumer --topic <topic-name> --from-beginning --bootstrap-server <anybroker>:9092 --consumer.config <consumer.properties>
    kafka-console-consumer --topic <topic-name> --from-beginning --bootstrap-server <anybroker>:9092 --consumer.config ./consumer.properties
    ```
  15. Achintya Kumar revised this gist Jun 20, 2019. 1 changed file with 1 addition and 2 deletions.
    3 changes: 1 addition & 2 deletions Kafka-console-consumer-with-kerberos.md
    View file
    Original file line number Diff line number Diff line change
    @@ -31,6 +31,5 @@ export KAFKA_OPTS="-Djava.security.auth.login.config=<path-to-jaas.conf>"

    ### 4. Execute the Kafka-console-consumer script:
    ```
    kafka-console-consumer --topic <topic-name> --from-beginning --bootstrap-server <anybroker>:9092 \
    --consumer.config <consumer.properties>
    kafka-console-consumer --topic <topic-name> --from-beginning --bootstrap-server <anybroker>:9092 --consumer.config <consumer.properties>
    ```
  16. Achintya Kumar revised this gist Jun 20, 2019. 1 changed file with 4 additions and 4 deletions.
    8 changes: 4 additions & 4 deletions Kafka-console-consumer-with-kerberos.md
    View file
    Original file line number Diff line number Diff line change
    @@ -1,4 +1,4 @@
    # 1. Create a jaas.conf file with the following contents:
    ### 1. Create a jaas.conf file with the following contents:
    ```
    KafkaClient {
    com.sun.security.auth.module.Krb5LoginModule required
    @@ -11,7 +11,7 @@ principal="<kafka-principal>";



    # 2. Create a properties file (say "consumer.properties") with the following contents:
    ### 2. Create a properties file (say "consumer.properties") with the following contents:
    ```
    security.protocol=SASL_PLAINTEXT
    sasl.kerberos.service.name=kafka
    @@ -20,7 +20,7 @@ sasl.kerberos.service.name=kafka



    # 3. Then at the terminal run the following command:
    ### 3. Then at the terminal run the following command:
    ```
    export KAFKA_OPTS="-Djava.security.auth.login.config=<path-to-jaas.conf>"
    ```
    @@ -29,7 +29,7 @@ export KAFKA_OPTS="-Djava.security.auth.login.config=<path-to-jaas.conf>"



    # 4. Execute the Kafka-console-consumer script:
    ### 4. Execute the Kafka-console-consumer script:
    ```
    kafka-console-consumer --topic <topic-name> --from-beginning --bootstrap-server <anybroker>:9092 \
    --consumer.config <consumer.properties>
  17. Achintya Kumar renamed this gist Jun 20, 2019. 1 changed file with 9 additions and 1 deletion.
    10 changes: 9 additions & 1 deletion Kafka-console-consumer-with-kerberos → Kafka-console-consumer-with-kerberos.md
    View file
    Original file line number Diff line number Diff line change
    @@ -1,28 +1,36 @@
    # 1. Create a jaas.conf file with the following contents:
    ```
    KafkaClient {
    com.sun.security.auth.module.Krb5LoginModule required
    useKeyTab=true
    keyTab="<path-to-the-keytab-file>"
    principal="<kafka-principal>";
    };
    ```




    # 2. Create a properties file (say "consumer.properties") with the following contents:
    ```
    security.protocol=SASL_PLAINTEXT
    sasl.kerberos.service.name=kafka
    ```




    # 3. Then at the terminal run the following command:
    ```
    export KAFKA_OPTS="-Djava.security.auth.login.config=<path-to-jaas.conf>"
    ```





    # 4. Execute the Kafka-console-consumer script:
    ```
    kafka-console-consumer --topic <topic-name> --from-beginning --bootstrap-server <anybroker>:9092 \
    --consumer.config <consumer.properties>
    --consumer.config <consumer.properties>
    ```
  18. Achintya Kumar revised this gist Jun 20, 2019. 1 changed file with 2 additions and 1 deletion.
    3 changes: 2 additions & 1 deletion Kafka-console-consumer-with-kerberos
    View file
    Original file line number Diff line number Diff line change
    @@ -24,4 +24,5 @@ export KAFKA_OPTS="-Djava.security.auth.login.config=<path-to-jaas.conf>"


    # 4. Execute the Kafka-console-consumer script:
    kafka-console-consumer --topic <topic-name> --from-beginning --bootstrap-server <anybroker>:9092 --consumer.config <consumer.properties>
    kafka-console-consumer --topic <topic-name> --from-beginning --bootstrap-server <anybroker>:9092 \
    --consumer.config <consumer.properties>
  19. Achintya Kumar created this gist Jun 20, 2019.
    27 changes: 27 additions & 0 deletions Kafka-console-consumer-with-kerberos
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,27 @@
    # 1. Create a jaas.conf file with the following contents:
    KafkaClient {
    com.sun.security.auth.module.Krb5LoginModule required
    useKeyTab=true
    keyTab="<path-to-the-keytab-file>"
    principal="<kafka-principal>";
    };




    # 2. Create a properties file (say "consumer.properties") with the following contents:
    security.protocol=SASL_PLAINTEXT
    sasl.kerberos.service.name=kafka




    # 3. Then at the terminal run the following command:
    export KAFKA_OPTS="-Djava.security.auth.login.config=<path-to-jaas.conf>"





    # 4. Execute the Kafka-console-consumer script:
    kafka-console-consumer --topic <topic-name> --from-beginning --bootstrap-server <anybroker>:9092 --consumer.config <consumer.properties>