Skip to content

Instantly share code, notes, and snippets.

@crowdmatt

crowdmatt/setting-up-a-golang-server-with-kafka-to-s3.md

Last active January 1, 2022 20:09
Show Gist options
  • Save crowdmatt/5357326 to your computer and use it in GitHub Desktop.
Save crowdmatt/5357326 to your computer and use it in GitHub Desktop.
Download ZIP

Revisions

  1. crowdmatt revised this gist Apr 13, 2013. 1 changed file with 6 additions and 0 deletions.
    6 changes: 6 additions & 0 deletions setting-up-a-golang-server-with-kafka-to-s3.md
    View file
    Original file line number Diff line number Diff line change
    @@ -111,6 +111,12 @@ vim ~/consumer.properties

    Change the consumer properties file to suit your needs.

    Optionally, you can put the consumer in `/usr/local/`, like:

    <pre>
    sudo mkdir /usr/local/crowdmob-kafka-consumer
    sudo mv kafka-s3-go-consumer /usr/local/crowdmob-kafka-consumer/
    </pre>


    About Producing Kafka Events from Go
  2. crowdmatt revised this gist Apr 13, 2013. 1 changed file with 6 additions and 0 deletions.
    6 changes: 6 additions & 0 deletions setting-up-a-golang-server-with-kafka-to-s3.md
    View file
    Original file line number Diff line number Diff line change
    @@ -88,6 +88,12 @@ Once you `source ~/.bash_profile`, you should be able to run `go` at the command

    Install the Consumer
    ---------------------

    First we need `git`, so let's install it.
    <pre>
    sudo yum install git
    </pre>

    We can install the consumer simply with:

    <pre>
  3. crowdmatt revised this gist Apr 12, 2013. 1 changed file with 2 additions and 2 deletions.
    4 changes: 2 additions & 2 deletions setting-up-a-golang-server-with-kafka-to-s3.md
    View file
    Original file line number Diff line number Diff line change
    @@ -110,15 +110,15 @@ Change the consumer properties file to suit your needs.
    About Producing Kafka Events from Go
    --------------------

    The most up-to-date Kafka library for Go seems to be this: https://github.com/jedsmith/kafka
    The most up-to-date Kafka library for Go seemed to be this: https://github.com/jedsmith/kafka , so we ended up forking it updating it a bit further, at https://github.com/crowdmob/kafka

    In your go file, simply add it to imports, like this:

    <pre>
    package main

    import (
    "github.com/jedsmith/kafka"
    "github.com/crowdmob/kafka"
    )

    func main() {
  4. crowdmatt revised this gist Apr 12, 2013. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion setting-up-a-golang-server-with-kafka-to-s3.md
    View file
    Original file line number Diff line number Diff line change
    @@ -132,7 +132,7 @@ I put that in a file called `~/gotest/producer.go`.
    You'll also need to go get the kafka go implementation:

    <pre>
    sudo /usr/local/go/bin/go get github.com/jedsmith/kafka
    sudo /usr/local/go/bin/go get github.com/crowdmob/kafka
    </pre>


  5. crowdmatt revised this gist Apr 12, 2013. 1 changed file with 2 additions and 2 deletions.
    4 changes: 2 additions & 2 deletions setting-up-a-golang-server-with-kafka-to-s3.md
    View file
    Original file line number Diff line number Diff line change
    @@ -165,9 +165,9 @@ go run ~/kafka-s3-go-consumer/consumer.go -c ~/consumer.properties
    Actually Produce From Go Script
    --------------------

    Now it's time for the fun part! Let's write our first message to go, and see if it ends up in our s3 bucket. Make sure you specify your AWS credentials and S3 bucket in `~/kafka-s3-consumer/kafka-s3-consumer.properties`.
    Now it's time for the fun part! Let's write our first message to go, and see if it ends up in our s3 bucket. Make sure you specify your AWS credentials and S3 bucket in `~/consumer.properties`.

    Also make sure `~/kafka-s3-consumer/kafka-s3-consumer.properties` specifies which Kafka Topic to listen to, and make sure that topic matches the topic you'll be producing to in Go.
    Also make sure `~/consumer.properties` specifies which Kafka Topic(s) to listen to, and make sure that topic matches the topic you'll be producing to in Go.

    If you've verified the buckets, credentials, and kafka topics, then you're ready to give the `go` producer a whirl.

  6. crowdmatt revised this gist Apr 12, 2013. 1 changed file with 2 additions and 5 deletions.
    7 changes: 2 additions & 5 deletions setting-up-a-golang-server-with-kafka-to-s3.md
    View file
    Original file line number Diff line number Diff line change
    @@ -105,10 +105,7 @@ vim ~/consumer.properties

    Change the consumer properties file to suit your needs.

    To run the consumer to listn, simply run:
    <pre>
    go run ~/kafka-s3-go-consumer/consumer.go -c ~/consumer.properties
    </pre>


    About Producing Kafka Events from Go
    --------------------
    @@ -162,7 +159,7 @@ In another shell, you'll need to run the kafka server as well:
    In another shell, we can now start up the s3 consumer that we created, using some properties file we put somewhere:

    <pre>
    java -jar /usr/local/apache-kafka/kafka-0.7.2/libs/kafka-s3-consumer-1.0.jar ~/kafka-s3-consumer/kafka-s3-consumer.properties
    go run ~/kafka-s3-go-consumer/consumer.go -c ~/consumer.properties
    </pre>

    Actually Produce From Go Script
  7. crowdmatt revised this gist Apr 12, 2013. 1 changed file with 26 additions and 51 deletions.
    77 changes: 26 additions & 51 deletions setting-up-a-golang-server-with-kafka-to-s3.md
    View file
    Original file line number Diff line number Diff line change
    @@ -55,86 +55,61 @@ Now source the bash_profile so you have access to the new env vars:
    source ~/.bash_profile
    </pre>

    Install Kafka S3 Consumer
    Find a Kafka S3 Consumer
    --------------------

    There seem to be a plethora of Kafka S3 Consumers, which is a good thing. Just take a look at this github search: https://github.com/search?q=kafka+s3
    There seem to be a small handful of Kafka S3 Consumers, which is a good thing. Just take a look at this github search: https://github.com/search?q=kafka+s3

    We're going to choose https://github.com/vinietje/kafka-s3-consumer because it's written in java and seems to be updated recently, and update it to use Kafka 0.7.2 (CrowdMob's made a fork of it at crowdmob/kafka-s3-consumer)
    I ended up writing a custom one, because I couldn't get the others to work very easily.

    First, we'll have to get git and maven setup:
    It's located at https://github.com/crowdmob/kafka-s3-go-consumer and we'll be using it in production.

    <pre>
    sudo yum install git
    cd ~
    curl -O http://mirror.olnevhost.net/pub/apache/maven/maven-3/3.0.5/binaries/apache-maven-3.0.5-bin.tar.gz
    tar -xvzf apache-maven-3.0.5-bin.tar.gz
    rm apache-maven-3.0.5-bin.tar.gz
    sudo mkdir /usr/local/apache-maven
    sudo cp -rv ~/apache-maven-3.0.5 /usr/local/apache-maven/
    vim ~/.bash_profile
    </pre>

    Add the following lines to your .bash_profile:

    <pre>
    export M2_HOME=/usr/local/apache-maven/apache-maven-3.0.5
    export M2=$M2_HOME/bin
    export PATH=$M2:$PATH
    </pre>
    Install Go / Golang on EC2
    --------------------

    Now source the bash_profile so you have access to the new env vars:
    Let's get go!

    <pre>
    source ~/.bash_profile
    cd ~
    curl -O https://go.googlecode.com/files/go1.0.3.linux-amd64.tar.gz
    sudo tar -C /usr/local -xzf ~/go1.0.3.linux-amd64.tar.gz
    rm ~/go1.0.3.linux-amd64.tar.gz
    vim ~/.bash_profile
    </pre>

    If it was successful, you should be able to run `mvn --version` and have the

    Now let's get back to the business of setting up the s3 consumer.

    First, we have to update maven to know about kafka 0.7.2 we installed.
    Of course, we have to add the path of go in our path again:

    <pre>
    mvn install:install-file -Dfile=/usr/local/apache-kafka/kafka-0.7.2/kafka-0.7.2.jar -DartifactId=kafka -Dversion=0.7.2 -DgroupId=kafka -Dpackaging=jar
    export PATH=$PATH:/usr/local/go/bin
    </pre>

    Now that maven knows about our kafka jar, we can build our s3 consumer.

    <pre>
    cd ~
    git clone https://github.com/crowdmob/kafka-s3-consumer
    cd kafka-s3-consumer
    mvn package
    </pre>
    Once you `source ~/.bash_profile`, you should be able to run `go` at the command line to make sure everything was installed!

    If build succeeds, which it should, then I personally like to copy the built jar file into kafka's lib dir:
    Install the Consumer
    ---------------------
    We can install the consumer simply with:

    <pre>
    sudo cp ~/kafka-s3-consumer/target/kafka-s3-consumer-1.0.jar /usr/local/apache-kafka/kafka-0.7.2/libs/
    cd ~
    git clone https://github.com/crowdmob/kafka-s3-go-consumer.git
    </pre>

    Install Go / Golang on EC2
    --------------------

    Let's get go!
    Then, configure it's properties file:

    <pre>
    cd ~
    curl -O https://go.googlecode.com/files/go1.0.3.linux-amd64.tar.gz
    sudo tar -C /usr/local -xzf ~/go1.0.3.linux-amd64.tar.gz
    rm ~/go1.0.3.linux-amd64.tar.gz
    vim ~/.bash_profile
    cp ~/kafka-s3-go-consumer/consumer.example.properties ~/consumer.properties
    vim ~/consumer.properties
    </pre>

    Of course, we have to add the path of go in our path again:
    Change the consumer properties file to suit your needs.

    To run the consumer to listn, simply run:
    <pre>
    export PATH=$PATH:/usr/local/go/bin
    go run ~/kafka-s3-go-consumer/consumer.go -c ~/consumer.properties
    </pre>

    Once you `source ~/.bash_profile`, you should be able to run `go` at the command line to make sure everything was installed!

    About Producing Kafka Events from Go
    --------------------

  8. crowdmatt revised this gist Apr 10, 2013. 1 changed file with 21 additions and 1 deletion.
    22 changes: 21 additions & 1 deletion setting-up-a-golang-server-with-kafka-to-s3.md
    View file
    Original file line number Diff line number Diff line change
    @@ -135,7 +135,7 @@ export PATH=$PATH:/usr/local/go/bin

    Once you `source ~/.bash_profile`, you should be able to run `go` at the command line to make sure everything was installed!

    Producing Kafka Events from Go
    About Producing Kafka Events from Go
    --------------------

    The most up-to-date Kafka library for Go seems to be this: https://github.com/jedsmith/kafka
    @@ -155,6 +155,8 @@ func main() {
    }
    </pre>

    I put that in a file called `~/gotest/producer.go`.

    You'll also need to go get the kafka go implementation:

    <pre>
    @@ -187,3 +189,21 @@ In another shell, we can now start up the s3 consumer that we created, using som
    <pre>
    java -jar /usr/local/apache-kafka/kafka-0.7.2/libs/kafka-s3-consumer-1.0.jar ~/kafka-s3-consumer/kafka-s3-consumer.properties
    </pre>

    Actually Produce From Go Script
    --------------------

    Now it's time for the fun part! Let's write our first message to go, and see if it ends up in our s3 bucket. Make sure you specify your AWS credentials and S3 bucket in `~/kafka-s3-consumer/kafka-s3-consumer.properties`.

    Also make sure `~/kafka-s3-consumer/kafka-s3-consumer.properties` specifies which Kafka Topic to listen to, and make sure that topic matches the topic you'll be producing to in Go.

    If you've verified the buckets, credentials, and kafka topics, then you're ready to give the `go` producer a whirl.

    Simply:

    <pre>
    cd gotest
    go run producer.go
    </pre>


  9. crowdmatt revised this gist Apr 10, 2013. 1 changed file with 20 additions and 1 deletion.
    21 changes: 20 additions & 1 deletion setting-up-a-golang-server-with-kafka-to-s3.md
    View file
    Original file line number Diff line number Diff line change
    @@ -165,6 +165,25 @@ sudo /usr/local/go/bin/go get github.com/jedsmith/kafka
    Okay! Now we're all setup, and just have to turn on all the services we just installed :)


    Run Kafka and S3 Consumer "Daemons"
    Run Zookeeper, Kafka Server, and S3 Consumer
    --------------------

    Let's run a simple kafka server, before we daemonize it:

    And of course, that requires first starting a zookeeper server.

    <pre>
    ~/kafka-0.7.2-incubating-src/bin/zookeeper-server-start.sh ~/kafka-0.7.2-incubating-src/config/zookeeper.properties
    </pre>

    In another shell, you'll need to run the kafka server as well:

    <pre>
    ~/kafka-0.7.2-incubating-src/bin/kafka-server-start.sh ~/kafka-0.7.2-incubating-src/config/server.properties
    </pre>

    In another shell, we can now start up the s3 consumer that we created, using some properties file we put somewhere:

    <pre>
    java -jar /usr/local/apache-kafka/kafka-0.7.2/libs/kafka-s3-consumer-1.0.jar ~/kafka-s3-consumer/kafka-s3-consumer.properties
    </pre>
  10. crowdmatt revised this gist Apr 10, 2013. 1 changed file with 15 additions and 1 deletion.
    16 changes: 15 additions & 1 deletion setting-up-a-golang-server-with-kafka-to-s3.md
    View file
    Original file line number Diff line number Diff line change
    @@ -153,4 +153,18 @@ func main() {
    broker := kafka.NewBrokerPublisher("localhost:9092", "events", 0)
    broker.Publish(kafka.NewMessage([]byte("tesing 1 2 3")))
    }
    </pre>
    </pre>

    You'll also need to go get the kafka go implementation:

    <pre>
    sudo /usr/local/go/bin/go get github.com/jedsmith/kafka
    </pre>


    Okay! Now we're all setup, and just have to turn on all the services we just installed :)


    Run Kafka and S3 Consumer "Daemons"
    --------------------

  11. crowdmatt revised this gist Apr 10, 2013. 1 changed file with 7 additions and 0 deletions.
    7 changes: 7 additions & 0 deletions setting-up-a-golang-server-with-kafka-to-s3.md
    View file
    Original file line number Diff line number Diff line change
    @@ -143,7 +143,14 @@ The most up-to-date Kafka library for Go seems to be this: https://github.com/je
    In your go file, simply add it to imports, like this:

    <pre>
    package main

    import (
    "github.com/jedsmith/kafka"
    )

    func main() {
    broker := kafka.NewBrokerPublisher("localhost:9092", "events", 0)
    broker.Publish(kafka.NewMessage([]byte("tesing 1 2 3")))
    }
    </pre>
  12. crowdmatt revised this gist Apr 10, 2013. 1 changed file with 20 additions and 0 deletions.
    20 changes: 20 additions & 0 deletions setting-up-a-golang-server-with-kafka-to-s3.md
    View file
    Original file line number Diff line number Diff line change
    @@ -114,6 +114,26 @@ If build succeeds, which it should, then I personally like to copy the built jar
    sudo cp ~/kafka-s3-consumer/target/kafka-s3-consumer-1.0.jar /usr/local/apache-kafka/kafka-0.7.2/libs/
    </pre>

    Install Go / Golang on EC2
    --------------------

    Let's get go!

    <pre>
    cd ~
    curl -O https://go.googlecode.com/files/go1.0.3.linux-amd64.tar.gz
    sudo tar -C /usr/local -xzf ~/go1.0.3.linux-amd64.tar.gz
    rm ~/go1.0.3.linux-amd64.tar.gz
    vim ~/.bash_profile
    </pre>

    Of course, we have to add the path of go in our path again:

    <pre>
    export PATH=$PATH:/usr/local/go/bin
    </pre>

    Once you `source ~/.bash_profile`, you should be able to run `go` at the command line to make sure everything was installed!

    Producing Kafka Events from Go
    --------------------
  13. crowdmatt revised this gist Apr 10, 2013. 1 changed file with 14 additions and 0 deletions.
    14 changes: 14 additions & 0 deletions setting-up-a-golang-server-with-kafka-to-s3.md
    View file
    Original file line number Diff line number Diff line change
    @@ -112,4 +112,18 @@ If build succeeds, which it should, then I personally like to copy the built jar

    <pre>
    sudo cp ~/kafka-s3-consumer/target/kafka-s3-consumer-1.0.jar /usr/local/apache-kafka/kafka-0.7.2/libs/
    </pre>


    Producing Kafka Events from Go
    --------------------

    The most up-to-date Kafka library for Go seems to be this: https://github.com/jedsmith/kafka

    In your go file, simply add it to imports, like this:

    <pre>
    import (
    "github.com/jedsmith/kafka"
    )
    </pre>
  14. crowdmatt revised this gist Apr 10, 2013. 1 changed file with 8 additions and 0 deletions.
    8 changes: 8 additions & 0 deletions setting-up-a-golang-server-with-kafka-to-s3.md
    View file
    Original file line number Diff line number Diff line change
    @@ -99,9 +99,17 @@ First, we have to update maven to know about kafka 0.7.2 we installed.
    mvn install:install-file -Dfile=/usr/local/apache-kafka/kafka-0.7.2/kafka-0.7.2.jar -DartifactId=kafka -Dversion=0.7.2 -DgroupId=kafka -Dpackaging=jar
    </pre>

    Now that maven knows about our kafka jar, we can build our s3 consumer.

    <pre>
    cd ~
    git clone https://github.com/crowdmob/kafka-s3-consumer
    cd kafka-s3-consumer
    mvn package
    </pre>

    If build succeeds, which it should, then I personally like to copy the built jar file into kafka's lib dir:

    <pre>
    sudo cp ~/kafka-s3-consumer/target/kafka-s3-consumer-1.0.jar /usr/local/apache-kafka/kafka-0.7.2/libs/
    </pre>
  15. crowdmatt revised this gist Apr 10, 2013. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion setting-up-a-golang-server-with-kafka-to-s3.md
    View file
    Original file line number Diff line number Diff line change
    @@ -60,7 +60,7 @@ Install Kafka S3 Consumer

    There seem to be a plethora of Kafka S3 Consumers, which is a good thing. Just take a look at this github search: https://github.com/search?q=kafka+s3

    We're going to choose https://github.com/QubitProducts/kafka-s3-consumer because it's written in java and seems to be updated recently, and update it to use Kafka 0.7.2 (CrowdMob's made a fork of it at crowdmob/kafka-s3-consumer)
    We're going to choose https://github.com/vinietje/kafka-s3-consumer because it's written in java and seems to be updated recently, and update it to use Kafka 0.7.2 (CrowdMob's made a fork of it at crowdmob/kafka-s3-consumer)

    First, we'll have to get git and maven setup:

  16. crowdmatt revised this gist Apr 10, 2013. 1 changed file with 8 additions and 3 deletions.
    11 changes: 8 additions & 3 deletions setting-up-a-golang-server-with-kafka-to-s3.md
    View file
    Original file line number Diff line number Diff line change
    @@ -47,7 +47,6 @@ Add the following lines to your .bash_profile:

    <pre>
    export KAFKA_HOME=/usr/local/apache-kafka/kafka-0.7.2
    export PATH=$KAFKA_HOME:$PATH
    </pre>

    Now source the bash_profile so you have access to the new env vars:
    @@ -61,7 +60,7 @@ Install Kafka S3 Consumer

    There seem to be a plethora of Kafka S3 Consumers, which is a good thing. Just take a look at this github search: https://github.com/search?q=kafka+s3

    We're going to choose https://github.com/QubitProducts/kafka-s3-consumer because it's written in java and seems to be updated recently.
    We're going to choose https://github.com/QubitProducts/kafka-s3-consumer because it's written in java and seems to be updated recently, and update it to use Kafka 0.7.2 (CrowdMob's made a fork of it at crowdmob/kafka-s3-consumer)

    First, we'll have to get git and maven setup:

    @@ -94,9 +93,15 @@ If it was successful, you should be able to run `mvn --version` and have the

    Now let's get back to the business of setting up the s3 consumer.

    First, we have to update maven to know about kafka 0.7.2 we installed.

    <pre>
    mvn install:install-file -Dfile=/usr/local/apache-kafka/kafka-0.7.2/kafka-0.7.2.jar -DartifactId=kafka -Dversion=0.7.2 -DgroupId=kafka -Dpackaging=jar
    </pre>

    <pre>
    cd ~
    git clone https://github.com/QubitProducts/kafka-s3-consumer
    git clone https://github.com/crowdmob/kafka-s3-consumer
    cd kafka-s3-consumer
    mvn package
    </pre>
  17. crowdmatt revised this gist Apr 10, 2013. 1 changed file with 1 addition and 0 deletions.
    1 change: 1 addition & 0 deletions setting-up-a-golang-server-with-kafka-to-s3.md
    View file
    Original file line number Diff line number Diff line change
    @@ -37,6 +37,7 @@ Command-C out of SBT, copy the resulting release-zip so we can work with it:
    <pre>
    cp ~/kafka-0.7.2-incubating-src/core/dist/kafka-0.7.2.zip ~/
    unzip ~/kafka-0.7.2.zip -d ~/kafka-0.7.2
    rm ~/kafka-0.7.2.zip
    sudo mkdir /usr/local/apache-kafka
    sudo cp -rv ~/kafka-0.7.2 /usr/local/apache-kafka/
    vim ~/.bash_profile
  18. crowdmatt revised this gist Apr 10, 2013. 1 changed file with 16 additions and 2 deletions.
    18 changes: 16 additions & 2 deletions setting-up-a-golang-server-with-kafka-to-s3.md
    View file
    Original file line number Diff line number Diff line change
    @@ -39,6 +39,20 @@ cp ~/kafka-0.7.2-incubating-src/core/dist/kafka-0.7.2.zip ~/
    unzip ~/kafka-0.7.2.zip -d ~/kafka-0.7.2
    sudo mkdir /usr/local/apache-kafka
    sudo cp -rv ~/kafka-0.7.2 /usr/local/apache-kafka/
    vim ~/.bash_profile
    </pre>

    Add the following lines to your .bash_profile:

    <pre>
    export KAFKA_HOME=/usr/local/apache-kafka/kafka-0.7.2
    export PATH=$KAFKA_HOME:$PATH
    </pre>

    Now source the bash_profile so you have access to the new env vars:

    <pre>
    source ~/.bash_profile
    </pre>

    Install Kafka S3 Consumer
    @@ -61,10 +75,10 @@ sudo cp -rv ~/apache-maven-3.0.5 /usr/local/apache-maven/
    vim ~/.bash_profile
    </pre>

    Add the following lines:
    Add the following lines to your .bash_profile:

    <pre>
    export M2_HOME=/usr/local/apache-maven/apache-maven-3.0.5/
    export M2_HOME=/usr/local/apache-maven/apache-maven-3.0.5
    export M2=$M2_HOME/bin
    export PATH=$M2:$PATH
    </pre>
  19. crowdmatt revised this gist Apr 10, 2013. 1 changed file with 10 additions and 0 deletions.
    10 changes: 10 additions & 0 deletions setting-up-a-golang-server-with-kafka-to-s3.md
    View file
    Original file line number Diff line number Diff line change
    @@ -16,6 +16,7 @@ rm kafka-0.7.2-incubating-src.tgz
    cd kafka-0.7.2-incubating-src
    </pre>


    Kafka setup requires `javac` which only comes in the `java-devel` package and isn't installed by default, so run:

    <pre>
    @@ -31,6 +32,15 @@ cd ~/kafka-0.7.2-incubating-src
    &gt; release-zip
    </pre>

    Command-C out of SBT, copy the resulting release-zip so we can work with it:

    <pre>
    cp ~/kafka-0.7.2-incubating-src/core/dist/kafka-0.7.2.zip ~/
    unzip ~/kafka-0.7.2.zip -d ~/kafka-0.7.2
    sudo mkdir /usr/local/apache-kafka
    sudo cp -rv ~/kafka-0.7.2 /usr/local/apache-kafka/
    </pre>

    Install Kafka S3 Consumer
    --------------------

  20. crowdmatt revised this gist Apr 10, 2013. 1 changed file with 4 additions and 1 deletion.
    5 changes: 4 additions & 1 deletion setting-up-a-golang-server-with-kafka-to-s3.md
    View file
    Original file line number Diff line number Diff line change
    @@ -22,12 +22,13 @@ Kafka setup requires `javac` which only comes in the `java-devel` package and is
    sudo yum install java-devel
    </pre>

    Then, to prevent errors, run SBT:
    Let's build it and package it up for release:
    <pre>
    cd ~/kafka-0.7.2-incubating-src
    ./sbt
    &gt; update
    &gt; package
    &gt; release-zip
    </pre>

    Install Kafka S3 Consumer
    @@ -71,4 +72,6 @@ Now let's get back to the business of setting up the s3 consumer.
    <pre>
    cd ~
    git clone https://github.com/QubitProducts/kafka-s3-consumer
    cd kafka-s3-consumer
    mvn package
    </pre>
  21. crowdmatt revised this gist Apr 10, 2013. 1 changed file with 43 additions and 0 deletions.
    43 changes: 43 additions & 0 deletions setting-up-a-golang-server-with-kafka-to-s3.md
    View file
    Original file line number Diff line number Diff line change
    @@ -28,4 +28,47 @@ cd ~/kafka-0.7.2-incubating-src
    ./sbt
    &gt; update
    &gt; package
    </pre>

    Install Kafka S3 Consumer
    --------------------

    There seem to be a plethora of Kafka S3 Consumers, which is a good thing. Just take a look at this github search: https://github.com/search?q=kafka+s3

    We're going to choose https://github.com/QubitProducts/kafka-s3-consumer because it's written in java and seems to be updated recently.

    First, we'll have to get git and maven setup:

    <pre>
    sudo yum install git
    cd ~
    curl -O http://mirror.olnevhost.net/pub/apache/maven/maven-3/3.0.5/binaries/apache-maven-3.0.5-bin.tar.gz
    tar -xvzf apache-maven-3.0.5-bin.tar.gz
    rm apache-maven-3.0.5-bin.tar.gz
    sudo mkdir /usr/local/apache-maven
    sudo cp -rv ~/apache-maven-3.0.5 /usr/local/apache-maven/
    vim ~/.bash_profile
    </pre>

    Add the following lines:

    <pre>
    export M2_HOME=/usr/local/apache-maven/apache-maven-3.0.5/
    export M2=$M2_HOME/bin
    export PATH=$M2:$PATH
    </pre>

    Now source the bash_profile so you have access to the new env vars:

    <pre>
    source ~/.bash_profile
    </pre>

    If it was successful, you should be able to run `mvn --version` and have the

    Now let's get back to the business of setting up the s3 consumer.

    <pre>
    cd ~
    git clone https://github.com/QubitProducts/kafka-s3-consumer
    </pre>
  22. crowdmatt revised this gist Apr 10, 2013. 1 changed file with 6 additions and 0 deletions.
    6 changes: 6 additions & 0 deletions setting-up-a-golang-server-with-kafka-to-s3.md
    View file
    Original file line number Diff line number Diff line change
    @@ -16,6 +16,12 @@ rm kafka-0.7.2-incubating-src.tgz
    cd kafka-0.7.2-incubating-src
    </pre>

    Kafka setup requires `javac` which only comes in the `java-devel` package and isn't installed by default, so run:

    <pre>
    sudo yum install java-devel
    </pre>

    Then, to prevent errors, run SBT:
    <pre>
    cd ~/kafka-0.7.2-incubating-src
  23. crowdmatt revised this gist Apr 10, 2013. 1 changed file with 8 additions and 0 deletions.
    8 changes: 8 additions & 0 deletions setting-up-a-golang-server-with-kafka-to-s3.md
    View file
    Original file line number Diff line number Diff line change
    @@ -14,4 +14,12 @@ curl -O http://apache.cs.utah.edu/incubator/kafka/kafka-0.7.2-incubating/kafka-0
    tar -xvzf kafka-0.7.2-incubating-src.tgz
    rm kafka-0.7.2-incubating-src.tgz
    cd kafka-0.7.2-incubating-src
    </pre>

    Then, to prevent errors, run SBT:
    <pre>
    cd ~/kafka-0.7.2-incubating-src
    ./sbt
    &gt; update
    &gt; package
    </pre>
  24. crowdmatt revised this gist Apr 10, 2013. 2 changed files with 17 additions and 7 deletions.
    17 changes: 17 additions & 0 deletions setting-up-a-golang-server-with-kafka-to-s3.md
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,17 @@
    How to setup a Go / Golang WebServer with Kafka Log Aggregation to S3
    =====================================================================

    This is a work in progress.

    Install Kafka
    --------------------

    Download the latest Kafka, which you can find at: http://kafka.apache.org/downloads.html

    <pre>
    cd ~
    curl -O http://apache.cs.utah.edu/incubator/kafka/kafka-0.7.2-incubating/kafka-0.7.2-incubating-src.tgz
    tar -xvzf kafka-0.7.2-incubating-src.tgz
    rm kafka-0.7.2-incubating-src.tgz
    cd kafka-0.7.2-incubating-src
    </pre>
    7 changes: 0 additions & 7 deletions setting-up-a-golang-server-with-kafka-to-s3.textile
    View file
    Original file line number Diff line number Diff line change
    @@ -1,7 +0,0 @@
    h1. How to setup a Go / Golang WebServer with Kafka Log Aggregation to S3

    This is a work in progress.

    h2. Install Kafka

    Go to http://kafka.apache.org/downloads.html
  25. crowdmatt created this gist Apr 10, 2013.
    7 changes: 7 additions & 0 deletions setting-up-a-golang-server-with-kafka-to-s3.textile
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,7 @@
    h1. How to setup a Go / Golang WebServer with Kafka Log Aggregation to S3

    This is a work in progress.

    h2. Install Kafka

    Go to http://kafka.apache.org/downloads.html