This is a work in progress.

Download the latest Kafka, which you can find at: http://kafka.apache.org/downloads.html

cd ~
curl -O http://apache.cs.utah.edu/incubator/kafka/kafka-0.7.2-incubating/kafka-0.7.2-incubating-src.tgz
tar -xvzf kafka-0.7.2-incubating-src.tgz
rm kafka-0.7.2-incubating-src.tgz
cd kafka-0.7.2-incubating-src

Kafka setup requires javac which only comes in the java-devel package and isn't installed by default, so run:

sudo yum install java-devel

Let's build it and package it up for release:

cd ~/kafka-0.7.2-incubating-src
./sbt
> update
> package
> release-zip

Command-C out of SBT, copy the resulting release-zip so we can work with it:

cp ~/kafka-0.7.2-incubating-src/core/dist/kafka-0.7.2.zip ~/
unzip ~/kafka-0.7.2.zip -d ~/kafka-0.7.2
sudo mkdir /usr/local/apache-kafka
sudo cp -rv ~/kafka-0.7.2 /usr/local/apache-kafka/
vim ~/.bash_profile

Add the following lines to your .bash_profile:

export KAFKA_HOME=/usr/local/apache-kafka/kafka-0.7.2
export PATH=$KAFKA_HOME:$PATH

Now source the bash_profile so you have access to the new env vars:

source ~/.bash_profile

There seem to be a plethora of Kafka S3 Consumers, which is a good thing. Just take a look at this github search: https://github.com/search?q=kafka+s3

We're going to choose https://github.com/QubitProducts/kafka-s3-consumer because it's written in java and seems to be updated recently.

First, we'll have to get git and maven setup:

sudo yum install git
cd ~
curl -O http://mirror.olnevhost.net/pub/apache/maven/maven-3/3.0.5/binaries/apache-maven-3.0.5-bin.tar.gz
tar -xvzf apache-maven-3.0.5-bin.tar.gz
rm apache-maven-3.0.5-bin.tar.gz
sudo mkdir /usr/local/apache-maven
sudo cp -rv ~/apache-maven-3.0.5 /usr/local/apache-maven/
vim ~/.bash_profile

Add the following lines to your .bash_profile:

export M2_HOME=/usr/local/apache-maven/apache-maven-3.0.5
export M2=$M2_HOME/bin
export PATH=$M2:$PATH

Now source the bash_profile so you have access to the new env vars:

source ~/.bash_profile

If it was successful, you should be able to run mvn --version and have the

Now let's get back to the business of setting up the s3 consumer.

cd ~
git clone https://github.com/QubitProducts/kafka-s3-consumer
cd kafka-s3-consumer
mvn package