# Kafka-node consumer benchmark

This code can be used to benchmark throughput for a kafka cluster. 
Args:
```
groupId -- (str) kafka consumer group id, default: bench
concurrency -- (int) Number of worker threads to spawn, defaults to number of cpus on current host
duration -- (int) How long to run the benchmark for, default: 20s
topic -- (str) the kafka topic to consume from, defaults to observations.json
zk -- (str) zookeeper url, defaults to localhost:2181
useHLC -- (bool) wether to use a simple consumer or a high level consumer, see the kafka docs for an explanation. Defaults to false
```
E.g:
```
node master.js --groupId=test --concurrency=2 --duration=60 --topic=test --zk=localhost:2181/zkchroot --useHLC=true
```
Or for just a single worker
```
node kafka_consumer_worker.js
```

## Prereqs:
 You need a few libs
 ```
 npm install kafka-node optimist
 ```
 If you get a error installing kafka-node from a failure to compile its 
 snappy dependency you can replace it with kafka-node--light which removes the dependency on snappy
 https://github.com/kaustavha/kafka-node--light
 And replace the require statements from `require('kafka-node')` to `require('kafka-node--light')`.
 Albeit if youre using snappy compression in kafka this wont be able to decompress your messages and you'll see a lower throughput since messages are chunked and compressed together

## Results:

All these benchmarks were run a 12 AMD Opteron(tm) Processor 4184 (800mhz) core server with 64 gigs of RAM and 64 kafka partitions. 

```
Processed 407402 messages
Start Time 1452886344.33
Elapsed Time 60.134000062942505
TPS: 6774.902710173456
Total metrics processed: 2076641
Mean metric count per obv: 5.097277382045252
Mode: 2
```
```
Processed 411449 messages
Start Time 1452886457.638
Elapsed Time 60.14800000190735
TPS: 6840.6098288713265
Total metrics processed: 2097194
Mean metric count per obv: 5.097093442929744
Mode: 2
```
```
Processed 474156 messages
Start Time 1452886560.338
Elapsed Time 60.01300001144409
TPS: 7900.888139396156
Total metrics processed: 2414438
Mean metric count per obv: 5.092075182007609
Mode: 2
```

### Using a simple consumer and variable worker counts:  
2 workers : `~16 TPS`  
4 workers: `~25k TPS`  
6 workers: `~40k TPS`  
12 workers: `~70k TPS`  
24 workers: `~57k TPS`  

### Using the High level consumer and variable worker counts:   
6 workers: `~16k TPS`   
12 workers: `~40k TPS`  
32 workers: `~45 TPS`  
64 workers: `~55k TPS`  
96 workers: `~50k TPS`