Alex Deimosster

Benchmark commands

#test-producer.conf
acks=1
bootstrap.servers=kafka.example.com:9092
buffer.memory=67108864
batch.size=128000

##Sample Command

export KAFKA_HEAP_OPTS='-Xmx6g -Xms6g -XX:MetaspaceSize=96m -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:G1HeapRegionSize=16M -XX:MinMetaspaceFreeRatio=50 -XX:MaxMetaspaceFreeRatio=80'

./kafka-mirror-maker.sh --consumer.config consumer.properties --producer.config producer.properties --num.streams 5 --whitelist="proc_output"

Increase replication factor for a topic or change its assigment:

1.Generate custom JSON file using provided tools by Kafka distribution (/bin/kafka-reassign-partitions.sh)

  kafka-reassign-partitions.sh --generate \
       --topics-to-move-json-file topics_to_move.json \
       --zookeeper $ZOOKEEPER_CONNECT \
       --broker-list "1,2,3"

2.Use a custom reassignment plan (see attached file partition_assign.json). In this case we are going from replication factor of 1 to 3.

	## Consumer Throughput: Single consumer thread, no compression
	## Consumer Throughput: 3 consumer thread, no compression

	bin/kafka-consumer-perf-test.sh --topic benchmark-3-3-none \
	--zookeeper kafka-zk-1:2181,kafka-zk-2:2181,kafka-zk-3:2181 \
	--messages 15000000 \
	--threads 1

	#!/usr/bin/env python



	from datetime import datetime

	import os