Kafka 0.11.0.0 (Confluent 3.3.0) added support to manipulate offsets for a consumer group via cli kafka-consumer-groups command.
kafka-consumer-groups --bootstrap-server <kafkahost:port> --group <group_id> --describeNote the values under "CURRENT-OFFSET" and "LOG-END-OFFSET". "CURRENT-OFFSET" is the offset where this consumer group is currently at in each of the partitions.
| hivesite where to get it ? |
| Producer | |
| Setup | |
| bin/kafka-topics.sh --zookeeper esv4-hcl197.grid.linkedin.com:2181 --create --topic test-rep-one --partitions 6 --replication-factor 1 | |
| bin/kafka-topics.sh --zookeeper esv4-hcl197.grid.linkedin.com:2181 --create --topic test --partitions 6 --replication-factor 3 | |
| Single thread, no replication | |
| bin/kafka-run-class.sh org.apache.kafka.clients.tools.ProducerPerformance test7 50000000 100 -1 acks=1 bootstrap.servers=esv4-hcl198.grid.linkedin.com:9092 buffer.memory=67108864 batch.size=8196 |
| App Assembly plugin: | |
| 1> Spring boot (Using parent pom and spring boot plugin. http://docs.spring.io/spring-boot/docs/current/reference/html/build-tool-plugins-maven-plugin.html) | |
| 2> Maven App Assembly - http://www.mojohaus.org/appassembler/appassembler-maven-plugin/ (Used in DC project.) | |
| 3> Maven Assembly - http://maven.apache.org/plugins/maven-assembly-plugin/ | |
| 4> Maven shade plugin - https://maven.apache.org/plugins/maven-shade-plugin/ | |
| sudo nano /etc/postgresql/9.3/main/postgresql.conf | |
| Enable or add: | |
| listen_addresses = '*' | |
| Restart the database engine: | |
| sudo service postgresql restart |
| Check for open ports. | |
| sudo nmap -sS -O 127.0.0.1 | |
| -sS | |
| TCP SYN scan | |
| -O | |
| Enable Operating System detection | |
| Maven Based | |
| 1. Camel https://github.com/apache/camel | |
| 2. Spring Boot https://github.com/spring-projects/spring-boot | |
| Gradle Based | |
| 1. Hibernate - https://github.com/hibernate |