Sergey Volkodav volkodavs

Avro serialization

There are 4 possible serialization format when using avro:

Avro Json encoding
Avro Data Serialization (https://avro.apache.org/docs/current/spec.html#Data+Serialization) Binary format with an header that contains the full schema, this is the format usually used when writing Avro files
Avro Single Object Encoding (https://avro.apache.org/docs/current/spec.html#single_object_encoding) Binary format with an header with only the fingerprint/id of the schema, this it the format used by Kafka (see this
Avro Binary Encoding (https://avro.apache.org/docs/current/spec.html#binary_encoding)

What do Etcd, Consul, and Zookeeper do?
- Service Registration:
  - Host, port number, and sometimes authentication credentials, protocols, versions numbers, and/or environment details.
- Service Discovery:
  - Ability for client application to query the central registry to learn of service location.
- Consistent and durable general-purpose K/V store across distributed system.
  - Some solutions support this better than others.
  - Based on Paxos or some derivative (i.e. Raft) algorithm to quickly converge to a consistent state.
Centralized locking can be based on this K/V store.

Processing JSON using jq

jq is useful to slice, filter, map and transform structured json data.

Installing jq

On Mac OS

brew install jq

Introductory Material

Add Graal JIT Compilation to Your JVM Language in 5 Steps, A Tutorial http://stefan-marr.de/2015/11/add-graal-jit-compilation-to-your-jvm-language-in-5-easy-steps-step-1/
The SimpleLanguage, an example of using Truffle with great JavaDocs. It is the officle getting-started project: https://github.com/graalvm/simplelanguage
Truffle Tutorial, Christan Wimmer, PLDI 2016, 3h recording https://youtu.be/FJY96_6Y3a4 Slides

Moved to git repository: https://github.com/denji/nginx-tuning

NGINX Tuning For Best Performance

For this configuration you can use web server you like, i decided, because i work mostly with it to use nginx.

Generally, properly configured nginx can handle up to 400K to 500K requests per second (clustered), most what i saw is 50K to 80K (non-clustered) requests per second and 30% CPU load, course, this was 2 x Intel Xeon with HyperThreading enabled, but it can work without problem on slower machines.

You must understand that this config is used in testing environment and not in production so you will need to find a way to implement most of those features best possible for your servers.

Latency numbers every programmer should know

L1 cache reference ......................... 0.5 ns
Branch mispredict ............................ 5 ns
L2 cache reference ........................... 7 ns
Mutex lock/unlock ........................... 25 ns
Main memory reference ...................... 100 ns             
Compress 1K bytes with Zippy ............. 3,000 ns  =   3 µs
Send 2K bytes over 1 Gbps network ....... 20,000 ns  =  20 µs
SSD random read ........................ 150,000 ns  = 150 µs

Read 1 MB sequentially from memory ..... 250,000 ns = 250 µs

	Latency Comparison Numbers (~2012)
	----------------------------------
	L1 cache reference 0.5 ns
	Branch mispredict 5 ns
	L2 cache reference 7 ns 14x L1 cache
	Mutex lock/unlock 25 ns
	Main memory reference 100 ns 20x L2 cache, 200x L1 cache
	Compress 1K bytes with Zippy 3,000 ns 3 us
	Send 1K bytes over 1 Gbps network 10,000 ns 10 us
	Read 4K randomly from SSD* 150,000 ns 150 us ~1GB/sec SSD

	# Run this in the project repo from the command-line
	# http://stackoverflow.com/a/4593065/99923

	git log --shortstat --author "Xeoncross" --since "2 weeks ago" --until "1 week ago" \| grep "files changed" \| awk '{files+=$1; inserted+=$4; deleted+=$6} END {print "files changed", files, "lines inserted:", inserted, "lines deleted:", deleted}'

	[server]
	root_url = %(protocol)s://%(domain)s:/grafana