fagossa fagossa

This post is Part 1 of a 3-part series about monitoring Apache Kafka. [Part 2][part2] is about collecting operational data from Kafka, and [Part 3][part3] details how to monitor Kafka with Datadog.

What is Kafka?

Kafka is a distributed, partitioned, replicated, [log][what-is-a-log] service developed by LinkedIn and open sourced in 2011. Basically it is a massively scalable pub/sub message queue architected as a distributed transaction log. It was created to provide "a unified platform for handling all the real-time data feeds a large company might have".^{[1][design-motivation]}

There are a few key differences between Kafka and other queueing systems like [RabbitMQ], [ActiveMQ], or [Redis's Pub/Sub][Redis]:

As mentioned above, it is fundamentally a replicated log service.
It does [not use AMQP][not-AMQP] or any other pre-existing protocol for communication. Instead, it uses a custom binary TCP-based protocol.
It is [very fast][kafka-benchmark], even in a small cluster.
It has str

	#!/usr/bin/env bash
	####################################################################################
	# Slack Bash console script for sending messages.
	####################################################################################
	# Installation
	# $ curl -s https://gist.github.com/andkirby/67a774513215d7ba06384186dd441d9e/raw --output /usr/bin/slack
	# $ chmod +x /usr/bin/slack
	####################################################################################
	# USAGE
	# Send message to slack channel/user

	lazy val root = project.in(file("."))
	.configs(IntegrationTest)
	.settings(Defaults.itSettings : _*)
	Revolver.settings

	lazy val startTask = TaskKey[Unit]("startDocker", "Start docker")
	startTask := {
	import sys.process._
	// EXECUTER DOCKER-COMPOSE UP
	println("waiting ...")

	package main

	import (
	"fmt"
	"math"
	)

	type Product struct {
	Price float64
	Name string

	import cats.data.Validated
	import Validated.{invalid, valid}
	import cats.data.{ValidatedNel, NonEmptyList => NEL}
	import cats.data._
	import cats.implicits._

	case class Person(mail: String, name: String)

	def isEmail(mail: String): Validated[NEL[String], String] =
	if (mail.contains("@")) {

	class SpliteratorExample {

	private static class GroupSpliterator<T> implements Spliterator<Stream<T>> {

	private final int grouping;
	private final Spliterator<T> spliterator;

	GroupSpliterator(int grouping, Spliterator<T> spliterator) {
	this.grouping = grouping;
	this.spliterator = spliterator;

	@RequestMapping("/employee")
	@Controller
	public class EmployeeSearchController {

	// some other code generating exceptions...

	@ResponseStatus(HttpStatus.BAD_REQUEST)
	@ExceptionHandler(value = {EmptyResultDataAccessException.class, EntityNotFoundException.class})
	public void handleNotFound() {}