sbacelic

heka.counter-output: Generated by CounterFilter. Usually picked up by a LogOutput for writing to stdout.
heka.plugin-report: Used inside the reporting infrastructure for plugins to provide report data to the dashboard, typically not injected into the router at all.
heka.input-report: Used inside the reporting infrastructure for the input recycle chan to provide report data to the dashboard, typically not injected into the router at all.
heka.inject-report: Used inside the reporting infrastructure for the inject recycle chan to provide report data to the dashboard, typically not injected into the router at all.
heka.router-report: Used inside the reporting infrastructure for the router to provide report data to the dashboard, typically not injected into the router at all.

numbat

An alerting engine for a metrics & monitoring system.

This is the same approach I wanted in my initial spike, only instead of writing a custom collector & using an existing alerting engine (riemann), I'm proposing using an existing collector (hekad) and writing the alerting engine.

The system

Running Docker on two hosts, bridged with Tinc

This Gist is similar to https://gist.github.com/noteed/8656989 which uses Open vSwitch instead of Tinc.

Setup

Download the install.sh script and run it:

> https://gist.github.com/noteed/11031504/raw/install.sh

	[mysql]
	host=localhost
	socket=/var/run/mysqld/mysqld.sock

	[client]
	host=localhost
	socket=/var/run/mysqld/mysqld.sock

	user nginx;
	worker_processes auto;
	error_log /var/log/nginx/error.log info;
	pid /var/run/nginx.pid;
	events { worker_connections 1024; }

	http {
	default_type text/html;
	log_format main '$remote_addr -> $request $status $body_bytes_sent bytes -> $upstream_addr';
	access_log /var/log/nginx/access.log main;

	<?php
	// app/AppKernel.php

	use Symfony\Component\Config\Loader\LoaderInterface;
	use Symfony\Component\HttpKernel\Kernel;

	class AppKernel extends Kernel
	{
	public function registerBundles()
	{

	require "cjson"

	local dt = require 'date_time'
	-- Generic decoder for JSON logs. This will extract all JSON
	-- keys and add them to the `Fields` variable of the created
	-- Heka message.
	--
	-- Example use:
	--
	-- [NginxJsonLogDecoder]

	[hekad]
	maxprocs = 4

	# Nginx access log reader
	[Nginxserver]
	type = "LogstreamerInput"
	log_directory = "/var/log/nginx"
	file_match = 'access\.log'
	decoder = "CombinedNginxDecoder"

	apt-get purge perl python python2.7 python2.7-minimal python-apt python-apt-common python-chardet python-debian python-debianbts python-fpconst python-minimal python-reportbug python-soappy python-support python-xapian exim4 exim4-base exim4-config exim4-daemon-light tasksel tasksel-data busybox anacron dnsutils w3m consolekit geoip-database ftp fuse curl accountsservice acl avahi-daemon ispell iamerican ibritish ienglish-common isc-dhcp-client isc-dhcp-common netcat netcat-traditional ntpdate os-prober parted rsync curl task-english tcpdump tcpd dbus dc console-common console-data console-setup console-setup-linux mlocate wamerican install-info installation-report bc binutils debian-faq doc-debian dosfstools ed eject hdparm heirloom-mailx xauth usbutils aptitude aptitude-common libboost-iostreams1.49.0 libept1.4.12 libsigc++-2.0-0c2a libsqlite3-0 libxapian22 debconf-i18n libtext-iconv-perl wget whois xz-utils time unzip texinfo arping gettext gettext-base manpages man-db

	apt-get autoremove

	dpkg

	require "cjson"

	-- Generic decoder for Canary JSON data. This will extract all JSON
	-- keys and add them to the `Fields` variable of the created
	-- Heka message.
	--
	-- Example use:
	--
	-- [CanaryDecoder]
	-- type = "SandboxDecoder"