Installing Graphite:
-------------------

**Graphite does two things**:

1. Store numeric time-series data
2. Render graphs of this data on demand

What Graphite does not do is collect data for you, however there are some
[tools](https://graphite.readthedocs.org/en/1.0/tools.html) out there that know
how to send data to graphite. Even though it often requires a little code,
[sending data](https://graphite.readthedocs.org/en/1.0/feeding-carbon.html) to
Graphite is very simple.

Architecture:
============

Graphite consists of 3 software components:

* carbon - a Twisted daemon that listens for time-series data
* whisper - a simple database library for storing time-series data (similar in design to RRD)
* graphite webapp - A Django webapp that renders graphs on-demand using Cairo

Installing:
==========

**Install Required repo**:

Redhat:

```
curl -o epel.rpm -L http://download.fedoraproject.org/pub/epel/6/$(arch)/epel-release-6-8.noarch.rpm
rpm -ivh epel.rpm


yum install pycairo Django14 python-ldap python-memcached python-sqlite2 bitmap bitmap-fonts-compat \
python-devel python-crypto pyOpenSSL gcc python-zope-filesystem python-zope-interface git gcc-c++ \
zlib-static MySQL-python python-txamqp python-setuptools python-psycopg2 mod_wsgi
```

Ubuntu:

```
apt-get install python-cairo python-twisted python-django python-django-tagging python-ldap \
python-memcache python-sqlite python-simplejson python-txamqp ca-certificates
```

Download and extract packages:

```
carbon_version="0.9.12"
whisper_version="0.9.12"
graphite_web_version="0.9.12"
build_dir="/opt"
cd ${build_dir}
curl -s -L https://github.com/graphite-project/carbon/archive/${carbon_version}.tar.gz | tar xz
curl -s -L https://github.com/graphite-project/whisper/archive/${whisper_version}.tar.gz | tar xz
curl -s -L https://github.com/graphite-project/graphite-web/archive/${graphite_web_version}.tar.gz | tar xz
easy_install django-tagging==0.3.1
easy_install twisted==11.1.0
easy_install txamqp==0.4
```

Install webapp (this requires djando-tagging)

```
cd /opt/graphite-web-0.9.12
python setup.py install # subscribe to untar webapp
```

Install carbon (this requires twisted)

```
cd /opt/carbon-0.9.12
python setup.py install
```

Install whisper (this requires twisted)

```
cd /opt/whisper-0.9.12
python setup.py install
```

Configuring:
===========

Install postgres as a backend for graphite:

```
yum -y install postgresql postgresql-server postgresql-devel

1. Yum Install PG9.4
 2. wget http://yum.postgresql.org/9.4/redhat/rhel-6-x86_64/pgdg-redhat94-9.4-1.noarch.rpm
 3. yum install pgdg-redhat94-9.4-1.noarch.rpm
 4. yum install postgresql94-server
 5. service postgresql-9.4 initdb
 6. chkconfig postgresql-9.4 on
```

Configure postgres to allow graphite user:

```
service postgresql initdb
grep "listen_addresses = '0.0.0.0'" /var/lib/pgsql/data/postgresql.conf
if [[ $? -ne 0 ]]; then
  echo "listen_addresses = '0.0.0.0'" >> /var/lib/pgsql/data/postgresql.conf
fi
service postgresql start
sudo -u postgres psql template1 <<END
create user graphite with password 'graphite';
create database graphite with owner graphite;
END
```

Configure graphite settings:

```
cat > /opt/graphite/webapp/graphite/local_settings.py <<EOF
SECRET_KEY = 'UNSAFE_DEFAULT'

TIME_ZONE = 'GMT'

#DOCUMENTATION_URL = "http://graphite.readthedocs.org/"

# Logging
#LOG_RENDERING_PERFORMANCE = True
#LOG_CACHE_PERFORMANCE = True
#LOG_METRIC_ACCESS = True

# Enable full debug page display on exceptions (Internal Server Error pages)
#DEBUG = True

# If using RRD files and rrdcached, set to the address or socket of the daemon
#FLUSHRRDCACHED = 'unix:/var/run/rrdcached.sock'

# Change only GRAPHITE_ROOT if your install is merely shifted from /opt/graphite
# to somewhere else
#GRAPHITE_ROOT = '/opt/graphite'

# Most installs done outside of a separate tree such as /opt/graphite will only
# need to change these three settings. Note that the default settings for each
# of these is relative to GRAPHITE_ROOT
#CONF_DIR = '/opt/graphite/conf'
#STORAGE_DIR = '/opt/graphite/storage'
#CONTENT_DIR = '/opt/graphite/webapp/content'

# To further or fully customize the paths, modify the following. Note that the
# default settings for each of these are relative to CONF_DIR and STORAGE_DIR
#
## Webapp config files
#DASHBOARD_CONF = '/opt/graphite/conf/dashboard.conf'
#GRAPHTEMPLATES_CONF = '/opt/graphite/conf/graphTemplates.conf'

## Data directories
# NOTE: If any directory is unreadable in DATA_DIRS it will break metric browsing
#WHISPER_DIR = '/opt/graphite/storage/whisper'
#RRD_DIR = '/opt/graphite/storage/rrd'
#DATA_DIRS = [WHISPER_DIR, RRD_DIR] # Default: set from the above variables
#LOG_DIR = '/opt/graphite/storage/log/webapp'
#INDEX_FILE = '/opt/graphite/storage/index'  # Search index file


DATABASES = {
   'default': {
        'ENGINE': 'django.db.backends.postgresql_psycopg2',
        'NAME': 'graphite',
        'USER': 'graphite',
        'PASSWORD': 'graphite',
        'HOST': 'localhost',
        'PORT': '5432',
    }
}
EOF
```

Modify PostgreSQL "Client Authentication" -- use md5:

```
cat > /var/lib/pgsql/data/pg_hba.conf <<EOM
# "local" is for Unix domain socket connections only
local   all         all                               md5
# IPv4 local connections:
host    all         all         127.0.0.1/32          md5
# IPv6 local connections:
host    all         all         ::1/128               md5
EOM
```

Restart PostrgreSQL to take effect:

```
service postgresql restart
```

Initialize Django DB creation (requires configuring /opt/graphite/webapp/graphite/local_settings.py):

```
cd /opt/graphite/webapp/graphite
python manage.py syncdb --noinput
```

Change the permission for webapp storage to httpd user

```
chown -R apache:apache /opt/graphite/storage/
chcon -R -h -t httpd_sys_content_t /opt/graphite/storage
```

Configure graphite web:

```
cat > /opt/graphite/conf/graphite.wsgi <<EOF
import os, sys
sys.path.append('/opt/graphite/webapp')
os.environ['DJANGO_SETTINGS_MODULE'] = 'graphite.settings'

import django.core.handlers.wsgi

application = django.core.handlers.wsgi.WSGIHandler()

# READ THIS
# Initializing the search index can be very expensive, please include
# the WSGIImportScript directive pointing to this script in your vhost
# config to ensure the index is preloaded before any requests are handed
# to the process.
from graphite.logger import log
log.info("graphite.wsgi - pid %d - reloading search index" % os.getpid())
import graphite.metrics.search
EOF

# Configuring whisper

cat > /opt/graphite/conf/storage-schemas.conf <<EOF
# Schema definitions for whisper files. Entries are scanned in order,
# and first match wins.
[default]
pattern = .*
retentions = 1s:30m,1m:1d,5m:2y
EOF

cat > /opt/graphite/conf/carbon.conf <<EOF
[cache]
ENABLE_LOGROTATION = True
USER =
MAX_CACHE_SIZE = inf
MAX_UPDATES_PER_SECOND = 500
MAX_CREATES_PER_MINUTE = 50
LINE_RECEIVER_INTERFACE = 0.0.0.0
LINE_RECEIVER_PORT = 2003
ENABLE_UDP_LISTENER = False
UDP_RECEIVER_INTERFACE = 0.0.0.0
UDP_RECEIVER_PORT = 2003
PICKLE_RECEIVER_INTERFACE = 0.0.0.0
PICKLE_RECEIVER_PORT = 2004
LOG_LISTENER_CONNECTIONS = True
USE_INSECURE_UNPICKLER = False
CACHE_QUERY_INTERFACE = 0.0.0.0
CACHE_QUERY_PORT = 7002
USE_FLOW_CONTROL = True
LOG_UPDATES = False
LOG_CACHE_HITS = False
LOG_CACHE_QUEUE_SORTS = True
CACHE_WRITE_STRATEGY = sorted
WHISPER_AUTOFLUSH = False
WHISPER_FALLOCATE_CREATE = True
CARBON_METRIC_INTERVAL = 60
ENABLE_AMQP = False

[relay]
ENABLE_LOGROTATION = True
USER =
LINE_RECEIVER_INTERFACE = 0.0.0.0
LINE_RECEIVER_PORT = 2013
PICKLE_RECEIVER_INTERFACE = 0.0.0.0
PICKLE_RECEIVER_PORT = 2014
LOG_LISTENER_CONNECTIONS = True
RELAY_METHOD = rules
REPLICATION_FACTOR = 1
DESTINATIONS = 127.0.0.1:2004
MAX_DATAPOINTS_PER_MESSAGE = 500
MAX_QUEUE_SIZE = 10000
USE_FLOW_CONTROL = True
CARBON_METRIC_INTERVAL = 60

[aggregator]
ENABLE_LOGROTATION = True
USER =
LINE_RECEIVER_INTERFACE = 0.0.0.0
LINE_RECEIVER_PORT = 2023
PICKLE_RECEIVER_INTERFACE = 0.0.0.0
PICKLE_RECEIVER_PORT = 2024
LOG_LISTENER_CONNECTIONS = True
FORWARD_ALL = True
DESTINATIONS = 127.0.0.1:2004
REPLICATION_FACTOR = 1
MAX_QUEUE_SIZE = 10000
USE_FLOW_CONTROL = True
MAX_DATAPOINTS_PER_MESSAGE = 500
MAX_AGGREGATION_INTERVALS = 5
CARBON_METRIC_INTERVAL = 60
EOF

cat > /opt/graphite/conf/storage-aggregation.conf <<EOF
[min]
pattern = \.min$
xFilesFactor = 0.1
aggregationMethod = min

[max]
pattern = \.max$
xFilesFactor = 0.1
aggregationMethod = max

[sum]
pattern = \.count$
xFilesFactor = 0
aggregationMethod = sum

[default_average]
pattern = .*
xFilesFactor = 0.5
aggregationMethod = average
EOF
```

On some systems, `hostname -f` can not work due to incorrect `/etc/hosts` entry. Fix:
```
if ! $(hostname -f > 2&>1 /dev/null); then
  cat > /etc/hosts <<EOF
127.0.0.1   $(hostname) localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
EOF
fi
```

Configuring `httpd`:

```
cat > /etc/httpd/conf.d/graphite.conf <<EOF
<IfModule !wsgi_module.c>
    LoadModule wsgi_module modules/mod_wsgi.so
</IfModule>

WSGISocketPrefix run/wsgi

<VirtualHost *:80>
        ServerName $(hostname -f)
        DocumentRoot "/opt/graphite/webapp"
        ErrorLog /var/log/httpd/graphite_error.log
        CustomLog /var/log/httpd/graphite_access.log common

        WSGIDaemonProcess graphite processes=5 threads=5 display-name='%{GROUP}' inactivity-timeout=120
        WSGIProcessGroup graphite
        WSGIApplicationGroup %{GLOBAL}
        WSGIImportScript /opt/graphite/conf/graphite.wsgi process-group=graphite application-group=%{GLOBAL}

        WSGIScriptAlias / /opt/graphite/conf/graphite.wsgi

        Alias /content/ /opt/graphite/webapp/content/
        <Location "/content/">
                SetHandler None
        </Location>

        Alias /media/ "@DJANGO_ROOT@/contrib/admin/media/"
        <Location "/media/">
                SetHandler None
        </Location>

        <Directory /opt/graphite/conf/>
                Order deny,allow
                Allow from all
        </Directory>

</VirtualHost>
EOF
```

```
cat > /etc/init.d/carbon-cache <<\EOF
#!/bin/bash
#
# This is used to start/stop the carbon-cache daemon

# chkconfig: - 99 01
# description: Starts the carbon-cache daemon

# Source function library.
. /etc/init.d/functions

RETVAL=0
prog="carbon-cache"

start_relay () {
    /usr/bin/python /opt/graphite/bin/carbon-relay.py start
        RETVAL=$?
        [ $RETVAL -eq 0 ] && success || failure
        echo
        return $RETVAL
}

start_cache () {
     /usr/bin/python /opt/graphite/bin/carbon-cache.py start
        RETVAL=$?
        [ $RETVAL -eq 0 ] && success || failure
        echo
        return $RETVAL
}

stop_relay () {
    /usr/bin/python /opt/graphite/bin/carbon-relay.py stop
        RETVAL=$?
        [ $RETVAL -eq 0 ] && success || failure
        echo
        return $RETVAL
}

stop_cache () {
          /usr/bin/python /opt/graphite/bin/carbon-cache.py stop
        RETVAL=$?
        [ $RETVAL -eq 0 ] && success || failure
        echo
        return $RETVAL
}

# See how we were called.
case "$1" in
  start)
    #start_relay
    start_cache
        ;;
  stop)
    #stop_relay
    stop_cache
        ;;
  restart)
    #stop_relay
    stop_cache
    sleep 5
    #start_relay
    start_cache
    ;;

  *)
        echo $"Usage: $0 {start|stop}"
        exit 2
        ;;
esac

EOF
chmod +x /etc/init.d/carbon-cache
```

```
service carbon-cache start
service httpd start
```


Install collectd:
================

```
yum install -y rrdtool rrdtool-devel perl rrdtool-perl libgcrypt-devel gcc make gcc-c++ perl-ExtUtils-CBuilder perl-ExtUtils-MakeMaker perl-ExtUtils-Embed
cd /opt
curl -s -L http://collectd.org/files/collectd-5.4.0.tar.bz2 | tar jx
cd collectd-5.4.0
./configure --prefix=/usr --sysconfdir=/etc --localstatedir=/var --libdir=/usr/lib --mandir=/usr/share/man --enable-cpu --enable-curl --enable-df --enable-exec --enable-load --enable-logfile --enable-memory --enable-network --enable-nginx --enable-syslog  --enable-rrdtool --enable-uptime --enable-write_graphite
make
make install
cp contrib/redhat/init.d-collectd /etc/init.d/collectd
chmod 755 /etc/init.d/collectd
chown root:root /etc/init.d/collectd
/etc/init.d/collectd start
chkconfig collectd on
```
Creation of collectd packages so that , if you have internal repos you could install it via package manager.

```
# Install tools required to build the rpms
yum install -y rpm-build yum-utils rpmdevtools

# Install EPEL
rpm -ivh http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm

# Go to /tmp to get and extract collectd source base
cd /tmp
wget -c http://collectd.org/files/collectd-5.5.0.tar.bz2

# Extract source
tar xvf collectd-5.5.0.tar.bz2

# Create build directory
mkdir -p /root/rpmbuild/SOURCES/

# Copy sources in build directory
cp collectd-5.5.0.tar.bz2 /root/rpmbuild/SOURCES/

# Install dependencies of the spec file
yum-builddep -y collectd-5.5.0/contrib/redhat/collectd.spec

# Create RPMS
rpmbuild -bb collectd-5.5.0/contrib/redhat/collectd.spec

# Copy RPMs back to the host
cp -rv /root/rpmbuild/RPMS/ /root

```


Configure collectd to write system metrics to graphite by editing `/etc/collectd.conf` and adding the following lines:

```
LoadPlugin syslog

LoadPlugin "logfile"
<Plugin "logfile">
  LogLevel "info"
  File "/var/log/collectd.log"
  Timestamp true
</Plugin>

LoadPlugin cpu
LoadPlugin interface
LoadPlugin load
LoadPlugin memory
LoadPlugin rrdtool

LoadPlugin write_graphite
<Plugin "write_graphite">
 <Carbon>
   Host "127.0.0.1"
   Port "2003"
   Prefix "collectd."
   #Postfix ""
   Protocol "tcp"
   EscapeCharacter "_"
   SeparateInstances true
   StoreRates false
   AlwaysAppendDS false
 </Carbon>
</Plugin>
```

Restart collectd to start sending metrics to graphite

```
/etc/init.d/collectd restart
```

Installing Grafana 2.10
==================
Grafana is a custom dashboard for graphite.


```
sudo yum install https://grafanarel.s3.amazonaws.com/builds/grafana-2.1.0-1.x86_64.rpm
sudo service grafana-server start
```

Configure grafana with httpd:

```
cat > /etc/httpd/conf.d/grafana.conf <<EOF
<VirtualHost *:80>
  ServerName graphana.host.com
  ServerAlias grafana.host.com
  ProxyPass / http://localhost:3000/
</VirtualHost
EOF
```

Restart httpd to refresh graphite:

```
service httpd restart
```