apache monitoring with grafana.md

1. CentOS7 docker 설치

설치 되었다면 삭제

옛 버전 삭제

$ sudo yum remove docker \
           docker-common \
           docker-selinux \
           docker-engine

필수 패키지 설치

$ sudo yum install -y yum-utils device-mapper-persistent-data lvm2

# docker repo를 등록한다.
$ sudo yum-config-manager \
    --add-repo \
    https://download.docker.com/linux/centos/docker-ce.repo

# yum package를 업데이트 
$ sudo yum makecache fast

docker 설치

$ sudo yum install docker-ce

# 설치확인 
$ docker --version
  Docker version 19.03.10, build 9424aeaee9

Docker-compose 설치

$ curl -L https://github.com/docker/compose/releases/download/1.14.0/docker-compose-`uname -s`-`uname -m` > /usr/local/bin/docker-compose
$ chmod +x /usr/local/bin/docker-compose

# 설치 확인
$ docker-compose --version

2. Grafa 설치 (docker)

[다운로드] git clone https://github.com/stefanprodan/dockprom

run

ADMIN_USER=admin ADMIN_PASSWORD=admin docker-compose up -d

stop

docker-compose down

[연결] http://host:3000

3. Apache Exporter Target to Prometheus

Prometheus로 apache의 server-status를 읽어서 보내주는 데몬을 설치한다

일반적으로 apache가 실행되는 서버에 설치해서 운용한다. 뒷단의 톰캣을 모니터링하는 방법과 달리 전면의 apache를 모니터링 한다.

먼저 server-status 활성화 조치가 필요하다. 설정을 수정후 재시작

$ sudo vi /etc/httpd/conf.d/server-status.conf

ExtendedStatus on
<Location /server-status>
    SetHandler server-status
    Order deny,allow
    Deny from all
    Allow from 127.0.0.1
</Location>

# Restart Apache:
$ sudo systemctl restart httpd

# Check the server-status page:
$ curl http://127.0.0.1/server-status

Install Go and Git

export를 운용하려면 GoLang설치가 필요하다

$ curl -O https://dl.google.com/go/go1.11.linux-amd64.tar.gz
$ sudo tar -C /usr/local -xvzf go1.11.linux-amd64.tar.gz
$ echo "export GOPATH=$HOME/go" >> ~/.profile
$ echo "export PATH=$PATH:/usr/local/go/bin" >> ~/.profile
$ source ~/.profile
$ go version
go version go1.11 linux/amd64

Install Git:

$ sudo yum install git

Apache Exporter: Install and Config

Create unprivileged apache_exporter user:

$ sudo useradd apache_exporter -s /sbin/nologin

Download and compile Apache Exporter:

$ go get -v github.com/Lusitaniae/apache_exporter
$ sudo cp ~/go/bin/apache_exporter /usr/sbin/

Create a systemd service file /etc/systemd/system/apache_exporter.service:

[Unit]
Description=Apache Exporter

[Service]
User=apache_exporter
EnvironmentFile=/etc/sysconfig/apache_exporter
ExecStart=/usr/sbin/apache_exporter $OPTIONS

[Install]
WantedBy=multi-user.target

Create a sysconfig file:

$ sudo mkdir -p /etc/sysconfig
$ sudo touch /etc/sysconfig/apache_exporter

Use this file to define options to provide, for example:

vi /etc/sysconfig/apache_exporter

OPTIONS="--scrape_uri='http://127.0.0.1/server-status/?auto'"

To check the all available options, run:

$ /usr/sbin/apache_exporter --help

Reload systemd configuration and configure apache_exporter to auto-start on system boot:

$ sudo systemctl daemon-reload
$ sudo systemctl enable apache_exporter

Start the apache_exporter service:

$ sudo systemctl start apache_exporter

Once the Apache Exporter is installed, verify that metrics are being exported:

$ curl http://localhost:9117/metrics

Add Apache Exporter Target to Prometheus

scrape_configs:
  - job_name: 'apache'
    static_configs:
      - targets: ['$NODE_IP:9117']
Reload the Prometheus config, and you should now be able to access Apache metrics through Prometheus!

docker환경이 있어서 사용해 봤지만 호환이 안되었다. exporter는 apache가 실행되는 서버에 설치 했다.

4. Tomcat 모니터링

JVM 모니터링을 위한 jmx exporter를 설치한다.

추가로 방화벽을 열어햐 하는점도 고려가 되어야 한다.

JMX Exporter 설치

[참조] https://github.com/prometheus/jmx_exporter

Javaagent download 및 config

$ mkdir jmx_exporter
$ wget https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.12.0/jmx_prometheus_javaagent-0.12.0.jar
$ touch config.yaml

$ vim config.yaml
---
startDelaySeconds: 0
ssl: false
lowercaseOutputName: false
lowercaseOutputLabelNames: false

적용하기

$ vim catalina.sh
CATALINA_OPTS="$CATALINA_OPTS -javaagent:/home/ec2-user/jmx_exporter/jmx_prometheus_javaagent-0.12.0.jar=8181:/home/ec2-user/jmx_exporter/config.yaml"

Prometheus 서버

prometheus.yml 파일에 jmx exporter 설정

# jmx_exporter
- job_name: 'jmx_exporter'
static_configs:
- targets: ['10.27.0.53:8181', '10.27.10.41:8181']

Grafana Dashboard

아래의 dashboard json 파일을 다운로드하여 Import 하였다.

환경에 맞게 Variables 을 수정하여 사용하면 된다.

mc3767/apache monitoring with grafana.md