DESCRIPTION

logstat.pl and logtop.pl

These scripts show current and average request-per-second counts based on apache access logs in real-time.

You can see the total requests-per-second as well as a breakdown by:
* vhost
* URL
* user agent
* or referrer.

You can sort by:
* number of requests in the last second
* time of last stats update
* average requests-per-second over the last one, five or fifteen minutes

The output shows:
* the date of the last stats update
* the last second's hits per second (hps)
* the one, five and fifteen minute hps average
* the seconds since last request
* an asterisk to mark requests hit in the last 5 seconds
* and the request's vhost, URL, user agent or referrer.

See also the apachetop tool - http://freshmeat.net/projects/apachetop/ and
http://packages.debian.org/search?keywords=apachetop . This script is not based on or intended to
mimic apachetop, but the functionality is broadly similar.

SYNOPSIS

To use:

In one screen, run the log parser. It writes out a stats file which the other scripts use.

$ cd /dev/shm # use shared memory
$ tail -F /var/log/httpd/access.log | ~/logstat.pl # centralized logging helps here

In another screen, view your stats:

$ cd /dev/shm # use shared memory

# defaults to vhosts sorted by five minute average
$ watch -n 5 -- ~/logtop.pl

or

# URLs sorted by five minute average, ignoring images, JS and CSS
$ watch -n 5 -- '~/logtop.pl urls \
  --skip '\''\.(jpg|png|gif|js|css)$|^\S+\s(/robots\.txt$|/favicon\.ico$)'\'''

or

# user agents sorted by requests in the last second
$ watch -n 5 -- ~/logtop.pl ua --sort hps

or

# referrers sorted by one minute average
$ watch -n 5 -- ~/logtop.pl referrer --sort one

or

# URLs sorted by last stats update
$ watch -n 5 -- ~/logtop.pl urls --sort time

etc.

EXAMPLE

Every 5.0s: ~/logtop.pl                                                                                                Fri Jun 24 03:31:59 2011

Fri Jun 24 03:31:58 2011 hps:      2,  average: 5.35, 4.45, 2.37,   0 seconds ago,  total
vhosts sorted by five
Fri Jun 24 03:31:52 2011 hps:      0,  average: 0.41, 0.59, 0.35,   6 seconds ago,     example.com
Fri Jun 24 03:31:54 2011 hps:      1,  average: 1.17, 0.58, 0.23,   4 seconds ago,  *  example.net
Fri Jun 24 03:31:34 2011 hps:      2,  average: 0.99, 0.33, 0.12,  24 seconds ago,     example.org
...

NOTES

* The script's tally sub is properly abstracted so this script could be quite easily modified to tally
and report averages on anything you can count (not just httpd logs)

* The log parsing regex matches the "v-combined" log format (combined with the virtual host at the front)

* The logtop script accepts "match" and/or "skip" arguments to only show events that match (or do
not match) a given regex.

TODO

* Persist historical data and reload on restart (this can be done asynchronously with another client script)

* Persist position in log file (using ftell) to recover from where we left off on restart (this should
account for log file rotation)