setup td-agent

about

• source: server#1 nginx_access.log
• dist: server#2 mongodb

install

• http://help.treasuredata.com/customer/portal/articles/1246904-installing-td-agent-on-rhel-and-centos

edit td-agent.conf

server#1

$ sudo vi /etc/td-agent/td-agent.conf
<source>
  type tail
  format ltsv
  time_key time_local # formatがltsvのときに必要
  path /var/log/nginx/access.log
  tag nginx.access
  pos_file /var/log/td-agent/nginx_access.pos
</source>

<match nginx.*>
  type forward
  buffer_type memory
  buffer_chunk_limit 256m
  buffer_queue_limit 128
  flush_interval 5s
  <server>
    host $host
    port $port
  </server>
</match>

server#2

<source>
  type forward
  port $port
</source>
<match nginx.access>
  type mongo
  database $db_name
  collection $collection_name
</match>

restart td-agent

// pidが残る or 2重起動するためstop, startで反映
$ sudo service td-agent stop
$ sudo service td-agent start

edit iptables

$ sudo cat /etc/sysconfig/iptables
+ -A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 24224   -j ACCEPT
+ -A RH-Firewall-1-INPUT -m state --state NEW -m udp -p udp --dport 24224   -j ACCEPT
$ sudo service iptables restart

reference

• http://blog.excale.net/index.php/archives/2682