I’m running a number of OpenVZ based Debian VPS accounts for my hosting needs and I’ve busted out into using GeoIP in BIND to direct clients of my hosted services to their nearest available server to maximize performance and add the ability to failover.

A crucial piece in this puzzle is logging. Most logging on Linux systems is done with syslogd and Debian uses sysklogd. syslogd for years has had remote logging capabilities and its very simple to setup.

First I setup some “log servers” which will receive logs from other hosts.

I edited /etc/default/syslogd so that the SYSLOGD variable was defined like:

SYSLOGD="-r -s example.com -l node1.example.com:node2.example.com:node3.example.com"

Because OpenVZ hosting providers are bodgy and not all do proper reverse DNS, I created an /etc/hosts file that accurately produced reverse DNS on the logging hosts. I actually made just one file and used rsync to keep it in sync on other “log servers”.

I also edited /etc/logrotate.conf so that two options were tweaked:

# keep 1 year (52 weeks) worth of backlogs

rotate 52

# uncomment this if you want your log files compressed

compress

So then my “log servers” were all setup. I thought with disk space so cheap in certain locations, I may as well keep lots of logs. As its centralized its easier to monitor for log over-runs.

I then went on to edit /etc/syslogd.conf on each node so that it contained:

*.*    @syslog1.example.com

*.*    @syslog2.example.com

*.*    @syslog3.example.com

With this configuration every node will log in the same files and hosts will have their short hostname printed at the start of each log entry. The data is transmitted over UDP.

By having numerous sites where logs are stored means that logging is redundant and a comprehensive log set is always available. Additionally I never disabled the local logging facilities, so each node still has its own logs in /var/log

My final step was to whinge to one provider as the clock is set by the hosting provider in OpenVZ VPS containers. One provider had a clock that was inaccurate by 67 seconds. Clearly accurate time is important in a logging application and this effectively renders that node useless until the hosting provider fixes it.

An example of the advantage of this configuration… now on one of the “log servers” when I `tail –f /var/log/mail.log` I receive the activity for every single mail server I operate.