Installing and configuring centralized syslog-ng

[8/12/2009 04:54:00 PM

Installing and configuring centralized syslog-ng

By: Gary Tay, gary_tay@platts.com

Updated: 8-Jul-2003

Purpose:

This document describes the steps involved in setting up syslog-ng, a replacement of UNIX/Linux syslog, system message logging system, on a centralized syslog-ng server, such that the syslog messages get logged into

1. Text files under /var/log/$HOST

2. MySQL database syslog.logs table

With the help of Apache/PHP and php-syslog-ng, these messages could be browsed or queried using Web Interface.

Freeware tools used:

· syslog-ng 1.6RC1- a replacement for UNIX/Linux syslog daemon, http://www.balabit.com/products/syslog_ng/

· Apache 2.0.46 - http://www.apache.org

· MySQL 4.0.13 - http://www.mysql.com

· PHP 4.3.3RC2 - http://www.php.net

· php-syslog-ng 1.3 - http://www.vermeer.org/projects/php-syslog-ng

· Instructions and scripts from http://www.vermeer.org

· Other information: see http://www.loganalysis.org

Testing Environment: currently “Proof of Concept” in development environment

Central Server:

nywplbb2.platts.mcgraw-hill.com

Monitored Clients:

From nyppldev10.platts.mcgraw-hill.com

To nyppldev19.platts.mcgraw-hill.com

Step 1: Install and configure syslog-ng at central server

- ftp syslog-ng-1.6.0rc1.tar to /var/tmp

- extract source files, configure and compile syslog-ng

- create a UNIX “pipe” file in /tmp for MySQL database row insertion

# mkfifo /tmp/mysql.pipe

- create /usr/local/etc/syslog-ng/syslog-ng.conf using Appendix 1.1

- note the key statement here:

# Log syslog-ng to mysql database

destination d_mysql {

pipe("/tmp/mysql.pipe" template("INSERT INTO logs (host, facility, priority, level, tag, date, time, program, msg) VALUES ( '$HOST', '$FACILITY', '$PRIORITY', '$LEVEL', '$TAG', '$YEAR-$MONTH-$DAY', '$HOUR:$MIN:$SEC', '$PROGRAM', '$MSG' );\n") template-escape(yes));

};

- disable current syslog server

# cd /etc/rc2.d

# mv S74syslog s74syslog

# /etc/init.d/syslog stop

- create /etc/init.d/syslog-ng using Appendix 1.2 and set permission

# chmod 740 /etc/init.d/syslog-ng

- enable new syslog-ng server

# cd /etc/rc2.d

# ln –s /etc/init.d/syslog-ng S74syslog-ng

# /etc/init.d/syslog-ng start

- for Solaris 9, modify /etc/logadm.conf for log files rotation, using Appendix 1.3

Step 2: Install and configure syslog-ng at monitored clients

- ftp syslog-ng-1.6.0rc1.tar to /var/tmp

- extract source files, configure and compile syslog-ng

- create /usr/local/etc/syslog-ng/syslog-ng.conf using Appendix 2.1

- disable current syslog server

# cd /etc/rc2.d

# mv S74syslog s74syslog

# /etc/init.d/syslog stop

- create /etc/init.d/syslog-ng using Appendix 1.2 and set permission

# chmod 740 /etc/init.d/syslog-ng

- enable new syslog-ng server

# cd /etc/rc2.d

# ln –s /etc/init.d/syslog-ng S74syslog-ng

# /etc/init.d/syslog-ng start

- for Solaris 9, modify /etc/logadm.conf for log files rotation, using Appendix 1.3

Step 3: Install and configure Apache, MySQL & PHP at Central Server

- ftp httpd-2.0.46.tar to /var/tmp

- extract source file, configure and compile Apache with DSO support

# ./configure –enable-so

# make

# make install

- change Apache files owner:group to nobody:nobody

# cd /usr/local

# chown –R nobody:nobody apache2

- change Apache startup owner:group and create a Virtual Server with Virtual Directory (Alias) /syslog pointing to /var/log/

edit /usr/local/apache2/conf/httpd.conf, add/modify the following lines

User nobody

#Group #-1

Group nobody

ServerAdmin gary_tay@platts.com

DocumentRoot /usr/local/apache2/htdocs

ServerName nywplbb2.platts.mcgraw-hill.com

ErrorLog logs/nywplbb2-error_log

CustomLog logs/nywplbb2-access_log common

Alias /syslog "/var/log/"

Options Indexes FollowSymLinks MultiViews IncludesNoExec

AddOutputFilter Includes html

AllowOverride None

Order allow,deny

Allow from all

- disable Solaris 9 built-in Apache (1.3.X) server

# cd /etc/rc3.d

# mv S50apache s50apache

# /etc/init.d/apache stop

- create /etc/init.d/apache2 using Appendix 3.1 and set permission

# chmod 740 /etc/init.d/apache2

- enable new Apache server

# cd /etc/rc3.d

# ln –s /etc/init.d/apache2 S50apache2

# /etc/init.d/apache2 start

- ftp mysql-standard-4.0.13-sun-solaris2.8-sparc.tar to /var/tmp

- extract source file, configure and compile MySQL

# ./configure

# make

# make install

- run “mysql_install_db” to initialize data directory and create grant tables

# cd /usr/local/mysql

# ./scripts/mysql_install_db

- change MySQL program + data files owner:group to nobody:nobody or mysql:mysql if you have created the account and group

# cd /usr/local

# chown –R nobody:nobody mysql

- install Perl DBI, DBD:mysql and CGI support

# perl –MCPAN –e shell

cpan> install DBI

cpan> install DBD:mysql

cpan> install CGI

- create /etc/init.d/mysql.server using sample from source files

# cp /usr/local/mysql/support-files/mysql.server /etc/init.d

# chmod 740 /etc/init.d/mysql.server

- edit /etc/my.cnf, change startup owner from “root” to “nobody”

[mysqld]

user = nobody

- enable new MySQL server

# /etc/init.d/mysql.server start

- ftp php4-STABLE-latest.tar (4.3.3RC2) to /var/tmp

- extract source files, configure and compile PHP with MySQL and Apache APXS2 support

# ./configure –with-mysql=/usr/local/mysql \

–with-apxs2=/usr/local/apache2/bin/apxs2

# make

# make install

- copy initial php.ini

# cp php.ini-dist /usr/local/lib/php.ini

- edit /usr/local/lib/php.ini and turn on Global Variable support (required by php-syslog-ng)

;register_globals = Off

register_globals = On

- edit /usr/local/apache2/conf/httpd.conf, add/modify PHP module support

LoadModule php4_module modules/libphp4.so

AddType application/x-httpd-php .php

AddType application/x-httpd-php .phtml

DirectoryIndex index.html index.html.var index.cgi index.pl index.php index.phtml

Also add/modify CGI support

AllowOverride None

Options None

Order allow,deny

Allow from all

AddHandler cgi-script .cgi

AddHandler cgi-script .pl

- restart Apache

# /etc/init.d/apache2 stop

# /etc/init.d/apache2 start

Step 4: Create syslog DB, logs table and syslog user at Central Server

- create 2x .sql and 2x .sh scripts

Content of create_syslog_table.sql:

CREATE DATABASE syslog;

USE syslog;

CREATE TABLE logs (

host varchar(32) default NULL,

facility varchar(10) default NULL,

priority varchar(10) default NULL,

level varchar(10) default NULL,

tag varchar(10) default NULL,

date date default NULL,

time time default NULL,

program varchar(15) default NULL,

msg text,

seq int(10) unsigned NOT NULL auto_increment,

PRIMARY KEY (seq),

KEY host (host),

KEY seq (seq),

KEY program (program),

KEY time (time),

KEY date (date),

KEY priority (priority),

KEY facility (facility)

) TYPE=MyISAM;

Content of create_syslog_user.sh:

/usr/local/mysql/bin/mysql -u root -p

Content of create_syslog_user.sql:

GRANT ALL ON syslog.* TO syslog@localhost IDENTIFIED BY 'secret';

Content of create_syslog_user.sh:

/usr/local/mysql/bin/mysql -u root -p

- run “/usr/local/etc/syslog-ng/create_syslog_table.sh”

- run “/usr/local/etc/syslog-ng/create_syslog_user.sh”

Step 5: Create and run syslog piping script in background

- create pipe_syslog_table.sh

Content of pipe_syslog_table.sh

#!/usr/bin/sh

#

# pipe_syslog_table.sh

#

# run this script in background mode

# i.e. nohup ./pipe_syslog_table.sh & >/dev/null 2>/dev/null

#

PATH=$PATH:/usr/local/mysql/bin; export PATH

if [ -p /tmp/mysql.pipe ]; then

while [ -p /tmp/mysql.pipe ]

do

mysql -u syslog --password=secret syslog

done

else

mkfifo /tmp/mysql.pipe

fi

- run “/usr/local/etc/pipe_syslog_table.sh” using “nohup” and background mode

# nohup /usr/local/etc/pipe_syslog_table.sh &

Step 6: Install and configure php-syslog-ng at Central Server

- ftp php-syslog-ng-1.3.tar to /var/tmp

- extract source files, move and rename it to /usr/local/apache2/htdocs

# mv /var/tmp/php-syslog-ng-1.3 /usr/local/apache2/htdocs/php-syslog-ng

- move images files one level up

# cd /usr/local/apache2/htdocs/php-syslog-ng

# mv images ..

- edit /usr/local/apache2/htdocs/php-syslog-ng/dbinfo.inc.php and add/modify password of MySQL “syslog” user

//$password = "db_password";

$password = "secret";

- optionally modify /usr/local/apache2/htdocs/php-syslog-ng/index.php and /usr/local/apache2/htdocs/php-syslog-ng/results.php to display your company logo file

- a version of results.php with color coded Priority Level display is shown in Appendix 6.1

Step 7: Restart syslog-ng at Central Server and verify logging

- perform “select” or “count” statement to verify if syslog messages are logged

# /usr/local/mysql/bin/mysql -u syslog -p

Enter password:

Welcome to the MySQL monitor. Commands end with ; or \g.

Your MySQL connection id is 191 to server version: 4.0.13-standard

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

mysql> use syslog

Database changed

mysql> select count(*) from logs;

+----------+

| count(*) |

+----------+

| 11484 |

+----------+

1 row in set (0.03 sec)

mysql>

- access the syslog-ng URLs:

Text logging to /var/log/$HOST

http://nywplbb2.platts.mcgraw-hill.com/syslog

or

MySQL logging to syslog.logs table

http://nywplbb2.platts.mcgraw-hill.com/php-syslog-ng

Appendix 1.1: /usr/local/etc/syslog-ng/syslog-ng.conf

(for Central Server)

# Severity levels: from less to more

# debug -> info -> notice -> warn -> err -> crit -> alert -> emerg

options { sync(0); };

#Un-comment the next line for Linux box

#source src_local { unix-stream("/dev/log"); internal(); };

#Un-comment the next line for Solaris box

source src { sun-streams("/dev/log" door("/etc/.syslog_door")); internal()

; };

# Define source for remote logging

source net { udp(ip(0.0.0.0) port(514));};

# Define destination for local logging

destination console { usertty("root"); };

destination messages { file("/var/adm/messages"); };

destination syslog { file("/var/log/syslog"); };

destination authlog { file("/var/log/auth.log"); };

destination cronlog { file("/var/log/cron.log"); };

destination daemonlog { file("/var/log/daemon.log"); };

destination kernlog { file("/var/log/kern.log"); };

destination maillog { file("/var/log/mail.log"); };

destination userlog { file("/var/log/user.log"); };

# Define destination for remote logging

destination net_messages {

file("/var/adm/$HOST/messages" owner(root) group(nobody) perm(0640) dir_perm(0750) create_dirs(yes));

};

destination net_syslog {

file("/var/log/$HOST/syslog" owner(root) group(nobody) perm(0640) dir_perm(0750) create_dirs(yes));

};

destination net_authlog {

file("/var/log/$HOST/auth.log" owner(root) group(nobody) perm(0640) dir_perm(0750) create_dirs(yes));

};

destination net_cronlog {

file("/var/log/$HOST/cron.log" owner(root) group(nobody) perm(0640) dir_perm(0750) create_dirs(yes));

};

destination net_daemonlog {

file("/var/log/$HOST/daemon.log" owner(root) group(nobody) perm(0640) dir_perm(0750) create_dirs(yes));

};

destination net_kernlog {

file("/var/log/$HOST/kern.log" owner(root) group(nobody) perm(0640) dir_perm(0750) create_dirs(yes));

};

destination net_maillog {

file("/var/log/$HOST/mail.log" owner(root) group(nobody) perm(0640) dir_perm(0750) create_dirs(yes));

};

destination net_userlog {

file("/var/log/$HOST/user.log" owner(root) group(nobody) perm(0640) dir_perm(0750) create_dirs(yes));

};

# Log syslog-ng to mysql database

destination d_mysql {

pipe("/tmp/mysql.pipe" template("INSERT INTO logs (host, facility, priority, level, tag, date, time, program, msg) VALUES ( '$HOST', '$FACILITY', '$PRIORITY', '$LEVEL', '$TAG', '$YEAR-$MONTH-$DAY', '$HOUR:$MIN:$SEC', '$PROGRAM', '$MSG' );\n") template-escape(yes));

};

# Define filters

filter f_console { level(alert..emerg); };

filter f_messages { level(err..emerg); };

filter f_syslog { level(info..warn); };

filter f_authlog { facility(auth); };

filter f_cronlog { facility(cron); };

filter f_daemonlog { facility(daemon); };

filter f_kernlog { facility(kern); };

filter f_maillog { facility(mail); };

filter f_userlog { facility(user); };

# Local logging

log { source(src); filter(f_console); destination(console); };

log { source(src); filter(f_messages); destination(messages); };

log { source(src); filter(f_syslog); destination(syslog); };

log { source(src); filter(f_authlog); destination(authlog); };

log { source(src); filter(f_cronlog); destination(cronlog); };

log { source(src); filter(f_daemonlog); destination(daemonlog); };

log { source(src); filter(f_kernlog); destination(kernlog); };

log { source(src); filter(f_maillog); destination(maillog); };

log { source(src); filter(f_userlog); destination(userlog); };

# Remote logging

log { source(net); filter(f_messages); destination(net_messages); };

log { source(net); filter(f_syslog); destination(net_syslog); };

log { source(net); filter(f_authlog); destination(net_authlog); };

log { source(net); filter(f_cronlog); destination(net_cronlog); };

log { source(net); filter(f_daemonlog); destination(net_daemonlog); };

log { source(net); filter(f_kernlog); destination(net_kernlog); };

log { source(net); filter(f_maillog); destination(net_maillog); };

log { source(net); filter(f_userlog); destination(net_userlog); };

# Remote logging into MySQL

log { source(net); destination(d_mysql); };

Appendix 1.2: /etc/init.d/syslog-ng

(same for central server and all monitored clients)

#!/sbin/sh

# syslog-ng

case "$1" in

'start')

if [ -f /usr/local/etc/syslog-ng/syslog-ng.conf -a -f /usr/local/sbin/syslog-ng ]; then

echo 'syslog-ng service starting.'

#

# Before syslog-ng starts, save any messages from previous

# crash dumps so that messages appear in chronological order.

#

/usr/bin/savecore -m

if [ -r /etc/dumpadm.conf ]; then

. /etc/dumpadm.conf

[ -n "$DUMPADM_DEVICE" -a \

"x$DUMPADM_DEVICE" != xswap ] && \

/usr/bin/savecore -m -f $DUMPADM_DEVICE

fi

/usr/local/sbin/syslog-ng

fi

;;

'stop')

if [ -f /var/run/syslog-ng.pid ]; then

syspid=`/usr/bin/cat /var/run/syslog-ng.pid`

[ "$syspid" -gt 0 ] && kill -15 $syspid

fi

;;

*)

echo "Usage: $0 { start | stop }"

exit 1

;;

esac

Appendix 1.3: /etc/logadm.conf

(same for central server and monitored client)

# gtay, 25-Apr-2003, syslog-ng implementation, disabled next 2 lines

#/var/log/syslog -C 8 -P 'Sat Apr 5 08:10:00 2003' -a 'kill -HUP `cat /var/run/syslog.pid`'

#/var/adm/messages -C 4 -P 'Wed Apr 9 07:10:00 2003' -a 'kill -HUP `cat /var/run/syslog.pid`'

# gtay, 25-Apr-2003, syslog-ng implementation, added next 8 lines

/var/log/syslog -C 8 -P 'Sat Jul 5 07:10:04 2003' -a 'kill -HUP `cat /var/run/syslog-ng.pid`'

/var/adm/messages -C 4 -P 'Tue Jul 1 07:10:02 2003' -a 'kill -HUP `cat /var/run/syslog-ng.pid`'

/var/log/auth.log -C 4 -P 'Tue Jul 1 07:10:02 2003' -a 'kill -HUP `cat /var/run/syslog-ng.pid`'

/var/log/cron.log -C 4 -P 'Mon Apr 21 09:37:16 2003' -a 'kill -HUP `cat /var/run/syslog-ng.pid`'

/var/log/daemon.log -C 4 -P 'Tue Jul 1 07:10:02 2003' -a 'kill -HUP `cat /var/run/syslog-ng.pid`'

/var/log/kern.log -C 4 -P 'Fri Jul 4 07:10:04 2003' -a 'kill -HUP `cat /var/run/syslog-ng.pid`'

/var/log/mail.log -C 4 -P 'Tue Jul 1 07:10:02 2003' -a 'kill -HUP `cat /var/run/syslog-ng.pid`'

/var/log/user.log -C 4 -P 'Thu Jul 3 07:10:02 2003' -a 'kill -HUP `cat /var/run/syslog-ng.pid`'

Appendix 2.1: /usr/local/etc/syslog-ng/syslog-ng.conf

(for all Monitored Clients)

# Severity levels: from less to more

# debug -> info -> notice -> warn -> err -> crit -> alert -> emerg

options { sync(0); };

#Un-comment the next line for Linux box

#source src_local { unix-stream("/dev/log"); internal(); };

#Un-comment the next line for Solaris box

source src { sun-streams("/dev/log" door("/etc/.syslog_door")); internal()

; };

# Define destination for remote logging

destination loghost { udp("nywplbb2.platts.mcgraw-hill.com" port(514));};