README/FAQ for logfilterd, written by Adam Spiers ============================================================================== Introduction and motivation --------------------------- This program is a log filtering "daemon" written in Perl. I decided to write it after evaluating several (all?) the log filtering programs currently available (swatch, logsurfer, logcheck etc.) and finding that to my mind, none of them provided the full flexibility I needed. I wanted something that would: - eliminate uninteresting verbosity in /var/log/messages, thereby making it very easy to spot anything out of the normal (which I consider a Very Good Thing for a sysadmin to be able to do). Simply tweaking syslogd.conf is an unacceptable solution for many reasons, but not least because the policies adopted by different software packages for what priority levels they use when logging with syslog vastly differ. - siphon off different types of log data into different files (e.g. one log file for all potential hacking attempts, one for all ftpd transactions etc.) - rapidly alert me in certain situations (e.g. detection of port scans and other attacks) - provide me with concise summaries of certain events, even if they occurred continuously over several minutes or even hours - be pretty quick - not refilter whole input log files if you altered the filtering algorithm (unless you really wanted it to) - have the power of Perl regexps and string handling in its full glory :-) I decided to make the daemon a single script rather than an immutable code base together with a complex configuration file, and in doing so obviously gained pretty much unlimited flexibility in how I wanted my logs filtered. Of course, the downside was that anyone else who wanted to use the code would need enough programming knowledge to be able to read Perl code and alter it in simple ways. However, I deliberately wrote the code so that this would pose very little difficulty for the average programmer/Linux enthusiast. I wrote it specifically for the machine thelonious.new.ox.ac.uk, which is a RedHat Linux box running a slightly unusual combination of services; hence it will need to be adapted to your machine before you can use it. However, the structure of the program should be very clear, especially if viewed in emacs with folding mode (folding.el) loaded. (Wow, never thought I hear myself come out with a 'best viewed in ...' line.) What does it actually do? ------------------------- In addition to filtering out a lot of common log messages of little interest, it splits the output into several different files (one for each daemon), sets up 'contexts' a la logsurfer (see http://www.cert.dfn.de/eng/team/wl/logsurf/ for information about 'contexts' to help you understand more precisely what this is all about) in order to compress noisy security events (e.g. a big portscan) into a readable summary, and sends e-mail alerts such as the one above immediately upon detection of something more serious like a portscan. The current version has whittled down a 10MB /var/log/messages to a 1MB /var/log/filtered-messages, which makes for additions which can be easily read every day without consuming too much time. The patterns it matches are found towards the end of the code. Here's a sample e-mail it might (and in fact did) generate. -------- 8< -------- 8< -------- Date: 20 Jul 1998 06:59:03 -0000 From: anonymous@thelonious.new.ox.ac.uk To: adam@thelonious.new.ox.ac.uk Subject: logfilterd ALERT: port scan detection Port scan (context id #3) detected from host: pigpen.pcl.ox.ac.uk/163.1.218.50, starting at Jul 20 05:20:00, and ending at Jul 20 05:20:00. SYN/Normal scan employed on TCP ports: 53 -------- 8< -------- 8< -------- And here are some of the lines which might appear in your /var/log/filtered-messages following port scans. The number in the [] parentheses is the context number (these numbers are allocated sequentially, but reclaimed as the contexts finish). -------- 8< -------- 8< -------- May 31 19:11:18 logfilterd[sentry/3]: Detected port scan initiating from DIALIN0165.UPENN.EDU/128.91.16.165 May 31 20:15:54 logfilterd[sentry/3]: UDP scan employed on UDP ports: 137 May 31 20:15:54 logfilterd[sentry/3]: Port scan from DIALIN0165.UPENN.EDU/128.91.16.165 finished at May 31 19:11:46 Jun 1 07:24:08 logfilterd[sentry/4]: Detected port scan initiating from james.eimc.brad.ac.uk/143.53.203.241 Jun 1 11:07:58 logfilterd[sentry/4]: SYN/Normal scan employed on TCP ports: 1--20, 24, 26--78, 81--109, 112, 114--122, 124--142, 145--176, 178--194 Jun 1 11:07:58 logfilterd[sentry/4]: Port scan from james.eimc.brad.ac.uk/143.53.203.241 finished at Jun 1 07:24:52 Jun 1 11:07:58 logfilterd[sentry/5]: Detected port scan initiating from colorgfx.demon.co.uk/158.152.162.35 Jun 1 13:10:15 logfilterd[sentry/5]: UDP scan employed on UDP ports: 137 Jun 1 13:10:15 logfilterd[sentry/5]: Port scan from colorgfx.demon.co.uk/158.152.162.35 finished at Jun 1 11:08:01 Jun 1 13:10:15 logfilterd[sentry/6]: Detected port scan initiating from hail.pipex.net/158.43.128.104 Jun 1 15:10:17 logfilterd[sentry/6]: UDP scan employed on UDP ports: 7 Jun 1 15:10:17 logfilterd[sentry/6]: Port scan from hail.pipex.net/158.43.128.104 finished at Jun 1 13:11:21 -------- 8< -------- 8< -------- How do I install it? -------------------- See INSTALL for rough installation instructions. I've tried installing it but I'm stuck! What now? -------------------------------------------------- You can try e-mailing me with a very detailed and precise description of the problem, and I may help, but I spent very little time computing these days and rarely check my e-mail anyway, so you might be better off asking elsewhere ... Needless to say, any questions which show no signs of consulting Perl man pages and/or FAQs where appropriate will be redirected straight to /dev/null. Gee, that's useful, thanks. But how about changing it like this? ----------------------------------------------------------------- If you've extended/changed it in a way which may be useful to others, wonderful --- let me know (long live the Bazaar!). This applies to any suggestions you might have too, even if it's about the slightly muddled documentation. Ideally I'd like someone to take it over and develop it properly. Building an .rpm would also be an easy and worthwhile job. Any offers?