LMF: Log Monitoring Framework By Max Schubert a.k.a. "perldork"

                                                                                     
LMF - A log monitoring framework
================================

1) Overview
2) Configuration file organization
3) Global options
4) Configuring a log file matcher
5) Installing LMF

1) Overview
===========

LMF is a flexible log monitoring framework that watches log files on
your system, matching patterns that you specify and taking actions
based on match thresholds you configure on a per-pattern basis.

LMF lets you monitor many log files with multiple patterns each at
once.  Configurations for these log file matchers are kept in 
configuration 'snippets' located in the config/ directory under the
LMF home directory or on a web server that can be accessed over HTTP
or HTTPS, with or without apache authentication.

LMF performs the equivalent of a 'tail -f' on each unique log file
specified in the log matchers you have chosen to use on your 
system.  For each log watched, a single daemon is forked, so if you
have 30 actions across 3 files, 3 separate watcher daemons will be
started, each just watching for matchers associated with the file.

2) Configuration file organization:
===================================

LMF can use local configurations or configurations stored on remote
servers.

For a local configuration, you might have:

config/
   main.conf
   apache.conf
   dns.conf
   ssh.conf

You can name your snippets however you like, just remember to use
'.conf' as the suffix for the name.

This version of LMF also supports loading its' configuration from a
remote HTTP/HTTPS source with or without HTTP AUTH protection.  This
allows you to have a number of LMF clients all using configuration
files stored and managed on a central server.  If each client is
restarted every night or every N hours via cron it would pick up
any new configurations or changed configurations on the remote server.

For more documentation and an example of how to do this, read the
README file and configuration example files in

misc/http-config.dist/

3) Global options
=================

Global options are found in config/main.conf, they are as follows:

[main]
#  Syslog level to use
syslog-level = LOCAL3
#  How often, in seconds, should each pattern be checked for in
#  the log files?  Set to a higher number to reduce the load LMF
#  puts on the system (5 should be fine for most systems), set
#  to a lower number to have it check more often.  Most people
#  will not need to change this.
interval = 5
#  Set to 1 to see verbose debugging information
debug = 0

[main-action-queue]
#  File that holds triggered actions, when they were started, and
#  program to run apon release.   lmf-released reads this file,
#  lmf-monitord appends to this file.
db = /usr/local/lmf/data/queue.db

#  Field separator for action-queue log file
field-separator = ^T^K^O

4) Configuring a log file matcher
=================================

NOTE:

Do not name your log file matchers with a prefix of "main-" as the
log file matcher parsing code skips those secions.

Each configuration has a 'name' section that is formatted like so:

[This is the name of my rule]

The name can contain any descriptive text you would like to use, for
example "SSH brute force login attempt" or "proftpd root login attempt."

Log matchers consist of a number of fields, listed under the name section,
each separated by a new line.  

Fields:
file      - Name of the file to watch for this pattern
pattern   - A perl regular expression that matches the pattern you are 
            interested in, use perl capturing parens "()" to store 
            information for use in the trigger, release, and message fields;
            similar to perl/sed, the first match is store in %1, the second 
            in %2, the third %3, and so on.
threshold - How many repetitions of this pattern constitute a match?
within    - How long is the window for matching?  For example, do we want
            to consider this matcher matched if we have 3 hits in 60 second?
            5 hits in an hour?  8 hits at any time?  If you want no limit
            on the duration of time to look for matches, set this to 0.
            You can use suffixes of 's', 'm', and 'h' to indicate 
            seconds, minutes, and hours, respectively.  If you use no
            suffix, seconds is assumed. (e.g. 60m == 60 minutes).
duration  - How long after a match should the release be called?  Can use
            the same suffixes as in 'within,' specify 0 for no release
            to be called.
trigger   - Program or script to be called when this matcher has a match;
            you can use variables matched in the pattern in this variable,
            e.g. '/sbin/iptables -I INPUT -j REJECT -s %1'
release   - Program or script to be called when this matcher's duration
            has ended (ignored if the duration is set to be 0).
            You can use variables matched in the pattern in this variable,
            e.g. '/sbin/iptables -I INPUT -j REJECT -s %1'
message   - Message to be output to syslog (/var/log/lmf.log by default).
            You can use variables matched in the pattern in this variable,
            e.g. '%name - brute force SSH attempt from %1'

The folling additional 'meta' variables can be used in the configuration
directives 'message', 'trigger', and 'release' (in addition to %1 .. %N
for captured information from the regular expression in the 'pattern' 
field:
* %count     - number of matches that triggered this action
* %threshold - number of matches that triggers this action
* %duration  - number of seconds before release is executed
               (0 if no release is set up)
* %within    - number of seconds threshold number of matches have
               to be found in for us to consider this a match
* %name      - Name of the rule that is matched
* %file      - File the match was found in
* %time      - Number of seconds it took for this match to be triggered
* %pct       - % of the time threshold (within) it took for this match
               to be triggered

Example usage:
message = %name - %1, %count hits, %time secs (%threshold/h %within/s): 

A complete configuration is shown below for a matcher that detects ssh brute 
force attempts, lines with # describe each field.

[SSH brute force attempt]
#  What file this matcher looks at
file = /var/log/secure
#  The pattern we are looking for in the file, using perl
#  regular expressions
pattern = Illegal user \S+ from (\S+)
#  How many times does this pattern have to match before
#  we take an action?
threshold = 6
#  What time frame are we looking at for number of hits on
#  this rule to trigger an action.  In this case, 6 matches
#  within 2 minutes would trigger an action.  Use 'm' for
#  minutes, 's' for seconds, or 'h' for hours when designating
#  times.  If you use no designtation, seconds is assumed :).
within = 2m
#  How long AFTER a triggered action should we run the release
#  action?  Use 0 for 'never'
duration = 3h
#  Script to run when 'threshold' matches are found within 
#  'within' period of time.  Use %N variables to substitute
#  in data you requested from the pattern.  In the pattern above,
#  %1 will match the IP address of the user '\S+ from (\S+)' .. the
#  parenthesis 'capture' that data from the matched entries .. just
#  as in perl.
trigger = /usr/local/lmf/actions/fw drop %1
#  Script to run to 'undo' whatever action was done when the match
#  was triggered.
release = /usr/local/lmf/actions/fw allow %1
#  Message to put in syslog when this action is triggered 
message = %name from %1 (%count attempts in %time seconds)

Additionally, the 'trigger' and 'release' directives can use %message
to use the fully formatted message for this match (same as the message
lmf-monitord sends to syslog).

5) Installing LMF
=================

!! Important: wherever you untarred/unzipped LMF *is* its home directory !!
   We recommend using /usr/local/ as the base directory.

* Get the latest version of lmf and untar/gzip it.

Example

# cd /usr/local
# elinks 'http://www.sf.net/project/showfiles.php?group_id=178711&package_id=206217&release_id=456171'
# tar zxvf ./lmf-0.5.tgz

*  cd to the LMF home directory
*  Set up your configuration snippets by creating them from scratch OR
   copying sample rules from config/dist/ to config/.  We will provide 
   an online configuration snippet repository at some point in the near 
   future to help with this process.
*  Run the install script from the base lmf directory, e.g.

# cd /usr/local/lmf-#version#/
# bash ./install.sh

Note: the installer will install perl modules using the CPAN module of
perl, if you have not used CPAN before you will be asked a series of
questions by the *CPAN* installer as it sets up CPAN for you.  If you
are not familiar with this process, you can learn about it from
www.perl.com or via google.com, or we will be glad to set this script
up for you for a 15 minute charge.

*  Start LMF!

/sbin/service lmf restart

Log file for lmf will be /var/log/lmf.log

Email any questions you have about this script to

lmf-general@lists.sourceforge.net