.:. METALOG .:.



           ------------------------ BLURB ------------------------


Metalog is a modern replacement for syslogd and klogd. The logged messages
can be dispatched according to their facility, urgency, program name and/or
Perl-compatible regular expressions. Log files can be automatically rotated
when they exceed a certain size or age. External shell scripts (e.g., mail)
can be launched when specific patterns are found.

Logs are written according to RFC3164, RFC5423 or a minimal format with
optional severity levels.

It can send its logs to remote syslog servers via UDP. It can gather info from
the kernel or /dev/log as well as from several Unix Domain Sockets. This allows
logging from chrooted or containerized applications.

Metalog is simple to configure, accepts unlimited number of rules and has
(switchable) memory bufferization for maximal performance.

http://metalog.sourceforge.net/

        ------------------------ COMPILATION ------------------------


In order to compile Metalog, you need to have the PCRE2 library. It's used
to process Perl-compatible regular expressions.

libpcre2 comes with many distributions, but you can also download it from:
                             https://www.pcre.org/

In case you want to use the optional native gzip compression for rotated
log files you need to have zlib. It's very likely, that this it's already
installed, but you can also download it from:
                             https://www.zlib.net/

Then, compiling and installing Metalab isn't rocket science:

./autogen.sh
./configure
make
make install


       ------------------------ CONFIGURATION ------------------------


A configuration file should be installed. Its default location is
/etc/metalog.conf (unless you tweaked --with-sysconfdir). You can find a
sample file in this directory, but it's certainly not perfect for your system
and your needs. So read on.

A configuration file has the following syntax:

[default values]

<section title>:
<section values>
...

<section title 2>:
<section 2 values>
...

[...]

To make it clear, here's an example:

maxsize  = 100000
maxtime  = 86400
maxfiles = 5

Kernel messages:
  facility = "kern"
  logdir   = "/var/log/kernel"

Crond:
  program  = "crond"
  logdir   = "/var/log/crond"

A section defines several things:

 - A title (useless for the software, it's just to make your configuration
file look better).

 - Filters: you can define facilities, program names, urgency levels and
regular expressions. An incoming message will pass through all filters. If
all conditions are matching, actions defined for the section are performed.

 - Actions: they are taken only when all previous conditions are met. Only
two actions are currently possible: write the message to a log file, and/or
launch an external script.

Here's a list of values that can be independently assigned for each section:

* minimum = <level>: only record a message if its urgency is inferior or
equal to <level>. Level '0' is the most critical one, while level '7' is for
debugging messages. 'minimum = 5' will strip all non-important messages. The
default minimum level is 7 (ie. keep all messages).

Example: record only critical messages to /var/log/important:

Critical messages:
  facility = "*"
  minimum  = 1
  logdir   = "/var/log/important"

* maximum = <level>: don't log if the message level exceeds that value. By
default, maximum is the largest possible level.

* facility = <facility>: only record a message if the application that issued
it uses syslog facility <facility>.

* break = 0|1 (default=0): if set to 1 and a section is matched,
perform action, but don't consider any more possible section matches
below this one in the config file.  This is useful for creating a config
where specific types of log messages are matched and dispatched, while a
"catch-all" section at the bottom of the config file handles the default
case.  Using break=0 would cause messages to be potentially handled by
multiple sections, causing message duplication, while using break=1
(default) on the sections above the "catch-all" would avoid duplication.

Facility names are: "auth", "authpriv", "cron", "daemon", "ftp", "kern",
"lpr", "mail", "news", "security", "syslog", "user", "uucp", "local0",
"local1" ... "local7". All kernel messages are logged with facility "kern".

A section can have several "facility = ..." lines to match more than one
facility. If <facility> is "*", it'll match all the facilities.

Example: record all authentication messages to /var/log/auth:

Authentication messages:
  facility = "auth"
  facility = "authpriv"
  logdir   = "/var/log/auth"

* program = <program>: only record messages signed by a specific daemon or
program.

You can use this as a modern alternative to syslog facilities: use "*" as a
facility, and set a program name with that directive.

Kernel messages can be caught with 'program = "kernel"'.

Example: record Pure-FTPd messages in a directory, and in.ftpd messages in
another directory (although the facility is the same: FTP):

In.FTPd messages:
  facility = "*"
  program  = "in.ftpd"
  logdir   = "/var/log/in.ftpd"

Pure-FTPd messages:
  facility = "*"
  program  = "pure-ftpd"
  logdir   = "/var/log/pure-ftpd"

* program_regex = <regex>: similar to program, but not just a string equal
match, but a regular expression.

Example: this would match program fields with [postfix/smtp], [postfix/smtpd]
and [postfix/cleanup] etc.

All Postfix messages:
 facility      = "*"
 program_regex = "postfix"
 logdir        = /var/log/postfix

* program_neg_regex = <regex>: log programs that do _not_ match the regex.

* regex = <regex>: a message must match the regular expression to pass that
filter. Multiple regexes are allowed for a single section. All Perl
extensions are allowed, and matching is case insensitive.

Example: record all authentication failures to /var/log/pwdfail:

Password failures:
  regex    = "(password|login|authentication)\s+(fail|invalid)"
  regex    = "(failed|invalid)\s+(password|login|authentication)"
  regex    = "ILLEGAL ROOT LOGIN"
  logdir   = "/var/log/pwdfail"

* neg_regex = <regex>: this is the opposite of the previous directive.
Logging will occur if the regex doesn't match. It can be useful to remove the
crap, and it can be freely mixed with "regex" directives. "regex" and
"neg_regex" directives are scanned in order.

Example:
  facility = "mail"
  neg_regex= "starting daemon"
  logdir   = "/var/log/mail"

* socket = <path to socketfile>: metalog will create a socket file on the
given path and listen on this Unix Domain Socket. That way applications
in a chroot jail can still send log messages that will be received by
metalog. Such a socket could also get bind mounted in LXC containers and
overmount /dev/log in there. Within the container then is no longer a own
logger necessary.
This option can be set in the default part of the config or within a section.
When used within a section it can we use as a selector for a specific log
file instead of flags like "program", "facility", "regex" and so on.

* maxsize = <size in bytes>: automatically rotate log files when their size
have exceeded that size. Messages are never truncated and no message can be
lost during a rotation. Every section can have a different maxsize value.

* maxfiles = <number of files>: how many files to keep after rotation. This
parameter can be set independently for each section. It defaults to 5.

* maxtime = <age in seconds>: automatically rotate log files when they are
older than this number of seconds. Every section can have a different
maxtime value. maxsize and maxtime can be combined together, so that rotation
occurs when any of these conditions occurs.

* compress = 0|1: With this active and when compiled with zlib support the
rotated log files will get compress with gzip.

* compress_delay = <number of files>: This will leave the amount of the
latest rotated log files uncompressed.

Example: rotate log files daily or when they are more than 1,000,000 bytes
long, keep only 3 history files with the first and second uncompressed and
the third one compressed:

  maxsize  = 1000000
  maxtime  = 86400
  maxfiles = 3
  compress = 1
  compress_delay = 2

* log_format = <format>: The log entries can be written in different formats:
"legacy", "legacy_timestamp", "rfc3164" and "rfc5424".
The legacy ones are quite similar to the format defined in obsolete RFC3164.
legacy_timestamp is the default format.

* log_severity = <number>: In case a legacy format is used, this will
introduce the severity info to a log entry. In both RFC formats this is
already included and can't get disabled.

* showrepeats = 0|1: disable printing the "last message repeated N times"
summary messages.  When showrepeats is set to 1, messages will be printed
directly to the log, without summarizing them based on number received (which
can cause problems for some log monitoring programs, as well as delaying
repeated messages if another unique message isn't received promptyly).
Defaults to 0 (i.e. repeated messages will be summarized)

* logdir = <directory>: record messages in the specified directory. If the
directory doesn't exist, it will be automatically created when the first
matching message will be logged (the parent directory has to exist, though).

* perms = <mode>  permissions for the log directory. Defaults to 0700

Example: Let those in the group with the GID of the process read the log.
Don't forget to run metalog as the group.

  perms    = 0770

* command = <path/to/command>: run a program or a shell-script when all
conditions are met. This directive is not incompatible with logdir: a
message can be both logged and passed to an external command. When the
command is launched, the first argument is filled with the date the message
was received, the second argument is the program name, and the last one is
the text of the message itself. Environment variables aren't cleared.

Example: send a mail to root when authentication failures occur:

Mail password failures:
  regex    = "(password|login|authentication)\s+(fail|invalid)"
  regex    = "(failed|invalid)\s+(password|login|authentication)"
  regex    = "ILLEGAL ROOT LOGIN"
  command  = "/usr/local/sbin/pwdfail.sh"

"pwdfail.sh" can be a simple shell script like this one:

#! /bin/sh
echo "$3" | mail -s "Password failure (program: $2)" root

  Don't forget to properly quote arguments to avoid security problems.

* postrotate_cmd = <path/to/script>: run a script after rotating
If specified, the postrotate_cmd is run after log-files are rotated.
The following parameters are provided, environments are not cleared:
  date: date of the last message received
  prog: program name of the last message received
  file: file name with path of the rotated log-file

Example: compress rotated log files with bzip2

  postrotate_cmd = /usr/local/bin/compress.sh

  "/usr/local/bin/compress.sh" provides bzip2 compression:
#!/bin/sh
#*
#* Copyright (c) 2006 by Lukas Ruf (lukas.ruf@lpr.ch),
#* Computer Engineering and Networks Laboratory (TIK),
#* Swiss Federal Institute of Technology (ETH) Zurich
#*

DATE="$1"
PRG="$2"
FILE="$3"

# tests to run bzip2
# parameter file is provided
# file exists
# file has a size greater than zero
if [ -n "${FILE}" -a -f "${FILE}" -a -s "${FILE}" ]; then
  bzip2 -9 ${FILE}
fi

* configdir = <path to directory>: This  optional  directory can contain
more config files that will get interpreted at metalog start. Only file
names ending with ".conf" are interpreted. This option is only allowed once,
all repetitions  will be ignored. The content of the additional config files
will be appended to the existing config in alphanumeric order of the file
names, to get a determined behaviour of metalog. The config files could
contain sections for specific services.

Example: Content of /etc/metalog.d/:

    00_kernel.conf
    10_metalog.conf
    11_crond.conf
    20_application_xy.conf
    21_apache.conf


         --------- CONFIGURATION: REMOTE SYSLOG SERVERS ------------


metalog can send log messages to syslog server(s) via UDP.

* remote_log = 0|1: Log messages will be sent to a remote syslog server.
If defined in the default settings section, all log files will be sent.
It can also be defined in the seperate sections, so remote logging can
get activated/deactivated for every single section.

* remote_host = <IP address or domain name>: This is the remote syslog
server.

* remote_port = <number>: This is the UDP the syslog server listens for
new log messages.

* remote_format = <format>: It makes sense to set the format to "rfc3164" or
"rfc5424", so the host name of the sender will be included in the message.

* remote_severity_level = <severity level>: This defines which log messages
will be sent to the remote syslog servers. It filters the less important
messages and so reduces the amount of messages sent.

Example: send all logs to a remote server:

  remote_host = 10.1.0.1
  remote_port = 514
  remote_format = rfc5424
  remote_severity_level = 3

This will send all log messages also to the remote syslog server, that have
a severity level of 3 [ERROR] or a higher (that have a lower number).

If several syslog servers should receive the logs, the above remote_*
commands can be replaced with remote_*[<name>], where <name> is a
alphanumeric value.

Example: This will send the logs to two servers: one called "main" and the
other one "backup".

remote_log[main] = 1
remote_host[main] = 10.1.0.1
remote_port[main] = 514
remote_format[main] = rfc5424
remote_severity_level[main] = 3

remote_log[backup] = 1
remote_host[backup] = 192.168.1.10
remote_port[backup] = 514
remote_format[backup] = rfc5424
remote_severity_level[backup] = 3


         ------------------------ LOG FILES ------------------------


With the "logdir" directive, messages from a specific section are recorded
in a directory. In that directory, the following files are created:

- "current": this file contains the latest recorded messages. It can be
incomplete (ie. data is being written to the file when you are reading it).

- ".timestamp": the creation date of "current". Needed for the "maxtime"
directive.

- "log-<year>-<month>-<day>-<hour>:<minutes>:<seconds>": old logs,
chronologically sorted.

If you ever delete these files by hand for some obscure reasons, it's not a
bad idea to restart the daemon (and even better: to stop it before you mess
the directories).


      ------------------------ RUNNING METALOG ------------------------


Kill "klogd" and "syslogd" first. Don't run Metalog while they are running.
Multiple programs listening for the same source of data is a silly idea.

Once these programs are killed (and you are *sure* they are killed), check
that /etc/metalog.conf is installed, and simply run:

/usr/local/sbin/metalog &

On GNU/Linux systems, two processes are spawned: one is the "MASTER", doing
much of the work. The other one (named "KERNEL") is needed for Linux and
replaces the "klogd" daemon. It is responsible for logging kernel messages.

By default, messages are immediately recorded into log files, and the I/O
cache is flushed to be sure that you don't loose any message if a fatal crash
or a power outage occurs.

But if you prefer speed, Metalog can work asynchronously, using the --async
switch. In this mode, to avoid disk I/O, and unlike traditional syslog
daemons, Metalog works in memory buffers, then flushes the data to disk by
blocks. It improves a lot overall performances.

If you temporarely want to switch to the asynchronous mode, send an USR2
signal to the process. Something like:

kill -USR2 $(cat /var/run/metalog.pid)

should do the trick.

Later, if you want to watch activity in real-time (like a good old "tail -f"
on a log file), you can disable buffering. Just send an USR1 signal to the
"MASTER" process. You can always re-enable buffering afterwards.


     ------------------------ RUN-TIME OPTIONS ------------------------


Metalog accepts some run-time options:

- '-a' or '--async': improve performance by using buffers (but log files
won't get updated in real time).

- '-B' or '--daemonize': have the server start in background
(daemonization).

- '-c <xxx>' or '--consolelevel=<xxx>': set the console log level on
Linux. Valid values are from 0 to 7. The default is 7.

- '-C <configuration file>' or '--configfile=<configuration file>': use an
alternative configuration file.

- '-g <group>' or '--group=<group>': change the GID of the metalog process.
Created files will be owned by this group.

- '-h' or '--help': show help and version number.

- '-p <filename>' or '--pidfile=<filename>': set the name of the file
that will hold the PID number. It defaults to /var/run/metalog.pid

- '-s' or '--sync': start in synchronous mode, with no bufferization.

- '-t' or '--test-config': Exit after parsing the config file(s).
Returns 0 if config is ok.




                        -Frank DENIS "Jedi/Sector One" <j@pureftpd.org> .