1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
|
fwanalog: A firewall log summarizer that uses Analog
http://tud.at/programm/fwanalog/
Balzs Brny, balazs@tud.at
Current version: 0.6.9
This program summarizes firewall logs and creates reports from them.
There are lots of programs that do this. But they weren't good enough for me:
- I use OpenBSD and GNU/Linux, so I need a solution that can handle both.
- I want pretty reports, see recent attacks, host names instead of IP
addresses, HTML output.
- I want a daily report mailed to me, of course in text format.
I use the excellent logfile analyzer Analog ( http://www.analog.cx/ ) a lot.
It is very flexible. So I thought I could convert firewall logs to web
server logs Analog can use.
This shell script does exactly that.
1. It parses the firewall log (I can choose which one) and converts it to a
logfile that Analog understands. The fields in this file are faked, of course;
e.g. the network interface name is the virtual host.
2. It calls Analog with some nice options and so creates different reports:
one about all data; one about the last week; one about the current day in
HTML format and a short one in ASCII format for a report e-mailed daily.
3. Optionally, it can create a separate report for each host and each
blocked packet so you can look at the actions of a "bad guy" or answer
the question "who scanned port 443?".
Requires: Perl, zegrep, awk, diff, sed, egrep, and of course Analog.
Most of these programs should be already installed on any Unix system.
You should really have the GNU versions somewhere and configure them in
fwanalog.opts.
Install Analog:
Debian GNU/Linux: "apt-get install analog"
Any modern BSD: "cd /usr/ports/www/analog; make"
Other GNU/Linux: probably there are RPMs for your distribution
Any other OS: Download from http://www.analog.cx/ , compile, install
Installation
============
1. Decompress the distribution in some directory, e.g. /usr/local/fwanalog
2. Symlink, move or copy the fwanalog.opts.{your OS} to "fwanalog.opts"
3. Edit fwanalog.opts if necessary (most settings should be OK, though)
4. If your Analog version is not the newest stable one, find a language
file for it in the langfiles/ directory and copy it over fwanalog.lng
5. On a non-free Unix (e.g. Solaris), modifiy the first line of the
fwanalog.sh script to "#! /bin/bash" or where your bash or ksh shell
is. Also, look if you have the GNU versions of the utilites listed in
fwanalog.opts.
6. Execute ./fwanalog.sh
7. There should be some HTML and text reports in the directory you specified
in fwanalog.opts ("$outdir").
Customizing
===========
You can edit fwanalog.analog.conf.local to suit your taste, e.g. add pretty
icons and style sheets, switch reports on and off (however, the deactivated
reports don't make much sense with firewall logs). It is better not to edit
the master fwanalog.analog.conf yourself as that file probably will be
updated by me in the next fwanalog release.
You can also edit fwanalog.sh and change the Analog command line options,
deactivate reports and create a conversion function for your firewall if it is
not supported. It's easy. If you think that your changes made the program
better, please send them to me so I can include them in the next version.
Troubleshooting
===============
Some frequent problems:
- The language file doesn't match the Analog version. This can happen with
new installations or after an upgrade of Analog or fwanalog.
Make sure that you use one the correct version of the language file; the
major and the first minor version numbers must match. (E.g. analog 5.32
works with the 5.3 langfile.)
- "It works perfectly when called from the command line but not when called
from cron!" - Search for differences between your shell's and the cron
shell's configuration. The cron path sometimes doesn't include
/usr/local/bin where Analog can be etc.
If you have a problem with fwanalog, go to the homepage and read through the
mailing list archives. Many common problems are already solved there. If
not, subscribe to the mailing list and ask there so more people can help
you. I don't have time to answer e-mails with problems that can be solved by
reading the documentation and/or the knowledge in the mailing list archives.
"One host" mode
===============
You can set "onehost=" to true in fwanalog.opts if you are analyzing the
logs of only one host. This will cause fwanalog to show each packet source
host (i.e. attacker) with the ports it tried.
However, setting this option loses the information about the target IP
address. So don't set this if your firewall protects an entire network.
This feature is based on an idea by Kenneth Vestergaard Schmidt, who is
also the Debian maintainer for fwanalog.
There is also an option "onehost=dynip" based on an idea by Ralph Niere.
This is useful if the address of your firewall changes often, e.g. because
you are on a dial-up connection with dynamic IPs.
Creating separate reports of hosts and packets
==============================================
In fwanalog.opts, set sep_hosts and/or sep_packets to true. Note that this
will cause analog to run once for each host and each packet in the "current"
log. This shouldn't be a problem on a modern machine when fwanalog is run
periodically (e.g. once a day).
As this processes the current log, it will probably run for a long time when
you run fwanalog. If you update from an older version of fwanalog, it will
only process the new log entries since the last invocation, so only a few
packets and hosts will be linked in the reports. You can call analog with
"-a host" or "-p packet" to create a report for a host or a packet you are
interested in. In the future, this host or packet will always be linked in
the reports.
Services
========
fwanalog includes a services.conf file for Analog to convert port numbers
like 21 into service names like ftp. If you think that your services list
is better, feel free to use support/mkservices.conf.sh with your list.
There is also a well_known_services.conf file in support/. It includes lots
of port definitions of more-or-less well known ports. You can include that
file by simply appending it to services.conf or by editing
fwanalog.analog.conf.local to include it. However, so many aliases make
analog slower.
Creating conversion functions for unsupported firewall formats
==============================================================
If your firewall is not supported, please contribute a conversion routine.
It is not very hard:
1. Add your format in fwanalog.opts to the known ones.
2. Copy the ipf or iptables function in fwanalog.sh into a new function with
the name of your firewall (the same you added to fwanalog.opts).
3. Grep the lines about blocked packets from your firewall log into the
fwlog.current file in the output directory
4. Call mkdateconvscript and sed if your log file doesn't contain years.
5. Change the long perl regexp (or use any other tool if you like) so it
changes all lines into the faked web server log format. It is not very
hard if you know regular expressions. Be careful with \$!
Language files
==============
Language files define the strings in the fwanalog output.
Most versions of analog require language files of a matching version.
You might find a language file for your Analog version (and your language,
if you prefer) in the langfiles/ subdirectory of the fwanalog distribution.
If you have a currently unsupported version of Analog, try the mklangfile
scripts in the support/ directory of the fwanalog distribution. Please
contribute language files you have created by submitting them to the author.
Report mappings
===============
fwanalog renames some Analog reports. Not all reports are switched on by default.
(See http://www.analog.cx/docs/output.html for details on Analog's reports)
Analog report fwanalog report remarks
------------- --------------- -------
GENERAL General summary The first report, gives an overview
YEARLY Yearly report Makes sense if you have firewall logs for more than a year
QUARTERLY Quarterly report Makes sense if you have firewall logs for more than 3 months
MONTHLY Monthly report
WEEKLY Weekly report See also the "WEEKBEGINSON" in fwanalog.analog.conf.local
DAILYREP Daily report
DAILYSUM Daily summary Summary by weekdays
HOURLYREP Hourly report
HOURLYSUM Hourly summary Summary by hour of day
WEEKHOUR Hour of the Week Summary
QUARTERREP Quarter-hour report
QUARTERSUM Quarter-hour summary
FIVEREP Five-minute report
FIVESUM Five-minute summary
HOST Packet Source Host Which hosts sent the packets that your firewall blocked
REDIRHOST - Doesn't make sense with firewall logs
FAILHOST - Doesn't make sense with firewall logs
ORGANISATION Organization report
DOMAIN Domain report top level domains
REQUEST - Not used: the directory report is better
suited for fwanalog.
DIRECTORY Blocked Packet Detailed report of blocked packets.
If onehost=false, the target address;
if onehost=true, the source address and the target port.
FILETYPE - Doesn't make sense with firewall logs
SIZE Packet Size Not many variations with some firewall settings
PROCTIME Processing time Not very interesting
REDIR - Doesn't make sense with firewall logs
FAILURE - Doesn't make sense with firewall logs
REFERRER Source Port Sometimes interesting, e.g. with port 21
REFSITE - Doesn't make sense with firewall logs
SEARCHQUERY - Doesn't make sense with firewall logs
SEARCHWORD - Doesn't make sense with firewall logs
INTSEARCHQUERY - Doesn't make sense with firewall logs
INTSEARCHWORD - Doesn't make sense with firewall logs
REDIRREF - Doesn't make sense with firewall logs
FAILREF - Doesn't make sense with firewall logs
BROWSERREP - MAC Address report (if your firewall logs them)
BROWSERSUM - Doesn't make sense with firewall logs
OSREP - Would be nice, but no firewall logs it 8-(
VHOST Interface Report You can turn it off if you have only one interface
REDIRVHOST - Doesn't make sense with firewall logs
FAILVHOST - Doesn't make sense with firewall logs
USER Log Prefix Report Only with iptables, if you set a log prefix
Analog ignores this if it sees no data.
REDIRUSER - Doesn't make sense with firewall logs
FAILUSER - Doesn't make sense with firewall logs
STATUS - Doesn't make sense with firewall logs
OpenBSD 3.x problem
===================
The developers of the new OpenBSD firewall "pf" decided that they log
blocked packets in a binary format instead of a text as usual on Unix.
This file can be only read by the OpenBSD version of tcpdump. So,
fwanalog must run on the OpenBSD 3.x machine itself in order to process
OpenBSD 3.x logfiles.
All other logfiles can be handled on each architecture, e.g. a Linux 2.4
machine can process the logfiles of Solaris, or FreeBSD the logs of
Linux 2.2 etc.
Other documentation
===================
See README.firewall for hints on configuring your firewall.
See README.sudo for information about running fwanalog as a non-root user.
Please mail your suggestions, patches, bugfixes etc. to balazs@tud.at .
$Id: README,v 1.30 2004/03/18 16:40:17 bb Exp $
|
