'\" -*- coding: us-ascii -*-
.if \n(.g .ds T< \\FC
.if \n(.g .ds T> \\F[\n[.fam]]
.de URL
\\$2 \(la\\$1\(ra\\$3
..
.if \n(.g .mso www.tmac
.TH awffull 1 2008-Dec-13 "" ""
.SH NAME
AWFFull \- A Webalizer Fork, Full o' features
.SH SYNOPSIS
'nh
.fi
.ad l
\fBawffull\fR \kx
.if (\nx>(\n(.l/2)) .nr x (\n(.l/5)
'in \n(.iu+\nxu
[...] [log-file]
'in \n(.iu-\nxu
.ad b
'hy
.SH DESCRIPTION
AWFFull is a web server log analysis
program based on The Webalizer.
AWFFull produces
usage statistics in HTML format for viewing with a browser. The results
are presented in both columnar and graphical format, which facilitates
interpretation. Yearly, monthly, daily and hourly usage statistics are
presented, along with the ability to display usage by site, URL, referrer,
user agent (browser), user name, search strings, entry/exit pages, and
country (some information may not be available if not present in the log
file being processed).
.PP
AWFFull supports the following log
formats shown in the following variable list:
.TP 
CLF
(common log format) log files
.TP 
Combined
log formats as defined by NCSA and others,
and variations of these which it attempts to handle
intelligently
.TP 
xferlog
wu-ftpd formatted log files
allowing analysis of ftp servers, and
squid proxy logs.
.RS 
\fBNote\fR
.PP
Logs may also be compressed, via \fBgzip\fR. If a
compressed log file is detected, it will be automatically uncompressed
while it is read. Compressed logs must have the standard
\fBgzip\fR extension of
\*(T<\fI\fI\&.gz\fR\fR\*(T>.
.RE
.PP
This documentation applies to
AWFFull Version
3.8.2
.SH "CHANGES FROM WEBALIZER"
AWFFull is based on The
Webalizer code and has a number of large and small changes.
These include:
.TP 0.2i
\(bu
Beyond the raw statistics: Making use of published formulae to
provide additional insights into site usage
.TP 0.2i
\(bu
GeoIP IP Address look-ups for more accurate country
detection
.TP 0.2i
\(bu
Resizable graphs
.TP 0.2i
\(bu
Integration with GNU gettext allowing for ease of translations.
Currently 32 languages are supported.
.TP 0.2i
\(bu
Display more than 12 months of the site history on the front
page.
.TP 0.2i
\(bu
Additional page count tracking and sort by same.
.TP 0.2i
\(bu
Some minor visual tweaks, including Geolizer's use of Kb, Mb etc
for Volumes
.TP 0.2i
\(bu
Additional Pie Charts for URL counts, Entry and Exit Pages, and
Sites
.TP 0.2i
\(bu
Horizontal lines on graphs that are more sensible and easier to
read
.TP 0.2i
\(bu
User Agent and Referral tracking is now calculated via PAGES not
HITS
.TP 0.2i
\(bu
GNU style long command line options are now supported (eg
\*(T<\fB\-\-help\fR\*(T>)
.TP 0.2i
\(bu
Can choose what is a page by excluding \(oqwhat isn't\(cq
vs the original \(oqwhat is\(cq method
.TP 0.2i
\(bu
Requests to the site being analysed are displayed with the
matching referring URL
.TP 0.2i
\(bu
A Table of 404 Errors, and the referring URL can be
generated
.TP 0.2i
\(bu
An external CSS file can be used with the generated html
.TP 0.2i
\(bu
Manual performance optimisation of the config file is now easier
with a post analysis summary output
.TP 0.2i
\(bu
Specified IP Addresses can be assigned to a given country
.TP 0.2i
\(bu
Additional Dump options for detailed analysis with other
tools
.TP 0.2i
\(bu
Lotus Domino v6 logs are now detected and processed
.PP
Additional changes and improvements are planned and undergoing
implementation. See the TODO file for details.
.SH "NEW REPORT MEASUREMENTS"
With version 3.8.1 of AWFFull, several
new measured results have been added to the detailed report monthly
page.
.TP 
Single Access
Single Access Pages - the only page seen within a given
visit
.TP 
Stickiness
How useful a given entry page is to draw Visitors deeper into
your site
.TP 
Popularity
The Ratio of Page Entries to Page Exits
.PP
These metrics can help towards improving insight in the usage of the
processed web site. And hence allow the site owner to make positive change
to make the site more useful to site visitors. All three metrics appear in
the \(oqEntry Pages\(cq Report. \(oqPopularity\(cq is also
on the \(oqExit Pages\(cq Report.

Single Access

More completely: \(oqSingle Access Pages\(cq. This is a
report on the number of times that a given page was the only page viewed
within a Visit. Or in English, Someone came to your website. They only
viewed one page. The number is the cumulative count of people who did this
for that particular page. Why is this useful? Identifying those entry
pages that don't draw visitors deeper into your site. Or seeing entry
pages that shouldn't be entry pages. It's also a reality check against the
next two values which are calculated from this number. The number
generated should be a subset of the \(oqEntry Page Views\(cq and/or
\(oqExit Page Views\(cq metric. If it isn't? Let me know, we have a
bug. :-)

Stickiness

Is calculated as 1 - (Single Access / Entry Page Views) expressed as
a percentage. In essence Stickiness describes how useful a given entry
page is to draw Visitors deeper into your site. The stickier the page, the
more folk are caught by it. :-) The closer to 100% the better. Generally.
Certain pages within YOUR website may not make sense to have a high
stickiness or even > 5%. This measurement is a clue to understanding
how your site is used, it is not a rule. How is this useful? How and where
are people entering your web site. Does that make sense? Should it be here
or there? What can you change to fix this and hence improve their use of
your website.

Popularity

Popularity is the Ratio of Page Entries to Page Exits. o If it
equals 1.0? Then the number of visitors to your site who started with that
page, equals the number who left at that page. o If greater then 1.0, then
more people entered here then left. o If less then 0? More people left
from here then entered. I personally find this metric one of the more
useful "At a Glance: How are Pages Performing" metrics. One of the
difficulties with using this particular metric is that certain numbers
will NOT make sense for YOUR site. In that a natural exit page would
expect to have a very low Popularity. It's an exit page, not an entry
page. So if an exit page has a high popularity, then you have a real
problem. Likewise, a low Popularity for an entry page is unlikely to be a
Good Thing(tm).
.PP
"Where & Why?" All three of these metrics are covered very
nicely in Hack #58 from "Web Site Measurement Hacks" [1]. Which is where,
credit where credit due, the inspiration to add these metrics came
from.
.SH "RUNNING AWFFULL"
AWFFull is
designed to be run from a Unix command line prompt or as a crond(8) job.
There is no need to run with super-user privleges, and indeed, is
preferable NOT to.
.PP
Once executed, the general flow of the program is:
.PP
A default configuration file is scanned for,
\*(T<\fI/usr/local/etc/awffull.conf\fR\*(T> and, if found, is
used.
.PP
Any command line arguments given to the program are parsed. This
may include the specification of one or more configuration files,
which are processed at the time it is encountered.
.PP
It can be useful to have multiple config files. A master used
for multiple sites, and individualised config files. Do be aware that
last option set wins. So last config file, or if after a config file,
command line options. Useful if you desire to send the output to an
alternate directory.
.PP
If a log file was specified, it is opened and made ready for
processing. If no log file was given, STDIN is used for input. If the
log filename '-' is specified, STDIN will be forced.
.PP
If an output directory was specified,
AWFFull changes to that directory in
preparation for generating output. If no output directory was given,
the current directory is used.
.PP
If no hostname was given, the program attempts to get the
hostname using a uname(2) system call. If
that fails, localhost is used.
.PP
A history file is searched for in the current directory (output
directory) and read if found. This file keeps totals for previous
months, which is used in the main index.html HTML document. Note: The
file location can now be specified with the HistoryName configuration
option.
.PP
If incremental processing was specified, a data file is searched
for and loaded if found, containing the 'internal state' data of the
program at the end of a previous run. Note: The file location can now
be specified with the IncrementalName configuration option.
.PP
Main processing begins on the log file. If the log spans
multiple months, a separate HTML document is created for each month.
After main processing, the main index.html page is created, which has
totals by month and links to each months HTML document.
.PP
A new history file is saved to disk, which includes totals
generated by AWFFull during the current
run.
.PP
If incremental processing was specified, a data file is written
that contains the 'internal state' data at the end of this run.
.SH "INCREMENTAL PROCESSING"
Version 1.2x of The
Webalizer added incremental run capability.
Simply put, this allows processing large log files by breaking them up
into smaller pieces, and processing these pieces instead. What this means
in real terms is that you can now rotate your log files as often as you
want, and still be able to produce monthly usage statistics without the
loss of any detail. Basically, AWFFull saves
and restores all internal data in a file named
\*(T<\fIawffull.current\fR\*(T>. This
allows the program to 'start where it left off' so to speak, and allows
the preservation of detail from one run to the next. The data file is
placed in the current output directory, and is a plain ASCII text file
that can be viewed with any standard text editor. It's location and name
may be changed using the IncrementalName
configuration keyword.
.PP
Some special precautions need to be taken when using the incremental
run capability of AWFFull. Configuration
options should not be changed between runs, as that could cause corruption
of the internal data stored. For example, changing the
MangleAgents level will cause different
representations of user agents to be stored, producing invalid results in
the user agents section of the report. If you need to change configuration
options, do it at the end of the month after normal processing of the
previous month and before processing the current month. You may also want
to delete the awffull.current file as
well.
.PP
AWFFull also
attempts to prevent data duplication by keeping track of the timestamp of
the last record processed. This timestamp is then compared to current
records being processed, and any records that were logged previous to that
timestamp are ignored. This, in theory, should allow you to re-process
logs that have already been processed, or process logs that contain a mix
of processed/not yet processed records, and not produce duplication of
statistics.
.PP
The only time this may break is if you have duplicate timestamps in
two separate log files. Any records in the second log file that do have
the same timestamp as the last record in the previous log file processed,
will be discarded as if they had already been processed. There are lots of
ways to prevent this however, for example, stopping the web server before
rotating logs will prevent this situation, or using a tool such as
cronolog (\(lahttp://cronolog.org/\(ra).
This setup also necessitates that you always process logs in chronological
order, otherwise data loss will occur as a result of the timestamp
compare.
.SH "REVERSE DNS LOOKUPS"
AWFFull no
longer supports DNS lookups. Please use an external program such as
DNShistory or
DNSTran instead.
.TP 0.2i
\(bu
\(lahttp://www.summary.net/soft/dnstran.html\(ra
.TP 0.2i
\(bu
\(lahttp://www.stedee.id.au/dnshistory\(ra
.PP
With version 3.7.1 of AWFFull,
GeoIP capability can be used for improved
country detection.
.SH "COMMAND LINE OPTIONS"
AWFFull supports many different
configuration options that will alter the way the program behaves and
generates output. Most of these can be specified on the command line,
while some can only be specified in a configuration file. The command line
options are listed below, with references to the corresponding
configuration file keywords. See also awffull.conf(5).

General Options
.TP 
\*(T<\fB\-h\fR\*(T>, \*(T<\fB\-\-help\fR\*(T>
Display all available command line options and exit
program
.TP 
\*(T<\fB\-V\fR\*(T>, \*(T<\fB\-\-version\fR\*(T>
Display program version and exit program
.TP 
-v --verbose
\*(T<\fBVerbosity\fR\*(T> Display debugging information for
errors and warnings. Multiple v's will increase the amount of
information displayed.
.TP 
--match_counts
Display optimisation useful information pertaining to the
number of matches against various Group, Hide and Ignore
options.
.TP 
-i --ignore_history
\*(T<\fBIgnoreHist\fR\*(T> Ignore history. USE WITH CAUTION.
This will cause AWFFull to ignore any
previous monthly history file only. Incremental data (if present) is
still processed.
.TP 
-p --preserve_state
\*(T<\fBIncremental\fR\*(T> Preserve internal data between
runs.
.TP 
-T --timing
\*(T<\fBTimeMe\fR\*(T>. Force display of timing information
at end of processing.
.TP 
-c --config=FILE
Use configuration file FILE
.TP 
-n NAME
\*(T<\fBHostName\fR\*(T>. Use the hostname
NAME.
.TP 
-o --output=DIR
\*(T<\fBOutputDir\fR\*(T>. Use output directory
DIR.
.TP 
-t NAME
\*(T<\fBReportTitle\fR\*(T>. Use
NAME for report title.
.TP 
F --logtype=TYPE
\*(T<\fBLogType\fR\*(T>. Specify log type to be processed.
Value can be one of: auto,
clf, combined,
domino, ftp or
squid format. If not specified, will
default to auto format. FTP logs must be
in standard wu-ftpd xferlog format. A
value of \(oqauto\(cq states that the log format
automatically ascertained.
.TP 
-f --fold
\*(T<\fBFoldSeqErr\fR\*(T>. Fold out of sequence log records
back into analysis, by treating as if they were the same date/time
as the last good record. Normally, out of sequence log records are
simply ignored.
.TP 
-Y
\*(T<\fBCountryGraph\fR\*(T>. Suppress country graph.
.TP 
-G
\*(T<\fBHourlyGraph\fR\*(T>. Suppress hourly graph.
.TP 
-x NAME
\*(T<\fBHTMLExtension\fR\*(T>. Defines the HTML file
extension to use on the created report files. If not specified,
defaults to html. Do not include the
leading period.
.TP 
-H
\*(T<\fBHourlyStats\fR\*(T>. Suppress hourly
statistics.
.TP 
-L
\*(T<\fBGraphLegend\fR\*(T>. Suppress color coded graph
legends.
.TP 
-l NUM
\*(T<\fBGraphLines\fR\*(T>. Use background lines. The number
of lines and where to place are automatically calculated. For
backwards compatibility, any number > 0 enables. Use zero ('0')
to disable the lines.
.TP 
-P NAME"
\*(T<\fBPageType\fR\*(T>. Specify file extensions that are
considered pages. Sometimes referred to
as pageviews.
.TP 
-m NUM
\*(T<\fBVisitTimeout\fR\*(T>. Specify the Visit timeout
period. Specified in number of seconds. Default is 1800 seconds (30
minutes). Sometimes referred to as
sessions.
.TP 
-I NAME
\*(T<\fBIndexAlias\fR\*(T>. Use the filename
name as an additional alias for
index.
.TP 
-M NUM
\*(T<\fBMangleAgents\fR\*(T>. Mangle user agent names
according to the mangle level specified by
num.

Mangle levels are:

5 - Browser name and major version

4 - Browser name, major and minor version

3 - Browser name, major version, minor version to two decimal
places

2 - Browser name, major and minor versions and
sub-version

1 - Browser name, version and machine type if possible

0 - All information (left unchanged).
.TP 
-g NUM
GroupDomains. Automatically group sites by domain. The
grouping level specified by num can be
thought of as 'the number of dots' to display in the grouping. The
default value of 0 disables any domain grouping.
.PP
Hide Options
.TP 
-a NAME
\*(T<\fBHideAgent\fR\*(T>. Hide user agents matching
name.
.TP 
-r NAME
\*(T<\fBHideReferrer\fR\*(T>. Hide referrer matching
name.
.TP 
-s NAME
\*(T<\fBHideSite\fR\*(T>. Hide site matching
name.
.TP 
-X NAME
\*(T<\fBHideAllSites\fR\*(T>. Hide all individual sites (only
display groups).
.TP 
-u NAME
\*(T<\fBHideURL\fR\*(T>. Hide URL matching
name.
.PP
Table size options
.TP 
-A --top_agents=NUM
\*(T<\fBTopAgents\fR\*(T>. Display the top
num user agents table.
.TP 
-R --top_refers=NUM
\*(T<\fBTopReferrers\fR\*(T>. Display the top
num referrers table.
.TP 
-S --top_sites=NUM
\*(T<\fBTopSites\fR\*(T>. Display the top
num sites table.
.TP 
-U --top_urls=NUM
\*(T<\fBTopURLs\fR\*(T>. Display the top
num URL's table.
.TP 
-C --top_countries=NUM
\*(T<\fBTopCountries\fR\*(T>. Display the top
num countries table.
.TP 
-e --top_entry=NUM
\*(T<\fBTopEntry\fR\*(T>. Display the top
num entry pages table.
.TP 
-E --top_exit=NUM
\*(T<\fBTopExit\fR\*(T>. Display the top
num exit pages table.
.PP
Other Options
.TP 
--use_geoip
Enables the use of the Maxmind GeoIP capability for more
accurate detection of countries.

\fINOTE! Do not enable GeoIP if you analyse files that
have had the IP Address translated to a Fully Qualified Host Name.
Use either raw IP Addresses and GeoIP, or Names and disable GeoIP.
ie. Don't use GeoIP AND DNShistory.\fR
.TP 
--match_counts
Display the various Group/Hide etc Match Counts. This option
is ideal for optimisation of the awffull.conf file. Just be careful
with optimising Agents in particular, as the order is typically
important.
.SH "CONFIGURATION FILES"
See the awffull.conf(5) man page for
complete details of all configuration options.
.PP
Configuration files are standard ASCII(7)
text files that may be created or edited using any standard editor.
.PP
Blank lines and lines that begin with a pound sign ('#') are
ignored.
.PP
Any other lines are considered to be configuration lines, and have
the form \(oqKeyword Value\(cq, where the \(oqKeyword\(cq is
one of the currently available configuration keywords (see
awffull.conf(5)), and \(oqValue\(cq is
the value to assign to that particular option.
.PP
Any text found after the keyword up to the end of the line is
considered the keyword's value, so you should not include anything after
the actual value on the line that is not actually part of the value being
assigned. The file
\*(T<\fIsample.conf\fR\*(T> provided with
the distribution contains lots of useful documentation and examples as
well.
.PP
Certain "Keywords" will accept a 2nd value. In those situations, the
first value may be enclosed in double quotes (") to allow for
whitespace.
.SH "SEE ALSO"
awffull.conf(5)
.SH BUGS
None currently known. YMMV....
.PP
Report bugs to \(lahttps://bugs.launchpad.net/awffull\(ra,
or use the email discussion list:
<\*(T<awffull@stedee.id.au\*(T>>
.SH NOTES
In case it is not obvious: AWFFull is a play/pun on the word
\(oqawful\(cq, and is pronounced the same way. Yes it was
deliberate.
.SH REFERENCES
[1] Web Site Measurement Hacks. Eric T. Peterson (and others).
O'Reilly. ISBN 0-596-00988-7.
.PP