$Id: NEWS,v 1.143 2002/02/14 15:05:54 flacoste Exp $

LogReport Lire NEWS - user visible changes (and some other changes also.)

Copyright (C) 2000-2001 Stichting LogReport Foundation LogReport@LogReport.org

version in 20020214

* Output when requesting HTML or XHTML was changed. The output is now
  always an uncompressed tar file (like when generating images
  previously). This makes HTML and XHTML output identical whether or
  not images are generated. Also, when report sections are used, the
  (X)HTML reports will be broken into several files.
* The HTML manuals are now splitted in several files.
* lr_config can now be run by any user (not only the Lire's
  administrative user).
* Reports can now be subdivided into sections. This can be achieved by
  using =section directives in the report's configuration file.
  Consult the User's Manual for the complete story.
* Sections can contain filters that will be applied to the input of
  all the section's subreports. Consult the User's Manual for the
  complete story.
* New proxy superservice, with support for MS ISA, WELF and squid logs
  and a lot of reports.
* New database superservice, with service MySQL and reports top-users,
  top-databases, actions-by-period, top-querytypes.
* New FTP service: IIS FTP Logs (iis_ftp).
* New firewall services: iptables, welf. 
  WELF is the WebTrends Enhanced Log Format supported by several
  firewall products (see
  http://www.webtrends.com/partners/firewall.htm for a list). Common
  firewalls using this format: SunScreen, SonicWall, Raptor, etc. This
  makes Lire support a lot of firewalls. We didn't test it with all of
  those products though. So we greatly appreciate all feedback
  regarding how Lire behaves with those firewalls' logs.
* The firewall superservice can also be used for network intrusion
  detection type of logs.
* The firewall superservice now includes a lot of new reports.
* The incompabitilities between the Cisco service and the other firewall 
  services were resolved.
* New ftp reports: bytes-by-user-by-period,
  bytes-by-dir-by-user-by-period, top-users-bytes.
* New www reports: requests-by-keywords, top-referers.
* New email service: Netscape Messaging Server (nms).
* New dns reports: requests-summary, requests-summary-by-method,
  requesttype-by-method, requests-by-timeslot,
  req-by-period-by-method, req-by-timeslot-by-method
* Support for Netscape Messaging Server log daemon.
* Improvements in postfix email service: more robust qmgr line
  handling (no longer creates bogus size fields in dlf), support for
  postfix/virtual lines. (Thanks Cedric Gross for helping us!)
* Online responder more robust against evil characters in subject of
  received email message. Directories in
  var/lib/lire/data/{email,log,report}, named after ID tags, which are
  contructed from these subjects, could contain \ in their names. This
  is ugly, and breaks non-robust scripts. It even might have had
  security implications. People running an online responder should
  upgrade. The change is in lr_processmail.
* Changes to the LRSML DTD: You may have to modify custom report
  specifications you wrote. 
    - The 'sort' attribute was removed from the filter-spec element.
* The WWW::* perl modules, like WWW::Useragent and WWW::Domain, are
  moved to Lire::WWW::*, in order to avoid future nameclashes.

version 20011205.1

* Fixed distclean target.
* Fixed typo in lr_xml2logml.
* Added missing doc/lire.ent in tarball.
* Fixed sendmail queue identifier parsing for very old and very new versions. 
* Fixed a timestamp bug in exim converter.

version 20011205

* Only RTF and PDF output now requires Jade. HTML and XHTML output can be 
  generated with xsltproc and Norman Walsh's DocBook XSL Stylesheets. 
  IMPORTANT: this means that you need to install Norman Walsh's XSL
  Stylesheets available from
  http://docbook.sourceforge.net/projects/xsl/index.html and
  libxslt version 1.0.4 or later to generate HTML output.
* New firewall service: Linux 2.2 ipchains log, as well as Darren Reed's
  IP Filter logs, as shipped with *BSD's.
  NOTE: there are some "known issues" with the IOS Cisco service in
  this release. The informations it gives isn't really equivalent to
  the other firewall services. You may consult
  http://www.nlnet.nl/projects/logreport/hypermail/logreport/development/0518.html
  for more informations.
* New superservice "print" with CUPS (page_log) and LPRng (lp-acct) plugins.
* Syslog parser now handles Solaris 8 syslog tag: [ID ...]
* lire:timegroup aggregator can now aggregate records by months or years.
* New aggregators available to build report specification: lire:summary, 
  lire:rangegroup and lire:timeslot
* New www service: W3C Extended Log Format (Used by Microsoft's Internet
  Information Server IIS 4.0 and IIS 5.0).
* New www reports: requests-summary, requests-by-size, requests-by-timeslot
* More www user session reports.
* Speed improvements to the www robot, country and OS analyzers.
* New robots and Nimda attack detected in www reports.
* New ftp reports: tracked-users, tracked-files.
* New email reports: deliveries-by-delay, deliveries-by-size, 
  tracked_senders, tracked_recipients.
* Sendmail converter is now more robust and interpret more lines. (Anti-spam
  messages will now get into the reports)
* The Lire client no longer is configured by running
  ./configure --disable-server.  If you want to run Lire as a client for an
  online responder, you just install Lire in the regular way.
* Various improvements to most DLF converters. (DNS converters were optimized, 
  all converters were updated to the new DLF API).
* All "by-day" reports have "by-period" equivalent: (www): bytes-by-period, 
  bytes-by-result-by-period, clienthost-by-period, requests-by-period, 
  requests-by-result-by-period; (ftp): bytes-by-period.
  The following report specification are now obsolete and will be removed in 
  a future Lire release: (www): bytes-by-day, bytes-by-result-by-day, 
  clienthost-by-day, requests-by-day, requests-by-result-by-day; 
  (ftp): bytes-by-day
  You should update your report configuration accordingly.
* Bytes, seconds and numbers can be scaled to more human readable format by 
  setting the variables LR_SCALE_BYTES, LR_SCALE_SEC and LR_SCALE_NUMBER to 
  'yes'. By default, bytes and seconds are scaled into '1k' and '2m' 
  in the reports.
* Fixed various small bugs (e.g. the one which caused .org to get skipped in
  the www 'Requests By Top Level Domain' report, as well as the one which
  caused requests from .net clients to get filed under .et, when using perl
  versions << 5.6.1 (5.005_03 is reported to behave in such a way))

version 20011017

* The www report is more exhaustive now: New www reports: requests-by-attack, 
  tracked_pages-by-period, top-last_page, user_sessions-by-period reports.
  Added detection of Code Red attack in WWW reports. Added detection of 
  country where client is located (WWW superservice). Added detection of 
  GNU/Hurd OS and Links browser in WWW reports.
* Added volume per hour email report.  Reordered email reports, as suggested
  by Thierry Montigneaux (thanks Thierry!)
* Added firewall superservice, added cisco service, for cisco acl logfiles
  (Thanks Joost Bekkers)
* Added FTP superservice. Xferlog file format is now supported.
* Overall performance improvement.
* This version introduces a complete overhaul of the reporting engine which
  will make customization and extension of Lire much easier. If you are 
  upgrading from a previous version, you probably want to read the "Upgrading
  From Lire's Version 20010903 or Earlier" section of the INSTALL
  file which has important informations about how to migrate your
  configurations.  Some changes in this release (especially the way 
  configuration files are handled) are _not_ backwards compatible.
* The xml report format has changed.  Therefore, if you install this version on
  a responder, submitters should install this version if they want to be able
  to typeset your reports, after deanonimizing them. 
* XML::Parser and expat (http://expat.sourceforge.net/) are now required
  to build and use Lire. You can install those components separately or you can
  get the lire-full tarball which includes and build them as part of the normal
  Lire installation process. The only supported XSLT processor with this 
  version is xsltproc which comes with the XSLT C library for Gnome 
  (http://xmlsoft.org/XSLT/)
* Reports are now written using the XML Lire's Report Specification Markup 
  Language (LRSML). DLF formats are specified using the XML Lire's DLF Schema 
  Markup Language.
* ASCII reports generated by the builtin formatter or the XSL tools are now 
  identical.
* Informations included with a report can now be tweaked according to the
  intended audience. You can set LR_TARGET_USER in your 
  $HOME/.lire/etc/defaults file to 'sysadmin' or 'manager'. There is also the 
  LR_USERLEVEL variable which can be set to either 'normal' or 'advanced'. 
  Those variables will modify the description accordingly. Note that not all 
  reports have different informations based on those variables.
* There is a LR_MAX_MEMORY variable which you can set to process big log 
  files without trashing. The default is 40Megs. For optimal
  performance, you should set this to half your available RAM (unless
  you are usually short on memory). 
* All Lire's programs now log performance information: time taken, 
  memory used (on Linux only).
* Reports which need missing information in the DLF file are now
  automatically skipped.
* Report dates are now in the local time zone.
* 8bits characters are escaped in the reports to ?.
* New supported output format: XHTML and RTF.
* Improvement to charts: Height increased to 300 pixels. X labels are drawn 
  vertically making it possible to read e.g. URLs.
* Charts for each report that support it are now generated when 
  INCLUDEIMAGES is set to 1. (Not only the first one).
* New chart type: histogram (bars now have space between the bars)
* When generating an HTML, XHTML or DocBook report with images, the result 
  is a tar gzipped file that contains the report file and the images.
  When generating an RTF report with images, the result is a zip file 
  that contains the report file and the images.
* The user's and developer's manuals are reorganized. There are still a lot 
  of the new intended sections uncompleted, though.
* The apachemodgzip superservice is now a regular www service called 
  'modgzip'. The apache service was split into three different ones: common, 
  combined, referer.
* Fixed bugs (as always ;)

version 20010903

* Lot of bugs removed
* BEWARE!  The file ~/.lirerc is obsolete.  If you use it, please move it
  to ~/.lire/etc/defaults.
* Fixed "bogus message: mkdir: cannot create directory `/usr/local/var':
  Permission denied"-bug.  (This one occured when one did a systeminstall,
  and ran the lire scripts as a non-priviliged user afterwards.)
* Now setting print-category and print-severity in named.conf is optional:
  a wider range of dns / bind9 logs gets accepted.
* Added Apache referer parser: now one can see which links brought visitors
  to your site.
* For www user agents newly detected is: operating system, browser type, user
  browser UI language and web robots
* Added 6 new "www" reports: http version, user agents, bytes per directory,
  top referer-page connection and http method
* More reports are now customizable
* Added new output formats: LogML (an experimental XML application) and DocBook
* Responder use munpack for all MIME messages now, not just multipart one.
* Security check can be disabled by setting SKIP_ID_TEST.
* Default report format (txt, xml, pdf or html) can be configured via
  DEFAULT_OUTPUT_FORMAT in $sysconfdir/defaults.
* XML/SGML environment can be reconfigured at runtime from $sysconfir/defaults.
* Support more XSLT processor : Sablotron and XsltProc (from libxslt). 
  Xalan-C can be found by more names (testXSLT or xalan-c)

version 20010629.1

* developer.html and developer.txt now distributed again with the tarball.

version 20010629

* Cleaned up configure.in, to offer more hooks via the environment.  This
  is nice for package builders.

version 20010626

* Moved /usr/local/lib/xml to /usr/local/share/lire, to adhere to FHS.
* Added reports for www superservice: report_bytesperresultperday (this was
  report_bytesperday, but one does no longer split things up) and
  report_totalperday and report_iptotalperday.
* Reduce warnings generated by postfix2dlf.
* Responder should reply with a proper error message when it fails to
  generate a report.
* XML and SGML support should now be more portable between various
  environment.
* PDF reports now can include images. This can be turned on in the
  .../etc/lire/defaults file, by setting INCLUDEIMAGES to 1.
* Fixed possible security problem: we now run with umask 037, so that copied
  logfiles no longer possibly get world readable.
* Fixed bug in apache common convertor: now processes lines with empty http
  request ok.
* Fixed bug which got sendmail2dlf spit evil perl error messages when
  fed an empty logfile.  Thanks to Claire Holleman for finding it.
* Fixed bug which caused postfix2dlf to produce bogus dlf from logs containing
  qmgr lines caused by failed deliveries, e.g. in case of 'invalid recipient
  syntax'.  Thanks to Mark 'Xaa' Huizer for reporting this.
* Keeps old reports and dlf's in a Lire archive, if variable ARCHIVE is
  set.
* Stores metainfo on logs and dlf's in a Lire database.
* Added report_sizeperfromdomain and report_sizepertodomain email subreports
* Added lr_xml2html script.
* Fixed problem with Jade (now uses xml declaration)
* Description blurbs in email report resurrected.  Added description blurbs
  to email report.
* Lire's own logfile now easier machineparsable: added LR_ID job tags
* Default spooldir for responder setups is moved from
  /usr/local/var/spool/logreport to /usr/local/var/spool/lire.  THIS CAN
  BREAK UPGRADES!  When you're upgrading a responder, i.e. a Lire system which
  automatically processes logfiles which get received in email messages, move
  your spool manually to the new location before restarting lr_spoold.  One
  should also fix the .forward's (or any other mechanism your mta uses) to
  deliver mail to Maildirs in the new spool location.

version 20010509

* Fixed long line problems in ascii.xsl and docbook.xsl.
* Added lr_xml2pdf script.
* Fixed bug in "bytes-per-http-result" report.
* Info from README, doc/developer.txt converted to xml.  Now shipped as 
  {developer,manual}.{html,pdf,txt}.  Per default gets installed in 
  /usr/local/share/doc/lire/.

version 20010418

* Pointrelease for debian package.

version 20010407

* Fixed bugs and improved time handling.  We now take year and timezone in 
  account, if available in the logfile.  Some code cleanup.
* Internals: raw intermediate format replaced by xml format.
* Fixed bug which blew up Apache.pm while dealing with clients who's ip
  resolves to a hostname only.  Tnx Wytze!
* Fixed some bugs and added new elements to the logreport xml stylesheet, to 
  gain flexibility.
* Added Exim2DLF: convertor for logfiles from the exim Mail Transport Agent. 
  ( http://www.exim.org/ .)
* Added per-user ~/.lirerc file.

version 20010318

* Changed package name from lr to lire.
* Fixed anoying bug in lr_anonimize: now cleans up tmpfiles.
* Added option to generate XML output.
* Added lr_config script, for easier configuration.
* Responder now is capable of sending reports to more than one Reply-To 
  address.
* Added lr_getbody manpage.
* More sane defaults for things like e.g. From-address for emails sent by 
  lr_log2mail.
* Various bugfixes, especially in dealing with bogus logs.
* Responder now sends informative message in case the submitted log was 
  hardly parsable.
* Added responder for access logs from the boa ( http://www.boa.org/ ) 
  webserver.
* More configuration hooks for www reports: one can specify not to get top 30,
  but top n.
* Added ability to track specific webpages.

version 20010219

* Added local site config file defaults.local, kept between upgrades.
* New www reports, added apache mod_gzip engine, added responder for
  apache "combined" logfiles.
* Added lr_run script: running the scripts will no longer spam your terminal
  with debug output.
* Client software now includes tools (lr_rawmail2mail and friends) to 
  automatically de-anonimize received report. Added notes about this to 
  README.lr-client.
* Added notes about running lr-client software on Solaris to README.lr-client.
* Some cleanup of filesystem layout: got rid of some directories, merged config
  files.
* Some finetuning of email reports.

version 20010116

* No longer depends on GNU make.
* Finds perl and sendmail path during configure: now builds, installs and
  runs out of the box on Solaris.
* Scripts respect --prefix and other configure flags. We're really using
  autoconf and automake now, which makes the package act sanely in other
  directory hierarchies. No longer uses LR_HOME and other environment 
  variables: no need to set them any longer.
* Now package behaves sanely when configure is run without any arguments.
* Integrated a client package. During configure, one can choose to install 
  just the client, or the complete suite. The client enables one to send and 
  receive logs and reports in an anonimized format.
* Bugfixes in postfix and apache convertors.
* Documentation updated.

version 20001216

* Now runs on SunOS 5.7: fixed /bin/sh scripts and awk invocations.

version 20001213

* Various bugfixes, especially in sendmail2dlf
* Documentation updated
* New email dlf format, so that relay's ip number and fqdn can be used
  separately 
* postfix engine supports nqmgr
* Fixed awk scripts, to be able to run with the original awk, as shipped
  with OpenBSD
* lr_log2mail no longer relies on mutt(1), but uses the more generally 
  available /usr/sbin/sendmail

version 20001211

* Various bugfixes
* Documentation updated

version 20001205

* Bugfixes in qmail convertor
* An IPfilter convertor has been added
* A postfix convertor and report engine are added

version 20001130

* The responder uses a new mail receiving engine 
* The install now uses automake and autoconf
* Extra reports for www added
* Added manpages
* New setup of report scripts
* Scheduler no longer needed; more lean design
* Fixed various bugs

version 0.0.1 - september 2000

* First public release