ARIS Extractor

README


WELCOME TO ARIS	ANALYZER AND ARIS EXTRACTOR

SecurityFocus is pleased to introduce ARIS, the Attack Registry and
Intelligence Service, a new database designed to aid IDS users in effective
Incident Detection and Incident Handling. By extracting, storing, analyzing
and comparing data supplied by the security community and home users alike,
ARIS and its Extractor and Analyzer components will provide a powerful means
of accurately identifying and effectively responding to network attacks on an
ongoing, real-time basis.



ABOUT ARIS

The Attack Registry and Intelligence Service (ARIS) is a free, user-integrated
attack-trending system hosted by SecurityFocus that allows administrators
and operators of Intrusion Detection Systems (IDSs) to track, evaluate and
respond to security alerts and attacks in a proactive manner. 



ABOUT ARIS EXTRACTOR

As an integral piece of the ARIS Analzyer service, SecurityFocus's
open-source ARIS Extractor utility distills data provided by IDS attack-list
logs to build client portfolios that provide meaningful, graphical analysis of
potentially malicious network incidents. By filtering out insignificant or
benign data and converting it to a common format (xml), ARIS Extractor
streamlines incident reporting for both security professionals and home users
in a way that allows IDS operators to focus only on relevant attacks and
incidents. Additionally, ARIS Extractor ensures client confidentiality through
secure file-transfer protocols and optional IP address suppression.



USER AND NETWORK BENEFITS

From user-supplied IDS data, ARIS builds and maintains a cumulative,
date-sensitive incident portfolio while providing a forum for accurate
evaluation and response. At your discretion, the system will determine who
should be notified at the offending network about attacks or attempted
attacks, and subsequently draft a detailed, customizable letter
referencing specific log snippets of documented incidents. Further to this
type of direct response, ARIS will put account holders in touch with
other users who may have been attacked by the same offender.



FILE TRANSFER SECURITY

All communication with the SecurityFocus ARIS server is performed over an
encrypted SSL connection using the HTTPS protocol. All user accounts are
authenticated when cleaned attack logs are uploaded to the server. 



USER PRIVACY

While it is required of ARIS users to provide broad demographic details before
activating an account, it is not necessary for users to divulge their
identities to SecurityFocus should they wish to suppress their host IP
address(es). See the SecurityFocus Privacy Statement at
http://www.securityfocus.com/account/privacy.html for more information.



SUPPORTED INTRUSION DETECTION SYSTEMS (IDSs)

ARIS currently supports the following IDSs for UNIX users:

* Snort (open source) - with or without portscan file (http://www.snort.org)
                      - including full, fast and system logging formats.

  Versions:

	1.6.3
	1.7.0
	1.8.0
	1.8.1

* Cisco Secure IDS (Netranger)

  Versions:

	2.5(1)S2
	2.5(1)S1
	2.5(1)S0
	2.5(0)S0
	2.2.1.7
	2.2.1.6
	2.2.1.5
	2.2.1.4
	2.2.1.3
	2.2.1.2
	2.2.1.1
	2.2.1.0

* Dragon IDS

  Versions:

	4.2 - 5.0

With the release of future versions of ARIS Extractor, SecurityFocus will
evaluate the suitability of expanding its portfolio of supported IDSs.



ACCOUNT SETUP

In order to gain access to the ARIS server, users must establish an ARIS
Analyzer account prior to downloading the ARIS Extractor utility. This account
is discrete from any existing accounts you may already use to access other
services offered by SecurityFocus. 

Account holders may remain anonymous, but must disclose a minimum of broad
demographic detail (e.g. country of origin, type of industry in which you work)
upon registration. This information is then used in a comparative analysis of
other relevant incidents logged with the ARIS database. (For example, it may be
pertinent for certain users to know that similar attacks were launched against
other ARIS account holders during a matching timeframe.) It is not, however,
necessary for users to divulge their identities should they wish to suppress
their host IP address(es).



GRAPHICAL REPORTING

Reports available to ARIS users will graph a range of statistical data, including: 

* attacks via country of origin
* attacks via ISP
* attacks according to time of day and day of week
* type and class of logged attacks
* target platform applications
* new activity since last report
* reports that show which of your incidents and/or attackers are also being reported by other ARIS users



GETTING STARTED WITH ARIS EXTRACTOR

The basic operation performed by ARIS Extractor is the cleaning and 
uploading of a specified IDS log. ARIS Extractor keeps track of the time and 
date of the last record you processed so that you can run the utility 
on your logs as often as you wish without uploading duplicate information.

After you have successfully built ARIS Extractor on your platform, the easiest
way to use the utility is to schedule a cron task that periodically cleans
and uploads your logs to the server. Selected non-scheduled tasks may also
be run at your discretion.


ALL COMMANDS

To view a complete list of valid command line options, type 'extractor' at
the command prompt:

Example:	

  $ extractor

SecurityFocus ARIS Extractor Version ###
Usage: extractor [options] -u <username> [-p password] <logfile>
       extractor [options] -a <username/password file> <logfile>
       extractor [options] -X <logfile>
       extractor [options] -L -u <username> [-p password] <xml file>
       extractor [options] -L -a <username/password file> <xml file>
Options:
  -u <username>         Username for upload server.
  -p <password>         Password for upload server.
                        (ARIS Extractor will prompt for password if omitted.)
  -a <username/password file>   Text file containing Username and Password for
                        Upload server (Newline separated)
  -f <portscan_logfile> Optionally specify a Snort portscan logfile to process.
  -c <mask1,mask2...>   Strip IP addresses in specified netmasks from output.
  -S <proxy_addr:port>  Use a proxy server and optionally specify a port.
  -U <proxy_user>       Specify a username for authenticated proxy.
  -P <proxy_password>   Specify a password for authenticated proxy.
  -i <sensor_id>        Optionally specify a unique sensor identifier.
  -X                    Parse log only.  Do not upload results to server.
  -v                    Detailed output from cURL.
  -V                    Detailed output about log processing.
  -D <debug_logfile>    Detailed upload information.
  -I <DD/MM/YYYY>       Only parse incidents after date input.
  -h <host>             Override default upload server with <host>.
  -n                    Parse Cisco Secure IDS (Netranger) Logfile.
  -d                    Parse Dragon IDS Logfiles. <logfile> should specify the
                        DB directory and driders.cfg file (Comma separated)
  -L                    Upload specified XML file to the server.
  -q                    Quiet logging. Everything except error messages are supressed.
  -g                    IDS incident timestamps are in GMT. For use with only with Snort
                        and Dragon which log to local time by default.
  -l                    IDS incident timestamps are in local time. This switch is
                        only used by NetRanger which logs to GMT by default.
  -N <255.255.255.0>    Specify netmask for your IDS. This helps SecurityFocus keep
                        accurate statistics.
  
  <logfile> is the path to an input logfile.


USING A PROXY SERVER

If you are required to access the SecurityFocus Incidents Database
through a proxy server, use the following command to specify the address
of the proxy server:

		-S proxy_server

A port may be specified by appending a colon and port number to the server
name.

Example:	-S my.proxy.com:8080

When using a proxy that requires authentication, the following command
line options are used to specify username and password:

		-U proxy_user
		-P proxy_user


REQUIRED COMMAND LINE OPTIONS

		-u username
		-p password

These required options specify the username and password for your
Incidents Database account.


OPTIONAL COMMANDS

-f portscan_file

This option is not required but specifies an optional Snort portscan log
to parse.


EXCLUDING ADDRESSES FROM THE CLEANED LOG

-c mask1,mask2...

This option is used to specify a comma delimited list of netmasks that
indicate ranges of IP addresses to strip from the IDS log information. To
exclude known or benign addresses present in the IDS log file from the
cleaned results, use this option. For example, internal IP addresses and
trusted servers do not need to be included in the cleaning process. IP
addresses matching the specified networks will be replaced with the ips
number in the range. For instance with a mask of 1.2.3.0/24, the ip 
1.2.3.4 will be changed to "4". This allows SecurityFocus to track how
many different addresses are being covered by your IDS.
Netmasks must be in CIDR block notation.

Example: 	-c 10.10.10.0/24,1.2.3.4/30


DATA FROM MULTIPLE IDS SENSORS

-i my_second_profile

If you are operating multiple IDS sensors that you wish to upload logs from you
can create multiple profiles so that you can view data separately on the ARIS 
server.  To create an additional profile, simply choose an identification string
for it and specify it with the (-i) command line option.  The first time you upload 
a log file with this option, a profile with the specified name will be created on
the server.  Additional uploads under this sensor ID will be associated with the
profile on the server.


PARSE LOG WITHOUT UPLOAD

-X

If you are diagnosing a problem or are curious about the output generated by ARIS
Extractor you can use the (-X) option to generate an output XML file without 
uploading it to the server.  The output file will be saved in the directory ~/.sfclean

CURL VERBOSE OUTPUT

-v

Verbose output from the cURL HTTPS transfer library. Outputs to the command line.


VERBOSE OUTPUT

-V

You can use the (-V) option if you would like Extractor to print out some statistics
about how many rows and events were processed and how many records have been written
to the output XML file.


QUIET OUTPUT

-q

Use the -q option to surpress all "Successful Upload" messages. '-q' does not override
-V, -v, or -X outputs.


IDS LOGS TO GMT

-g

Only used by Snort and Dragon. Indicates the IDS's default logging to local time has 
been overiden, and that the IDS now logs to GMT. Lets Extrator know that the conversion to GMT 
has already been done.


IDS LOGS TO LOCAL TIME

-l

Only used by NetRanger. Indicates the IDS's default logging to GM time has 
been overiden, and that the IDS now logs to local time. Lets Extrator know that the 
conversion to GMT must be done.


SPECIFY NETMASK

-N <255.255.255.0>

To help SecurityFocus keep more accurate statistics, specify the netmask your IDS listens on. 
Extractor attempts to poll the system for the netmask, and then puts it in the XML
header. If there is more than 1 internet interface or you are running OpenBSD, the automatic 
grabbing of the Netmask doesn't work. The -N switch is provided to allow the user to provide the Netmask.


UPLOAD DEBUGGING INFORMATION

-D debug.log

General information and statistics that help with diagnosing common problems that 
occur during uploading. The output is sent to a log file. 


INPUT TIMESTAMP 

-I 1/7/2001

Extractor will skip all incidents that occur prior to the input date. 
The format is Day/Month/Year.
By default, Extractor saves a timestamp of the last successfully uploaded incident 
in $HOME/.sfclean/lasttime. This allows Extractor to only parse new incidents and 
keep upload file sizes to a minimum. The -I option overrides the timestamp in the 
$HOME/.sfclean/lasttime file. 
NOTE: This option _should_not_ be used for everday operation of Extractor. As your IDS 
log file grows, so will the parsed xml upload file, which will eventually result in 
upload files that are too large (the current limit is 40 megs). The purpose of the -I 
option is to force Extractor to reparse an IDS log file that it might have encountered 
errors parsing/uploading.


SPECIFY UPLOAD HOST

-h 1.2.3.4

If you would like to override the hardcoded upload server address, you can use the
(-h) command line option to specify an alternate hostname or IP address.


CLEANING AND UPLOADING A LOG

The following example demonstrates the command line used to clean a
specified Snort alert file:

	$ extractor -u username - p password alert.ids

If the cleaning and uploading process is successful, the following output
will appear:

	Upload completed

The alert input file can be in any of the alert logging formats supported
by the Snort IDS including syslog.  If you specify the '-n' option the
input file must be a log file generated by Cisco Secure IDS (Netranger).


COMMON LOG FORMAT

ARIS Extractor parses all log file data and converts it into a common log format
(xml) that references only information relevant to the Incidents Database.

In addition to a common log header (ARIS Extractor version number; IDS product
and version; multiple IDS upload source tag), the cleaned logs will
contain the following record for each attack (assuming the Source IP has
not been stripped):

* Attack ID reported by the IDS
* Date and time of event reported in GMT
* IP layer protocol number
* IP address of event source
* TCP or UDP port (if applicable) for event source
* TCP or UDP port (if applicable) for event destination
* Ancillary information


The following is an example portion of a cleaned xml file: 

?xml version="1.0"?>
<!DOCTYPE idslog [
        <!ELEMENT idslog (logheader, logentries)>

        <!ELEMENT logheader (sfclientver, seriallabel, logcleandatetime, localdatetime, timezone, productid, productname, pro
ductmajorver, productminorver, productreleasever, netmask)>
        <!ELEMENT sfclientver (#PCDATA)>
        <!ELEMENT seriallabel (#PCDATA)>
        <!ELEMENT logcleandatetime (#PCDATA)>
        <!ELEMENT localdatetime (#PCDATA)>
        <!ELEMENT timezone (#PCDATA)>
        <!ELEMENT productid (#PCDATA)>
        <!ELEMENT productname (#PCDATA)>
        <!ELEMENT productmajorver (#PCDATA)>
        <!ELEMENT productminorver (#PCDATA)>
        <!ELEMENT productreleasever (#PCDATA)>
        <!ELEMENT netmask (#PCDATA)>

        <!ELEMENT logentries (logentry*)>
        <!ELEMENT logentry (entry, addlinfo*)>
                <!ELEMENT entry EMPTY>
                        <!ATTLIST entry num CDATA #REQUIRED
                                datetime CDATA #REQUIRED
                                id CDATA #REQUIRED
                                fromip CDATA #REQUIRED
                                toip CDATA #REQUIRED
                                count CDATA #REQUIRED
                                protocol CDATA #IMPLIED
                                fromport CDATA #IMPLIED
                                toport CDATA #IMPLIED
                                portscan CDATA #IMPLIED>
                <!ELEMENT addlinfo (#PCDATA)>
                        <!ATTLIST addlinfo type CDATA #REQUIRED>
]>

<idslog>

<logheader>
<sfclientver>1.6</sfclientver>
<seriallabel></seriallabel>
<logcleandatetime>37213.21819444</logcleandatetime>
<localdatetime>37212.92652778</localdatetime>
<timezone>MST</timezone>
<productid>2</productid>
<productname>Unix Snort</productname>
<productmajorver>1</productmajorver>
<productminorver>6</productminorver>
<productreleasever>or greater</productreleasever>
<netmask>255.0.0.0</netmask>
</logheader>

<logentries>

<logentry>
<entry num="0" datetime="37190.73333277" id="ICMP Destination Unreachable" fromip="192.168.1.101" toip="62.125.37.54" count="1" prot
ocol="1" />
</logentry>

<logentry>
<entry num="1" datetime="37190.95367464" id="IDS177/netbios-name-query" fromip="24.8.42.41" toip="192.168.1.101" count="1" protocol=
"17" fromport="137" toport="137" />
</logentry>

<logentry>
<entry num="2" datetime="37190.95368954" id="IDS339/NETBIOS-SMB-C" fromip="24.8.42.41" toip="192.168.1.101" count="1" protocol="6" f
romport="3910" toport="139" />
</logentry>

</logentries>
</idslog>



HELP WITH ARIS EXTRACTOR

If you have any comments or questions regarding the SecurityFocus ARIS
Extractor, please mail aris-bugs@securityfocus.com.

You can also join the ARIS-USERS mailing list at
http://www.securityfocus.com/forums/aris-users/



CONTACT SecurityFocus

SecurityFocus
1660 South Amphlett Blvd.
Suite 128
San Mateo, CA  94402

(650) 655-2000 tel
(650) 655-2099 fax

info@securityfocus.com

For Public Relations, please contact our agency:
Tara Dugan
Schwartz Communications, Inc.
595 Market Street Suite 2000,
San Francisco, CA, 94105
(415) 512-0770 tel
(415) 882-5787 fax