#  userguide.pod - Torrus user guide
#  Copyright (C) 2003  Stanislav Sinyagin
#
#  This program is free software; you can redistribute it and/or modify
#  it under the terms of the GNU General Public License as published by
#  the Free Software Foundation; either version 2 of the License, or
#  (at your option) any later version.
#
#  This program is distributed in the hope that it will be useful,
#  but WITHOUT ANY WARRANTY; without even the implied warranty of
#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
#  GNU General Public License for more details.
#
#  You should have received a copy of the GNU General Public License
#  along with this program; if not, write to the Free Software
#  Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.

# Stanislav Sinyagin <ssinyagin@k-open.com>
#
#

=head1 Torrus User Guide


=head2 Quick start guide

The steps below will explain you how to make the thing running.

B<Note to Debian users:> please read the README.Debian file in 
/usr/share/doc/torrus-common directory for some Debian-specific
information.

B<What is where>. The executables reside in F<@pkgbindir@/>. You
normally don't need to access this directory, because the commandline
wrapper, C<torrus>, is installed in a usual execution path
(F</usr/sbin>).  All site-specific behaviour is controlled by
configuration files in F<@siteconfdir@/>.  Usually you need to change
F<torrus-siteconfig.pl> only. In this file, you must list your XML
configuration sources. The datasource trees configuration is read out of
XML files. They are searched in several directories, normally
F<@distxmldir@/> and F<@sitexmldir@/>. The first one contains files that
come with Torrus distribution, and the second one is for your local
site-specific XML files. Global site-specific XML configuration
parameters may be defined in F<site-global.xml>.  XML configuration is
compiled into internal database representation by C<torrus compilexml>
command. The database itself is a Git repository, and it is resided in
F<@gitrepodir@/>. It is safe to re-compile the configuration while the
Torrus daemons are running.

B<The datasource trees>. Torrus configuration consists of a number of I<trees>.
Each tree is independent from the others. A tree may run multple Collector and
one Monitor processes. Also the web interface access control lists
differentiate the user rights by datasource trees.

B<Inside the tree>. A tree defines the hierarchy of Torrus datasources.
The structure of the tree is solely defined by XML configuration files.
The tree consists of I<nodes>, each being either a I<subtree> or a I<leaf>.
Subtrees contain child subtrees and/or leaves. The leaf represents a
datasource: normally this is a numerical value that changes over time.
The leaf is the entity that may be presented as a graph.
There are leaves of special type: I<multigraph>.
They are not numerical values, and are designed for
drawing several values in one graph. Each node has I<path>, a
string that consists of slashes and node names, and uniquely identifies this
node. The path of a subtree always ends with slash, and the root of the tree
has the path consisting of a single slash.

B<Trees configuration>. The trees are defined in F<torrus-siteconfig.pl>.
See I<Torrus Installation Instructions> for a basic example of tree
configuration.

B<Round-robin databases>. RRDtool is the primary type of storage for the
collector data. Each leaf represents a datasource stored in an RRD
file. Of course, several leaves may refer to different datasources
within the same RRD file. Even more, more than one leaf may refer to the
same datasource within an RRD file. RRD files are created and updated
either by C<collector>, or by some other external programs.

B<Define the targets>. If you only want to collect some standard SNMP
counters from network devices, there are tools called C<torrus genddx>
and C<torrus devdisover>.

C<torrus genddx> creates a basic discovery instructions file, and it's
designed to be run only once for an initial DDX file. Further on, you
etither edit the DDX files manually, or generate them by some other
tools.

C<torrus devdisover> uses the discovery instructions to explore the SNMP
device capabilities and information: interface names, input/output
counters, CPU and memory usage, temperature sensors, and many other
vendor-specific statistics sources.

Torrus is much more than just an SNMP collector. So, when you decide
to use it in a more advanced way, you will have to read the whole bit of
this guide, and also I<Torrus XML Configuration Guide> and probably some
other documents too.

B<Build the hierarchy>. By default, C<torrus genddx> will put all your
devices into one hierarchy: C</Routers/E<lt>hostnameE<gt>/...>.
The subtree name, C<Routers>, may be changed with a command line option
of C<torrus genddx>. This program may also read the device names
(or IP addresses in case if you don't use DNS) from space-delimited text files.

  torrus genddx \
    --hostfile=myrouters.txt \
    --domain=example.net \
    --community=MySecretSNMPCommunity \
    --out=myrouters.ddx \
    --discout=myrouters.xml \
    --subtree=/My_Routers \
    --datadir=/data1/torrus/collector_rrd

  torrus genddx \
    --hostfile=myswitches.txt \
    --domain=example.net \
    --community=MySecretSNMPCommunity \
    --out=myswitches.ddx \
    --discout=myswitches.xml \
    --subtree=/My_Switches \
    --datadir=/data1/torrus/collector_rrd

  torrus devdiscover  --in=myrouters.ddx

  torrus devdiscover  --in=myswitches.ddx

In the example above, the routers' and switches' names are read from
F<myrouters.txt> and F<myswitches.txt> in the user's current directory.
They form a hierarchy with two subtrees: C</My_Routers/> and C</My_Switches/>.
C<genddx> creates the discovery instruction XML files into
F<myrouters.ddx> and F<myswitches.ddx> accordingly. By default,
you would find them in F<@sitedir@/discovery/>.
The result of C<devdiscover> is the Torrus configuration files:
F<myrouters.xml> and F<myswitches.xml>, placed into
F<@sitexmldir@/>. The C<collector> will place the
RRD files into F</data1/torrus/collector_rrd>. Make sure that this directory
exists, has enough free space, and is writable by C<torrus> user.

B<Note:> the C<genddx> utility is designed as a one-time helper, so
that you create your basic discovery instructions files from scratch.
Further on, the discovery files should be maintained separately.

Another useful utility is called C<ttproclist>. It can be used to generate
a DDX file from a template and a list of SNMP hosts. It is very useful if
you want to monitor many devices of similar type or function.

You can also define a I<bundle> file in your DDX file. C<Genddx> will
create it after all devices would discovered, and it will contain
E<lt>includeE<gt> statements for all XML files. This makes it practical to
use one XML file per SNMP host, and use the bundle file for inclusion
in the tree configuration.

B<Add your XML files to the tree configuration>. For each tree,
F<@siteconfdir@/torrus-siteconfig.pl> lists the XML files that have to be
compiled for it. In the example above, you would add F<myrouters.xml> and
F<myswitches.xml> into C<xmlfiles> array in the tree configuration.

See I<Torrus SNMP Discovery User Guide> for more details on how
C<genddx> and C<devdisover> interact and how you can customize
the discovery process.

B<Tip>: in most cases, your hierarchy division will be different.
It might be arranged by geographical locations, or by customer names.
There is a configuration statement that allows you to include other
XML files into configuration, thus giving you a big flexibility
in building the data hierarchies.

B<Compile the configuration>. After the XML configuration is prepared,
you need to execute the compiler:

  torrus compile --tree=treename --verbose

For most of the processes that you run within Torrus, you need to specify
the tree name with C<--tree> option. Some programs accept C<--all> option,
which causes them to process all existing trees.
With C<--verbose> option, the compiler tells you about the files being
processed, and about some other actions that may take quite a long time.
It will also tell you if there's any error in your configuration.

B<Build the search database>. The search database is updated by executing
the following command:

  torrus bs --all --verbose

For users that are allowed to display all the trees, you can enable the
global search across all trees:

  torrus acledit --addgroup=staff --permit=GlobalSearch --for='*'

B<Launch the collector>. Assuming that compilation went smoothly,
you may now launch the data collector:

  torrus collector --tree=treename

Without additional options, the collector will fork as a daemon
process, and write only error messages in its log file,
F<@logdir@/collector.treename.log>.

See the Torrus installation guide for details about startup scripts.

The C<monitor> daemon is used for monitoring the thresholds in the
data files. For more details, see the I<Torrus XML configuration guide>,
in the section about monitor definitions.

B<Define the ACLs>. By default, user authentication is enabled in the web
interface. You can change this by setting
C<$Torrus::CGI::authorizeUsers = 0> in your F<torrus-siteconfig.pl>.
In order to get use of user authentication, you need to create I<groups>
and I<user> accounts. Each user belongs to one or more groups, and each group
has access to a set of datasource trees. See
I<Torrus Installation Instructions> for a basic example.

B<Browse with your browser>. Provided that you followed the
installation guide to the end, and your HTTP server is running,
your Torrus hierarchy must be visible  with your favorite web browser.


=head2 Configuration guidelines

In complete detail, the XML configuration is described in
I<Torrus XML Configuration Guide>. The guidelines below will help
you to read that document.

B<Tree structure>. The tree structure is defined by the structure of
C<E<lt>subtreeE<gt>> and C<E<lt>leafE<gt>> XML elements. The rule is simple:
child XML elements of a C<E<lt>subtreeE<gt>> element define the child
nodes in the configuration tree.

B<Parameters>. Each node has a number of parameters. They are defined
by C<E<lt>paramE<gt>> XML element. Parameters are inherited:
the child node has all its parent's parameters, some of which may be
overridden.

B<Additive configuration>. The whole XML configuration is additive.
It means that you may define your subtree several times across
your XML configuration, and the new parameters and child nodes will
be added to previously defined ones.

B<Templates>. Some pieces of configuration may be written as templates,
and then re-used in multiple places.

B<Incremental compiler>. Subsequent running of C<torrus compile> will
only pricess the changes in XML files, thus saving the time if the
configuration has only partially changed.

For most of the processes that you run within Torrus, you need to specify
the tree name with C<--tree> option. Some programs accept C<--all> option,
which causes them to process all existing trees.
With C<--verbose> option, the compiler tells you about the files being
processed, and about some other actions that may take quite a long time.
It will also tell you if there's any error in your configuration.

B<Build the search database>. The search database is updated by executing
the following command:

  torrus bs --all --verbose

For users that are allowed to display all the trees, you can enable the
global search across all trees:

  torrus acledit --addgroup=staff --permit=GlobalSearch --for='*'

B<Launch the collector>. Assuming that compilation went smoothly,
you may now launch the data collector:

  torrus collector --tree=treename

Without additional options, the collector will fork as a daemon
process, and write only error messages in its log file,
F<@logdir@/collector.treename.log>.

See the Torrus installation guide for details about startup scripts.

The C<monitor> daemon is used for monitoring the thresholds in the
data files. For more details, see the I<Torrus XML configuration guide>,
in the section about monitor definitions.

B<Define the ACLs>. By default, user authentication is enabled in the web
interface. You can change this by setting
C<$Torrus::CGI::authorizeUsers = 0> in your F<torrus-siteconfig.pl>.
In order to get use of user authentication, you need to create I<groups>
and I<user> accounts. Each user belongs to one or more groups, and each group
has access to a set of datasource trees. See
I<Torrus Installation Instructions> for a basic example.

B<Browse with your browser>. Provided that you followed the
installation guide to the end, and your HTTP server is running,
your Torrus hierarchy must be visible  with your favorite web browser.



=head2 Configuration guidelines

In complete detail, the XML configuration is described in
I<Torrus XML Configuration Guide>. The guidelines below will help
you to read that document.

B<Tree structure>. The tree structure is defined by the structure of
C<E<lt>subtreeE<gt>> and C<E<lt>leafE<gt>> XML elements. The rule is simple:
child XML elements of a C<E<lt>subtreeE<gt>> element define the child
nodes in the configuration tree.

B<Parameters>. Each node has a number of parameters. They are defined
by C<E<lt>paramE<gt>> XML element. Parameters are inherited:
the child node has all its parent's parameters, some of which may be
overridden.

B<Additive configuration>. The whole XML configuration is additive.
It means that you may define your subtree several times across
your XML configuration, and the new parameters and child nodes will
be added to previously defined ones.

B<Templates>. Some pieces of configuration may be written as templates,
and then re-used in multiple places.

B<Incremental compiler>. Subsequent running of C<torrus compile> will
only process changed files, so the compilation time will be much
shorter. Also the collector will pick up the changes incrementally, so
that its schedule is not disturbed by re-initialization.

The C<configsnapshot> utility generates one large XML file back from
the compiled configuration. Its main purpose is backup of the configuration,
but it can also be used for studying the relationships between templates
and input files.

=head2 Handling SNMP errors

During SNMP discovery process, some SNMP devices may not be reachable.
By default, C<devdiscover> reports the error, and does not write the output
XML file containing that device. It also skips writing the bundle files that
contain the output file affected.

When C<devdiscover> is executed with C<--forcebundle> option, the bundle
files are written, and the output files related to the unreachable
devices are skipped from the bundles. This ensures that we always get
a configuration that may compile and run the collector.

Another option, C<--fallback=DAYS>, if given together with C<--forcebundle>,
tells the discovery engine to reuse old XML files if the related SNMP devices
are not reachable and the files are not older than DAYS.

If an SNMP device is unreachable by the moment of the collector initialization,
the collector reports the error and waits for a period of time specified in
C<$Torrus::Collector::SNMP::unreachableRetryDelay>, which is 10 minutes by
default. It then tries to reach the device with the specified retry interval
during some period of time, defined in
C<$Torrus::Collector::SNMP::unreachableTimeout>, by default 6 hours.
If the device is not available within the specified timeout, it is excluded
from collection. It would be tried again on collector initialization
only (at the collector process start or after recompiling the configuration).

If a device is not reachable during the normal collector running cycle,
it is retried in every collector's cycle (usually every 5 minutes),
during the period defined in C<$Torrus::Collector::SNMP::unreachableTimeout>.
It will be then excluded from configuration after the timeout.

If a device hardware configuration changes after the C<devdiscover>
execution, the collector may not find some values in SNMP tables,
such as interface names in ifTable. It then excludes such datasources from
collection immediately.




=head2 Tips and tricks


=head3 Comments, descriptions, and legends

C<torrus devdiscover> will extract some useful information from
your SNMP devices, and place it in the XML configuration:

=over 4

=item * Interface descriptions

The value of the SNMP variable C<ifAlias> (C<1.3.6.1.2.1.31.1.1.1.18>)
will be used as interface comment. In Cisco IOS, this is controlled by
C<description> interface configuration command.

=item * Location and contact

Two other SNMP values: C<sysLocation> (C<1.3.6.1.2.1.1.6.0>) and
C<sysContact> (C<1.3.6.1.2.1.1.4.0>) will be used in the legend text
for each device. In Cisco IOS, their values are controlled by
C<snmp-server location> and C<snmp-server contact> global configuration
commands.

=back


=head3 Grouping the datasources alternatively

In most cases, you would want to have several different groupings of
your datasources.

For instance, the default C<devdiscover> gives only one level of freedom:
the subtree name above the host level. It's reasonable to use this name for
grouping by geographical location . Thus, the hierarchy
would be characterised as
C</[location]/[hostname]/[interface]/[counter]>.

Let's say you would like to have alternative grouping, such as:

=over 4

=item * by customer connection:

Each customer is identified by name, and you'd like to see statistics
for all interfaces connected to a given customer;

=item * by service:

Your network is designed to provide various services, and you'd like to
group your devices or interfaces by service;

=item * by customer and location:

For each customer, group the connection by geographical location.

=back

Torrus provides two different ways for organising your datasources:

=over 4

=item * ds-type=rrd-file

You can create a leaf in some arbitrary place of your hierarchy that
points to an existing RRD file. This RRD file may be updated by
other datasource in your hierarchy. The advantage of such approach is
that this leaf may have its own I<legend> and I<comment> parameters,
alternative view parameters, etc.

    <leaf name="FoobarIn">
      <param name="ds-type"           value="rrd-file" />
      <param name="leaf-type"         value="rrd-def" />
      <param name="data-file"         value="rtr01_Fa0_1.rrd" />
      <param name="rrd-cf"            value="AVERAGE" />
      <param name="rrd-ds"            value="locIfInBitsSec" />
      <param name="comment"
             value="Foobar input traffic"/>
      <param name="graph-legend"      value="Bits in" />
      <param name="legend">
         Switch name: rtr01; Interface: Fa0/1;
      </param>
    </leaf>

In the example above, this leaf is defined somewhere in the hierarchy.
It refers to the RRD file updated by Torrus SNMP collector.
For more examples, see the template I<read-cisco-interface-counters>
in F<vendor/cisco.ios.xml>.

=item * Tokensets

Tokenset is an arbitrary collection of datasource leaves. It is characterised
by its name and description. There are two ways to add a leaf to a tokenset:
by the parameter I<tokenset-member>, or by defining a monitor action.
A tokenset is normally displayed in compact form: by default, 6-hour graphs
are put by two in a row.

=back



=head3 Automating XML generation

It is quite common task that you want Torrus to monitor certain set of
devices, and C<devdiscover> does not (yet) support them. Of course,
it's quite a pain to maintain a manually written XML file, especially if
the there are more than one devices of the same type.

In such case you may benefit from the approach suggested by
Christian Schnidrig:

Imagine you have 50 I<gizmos> which are able to speak SNMP and which you would
like to put into some Torrus tree structure. A good designer's approach would
be to keep the data and the presentation separately. In addition, changing
the presentation once would produce 50 changes accordingly.
To do that, let's create two files: F<gizmos.data> and F<gizmos.tmpl>.
The first one would contain data about our devices:

    [%
      gizmos = [
        {
          name => 'atwork'
          color => 'blue',
          location => 'Javastrasse 2, 8604 Hegnau'
          description => 'My gizmo @ Sun'
          community => 'blabla',
          hands => [
              {name => 'Left'}
              {name => 'Right'}
            ],
        }
        {
          name => 'athome'
          color => 'gray',
          location => 'Riedstrasse 120, 8604 Hegnau'
          description => 'My gizmo @ Home'
          community => 'blabla',
          hands => [
              {name => 'Upper'}
              {name => 'Lower'}
            ],
        }
      ]

    %]

Then F<gizmos.tmpl> would contain the XML template that would produce
the Torrus configuration file:

    [% PROCESS $data %]
    <?xml version="1.0"?>
    <configuration>
      <datasources>
        <subtree name="SNMP">
          <subtree name="Gizmos">
          [% FOREACH g = gizmos %]
          <!-- ******************************************************* -->
          <!-- [% g.name %] -->
          <subtree name="[% g.color %]">
              <alias>/ByName/[% g.name %]/</alias>

              <param name="snmp-community"  value="[% g.community %]" />
              <param name="comment"         value="[% g.description %]" />
              <param name="snmp-host"       value="[% g.name %]" />
              <param name="legend">
                Description: [% g.description %]
                Location:    [% g.location %]
              </param>

              [% FOREACH h=$g.hands %]
              <leaf name="[% h.name %]Hand">
                <!-- do something, my fantasy exhausted here -->
              </leaf>
          </subtree>
          [% END %]
        </subtree>
      </subtree>
    </datasources>
    </configuration>

See F<xmlconfig/examples/servers.data> and F<xmlconfig/examples/servers.tmpl>
for a more useful example of the described approach.

At the end, you will generate the Torrus config with the C<tpage> utility,
which is the standard part of Template-Toolkit package:

  tpage --define data=gizmos.data gizmos.tmpl > gizmos.xml


=head3 Several Torrus instances on one server

B<Note to Debian users:> in the current packaging scheme only one Torrus
instance per server is supported.

Sometimes it is necessary to have a separate instance of Torrus for testing
purposes on the same server as the production installation.
In the example below, a completely autonomous installation of Torrus is
installed in F</usr/testtorrus> directory on a FreeBSD system.

=over 4

=item * Directory structure

All files are located in subdirectories of F</usr/testtorrus>. No other
directories are affected. This ensures that deinstallation would be easy
and safe.

Four subdirectories are created:

=over 8

=item * F</usr/testtorrus/apache>

This directory contains Apache HTTP daemon configuration and logs. Create 3
subdirectories here: F<etc>, F<htdocs>, and F<var>.

=item * F</usr/testtorrus/home>

This is the installation directory of Torrus.

=item * F</usr/testtorrus/etc>

Directory for configuration files.

=item * F</usr/testtorrus/var>

Directory for logs, database and PID files.

=item * F</usr/testtorrus/collector_rrd>

Collector will store RRD files here.

=item * F</usr/testtorrus/src>

Distribution files will be stored and unpacked here.

=back


=item * Installation procedure

  cd /usr/testtorrus/src
  gzip -dc torrus-X.X.tar.gz | tar xvf -
  cd torrus-X.X
  ./configure pkghome=/usr/testtorrus/home \
    sitedir=/usr/testtorrus/etc \
    logdir=/usr/testtorrus/var/log \
    piddir=/usr/testtorrus/var/run \
    varprefix=/usr/testtorrus/var \
    wrapperdir=/usr/testtorrus
  make install

=item * Devdiscover configuration

Use devdiscover as usual. Place your discovery instruction files in
F</usr/testtorrus/etc/discovery/>, and make sure that
C<data-dir> is set to F</usr/testtorrus/collector_rrd>.

=item * Apache configuration

An additional virtual host section can be added to the production Apache
server, and ScriptAlias should point to the proper location.

=back


=head3 Watching the collector failures

There is a script in Torrus distribution in F<examples/rrdup_notify.sh>,
which provides a simple way of telling if the collector runs right: it checks
the modification time of RRD files, and if any file is older than given
threshold, it sends an e-mail warning.

Copy the script file to some place in your system and edit it so that it fits
your requirements: you might want to change the maximum age
parameter (default is 1 hour), the notification e-mail address, and the
directory paths where to look for RRD files. Then I<chmod> it so that it's
executable, and add it to I<crontab>. Depending on your operation requirements,
it might run every hour, or few times a day, or even at business hours only.

The script writes the number of aged files in the e-mail subject, and lists
the file names in the body. In case of relatively large installation,
you might want to amend the script, in order to avoid too large email messages.


=head3 Viewing external RRD files

Some external program may create its own RRD files, and you
may want to display and monitor them in Torrus.

Also some collector-generated RRDs may become outdated -- for example, after
a module is removed from a router, and the interface counters not
being updated any more.

The easiest way to use such files would be to utilize the
C<torrus rrddir2xml> command. It generates the XML configuration file
that represents all RRD files found in a given directory. It can also
scan the directory recursively.

See also few examples in Torrus distribution. There are some
templates for use with Smokeping, OpenNMS, and Flowscan.


=head2 Torrus usage scenarios


=head3 Scenario 1. Netflow Traffic Analyser

Cisco routers are capable of exporting the traffic statistics
data in I<Netflow> UDP packets.

A I<cflowd> or I<flow-tools> daemon collects Netflow packets into flow files.

I<FlowScan> software analyses the flow files and stores the
statistics into numerous RRD files.

Torrus is used to monitor the thresholds and diplay the graphs
in convenient form.


=head3 Scenario 2. Backbone Traffic Statistics

I<CiscoWorks2000> or I<NMSTOOLS> software is used to provide
the list of all devices in the network.

Torrus's C<devdiscover> buids the XML configuration to monitor the
router interfaces, CPU and memory usage, and temperature sensors.

Data importing scripts generate configuration for alternative
grouping of the datasources: by location; by customer connection;
by device type; by service type; etc...


=head2 Troubleshooting guidelines

=head3 SNMP Error: Received tooBig(1)

For some devices, the collector may issue the following error messages:

 [27-May-2004 10:15:17*] SNMP Error for XX.XX.XX.XX:161:public: Received
 tooBig(1) error-status at error-index 0

For better performance, SNMP collector sends several SNMP requests in one
UDP datagram. The SNMP agent then tries to send the reply to all requests
in a single datagram, and this error indicates the failure. In most cases,
this is caused by the agent software limitations or bugs.

The number of requests per datagram is controlled by the parameter
C<snmp-oids-per-pdu>, and it may be set in the discovery input XML or
in Torrus configuration XML. The default value is 40, and setting it to 10
generally works.



=head1 Author

Copyright (c) 2002-2017 Stanislav Sinyagin E<lt>ssinyagin@k-open.comE<gt>