File: ganglia.html

package info (click to toggle)
ganglia-monitor-core 2.5.7-3.1etch1
links: PTS
area: main
in suites: etch
size: 3,296 kB
ctags: 3,901
sloc: ansic: 27,894; sh: 11,134; makefile: 148
file content (1149 lines) | stat: -rw-r--r-- 51,675 bytes
parent folder | download | duplicates (4)
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<title>Ganglia</title>
<link rev="made" href="mailto:root@localhost" />
</head>

<body style="background-color: white">

<p><a name="__index__"></a></p>
<!-- INDEX BEGIN -->

<ul>

	<li><a href="#name">Name</a></li>
	<li><a href="#version">Version</a></li>
	<li><a href="#synopsis">Synopsis</a></li>
	<li><a href="#installation">Installation</a></li>
	<ul>

		<li><a href="#monitoring_core_installation">Monitoring Core Installation</a></li>
		<li><a href="#php_web_frontend_installation">PHP Web Frontend Installation</a></li>
	</ul>

	<li><a href="#configuration">Configuration</a></li>
	<ul>

		<li><a href="#gmond_configuration">Gmond Configuration</a></li>
		<li><a href="#gmetad_configuration">Gmetad Configuration</a></li>
		<li><a href="#php_web_frontend_configuration">PHP Web Frontend Configuration</a></li>
	</ul>

	<li><a href="#commandline_tools">Commandline Tools</a></li>
	<ul>

		<li><a href="#gmetric">Gmetric</a></li>
		<li><a href="#gstat">Gstat</a></li>
	</ul>

	<li><a href="#frequently_asked_questions_(faq)">Frequently Asked Questions (FAQ)</a></li>
	<li><a href="#getting_support">Getting Support</a></li>
	<li><a href="#copyright">Copyright</a></li>
	<li><a href="#authors">Authors</a></li>
	<li><a href="#contributors">Contributors</a></li>
</ul>
<!-- INDEX END -->

<hr />
<p>
</p>
<hr />
<h1><a name="name">Name</a></h1>
<p><strong>ganglia</strong> - distributed monitoring system</p>
<p>
</p>
<hr />
<h1><a name="version">Version</a></h1>
<p><strong>ganglia</strong> 2.5.7</p>
<p>The latest version of this software and document will always be found at 
<a href="http://ganglia.sourceforge.net/.">http://ganglia.sourceforge.net/.</a>  You are currently reading
$Revision: 1.12 $ of this document.</p>
<p>
</p>
<hr />
<h1><a name="synopsis">Synopsis</a></h1>
<pre>
     ______                  ___
    / ____/___ _____  ____ _/ (_)___ _
   / / __/ __ `/ __ \/ __ `/ / / __ `/
  / /_/ / /_/ / / / / /_/ / / / /_/ /
  \____/\__,_/_/ /_/\__, /_/_/\__,_/
                   /____/ Distributed Monitoring System</pre>
<p>Ganglia is a scalable distributed monitoring system for high-performance computing systems 
such as clusters and Grids. It is based on a hierarchical design targeted at federations of 
clusters. It relies on a multicast-based listen/announce protocol to monitor state within 
clusters and uses a tree of point-to-point connections amongst representative cluster nodes 
to federate clusters and aggregate their state. It leverages widely used technologies such as 
XML for data representation, XDR for compact, portable data transport, and RRDtool for data 
storage and visualization. It uses carefully engineered data structures and algorithms to 
achieve very low per-node overheads and high concurrency. The implementation is robust, has 
been ported to an extensive set of operating systems and processor architectures, and is 
currently in use on over 500 clusters around the world. It has been used to link clusters 
across university campuses and around the world and can scale to handle clusters with 2000 nodes.</p>
<p>The ganglia system is comprised of two unique daemons, a PHP-based web frontend and a few other
small utility programs.</p>
<dl>
<dt><strong><a name="item_daemon"><strong>Ganglia Monitoring Daemon (gmond)</strong></a></strong><br />
</dt>
<dd>
Gmond is a multi-threaded daemon which runs on each cluster node you want to monitor. 
Installation is easy. You don't have to have a common NFS filesystem or a database backend, 
install special accounts, maintain configuration files or other annoying hassles. Gmond is 
its own redundant, distributed database.
</dd>
<dd>
<p>Gmond has four main responsibilities: monitor changes in host state, multicast relevant changes, 
listen to the state of all other ganglia nodes via a multicast channel and answer requests for 
an XML description of the cluster state.</p>
</dd>
<dd>
<p>Each gmond transmits in information in two different ways: multicasting host state in external data 
representation (XDR) format or sending XML over a TCP connection.</p>
</dd>
<p></p>
<dt><strong><strong>Ganglia Meta Daemon (gmetad)</strong></strong><br />
</dt>
<dd>
Federation in Ganglia is achieved using a tree of point-to-point
connections amongst representative cluster nodes to aggregate the
state of multiple clusters.  At each node in the tree, a Ganglia Meta
Daemon (<code>gmetad</code>) periodically polls a collection of child data
sources, parses the collected XML, saves all numeric, volatile metrics
to round-robin databases and exports
the aggregated XML over a TCP sockets to clients.  Data sources may be either
<code>gmond</code> daemons, representing specific clusters, or other
<code>gmetad</code> daemons, representing sets of clusters. Data sources
use source IP addresses for access control and can be specified using
multiple IP addresses for failover. The latter capability is natural
for aggregating data from clusters since each <code>gmond</code> daemon
contains the entire state of its cluster.
</dd>
<p></p>
<dt><strong><a name="item_ganglia_php_web_frontend"><strong>Ganglia PHP Web Frontend</strong></a></strong><br />
</dt>
<dd>
The Ganglia web frontend provides a view of the gathered information 
via real-time dynamic web pages. Most importantly, it displays Ganglia 
data in a meaningful way for system administrators and computer users. 
Although the web frontend to ganglia started as a simple HTML view of the 
XML tree, it has evolved into a system that keeps a colorful history of 
all collected data.
</dd>
<dd>
<p>The Ganglia web frontend caters to system administrators and users. For 
example, one can view the CPU utilization over the past hour, day, week, 
month, or year. The web frontend shows similar graphs for Memory usage, 
disk usage, network statistics, number of running processes, and all other 
Ganglia metrics.</p>
</dd>
<dd>
<p>The web frontend depends on the existence of the <code>gmetad</code> which provides it 
with data from several Ganglia sources. Specifically, the web frontend will 
open the local port 8651 (by default) and expects to receive a Ganglia XML tree. 
The web pages themselves are highly dynamic; any change to the Ganglia data appears 
immediately on the site. This behavior leads to a very responsive site, but requires 
that the full XML tree be parsed on every page access. Therefore, the Ganglia web 
frontend should run on a fairly powerful, dedicated machine if it presents a large 
amount of data.</p>
</dd>
<dd>
<p>The Ganglia web frontend is written in the PHP scripting language, and uses graphs 
generated by <code>gmetad</code> to display history information. It has been tested on many 
flavours of Unix (primarily Linux) with the Apache webserver and the PHP 4.1 module.</p>
</dd>
<p></p></dl>
<p>
</p>
<hr />
<h1><a name="installation">Installation</a></h1>
<p>The latest version of all ganglia software can always be downloaded from 
<a href="http://ganglia.sourceforge.net/downloads.php">http://ganglia.sourceforge.net/downloads.php</a></p>
<p>Ganglia runs on Linux (i386, ia64, sparc, alpha, powerpc, m68k, mips, arm, hppa, s390), 
Solaris, FreeBSD, AIX, IRIX, Tru64, HPUX, MacOS X and Windows (cygwin beta) making it as 
portable as it is scalable.</p>
<p>
</p>
<h2><a name="monitoring_core_installation">Monitoring Core Installation</a></h2>
<p>If you use the Linux RPMs provided on the ganglia web site, you can skip 
to the end of this section.</p>
<p>Ganglia uses the GNU autoconf so compilation and installation of the monitoring core is basically</p>
<pre>
  % ./configure
  % make
  % make install</pre>
<p>but there are some issues that you need to take a look at first.</p>
<dl>
<dt><strong><a name="item_kernel_multicast_support"><strong>Kernel multicast support</strong></a></strong><br />
</dt>
<dd>
Currently ganglia will only run on machines with multicast support.  The
vast majority of machines have multicast support by default.  If you have
problems with ganglia this is a core issue.  Later versions of ganglia will not
have the multicast requirement.
</dd>
<p></p>
<dt><strong><a name="item_gmetad_is_not_installed_by_default"><strong>Gmetad is not installed by default</strong></a></strong><br />
</dt>
<dd>
Since <code>gmetad</code> relies on the Round-Robin Database Tool ( see <a href="http://www.rrdtool.com/">http://www.rrdtool.com/</a> )
it will not be compiled unless you explicit request it by using a <strong>--with-gmetad</strong> flag.
</dd>
<dd>
<pre>
  % ./configure --with-gmetad</pre>
</dd>
<dd>
<p>The configure script will fail if it cannot find the rrdtool library and header files.
By default, it expects to find them at /usr/include/rrd.h and /usr/lib/librrd.a.  If 
you installed them in different locations then you need to add the following configure
flags</p>
</dd>
<dd>
<pre>
  % ./configure CFLAGS=&quot;-I/rrd/header/path&quot; CPPFLAGS=&quot;-I/rrd/header/path&quot; \
     LDFLAGS=&quot;-L/rrd/library/path&quot; --with-gmetad</pre>
</dd>
<dd>
<p>of course, you need to substitute <code>/rrd/header/path</code> and <code>/rrd/library/path</code> with the
real location of the rrd tool header file and library respectively.</p>
</dd>
<p></p>
<dt><strong><a name="item_aix_should_not_be_compiled_with_shared_libraries"><strong>AIX should not be compiled with shared libraries</strong></a></strong><br />
</dt>
<dd>
You must add the <code>--disable-shared</code> and <code>--enable-static</code> configure flags if you running on AIX
</dd>
<dd>
<pre>
  % ./configure --disable-shared --enable-static</pre>
</dd>
<p></p>
<dt><strong><a name="item_gexec_confusion"><strong>GEXEC confusion</strong></a></strong><br />
</dt>
<dd>
GEXEC is a scalable cluster remote execution system which provides fast, RSA authenticated 
remote execution of parallel and distributed jobs. It provides transparent forwarding of 
stdin, stdout, stderr, and signals to and from remote processes, provides local environment 
propagation, and is designed to be robust and to scale to systems over 1000 nodes. Internally, 
GEXEC operates by building an n-ary tree of TCP sockets and threads between gexec daemons and 
propagating control information up and down the tree. By using hierarchical control, GEXEC 
distributes both the work and resource usage associated with massive amounts of parallelism 
across multiple nodes, thereby eliminating problems associated with single node resource limits 
(e.g., limits on the number of file descriptors on front-end nodes). 
(from <a href="http://www.theether.org/gexec">http://www.theether.org/gexec</a> )
</dd>
<dd>
<p><code>gexec</code> is a great cluster execution tool but integrating it with ganglia is very clumsy to
say the least.  GEXEC can run standalone without access to a ganglia <code>gmond</code>.  In standalone
mode gexec will use the hosts listed in your GEXEC_SVRS variable to run on.  For example,
say I want to run <code>hostname</code> on three machines in my cluster: <code>host1</code>, <code>host2</code> and <code>host3</code>.
I use the following command line.</p>
</dd>
<dd>
<pre>
  % GEXEC_SVRS=&quot;host1 host2 host3&quot; gexec -n 3 hostname</pre>
</dd>
<dd>
<p>and gexec would build an n-ary tree (binary tree by default) of TCP sockets to those machines
and run the command <code>hostname</code></p>
</dd>
<dd>
<p>As an added feature, you can have <code>gexec</code> pull a host list from a locally running gmond and
use that as the host list instead of GEXEC_SVRS.  The list is load balanced and <code>gexec</code> will
start the job on the <em>n</em> least-loaded machines.</p>
</dd>
<dd>
<p>For example..</p>
</dd>
<dd>
<pre>
  % gexec -n 5 hostname</pre>
</dd>
<dd>
<p>will run the command <code>hostname</code> on the five least-loaded machines in a cluster.</p>
</dd>
<dd>
<p>To turn on the <code>gexec</code> feature in ganglia you must configure ganglia with the <code>--enable-gexec</code>
flag</p>
</dd>
<dd>
<pre>
  % ./configure --enable-gexec</pre>
</dd>
<dd>
<p>Enabling <code>gexec</code> means that by default any host running gmond will send a special multicast
message announcing that gexec is installed on it and open for requests.</p>
</dd>
<dd>
<p>Now the question is, what if I don't want gexec to run on every host in my cluster?  For
example, you may not want to have <code>gexec</code> run jobs on your cluster frontend nodes.</p>
</dd>
<dd>
<p>You simply add the following line to your <code>gmond</code> configuration file (<code>/etc/gmond.conf</code> by
default)</p>
</dd>
<dd>
<pre>
  no_gexec on</pre>
</dd>
<dd>
<p>Simple huh?  I know the configuration file option, <code>no_gexec</code>, seems crazy (and it is).  Why
have an option that says ``yes to no gexec''?  The early versions of gmond didn't use a configuration
file but instead commandline options.  One of the commandline options was simply <code>--no-gexec</code>
and the default was to announce gexec as on.</p>
</dd>
<p></p></dl>
<p>Once you have successfully run</p>
<pre>
  % ./configure
  % make
  % make install</pre>
<p>you should find the following files installed in <code>/usr</code> (by default).</p>
<pre>
  /usr/bin/gstat
  /usr/bin/gmetric
  /usr/sbin/gmond
  /usr/sbin/gmetad</pre>
<p>If you installed ganglia using RPMs then these files will be installed when you install
the RPM.  The RPM is installed simply by running</p>
<pre>
  % rpm -Uvh ganglia-monitor-core-2.5.7.tar.gz</pre>
<p>Once you have the necessary binaries installed, you can test your installation by running</p>
<pre>
   % ./gmond</pre>
<p>This will start the ganglia monitoring daemon.  You should then be able to run</p>
<pre>
   % telnet localhost 8649</pre>
<p>And get an XML description of the state of your machine (and any other hosts running gmond
at the time).</p>
<p>If you are installing by source on Linux, scripts are provided to start <code>gmetad</code> and <code>gmond</code>
at system startup.  They are easy to install from the source root.</p>
<pre>
   % cp ./gmond/gmond.init /etc/rc.d/init.d/gmond
   % chkconfig --add gmond
   % chkconfig --list gmond
     gmond              0:off   1:off   2:on    3:on    4:on    5:on    6:off
   % /etc/rc.d/init.d/gmond start
     Starting GANGLIA gmond:                                    [  OK  ]</pre>
<p>Repeat this step with gmetad.</p>
<p>
</p>
<h2><a name="php_web_frontend_installation">PHP Web Frontend Installation</a></h2>
<ol>
<li></li>
Unzip the webfrontend distribution in your website tree. This is often under the directory 
<code>/var/www/html</code>, however look for the variable <code>DocumentRoot</code> in your Apache configuration 
files to be sure. All the PHP script files use relative URLs in their links, so you may 
place the <code>ganglia/</code> directory anywhere convenient. I like to unzip *tar.gz 
files with one tar command:
<pre>
  % cd /var/www/html
  % tar xvzf gmetad-webfrontend-2.5.0.tar.gz</pre>
<p></p>
<li></li>
Ensure your webserver understands how to process PHP script files. Currently, the 
web frontend contains certain php language that requires PHP version 4 or greater. 
Processing PHP script files usually requires a webserver module, such as the 
<code>mod_php</code> for the popular Apache webserver. In RedHat Linux, the RPM package that 
provides this module is called simply ``php''.
<p>For Apache, <code>mod_php</code> module must be enabled. The following lines should appear 
somewhere in Apache's *conf files. This example applies to RedHat and Mandrake 
Linux. The actual filenames may vary on your system. If you installed the php 
module using an RPM package, this work will have been done automatically.</p>
<pre>
  &lt;IfDefine HAVE_PHP4&gt;
  LoadModule php4_module    extramodules/libphp4.so
  AddModule mod_php4.c
  &lt;/IfDefine&gt;</pre>
<pre>
  AddType  application/x-httpd-php         .php .php4 .php3 .phtml
  AddType  application/x-httpd-php-source  .phps</pre>
<p></p>
<li></li>
The webfrontend requires the existance of the gmetad package on the webserver. 
Follow the installation instructions on the gmetad page. Specifically, the webfrontend 
requires the rrdtool and the <code>rrds/</code> directory from gmetad. If you are a power user, 
you may use NFS to simulate the local existance of the rrds.
<p></p>
<li></li>
Test your installation. Visit the URL:
<pre>
  <a href="http://localhost/ganglia/">http://localhost/ganglia/</a></pre>
<p>With a web-browser, where localhost is the address of your webserver.</p>
<p></p></ol>
<p>Installation of the web frontend is simplified on Linux by using rpm.</p>
<pre>
  % rpm -Uvh gmetad-webfrontend-2.5.7-1.i386.rpm
  Preparing...                ########################################### [100%]
     1:gmetad-webfrontend     ########################################### [100%]</pre>
<p>
</p>
<hr />
<h1><a name="configuration">Configuration</a></h1>
<p>
</p>
<h2><a name="gmond_configuration">Gmond Configuration</a></h2>
<p>While the default options for gmond will work for most clusters, gmond is very 
flexible and can be customize with the configuration file: <code>/etc/gmond.conf</code>.</p>
<p><code>/etc/gmond.conf</code> is not required as its absence will only cause gmond to start 
in a default configuration. Here is a sample of a gmond.conf configuration file
with comment to help you configure gmond</p>
<pre>
   # This is the configuration file for the Ganglia Monitor Daemon (gmond)
   # Documentation can be found at <a href="http://ganglia.sourceforge.net/docs/">http://ganglia.sourceforge.net/docs/</a>
   #
   # To change a value from it's default simply uncomment the line
   # and alter the value
   #####################
   #
   # The name of the cluster this node is a part of
   # default: &quot;unspecified&quot;
   # name  &quot;My Cluster&quot;
   #
   # The owner of this cluster. Represents an administrative
   # domain. The pair name/owner should be unique for all clusters
   # in the world.
   # default: &quot;unspecified&quot;
   # owner &quot;My Organization&quot;
   #
   # The latitude and longitude GPS coordinates of this cluster on earth.
   # Specified to 1 mile accuracy with two decimal places per axis in Decimal
   # DMS format: &quot;N61.18 W130.50&quot;.
   # default: &quot;unspecified&quot;
   # latlong &quot;N32.87 W117.22&quot;
   #
   # The URL for more information on the Cluster. Intended to give purpose,
   # owner, administration, and account details for this cluster.
   # default: &quot;unspecified&quot;
   # url &quot;<a href="http://www.mycluster.edu/&quot">http://www.mycluster.edu/&quot</a>;
   #
   # The location of this host in the cluster. Given as a 3D coordinate:
   # &quot;Rack,Rank,Plane&quot; that corresponds to a Euclidean coordinate &quot;x,y,z&quot;.
   # default: &quot;unspecified&quot;
   # location &quot;0,0,0&quot;
   #
   # The multicast channel for gmond to send/receive data on
   # default: 239.2.11.71
   # mcast_channel 239.2.11.71
   #
   # The multicast port for gmond to send/receive data on
   # default: 8649
   # mcast_port    8649
   #
   # The multicast interface for gmond to send/receive data on
   # default: the kernel decides based on routing configuration
   # mcast_if  eth1
   #
   # The multicast Time-To-Live (TTL) for outgoing messages
   # default: 1
   # mcast_ttl  1
   #
   # The number of threads listening to multicast traffic
   # default: 2
   # mcast_threads 2
   #
   # Which port should gmond listen for XML requests on
   # default: 8649
   # xml_port     8649
   #
   # The number of threads answering XML requests
   # default: 2
   # xml_threads   2
   #
   # Hosts ASIDE from &quot;127.0.0.1&quot;/localhost and those multicasting
   # on the same multicast channel which you will share your XML
   # data with.  Multiple hosts are allowed on multiple lines.
   # Can be specified with either hostnames or IP addresses.
   # default: none
   # trusted_hosts 1.1.1.1 1.1.1.2 1.1.1.3 \
   # 2.3.2.3 3.4.3.4 5.6.5.6
   #
   # The number of nodes in your cluster.  This value is used in the
   # creation of the cluster hash.
   # default: 1024
   # num_nodes  1024
   #
   # The number of custom metrics this gmond will be storing.  This
   # value is used in the creation of the host custom_metrics hash.
   # default: 16
   # num_custom_metrics 16
   #
   # Run gmond in &quot;mute&quot; mode.  Gmond will only listen to the multicast
   # channel but will not send any data on the channel.
   # default: off
   # mute on
   #
   # Run gmond in &quot;deaf&quot; mode.  Gmond will only send data on the multicast
   # channel but will not listen/store any data from the channel.
   # default: off
   # deaf on
   #
   # Run gmond in &quot;debug&quot; mode.  Gmond will not background.  Debug messages
   # are sent to stdout.  Value from 0-100.  The higher the number the more
   # detailed debugging information will be sent.
   # default: 0
   # debug_level 10
   #
   # If you don't want gmond to setuid, set this to &quot;on&quot;
   # default: off
   # no_setuid  on
   #
   # Which user should gmond run as?
   # default: nobody
   # setuid     nobody
   #
   # If you do not want this host to appear in the gexec host list, set
   # this value to &quot;on&quot;
   # default: off
   # no_gexec   on
   #
   # If you want any host which connects to the gmond XML to receive
   # data, then set this value to &quot;on&quot;
   # default: off
   # all_trusted on</pre>
<p>If you want to customize the operation of gmond, simply edit this file and save it to 
<code>/etc/gmond.conf</code>. You can create multiple gmond configurations by writing the 
configuration file to a different file, say <code>/etc/gmond_test.conf</code>, and the using the 
<code>--conf</code> option of gmond to specify which configuration file to use.</p>
<pre>
  % ./gmond --conf=/etc/gmond_test.conf</pre>
<p>would start gmond with the settings in /etc/gmond_test.conf</p>
<p>
</p>
<h2><a name="gmetad_configuration">Gmetad Configuration</a></h2>
<p>The behavior of the Ganglia Meta Daemon is completely controlled by a single configuration 
file which is by default <code>/etc/gmetad.conf</code>. For gmetad to do anything useful you much 
specify at least one <code>data_source</code> in the configuration. The format of the data_source 
line is as follows</p>
<pre>
  data_source &quot;Cluster A&quot; 127.0.0.1  1.2.3.4:8655  1.2.3.5:8625
  data_source &quot;Cluster B&quot; 1.2.4.4:8655</pre>
<p>In this example, there are two unique data sources: ``Cluster A'' and ``Cluster B''. The 
Cluster A data source has three redundant sources. If gmetad cannot pull the data 
from the first source, it will continue trying the other sources in order.</p>
<p>If you do not specify a port number, gmetad will assume the default ganglia port which
is 8649 (U*N*I*X on a phone key pad)</p>
<p>Here is a sample gmetad configuration file with comments</p>
<pre>
   # This is an example of a Ganglia Meta Daemon configuration file
   #                <a href="http://ganglia.sourceforge.net/">http://ganglia.sourceforge.net/</a>
   #
   #-------------------------------------------------------------------------------
   # Setting the debug_level to 1 will keep daemon in the forground and
   # show only error messages. Setting this value higher than 1 will make 
   # gmetad output debugging information and stay in the foreground.
   # default: 0
   # debug_level 10
   #
   #-------------------------------------------------------------------------------
   # What to monitor. The most important section of this file. 
   #
   # The data_source tag specifies either a cluster or a grid to
   # monitor. If we detect the source is a cluster, we will maintain a complete
   # set of RRD databases for it, which can be used to create historical 
   # graphs of the metrics. If the source is a grid (it comes from another gmetad),
   # we will only maintain summary RRDs for it.
   #
   # Format: 
   # data_source &quot;my cluster&quot; [polling interval] address1:port addreses2:port ...
   # 
   # The keyword 'data_source' must immediately be followed by a unique
   # string which identifies the source, then an optional polling interval in 
   # seconds. The source will be polled at this interval on average. 
   # If the polling interval is omitted, 15sec is asssumed. 
   #
   # A list of machines which service the data source follows, in the 
   # format ip:port, or name:port. If a port is not specified then 8649
   # (the default gmond port) is assumed.
   # default: There is no default value
   #
   # data_source &quot;my cluster&quot; 10 localhost  my.machine.edu:8649  1.2.3.5:8655
   # data_source &quot;my grid&quot; 50 1.3.4.7:8655 grid.org:8651 grid-backup.org:8651
   # data_source &quot;another source&quot; 1.3.4.7:8655  1.3.4.8
   
   data_source &quot;my cluster&quot; localhost
   
   #
   #-------------------------------------------------------------------------------
   # Scalability mode. If on, we summarize over downstream grids, and respect
   # authority tags. If off, we take on 2.5.0-era behavior: we do not wrap our output
   # in &lt;GRID&gt;&lt;/GRID&gt; tags, we ignore all &lt;GRID&gt; tags we see, and always assume
   # we are the &quot;authority&quot; on data source feeds. This approach does not scale to
   # large groups of clusters, but is provided for backwards compatibility.
   # default: on
   # scalable off
   #
   #-------------------------------------------------------------------------------
   # The name of this Grid. All the data sources above will be wrapped in a GRID
   # tag with this name.
   # default: Unspecified
   # gridname &quot;MyGrid&quot;
   #
   #-------------------------------------------------------------------------------
   # The authority URL for this grid. Used by other gmetads to locate graphs
   # for our data sources. Generally points to a ganglia/
   # website on this machine.
   # default: &quot;<a href="http://hostname/ganglia/&quot">http://hostname/ganglia/&quot</a>;,
   #   where hostname is the name of this machine, as defined by gethostname().
   # authority &quot;<a href="http://mycluster.org/newprefix/&quot">http://mycluster.org/newprefix/&quot</a>;
   #
   #-------------------------------------------------------------------------------
   # List of machines this gmetad will share XML with. Localhost
   # is always trusted. 
   # default: There is no default value
   # trusted_hosts 127.0.0.1 169.229.50.165 my.gmetad.org
   #
   #-------------------------------------------------------------------------------
   # If you want any host which connects to the gmetad XML to receive
   # data, then set this value to &quot;on&quot;
   # default: off
   # all_trusted on
   #
   #-------------------------------------------------------------------------------
   # If you don't want gmetad to setuid then set this to off
   # default: on
   # setuid off
   #
   #-------------------------------------------------------------------------------
   # User gmetad will setuid to (defaults to &quot;nobody&quot;)
   # default: &quot;nobody&quot;
   # setuid_username &quot;nobody&quot;
   #
   #-------------------------------------------------------------------------------
   # The port gmetad will answer requests for XML
   # default: 8651
   # xml_port 8651
   #
   #-------------------------------------------------------------------------------
   # The port gmetad will answer queries for XML. This facility allows
   # simple subtree and summation views of the XML tree.
   # default: 8652
   # interactive_port 8652
   #
   #-------------------------------------------------------------------------------
   # The number of threads answering XML requests
   # default: 4
   # server_threads 10
   #
   #-------------------------------------------------------------------------------
   # Where gmetad stores its round-robin databases
   # default: &quot;/var/lib/ganglia/rrds&quot;
   # rrd_rootdir &quot;/some/other/place&quot;</pre>
<p><code>gmetad</code> has a <code>--conf</code> option to allow you to specify alternate configuration files</p>
<pre>
  % ./gmetad -conf=/tmp/my_custom_config.conf</pre>
<p>
</p>
<h2><a name="php_web_frontend_configuration">PHP Web Frontend Configuration</a></h2>
<p>Most configuration parameters reside in the <code>ganglia/conf.php</code> file. Here you 
may alter the template, gmetad location, RRDtool location, and set the default time range 
and metrics for graphs.</p>
<p>The static portions of the Ganglia website are themable. This means you can alter elements 
such as section lables, some links, and images to suit your individual tastes and environment. 
The <code>template_name</code> variable names a directory containing the current theme. Ganglia uses 
TemplatePower to implement themes. A user-defined skin must conform to the template interface 
as defined by the default theme. Essentially, the variable names and START/END blocks in a 
custom theme must remain the same as the default, but all other HTML elements may be changed.</p>
<p>Other configuration variables in <code>conf.php</code> specify the location of gmetad's files, and where 
to find the rrdtool program. These locations need only be changed if you do not run gmetad on 
the webserver. Otherwise the default locations should work fine. The <code>default_range</code> variable 
specifies what range of time to show on the graphs by default, with possible values of hour, 
day, week, month, year. The <code>default_metric</code> parameter specifies which metric to show on 
the cluster view page by default.</p>
<p>
</p>
<hr />
<h1><a name="commandline_tools">Commandline Tools</a></h1>
<p>There are two commandline tools that work with <code>gmond</code> to add custom metrics and
query the current state of a cluster: <code>gmetric</code> and <code>gstat</code> respectively.</p>
<p>
</p>
<h2><a name="gmetric">Gmetric</a></h2>
<p>The <strong>Ganglia Metric Tool (gmetric)</strong> allows you to easily monitor any arbitrary 
host metrics that you like expanding on the core metrics that gmond measures by default.</p>
<p>If you want help with the gmetric sytax, simply use the ``help'' commandline option</p>
<pre>
  % gmetric --help
    gmetric 2.5.7</pre>
<pre>
   Purpose:
     The Ganglia Metric Client (gmetric) announces a metric
     value to all Ganglia Monitoring Daemons (gmonds) that are listening
     on the cluster multicast channel.</pre>
<pre>
   Usage: gmetric [OPTIONS]...
      -h         --help                  Print help and exit
      -V         --version               Print version and exit
      -nSTRING   --name=STRING           Name of the metric
      -vSTRING   --value=STRING          Value of the metric
      -tSTRING   --type=STRING           Either string|int8|uint8|int16|uint16|int32|uint32|float|double
      -uSTRING   --units=STRING          Unit of measure for the value e.g. Kilobytes, Celcius
      -sSTRING   --slope=STRING          Either zero|positive|negative|both (default='both')
      -xINT      --tmax=INT              The maximum time in seconds between gmetric calls (default=60)
      -dINT      --dmax=INT              The lifetime in seconds of this metric (default=0)
      -cSTRING   --mcast_channel=STRING  Multicast channel to send/receive on (default='239.2.11.71')
      -pINT      --mcast_port=INT        Multicast port to send/receive on (default=8649)
      -iSTRING   --mcast_if=STRING       Network interface to multicast on e.g. 'eth1' (default='kernel decides') 
      -lINT      --mcast_ttl=INT         Multicast Time-To-Live (TTL) (default=1)</pre>
<p>The gmetric tool formats a special multicast message and sends it to all gmonds that are listening.</p>
<p>All metrics in ganglia have a name, value, type and optionally units. For example, say I wanted 
to measure the temperature of my CPU (something gmond doesn't do by default) then I could 
multicast this metric with name=``temperature'', value=``63'', type=``int16'' and units=``Celcius''.</p>
<p>Assume I have a program called <code>cputemp</code> which outputs in text the temperature of the CPU</p>
<pre>
  % cputemp
  63</pre>
<p>I could easily send this data to all listening gmonds by running</p>
<pre>
  % gmetric --name temperature --value `cputemp` --type int16 --units Celcius</pre>
<p>Check the exit value of gmetric to see if it successfully sent the data: 0 on success and -1 on failure.</p>
<p>To constantly sample this temperature metric, you just need too add this command to your cron table.</p>
<p>
</p>
<h2><a name="gstat">Gstat</a></h2>
<p>The Ganglia Cluster Status Tool (gstat) is a commandline utility that allows you to get status 
report for your cluster.</p>
<p>To get help with the commandline options, simply pass <code>gstat</code> the <code>--help</code> option</p>
<pre>
  % gstat --help
  gstat 2.5.7</pre>
<pre>
  Purpose:
    The Ganglia Status Client (gstat) connects with a
    Ganglia Monitoring Daemon (gmond) and output a load-balanced list
    of cluster hosts</pre>
<pre>
  Usage: gstat [OPTIONS]...
     -h         --help             Print help and exit
     -V         --version          Print version and exit
     -a         --all              List all hosts.  Not just hosts running gexec (default=off)
     -d         --dead             Print only the hosts which are dead (default=off)
     -m         --mpifile          Print a load-balanced mpifile (default=off)
     -1         --single_line      Print host and information all on one line (default=off)
     -l         --list             Print ONLY the host list (default=off)
     -iSTRING   --gmond_ip=STRING  Specify the ip address of the gmond to query (default='127.0.0.1')
     -pINT      --gmond_port=INT   Specify the gmond port to query (default=8649)</pre>
<p>
</p>
<hr />
<h1><a name="frequently_asked_questions_(faq)">Frequently Asked Questions (FAQ)</a></h1>
<dl>
<dt><strong><a name="item_what_metrics_does_ganglia_collect_on_platform_x%3f"><strong>What metrics does ganglia collect on platform x?</strong></a></strong><br />
</dt>
<dd>
This table describes all the metrics that ganglia collects and
shows what platforms the metric are supported on.  (The following table
is not yet complete).
</dd>
<dd>
<pre>
  Metric Name    Description                             Platforms
  -----------------------------------------------------------------------
  boottime      System boot timestamp                    l,f
  bread_sec
  bwrite_sec
  bytes_in      Number of bytes in per second            l,f
  bytes_out     Number of bytes out per second           l,f
  cpu_aidle     Percent of time since boot idle CPU      l
  cpu_arm
  cpu_avm
  cpu_idle      Percent CPU idle                         l,f
  cpu_intr
  cpu_nice      Percent CPU nice                         l,f
  cpu_num       Number of CPUs                           l,f
  cpu_rm
  cpu_speed     Speed in MHz of CPU                      l,f
  cpu_ssys
  cpu_system    Percent CPU system                       l,f
  cpu_user      Percent CPU user                         l,f
  cpu_vm
  cpu_wait
  cpu_wio
  disk_free     Total free disk space                    l,f
  disk_total    Total available disk space               l,f
  load_fifteen  Fifteen minute load average              l,f
  load_five     Five minute load average                 l,f
  load_one      One minute load average                  l,f
  location      GPS coordinates for host                 e
  lread_sec
  lwrite_sec
  machine_type
  mem_buffers   Amount of buffered memory                l,f
  mem_cached    Amount of cached memory                  l,f
  mem_free      Amount of available memory               l,f
  mem_shared    Amount of shared memory                  l,f
  mem_total     Amount of available memory               l,f
  mtu           Network maximum transmission unit        l,f
  os_name       Operating system name                    l,f
  os_release    Operating system release (version)       l,f
  part_max_used Maximum percent used for all partitions  l,f
  phread_sec
  phwrite_sec
  pkts_in       Packets in per second                    l,f
  pkts_out      Packets out per second                   l,f
  proc_run      Total number of running processes        l,f
  proc_total    Total number of processes                l,f
  rcache
  swap_free     Amount of available swap memory          l,f
  swap_total    Total amount of swap memory              l,f
  sys_clock     Current time on host                     l,f
  wcache</pre>
</dd>
<dd>
<pre>
  Platform key:
  l = Linux, f = FreeBSD, a = AIX, c = Cygwin
  m = MacOS, i = IRIX, h = HPUX,  t = Tru64
  e = Every Platform</pre>
</dd>
<dd>
<p>If you are interested in <strong>how</strong> the metrics are collected, just
take a look in directory <code>./gmond/machines</code> in the source distribution.
There is a single source file in the directory for each platform that is supported.</p>
</dd>
<p></p>
<dt><strong><a name="item_xml"><strong>What does the error ``Process XML (x): XML_ParseBuffer() error at line x: not well-formed''</strong></a></strong><br />
</dt>
<dd>
This is an error that occurs when a ganglia components reads data from another ganglia component
and finds that the XML is not well-formed.  The most common time this is a problem is when the PHP
web frontend tries to read the XML stream from gmetad.
</dd>
<dd>
<p>To troubleshoot this problem, capture an XML from the ganglia component in question (gmetad/gmond).
This is easy to do if you have telnet installed.  Simply login to the machine running the 
component and run.</p>
</dd>
<dd>
<pre>
  % telnet localhost 8651</pre>
</dd>
<dd>
<p>By default, gmetad exports its XML on port 8651 and gmond exports its XML on port 8649.  Modify
the port number above to suite your configuration.</p>
</dd>
<dd>
<p>When you connect to the port you should get an XML stream.  If not, look in the process table on
the machine to ensure that the component is actually running.</p>
</dd>
<dd>
<p>Once you are getting an XML stream, capture it to a file by running.</p>
</dd>
<dd>
<pre>
  % telnet localhost 8651 &gt; XML.txt
  Connection closed by foreign host.</pre>
</dd>
<dd>
<p>If you open the file <code>XML.txt</code>, you will see the captured XML stream.  You will need to remove
the first three lines of the <code>XML.txt</code> which will read...</p>
</dd>
<dd>
<pre>
  Trying 127.0.0.1...
  Connected to localhost.
  Escape character is '^]'.</pre>
</dd>
<dd>
<p>Those lines are output from <code>telnet</code> and not the ganglia component (I wish telnet would send
those messages to <code>stderr</code> but they are send to <code>stdout</code>).</p>
</dd>
<dd>
<p>There are many ways that XML can be misformed.  The great tool for validating XML is <code>xmllint</code>.
<code>xmllint</code> will read the file and find the line containing the error.</p>
</dd>
<dd>
<pre>
  % xmllint --valid XML.txt</pre>
</dd>
<dd>
<p>will read your captured XML stream, validate it against the ganglia DTD and check that it is 
well-formed XML.  <code>xmllint</code> will print the entire XML stream if there are no errors.  If 
there are errors they will be reported with line numbers.  For example...</p>
</dd>
<dd>
<pre>
  /tmp/XML.txt:3393: error: Opening and ending tag mismatch: HOST and CLUSTER
  &lt;/CLUSTER&gt;
         ^
  /tmp/XML.txt:3394: error: Opening and ending tag mismatch: CLUSTER and GANGLIA_XML
  &lt;/GANGLIA_XML&gt;
             ^
  /tmp/XML.txt:3395: error: Premature end of data in tag GANGLIA_XML</pre>
</dd>
<dd>
<p>If you get errors, open <code>XML.txt</code> and go to the line numbers in question.  See if you can
understand based on your configuration how these errors could occur.  If you cannot fix the
problem yourself, please email your <code>XML.txt</code> and output from <code>xmllint</code> to
<code>ganglia-developers@lists.sourceforge.net</code>.  Please include information about the version
of each component in question along with the operating system they are running on.  The
more details we have about your configuration the more likely it is we will be able to help
you.  Also, all mailing to <code>ganglia-developers</code> is archiving and available to read on the
web.  You may want to modify <code>XML.txt</code> to remove any sensitive information.</p>
</dd>
<p></p>
<dt><strong><a name="item_how_do_i_remove_a_host_from_the_list%3f"><strong>How do I remove a host from the list?</strong></a></strong><br />
</dt>
<dd>
A common problem that people have is not being able to remove a host from the ganglia
web frontend.
</dd>
<dd>
<p>Here is a common scenario</p>
</dd>
<ol>
<li><strong><a name="item_all_hosts_in_a_cluster_are_multicasting_on_the_gan">All hosts in a cluster are multicasting on the ganglia channel.</a></strong><br />
</li>
<li><strong><a name="item_one_of_the_hosts_fails_or_is_moved_for_whatever_re">One of the hosts fails or is moved for whatever reason.</a></strong><br />
</li>
<li><strong><a name="item_all_the_hosts_in_the_cluster_report_that_the_host_">All the hosts in the cluster report that the host is ``dead'' or ``expired''.</a></strong><br />
</li>
<li><strong><a name="item_the_sysadmin_wants_to_removed_this_host_from_the_%">The sysadmin wants to removed this host from the ``dead'' list.</a></strong><br />
</li>
</ol>
<p>Unfortunately there is currently no nice way to remove a single dead host from the list.
All data in gmond is soft state so you will need to restart all gmond and gmetad processes.
It is important to note that ALL dead hosts will be flushed from the record by restarting
the processes (since they have to hear the host at least once to know it is expired).</p>
<dt><strong><a name="item_how_good_is_solaris%2c_irix%2c_tru64_support%3f"><strong>How good is Solaris, IRIX, Tru64 support?</strong></a></strong><br />
</dt>
<dd>
Here is an email from Steve Wagner about the state of the ganglia on Solaris, 
IRIX and Tru64. Steve is to thank for porting ganglia to Solaris and Tru64. He also 
helped with the IRIX port.
</dd>
<dd>
<pre>
   State of the IRIX port:
   
   *  CPU percentage stuff hasn't improved despite my efforts.  I fear there
      may be a flaw in the way I'm summing counters for all the CPUs.
   *  Auto-detection of network interfaces apparently segfaults.
   *  Memory and load reporting appear to be running properly.
   *  CPU speed is not being reported properly on multi-proc machines.
   *  Total/running processes are not reported.
   *  gmetad untested.
   *  Monitoring core apparently stable in foreground, background being tested
   (had a segfault earlier).
   
   State of the Tru64 port:
   
   *  CPU percentage stuff here works perfectly.
   *  Memory and swap usage stats are suspected to be inaccurate.
   *  Total/running processes are not reported.
   *  gmetad untested.
   *  Monitoring core apparently stable in foreground and background.
   
   State of the Solaris port:
   *  CPU percentages are slightly off, but correct enough for trending
      purposes.
   *  Load, ncpus, CPU speed, breads/writes, lreads/writes, phreads/writes,
      and rcache/wcache are all accurate.
   *  Memory/swap statistics are suspiciously flat, but local stats bear
      this out (and they *are* being updated) so I haven't investigated
      further.
   *  Total processes are counted, but not running ones.
   *  gmetad appears stable
   
   Anyway, all three ports I've been messing with are usable and fairly
   stable.  Although there are areas for improvement I think we really can't
   keep hogging all this good stuff - what I'm looking at is ready for
   release.</pre>
</dd>
<p></p>
<dt><strong><a name="item_where_are_the_debian_packages%3f"><strong>Where are the debian packages?</strong></a></strong><br />
</dt>
<dd>
Here is an email message from Preston Smith for Debian users
</dd>
<dd>
<pre>
 Debian packages for Debian 3.0 (woody) are available at
  <a href="http://www.physics.purdue.edu/~psmith/ganglia">http://www.physics.purdue.edu/~psmith/ganglia</a>
 (i386, sparc, and powerpc are there presently, more architectures will
  appear when I get them built.)
 Packages for &quot;unstable&quot; (sid) will be available in the main Debian
  archive soon.</pre>
</dd>
<dd>
<pre>
 Also, a CVS note: I checked in the debian/ directory used to create
 debian packages.</pre>
</dd>
<p></p>
<dt><strong><a name="item_how_should_i_configure_multihomed_machines%3f"><strong>How should I configure multihomed machines?</strong></a></strong><br />
</dt>
<dd>
Here is an email that Matt Massie sent to a user having problems with
multihomed machines
</dd>
<dd>
<pre>
   i need to add a section in the documentation talking about this since it 
   seems to be a common question.
   
   when you use...
   
   mcast_if eth1
   
   .. in /etc/gmond.conf that tells gmond to send its data out the &quot;eth1&quot;
   network interface but that doesn't necessarily mean that the source
   address of the packets will match the &quot;eth1&quot; interface.  to make sure that
   data sent out eth1 has the correct source address run the following...
   
   % route add -host 239.2.11.71 dev eth1
   
   ... before starting gmond.  that should do the trick for you.
   
   -matt
   
   &gt; I have seen some post related to some issues
   &gt; with gmond + multicast running on a dual nic
   &gt; frontend.
   &gt; 
   &gt; Currently I am experiencing a weird behavior
   &gt; 
   &gt; I have the following setup:
   &gt; 
   &gt;   -----------------------
   &gt;   | web server + gmetad |
   &gt;   -----------------------
   &gt;              |
   &gt;              |
   &gt;              |
   &gt;     ----------------------
   &gt;     |   eth0 A.B.C.112   |
   &gt;     |                    |
   &gt;     |  Frontend + gmond  |
   &gt;     |                    |
   &gt;     | eth1 192.168.100.1 |
   &gt;     ----------------------
   &gt;              |
   &gt;              |
   &gt; 
   &gt;        26 nodes each
   &gt;           gmond
   &gt; 
   &gt; In the frontend /etc/gmond.conf I have the
   &gt; following statement: mcast_if  eth1
   &gt; 
   &gt; The 26 nodes are correctly reported. 
   &gt; 
   &gt; However the Frontend is never reported.
   &gt; 
   &gt; I am running iptables on the Frontend, and I am seing
   &gt; things like:
   &gt; 
   &gt; INPUT packet died: IN=eth1 OUT= MAC= SRC=A.B.C.112 DST=239.2.11.71 
   &gt; LEN=36 TOS=0x00 PREC=0x00 TTL=1 ID=53740 DF PROTO=UDP SPT=41608 DPT=8649
   &gt; LEN=16 
   &gt; 
   &gt; I would have expected the source to be 192.168.100.1 with mcast_if eth1
   &gt; 
   &gt; Any idea ?</pre>
</dd>
<p></p>
<dt><strong><a name="item_how_should_i_configure_my_cisco_catalyst_switches%"><strong>How should I configure my Cisco Catalyst Switches?</strong></a></strong><br />
</dt>
<dd>
Perhaps information regarding gmond on networks set up through cisco
catalyst switches should be mentioned in the ganglia documentation. I think
by default multicast traffic on the catalyst will flood all devices unless
configured properly. Here is a relavent snipet from a message forum, with a
link to cisco document.
</dd>
<dd>
<p>If what you are trying to do, is minimizing the impact on your network due
to a multicast application, this link may describe what you want to do:
<a href="http://www.cisco.com/warp/public/473/38.html">http://www.cisco.com/warp/public/473/38.html</a></p>
</dd>
<dd>
<p>We set up our switches according to this after a consultant came in and
installed an application multicasting several hundred packets per second.
This made the network functional again.</p>
</dd>
<p></p></dl>
<p>
</p>
<hr />
<h1><a name="getting_support">Getting Support</a></h1>
<pre>
  The tired and thirsty prospector threw himself down at the edge of the 
  watering hole and started to drink. But then he looked around and saw 
  skulls and bones everywhere. &quot;Uh-oh,&quot; he thought. &quot;This watering hole 
  is reserved for skeletons.&quot; --Jack Handey</pre>
<p>There are three mailing lists available to you: <code>ganglia-general</code>, <code>ganglia-developers</code> and 
<code>ganglia-announce</code>.  You can join these lists or read their archives by 
visiting <a href="https://sourceforge.net/mail/?group_id=43021">https://sourceforge.net/mail/?group_id=43021</a></p>
<p><code>All of the ganglia mailing lists are closed</code>.  That means that in order to
post to the lists, you must be subscribed to the list.  We're sorry for the 
inconvenience however it is very easy to subscribe and unsubscribe from the 
lists.  We had to close the mailing lists because of SPAM problems.</p>
<p>When you need help please follow these steps until your problem is resolved.</p>
<ol>
<li></li>
completely read the documentation
<p></p>
<li></li>
check the <code>ganglia-general</code> archive to see if other people have had the same problem
<p></p>
<li></li>
post your support request to the <code>ganglia-general</code> mailing list
<p></p>
<li></li>
check the <code>ganglia-developers</code> archive
<p></p>
<li></li>
post your question to the <code>ganglia-developers</code> list
<p></p></ol>
<p>please send all bugs, patches, and feature requests to the <code>ganglia-developers</code> list 
after you have checked the <code>ganglia-developers</code> archive to see if the question has 
already been asked and answered.</p>
<p>
</p>
<hr />
<h1><a name="copyright">Copyright</a></h1>
<pre>
  Copyright (C) 2002,2003 University of California, Berkeley
 
The ganglia source tree incorporated great source code from other projects as well.</pre>
<pre>
  Copyright (c) 2000 Dug Song &lt;dugsong@monkey.org&gt;
  Copyright (C) 1999,2000,2001,2002 Lukas Schroeder &lt;lukas@azzit.de&gt;,
   and others.
  Copyright (C) 1991, 1992, 1996, 1998, 1999 Free Software Foundation, Inc.
  Copyright (C) 2000  David Helder
  Copyright (C) 2000  Andrew Lanoix</pre>
<p>
</p>
<hr />
<h1><a name="authors">Authors</a></h1>
<pre>
  Matt Massie &lt;massie@CS.Berkeley.EDU&gt;</pre>
<p>and the <strong>Ganglia Development Team</strong>...</p>
<pre>
 Bas van der Vlies      basv               Developer    basv at users.sourceforge.net 
 Neil T. Spring         bluehal            Developer    bluehal at users.sourceforge.net
 Brooks Davis           brooks_en_davis    Developer    brooks_en_davis at users.sourceforge.net
 Eric Fraser            fraze              Developer    fraze at users.sourceforge.net 
 greg bruno             gregbruno          Developer    gregbruno at users.sourceforge.net
 Jeff Layton            laytonjb        Developer       laytonjb at users.sourceforge.net       
 Doc Schneider          maddocbuddha    Developer       maddocbuddha at users.sourceforge.net 
 Mason Katz             masonkatz       Developer       masonkatz at users.sourceforge.net      
 Mike Howard            mhoward         Developer       mhoward at users.sourceforge.net        
 Oliver Mössinger      olivpass        Developer       olivpass at users.sourceforge.net       
 Preston Smith          pmsmith         Developer       pmsmith at users.sourceforge.net        
 Federico David Sacerdoti sacerdoti     Developer       sacerdoti at users.sourceforge.net      
 Tim Cera               timcera         Developer       timcera at users.sourceforge.net        
 Mathew Benson          wintermute11    Developer       wintermute11 at users.sourceforge.net</pre>
<p>
</p>
<hr />
<h1><a name="contributors">Contributors</a></h1>
<p>There have been dozens of contributors who have provided patches and helpful bug reports.  
We need to list them here later.</p>

</body>

</html>