1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
|
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<html>
<head>
<title>How to setup custom graphs</title>
</head>
<body>
<h1>How to setup custom graphs</h1>
This document walks you through the setup of custom graphs in
your Xymon installation. Although Xymon comes with pre-defined
setups for a lot of common types of graphs, it is also extensible
allowing you to add your own tests. For many kinds of tests, it
is nice to view them over a period of time in a graph - this
document tells you how to do that.
<h1>The NCV (Name-Colon-Value) and SPLITNCV methods</h1>
<p>In many cases it is quite trivial to collect some data and
send them to Xymon in format like this:</p>
<pre>
Name: Value
</pre>
<p>e.g. if you have a device that collects weather information,
it might look like</p>
<pre>
Temperature: 19.2
Humidity: 53
Wind: 9.6
</pre>
<p>In Xymon, this is known as <b>NCV</b> - Name, Colon, Value -
formatted data, and Xymon has built-in support to pick up data
formatted this way, and collect the data for graphs.</p>
<h2>Make a script to collect the data</h2>
<p>First create your test data. Typically, this is an extension
script that sends in some data to Xymon, using a <b>status</b> or
<b>data</b> command. If you use <b>status</b>, it will show up as a
separate column on the display, with a green/yellow/red color
that can trigger alerts. If you use <b>data</b>, Xymon just collects
the data into a graph - you must go to the <b>trends</b> column to
see the graph. For this example, we'll use <b>status</b>.</p>
<p>So we create an extension script. Here is an example script;
it picks two numbers out of the Linux kernel's memory statistics,
and reports these to Xymon.
<pre><tt>
#!/bin/sh
cat /proc/slabinfo | \
egrep "^dentry_cache|^inode_cache" | \
awk '{print $1 " : " $3*$4}' >/tmp/slab.txt
$XYMON $XYMSRV "status $MACHINE.slab green `date`
`cat /tmp/slab.txt`
"
exit 0
</tt></pre>
<h2>Get xymonlaunch to run the script</h2>
<p>Save this script in ~xymon/client/ext/slab, and add a section to
the ~xymon/client/etc/clientlaunch.cfg to run it every 5 minutes:
<pre><tt>
[slabinfo]
ENVFILE /usr/lib/xymon/client/etc/xymonclient.cfg
CMD /usr/lib/xymon/client/ext/slab
INTERVAL 5m
</tt></pre>
(On the Xymon server itself, you must add this to the file
~xymon/server/etc/tasks.cfg)
<h2>Check that the script data arrives in Xymon</h2>
<p>After a few minutes, a <b>slab</b> column should appear on your Xymon view
of this host, with the data it reports. The output looks like this:
<pre><tt>
Sun Nov 20 09:03:44 CET 2005
inode_cache : 330624
dentry_cache : 40891068
</tt></pre>
<h2>Arrange for the data to be collected into an RRD file</h2>
<p>This is obviously a name-colon-value formatted report, so we'll use the
NCV module in Xymon to handle it. Xymon will find two datasets here:
The first will be called <b>inodecache</b>, and the second <b>dentrycache</b>
(note that Xymon strips off any part of the name that is not a letter
or a number; Xymon also limits the length of the dataset name to 19
letters max. since RRD will not handle longer names).
To enable this, on the Xymon server edit the ~xymon/server/etc/xymonserver.cfg
file. The TEST2RRD setting defines how Xymon tests (status columns) map to RRD
datafiles. So you add the new test to this setting, by adding <b>slab=ncv</b> at the
end:
<pre><tt>
TEST2RRD="cpu=la,disk,<...lots more stuff...>,xymond,mysql=ncv,slab=ncv"
</tt></pre>
<p><b>slab</b> is the status column name, and <b>=ncv</b> is a token that tells Xymon
to send these data through the built-in NCV module.</p>
<p>By default, the Xymon NCV module expects data to be some sort of counter,
e.g. number of bytes sent over a network - it uses the RRD <b>DERIVE</b> datatype
by default, which is for data that is continuously increasing in value.
Some data are not like that - the data in our test script is not - and for
those data you'll have to make an extra setting to tell Xymon what RRD data
type to use. The RRDtool rrdcreate(1) man-page has a detailed description
of the various RRD datatypes. It is available online at
<a href="http://www.mrtg.org/rrdtool/doc/rrdcreate.en.html">
http://www.mrtg.org/rrdtool/doc/rrdcreate.en.html</a></p>
<p>Our test script provides data that goes up and down in value
(it is the number of bytes of memory used for a Linux kernel bufffer), and
for that kind of data we'll use the RRD <b>GAUGE</b> datatype. So we add an extra
setting to xymonserver.cfg:</p>
<pre><tt>
NCV_slab="inodecache:GAUGE,dentrycache:GAUGE"
</tt></pre>
<p>This tells the xymond_rrd module that it should create an RRD file with
two datasets of type GAUGE instead of the default (DERIVE). The setting must
be named NCV_<columnname>.</p>
<p>In some cases it can be useful to use multiple RRD files - one for
each dataset - instead of putting all of the datasets into one RRD file.
E.g. if you want to add or remove datasets over time - if they are
all stored in one RRD file then you have to dump, modify and reload the
data from the RRD file. If you store each dataset in a separate file,
then you can just delete the file. Xymon supports this if you call this
setting <b>SPLITNCV</b> instead of NCV. Then Xymon will store each
dataset in an RRD file <tt>column,dataset.rrd</tt>, e.g. <tt>slab,inodecache.rrd</tt>,
and the name of the dataset will be "<b>lambda</b>".
Apart from this, the setup is identical for the NCV- and SPLITNCV-methods.</p>
<p>The xymonserver.cfg file is not reloaded automatically, so you must restart
Xymon after making these changes. Or at least, kill the xymond_rrd processes
(there are usually two) - xymonlaunch will automatically restart them, and
they will then pick up the new settings.</p>
<h2>Check that the RRD collects data</h2>
<p>The next time the <b>slab</b> status is updated, Xymon will begin to collect
the data. You can check this by looking for the <b>slab.rrd</b> file in the
~xymon/data/rrd/HOSTNAME/ directory. If you want to check the data it
collects, the <b>rrdtool dump ~xymon/data/rrd/HOSTNAME/slab.rrd</b> will
tell you what it got:
<pre><tt>
<!-- Round Robin Database Dump -->
<rrd>
<version> 0001 </version>
<step> 300 </step> <!-- Seconds -->
<lastupdate> 1132474725 </lastupdate> <!-- 2005-11-20 09:18:45 CET -->
<ds>
<name> inodecache </name>
RRD datatype------> <type> GAUGE </type>
<minimal_heartbeat> 600 </minimal_heartbeat>
<min> 0.0000000000e+00 </min>
<max> NaN </max>
<!-- PDP Status -->
current value-----> <last_ds> 330624 </last_ds>
<value> 0.0000000000e+00 </value>
<unknown_sec> 0 </unknown_sec>
</ds>
</tt></pre>
<p>If you go and look at the status page for the <b>slab</b> column, you should
not see any graph yet, but a link to <b>xymon graph ncv:slab</b>. One final
step is missing.</p>
<h2>Setup a graph definition</h2>
<p>The final step is to tell Xymon how to create a graph from the data
in the RRD file. This is done in the ~xymon/server/etc/graphs.cfg
file.
<pre><tt>
[slab]
TITLE Slab info
YAXIS Bytes
DEF:inode=slab.rrd:inodecache:AVERAGE
DEF:dentry=slab.rrd:dentrycache:AVERAGE
LINE2:inode#00CCCC:Inode cache
LINE2:dentry#FF0000:Dentry cache
COMMENT:\n
GPRINT:inode:LAST:Inode cache \: %5.1lf%s (cur)
GPRINT:inode:MAX: \: %5.1lf%s (max)
GPRINT:inode:MIN: \: %5.1lf%s (min)
GPRINT:inode:AVERAGE: \: %5.1lf%s (avg)\n
GPRINT:dentry:LAST:Dentry cache\: %5.1lf%s (cur)
GPRINT:dentry:MAX: \: %5.1lf%s (max)
GPRINT:dentry:MIN: \: %5.1lf%s (min)
GPRINT:dentry:AVERAGE: \: %5.1lf%s (avg)\n
</tt></pre>
<p><b>[slab]</b> is the name of this graph, and it must match the name of your
status column if you want the graph to appear together with the status.
The TITLE and YAXIS settings define the graph title and the legend on
the Y-axis. The rest are definitions for the rrdgraph(1) tool - you should
read the RRDtool docs if you want to know in detail how it works.
For now, all you need to know is that you must pick out the data you
want from the RRD file with a <b>DEF</b> line, like
<pre><tt>
DEF:inode=slab.rrd:inodecache:AVERAGE
</tt></pre>
which gives you an "inode" definition that has the value from the
<b>inodecache</b> dataset in the slab.rrd file. This is then used to draw
a line on the graph:
<pre><tt>
LINE2:inode#00CCCC:Inode cache
</tt></pre>
The line gets the color <b>#00CCCC</b> (red-green-blue), which is a light
greenish-blue color. Note that you can have several lines in one graph,
if it makes sense to compare them. You can also use other types of
visual effects, e.g. stack values on top of each other (like the <b>vmstat</b>
graphs do) - this is described in the rrdgraph man-page. An online version
is at <a href="http://www.mrtg.org/rrdtool/doc/rrdgraph.en.html">
http://www.mrtg.org/rrdtool/doc/rrdgraph.en.html</a>.</p>
<p>The GPRINT lines at the end of the graph definition also uses the <b>inode</b>
value to print a summary line showing the current, maximum, minimum and
average values from the data that has been collected.</p>
<p>Once you have added this section to graphs.cfg, refresh the
status page in your browser, and the graph should show up.</p>
<h2>Add the graph to the collection of graphs on the trends column</h2>
<p>If you want the graph included with the other graphs on the <b>trends</b>
column, you must add it to the GRAPHS setting in the
~xymon/server/etc/xymonserver.cfg file.</p>
<pre><tt>
GRAPHS="la,disk,<... lots more ...>,xymonproxy,xymond,slab"
</tt></pre>
Save the file, and when you click on the <b>trends</b> column you should
see the slab graph at the bottom of the page.
<h2>Common problems and pitfalls</h2>
<h3>If your graph nearly always shows 0</h3>
<p>You probably used the wrong RRD datatype for your data - see step 4.
By default, the RRD file expects data that is increasing constantly;
if you are tracking some data that just varies up and down, you must
use the RRD GAUGE datatype. Note that when you change the RRD datatype,
you must delete any existing RRD files - the RRD datatype is defined
when the RRD file is created, and cannot be changed on the fly.</p>
<h3>No graph on the status page, but OK on the trends page</h3>
<p>Make sure you have <b>ncv</b> listed in the GRAPHS setting in xymonserver.cfg.
(Don't ask why - just take my word that it must be there).</p>
<h1>More advanced: Sending a "trends" message</h1>
<p>The NCV method works fine in many cases, but you may run
into a situation where it isn't really suitable. One possible
problem with this method is that you can only store data in
a single RRD file using the NCV method, and there may be
situations where you want to use multiple RRD files for
flexibility - e.g. if you are reporting performance for a
number of applications, then it is useful to have one RRD
file for each application. And since there are different
performance metrics for each application, you cannot use
the SPLITNCV method.</p>
<p>Xymon supports a different method for collecting data in
these cases - you can send a "trends" message to Xymon with
the data you want to put into a graph. All hosts appearing
on the Xymon server already have a "trends" status column,
but if you send a "trends" status message to Xymon, it will
not show up in the "trends" column - instead, it will be used
to create/update an RRD file with data. And inside the "trends"
message, you can define exactly what RRD files you want to
use, how they are formatted, what data goes into the RRD
file and so on. Here is an example of a "trends" message,
using the same data that we used in the example above for
collecting some data about the current weather:</p>
<pre>
data berlin.trends
[weather.rrd]
DS:temperature:GAUGE:600:U:U 19.2
DS:humidity:GAUGE:600:0:U 72
DS:wind:GAUGE:600:0:U 9.6
</pre>
<p>This creates an RRD file for the host "berlin", called "weather.rrd".
The RRD file has three datasets: temperature, humidity and wind,
all of which are of the GAUGE datatype. For more information about
the DS definitions, see the
.I rrdcreate(1)
manpage. The current values for each of the datasets is specified
after the DS definition.</p>
<p>If the RRD file already exists, then it just updates the data.</p>
<p>You still need to create graph definitions on the Xymon server,
and if you want the trend graphs displayed on the "trends" status
page then you must also add the graph name to the "TEST2RRD" and
"GRAPHS" settings in
.I xymonserver.cfg(5)
but it is not necessary to restart the xymond_rrd program.</p>
<p>If you want to create more than one RRD file, you just
add more data in the "trends" status message. E.g. if you have
three applications with performance metrics:</p>
<pre>
data appserver.trends
[salaryapp.rrd]
DS:usercount:GAUGE:600:0:U 210
DS:payrollgeneration:GAUGE:600:0:U 29
[bulkmailapp.rrd]
DS:customercount:GAUGE:600:0:U 1922
DS:mailsize:GAUGE:600:0:U 61293
DS:transmissiontime:GAUGE:600:0:U 88
</pre>
<p>This will create 2 RRD files, <b>salaryapp.rrd</b> with performance
metrics from the Salary application, and <b>bulkmailapp.rrd</b> with
performance metrics from the bulkmail application. Note that the
metrics collected are quite different - you have total control over
which RRD files get created, what datasets they contain etc.</p>
<p>To view the graphs, you still need to setup a definition for
generating the graphs in the <b>graphs.cfg</b> configuration file,
and the graph definition must be listed in both the TEST2RRD- and
GRAPHS-settings in xymonserver.cfg - then the graph will appear
on the "trends" page.</p>
</body>
</html>
|
