1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
|
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<title>7.15. Performance Data</title>
<link rel="stylesheet" href="../stylesheets/icinga-docs.css" type="text/css">
<meta name="generator" content="DocBook XSL Stylesheets V1.75.1">
<meta name="keywords" content="Supervision, Icinga, Nagios, Linux">
<link rel="home" href="index.html" title="Icinga Version 1.14 Documentation">
<link rel="up" href="ch07.html" title="Chapter 7. Advanced Topics">
<link rel="prev" href="stalking.html" title="7.14. State Stalking">
<link rel="next" href="downtime.html" title="7.16. Scheduled Downtime">
<script src="../js/jquery-min.js" type="text/javascript"></script><script src="../js/icinga-docs.js" type="text/javascript"></script>
</head>
<body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF">
<CENTER><IMG src="../images/logofullsize.png" border="0" alt="Icinga" title="Icinga"></CENTER>
<div class="navheader">
<table width="100%" summary="Navigation header">
<tr><th colspan="3" align="center">7.15. Performance Data</th></tr>
<tr>
<td width="20%" align="left">
<a accesskey="p" href="stalking.html">Prev</a> </td>
<th width="60%" align="center">Chapter 7. Advanced Topics</th>
<td width="20%" align="right"> <a accesskey="n" href="downtime.html">Next</a>
</td>
</tr>
</table>
<hr>
</div>
<div class="section" title="7.15. Performance Data">
<div class="titlepage"><div><div><h2 class="title" style="clear: both">
<a name="perfdata"></a>7.15. <a name="perf_data"></a>Performance Data</h2></div></div></div>
<div class="toc"><dl>
<dt><span class="section">7.15.1. <a href="perfdata.html#introduction">Introduction</a></span></dt>
<dt><span class="section">7.15.2. <a href="perfdata.html#typesperfdata">Types of Performance Data</a></span></dt>
<dt><span class="section">7.15.3. <a href="perfdata.html#pluginperfdata">Plugin Performance Data</a></span></dt>
<dt><span class="section">7.15.4. <a href="perfdata.html#processing">Processing Performance Data</a></span></dt>
<dt><span class="section">7.15.5. <a href="perfdata.html#processingcommands">Processing Performance Data Using Commands</a></span></dt>
<dt><span class="section">7.15.6. <a href="perfdata.html#writingtofiles">Writing Performance Data To Files</a></span></dt>
<dt><span class="section">7.15.7. <a href="perfdata.html#formatperfdata">Performance Data Format</a></span></dt>
</dl></div>
<div class="section" title="7.15.1. Introduction">
<div class="titlepage"><div><div><h3 class="title">
<a name="introduction"></a>7.15.1. Introduction</h3></div></div></div>
<p>Icinga is designed to allow <a class="link" href="plugins.html" title="5.1. Icinga Plugins">plugins</a> to return optional performance data in addition to
normal status data, as well as allow you to pass that performance data to external applications for processing. A description of the
different types of performance data, as well as information on how to go about processing that data is described below...</p>
</div>
<div class="section" title="7.15.2. Types of Performance Data">
<div class="titlepage"><div><div><h3 class="title">
<a name="typesperfdata"></a>7.15.2. Types of Performance Data</h3></div></div></div>
<p>There are two basic categories of performance data that can be obtained from Icinga:</p>
<div class="orderedlist"><ol class="orderedlist" type="1">
<li class="listitem">
<p>Check performance data</p>
</li>
<li class="listitem">
<p>Plugin performance data</p>
</li>
</ol></div>
<p>Check performance data is internal data that relates to the actual execution of a host or service check. This might include things
like service check latency (i.e. how "late" was the service check from its scheduled execution time) and the number of seconds a host or
service check took to execute. This type of performance data is available for all checks that are performed. The <a class="link" href="macrolist.html#macrolist-hostexecutiontime">$HOSTEXECUTIONTIME$</a> and <a class="link" href="macrolist.html#macrolist-serviceexecutiontime">$SERVICEEXECUTIONTIME$</a> <a class="link" href="macros.html" title="5.2. Understanding Macros and How They Work">macros</a> can be used to determine the
number of seconds a host or service check was running and the <a class="link" href="macrolist.html#macrolist-hostlatency">$HOSTLATENCY$</a> and <a class="link" href="macrolist.html#macrolist-servicelatency">$SERVICELATENCY$</a> macros can be used to determine how "late" a regularly-scheduled host or
service check was.</p>
<p>Plugin performance data is external data specific to the plugin used to perform the host or service check. Plugin-specific data
can include things like percent packet loss, free disk space, processor load, number of current users, etc. - basically any type of
metric that the plugin is measuring when it executes. Plugin-specific performance data is optional and may not be supported by all
plugins. Plugin-specific performance data (if available) can be obtained by using the <a class="link" href="macrolist.html#macrolist-hostperfdata">$HOSTPERFDATA$</a> and <a class="link" href="macrolist.html#macrolist-serviceperfdata">$SERVICEPERFDATA$</a> <a class="link" href="macros.html" title="5.2. Understanding Macros and How They Work">macros</a>. Read on for more information on how plugins can return performance data to Icinga for inclusion
in the $HOSTPERFDATA$ and $SERVICEPERFDATA$ macros.</p>
</div>
<div class="section" title="7.15.3. Plugin Performance Data">
<div class="titlepage"><div><div><h3 class="title">
<a name="pluginperfdata"></a>7.15.3. Plugin Performance Data</h3></div></div></div>
<p>At a minimum, Icinga plugins must return a single line of human-readable text that indicates the status of some type of
measurable data. For example, the check_ping plugin might return a line of text like the following:</p>
<pre class="screen"> PING ok - Packet loss = 0%, RTA = 0.80 ms </pre>
<p>With this simple type of output, the entire line of text is available in the $HOSTOUTPUT$ or $SERVICEOUTPUT$ <a class="link" href="macros.html" title="5.2. Understanding Macros and How They Work">macros</a> (depending on whether this plugin was used as a host check or service check).</p>
<p>Plugins can return optional performance data in their output by sending the normal, human-readable text string that they usually
would, followed by a pipe character (|), and then a string containing one or more performance data metrics. Let's take the check_ping
plugin as an example and assume that it has been enhanced to return percent packet loss and average round trip time as performance data
metrics. Sample output from the plugin might look like this:</p>
<pre class="screen"> PING ok - Packet loss = 0%, RTA = 0.80 ms | percent_packet_loss=0 rta=0.80</pre>
<p>When Icinga sees this plugin output format it will split the output into two parts:</p>
<div class="orderedlist"><ol class="orderedlist" type="1">
<li class="listitem">
<p>Everything before the pipe character is considered to be the "normal" plugin output and will be stored in either the
$HOSTOUTPUT$ or $SERVICEOUTPUT$ macro</p>
</li>
<li class="listitem">
<p>Everything after the pipe character is considered to be the plugin-specific performance data and will be stored in the
$HOSTPERFDATA$ or $SERVICEPERFDATA$ macro</p>
</li>
</ol></div>
<p>In the example above, the $HOSTOUTPUT$ or $SERVICEOUTPUT$ macro would contain "<span class="emphasis"><em>PING ok - Packet loss = 0%, RTA = 0.80
ms</em></span>" (without quotes) and the $HOSTPERFDATA$ or $SERVICEPERFDATA$ macro would contain "<span class="emphasis"><em>percent_packet_loss=0
rta=0.80</em></span>" (without quotes).</p>
<p>Multiple lines of performance data (as well as normal text output) can be obtained from plugins, as described in the <a class="link" href="pluginapi.html" title="11.1. Icinga Plugin API">plugin API documentation</a>.</p>
<div class="note" title="Note" style="margin-left: 0.5in; margin-right: 0.5in;"><table border="0" summary="Note">
<tr>
<td rowspan="2" align="center" valign="top" width="25"><img alt="[Note]" src="../images/note.png"></td>
<th align="left">Note</th>
</tr>
<tr><td align="left" valign="top">
<p>The Icinga daemon doesn't directly process plugin performance data, so it doesn't really care what the performance data
looks like. The developer guidelines contain several rules (see <a class="link" href="perfdata.html#perfdata-format">performance data format</a> below
for details). However, if you are using an external addon to process the performance data (i.e. PerfParse), the addon may be expecting
that the plugin returns performance data in a specific format. Check the documentation that comes with the addon for more
information.</p>
</td></tr>
</table></div>
</div>
<div class="section" title="7.15.4. Processing Performance Data">
<div class="titlepage"><div><div><h3 class="title">
<a name="processing"></a>7.15.4. Processing Performance Data</h3></div></div></div>
<p>If you want to process the performance data that is available from Icinga and the plugins, you'll need to do the
following:</p>
<div class="orderedlist"><ol class="orderedlist" type="1">
<li class="listitem">
<p>Enable the <a class="link" href="configmain.html#configmain-process_performance_data">process_performance_data</a> option.</p>
</li>
<li class="listitem">
<p>Configure Icinga so that performance data is either written to files and/or processed by executing commands.</p>
</li>
</ol></div>
<p>Read on for information on how to process performance data by writing to files or executing commands.</p>
</div>
<div class="section" title="7.15.5. Processing Performance Data Using Commands">
<div class="titlepage"><div><div><h3 class="title">
<a name="processingcommands"></a>7.15.5. Processing Performance Data Using Commands</h3></div></div></div>
<p>The most flexible way to process performance data is by having Icinga execute commands (that you specify) to process or
redirect the data for later processing by external applications. The commands that Icinga executes to process host and service
performance data are determined by the <a class="link" href="configmain.html#configmain-host_perfdata_command">host_perfdata_command</a> and <a class="link" href="configmain.html#configmain-service_perfdata_command">service_perfdata_command</a> options, respectively.</p>
<p>An example command definition that redirects service check performance data to a text file for later processing by another
application is shown below:</p>
<pre class="programlisting"> define command{
command_name store-service-perfdata
command_line /bin/echo -e "$LASTSERVICECHECK$\t$HOSTNAME$\t$SERVICEDESC$\t$SERVICESTATE$\t$SERVICEATTEMPT$\t$SERVICESTATETYPE$\t$SERVICEEXECUTIONTIME$\t$SERVICELATENCY$\t$SERVICEOUTPUT$\t$SERVICEPERFDATA$" >> /usr/local/icinga/var/service-perfdata.dat
}</pre>
<div class="tip" title="Tip" style="margin-left: 0.5in; margin-right: 0.5in;"><table border="0" summary="Tip">
<tr>
<td rowspan="2" align="center" valign="top" width="25"><img alt="[Tip]" src="../images/tip.png"></td>
<th align="left">Tip</th>
</tr>
<tr><td align="left" valign="top">
<p>This method, while flexible, comes with a relatively high CPU overhead. If you're processing performance data for a large number
of hosts and services, you'll probably want Icinga to write performance data to files instead. This method is described in the
next section.</p>
</td></tr>
</table></div>
</div>
<div class="section" title="7.15.6. Writing Performance Data To Files">
<div class="titlepage"><div><div><h3 class="title">
<a name="writingtofiles"></a>7.15.6. Writing Performance Data To Files</h3></div></div></div>
<p>You can have Icinga write all host and service performance data directly to text files using the <a class="link" href="configmain.html#configmain-host_perfdata_file">host_perfdata_file</a> and <a class="link" href="configmain.html#configmain-service_perfdata_file">service_perfdata_file</a> options. The format in which host and service performance data
is written to those files is determined by the <a class="link" href="configmain.html#configmain-host_perfdata_file_template">host_perfdata_file_template</a>
and <a class="link" href="configmain.html#configmain-service_perfdata_file_template">service_perfdata_file_template</a> options.</p>
<p>An example file format template for service performance data might look like this:</p>
<pre class="programlisting"> service_perfdata_file_template=[SERVICEPERFDATA]\t$TIMET$\t$HOSTNAME$\t$SERVICEDESC$\t$SERVICEEXECUTIONTIME$\t$SERVICELATENCY$\t$SERVICEOUTPUT$\t$SERVICEPERFDATA$
</pre>
<div class="note" title="Note" style="margin-left: 0.5in; margin-right: 0.5in;"><table border="0" summary="Note">
<tr>
<td rowspan="2" align="center" valign="top" width="25"><img alt="[Note]" src="../images/note.png"></td>
<th align="left">Note</th>
</tr>
<tr><td align="left" valign="top">
<p>The above is just one example of how to define a template. This definition will NOT work for PNP!</p>
</td></tr>
</table></div>
<p>By default, the text files will be opened in "append" mode. If you need to change the modes to "write" or "non-blocking
read/write" (useful when writing to pipes), you can use the <a class="link" href="configmain.html#configmain-host_perfdata_file_mode">host_perfdata_file_mode</a> and <a class="link" href="configmain.html#configmain-service_perfdata_file_mode">service_perfdata_file_mode</a> options.</p>
<p>Additionally, you can have Icinga periodically execute commands to periocially process the performance data files (e.g.
rotate them) using the <a class="link" href="configmain.html#configmain-host_perfdata_file_processing_command">host_perfdata_file_processing_command</a> and
<a class="link" href="configmain.html#configmain-service_perfdata_file_processing_command">service_perfdata_file_processing_command</a> options. The
interval at which these commands are executed are governed by the <a class="link" href="configmain.html#configmain-host_perfdata_file_processing_interval">host_perfdata_file_processing_interval</a> and <a class="link" href="configmain.html#configmain-service_perfdata_file_processing_interval">service_perfdata_file_processing_interval</a> options,
respectively.</p>
</div>
<div class="section" title="7.15.7. Performance Data Format">
<div class="titlepage"><div><div><h3 class="title">
<a name="formatperfdata"></a>7.15.7. <a name="perfdata-format"></a>Performance Data Format</h3></div></div></div>
<p>The Monitoring Plugins Developer Guidelines (<a class="link" href="https://www.monitoring-plugins.org/doc/guidelines.html" target="_top">https://www.monitoring-plugins.org/doc/guidelines.html</a>) contain in <a class="link" href="https://www.monitoring-plugins.org/doc/guidelines.html#AEN200" target="_top">section 2.6</a> the specification and an explanation of
the format of performance data. It is repeated below for your convenience:</p>
<p>----- 8< -----</p>
<p><span class="emphasis"><em>2.6.Performance data</em></span></p>
<p>Nagios 3 and newer will concatenate the parts following a "|" in a) the first line output by the plugin, and b) in the second to
last line, into a string it passes to whatever performance data processing it has configured. (Note that it currently does not insert
additional whitespace between both, so the plugin needs to provide some to prevent the last pair of a) and the first of b) getting run
together.) Please refer to the Nagios documentation for information on how to configure such processing. However, it is the
responsibility of the plugin writer to ensure the performance data is in a "Nagios Plugins" format. This is the expected format:</p>
<p>'label'=value[UOM];[warn];[crit];[min];[max]</p>
<p>Notes:</p>
<div class="orderedlist"><ol class="orderedlist" type="1">
<li class="listitem">
<p>space separated list of label/value pairs</p>
</li>
<li class="listitem">
<p>label can contain any characters</p>
</li>
<li class="listitem">
<p>the single quotes for the label are optional. Required if spaces, = or ' are in the label</p>
</li>
<li class="listitem">
<p>label length is arbitrary, but ideally the first 19 characters are unique (due to a limitation in RRD). Be aware of a
limitation in the amount of data that NRPE returns to Nagios</p>
</li>
<li class="listitem">
<p>to specify a quote character, use two single quotes</p>
</li>
<li class="listitem">
<p>warn, crit, min, and/or max, respectively, may be null (for example, if the threshold is not defined or min and max do not
apply). Trailing unfilled semicolons can be dropped</p>
</li>
<li class="listitem">
<p>min and max are not required if UOM=%</p>
</li>
<li class="listitem">
<p>value, min and max in class [-0-9.]. Must all be the same UOM</p>
</li>
<li class="listitem">
<p>warn and crit are in the range format (see Section 2.5). Must be the same UOM</p>
</li>
<li class="listitem">
<p>UOM (unit of measurement) is one of:</p>
<div class="itemizedlist"><ul class="itemizedlist" type="disc">
<li class="listitem">
<p>no unit specified - assume a number (int or float) of things (eg, users, processes, load averages)</p>
</li>
<li class="listitem">
<p>s - seconds (also us, ms)</p>
</li>
<li class="listitem">
<p>% - percentage</p>
</li>
<li class="listitem">
<p>B - bytes (also KB, MB, TB, GB?)</p>
</li>
<li class="listitem">
<p>c - a continous counter (such as bytes transmitted on an interface)</p>
</li>
</ul></div>
</li>
</ol></div>
<p>It is up to third party programs to convert the Nagios plugins performance data into graphs.</p>
<p>----- 8< -----</p>
<p>Every now and then people are curious if their performance data is valid so we'll give some examples showing the data following
the pipe symbol:</p>
<pre class="screen"> loss=0 rta=0.80ms # valid (rule 1,10)
loss=0, rta=0.80ms # invalid (rule 1, no comma allowed)
loss=0 rta=0,80ms # invalid (rule 8, no comma allowed) *1
packet loss=0 rta=0.80 # invalid (rule 3, single quotes needed)
'packet loss'=0 rta=0.80 # valid version of line above
'john''s disk'=83% # valid (rule 5,10)
'disk usage'=78%;80;90;;; # invalid (trailing semicolon after max value)
'disk usage'=78%;80;90 # valid (rule 6)
'data packets'=11345234c # valid (rule 6,10)
drum=153482pages # invalid (rule 10, wrong UOM) *2
temperature=23;;;20;30 # valid (rule 6)</pre>
<div class="note" title="Note" style="margin-left: 0.5in; margin-right: 0.5in;"><table border="0" summary="Note">
<tr>
<td rowspan="2" align="center" valign="top" width="25"><img alt="[Note]" src="../images/note.png"></td>
<th align="left">Note</th>
</tr>
<tr><td align="left" valign="top">
<p>*1: Having a decimal comma instead of a decimal point may be a result of using your local language settings. It might be fixed
internally by your graphing tool but don't rely on it.</p>
<p>*2: Having "pages" or other non-standard UOMs (e.g. °C/°F) will probably result in errors or discarded performance data when
using addon like <a class="link" href="http://docs.pnp4nagios.org/" target="_top">PNP4Nagios</a> or other tools which try to graph your performance
data.</p>
</td></tr>
</table></div>
<a class="indexterm" name="idm140381624193040"></a>
<a class="indexterm" name="idm140381624194176"></a>
</div>
</div>
<div class="navfooter">
<hr>
<table width="100%" summary="Navigation footer">
<tr>
<td width="40%" align="left">
<a accesskey="p" href="stalking.html">Prev</a> </td>
<td width="20%" align="center"><a accesskey="u" href="ch07.html">Up</a></td>
<td width="40%" align="right"> <a accesskey="n" href="downtime.html">Next</a>
</td>
</tr>
<tr>
<td width="40%" align="left" valign="top">7.14. State Stalking </td>
<td width="20%" align="center"><a accesskey="h" href="index.html">Home</a></td>
<td width="40%" align="right" valign="top"> 7.16. Scheduled Downtime</td>
</tr>
</table>
</div>
<P class="copyright">© 1999-2009 Ethan Galstad, 2009-2017 Icinga Development Team, https://www.icinga.com</P>
</body>
</html>
|
