1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
|
<html>
<head>
<title>Service Check Scheduling</title>
<STYLE type="text/css">
<!--
.Default { font-family: verdana,arial,serif; font-size: 8pt; }
.PageTitle { font-family: verdana,arial,serif; font-size: 12pt; font-weight: bold; }
-->
</STYLE>
</head>
<body bgcolor="#FFFFFF" text="black" class="Default">
<p>
<div align="center">
<h2 class="PageTitle">Service Check Scheduling</h2>
</div>
</p>
<hr>
<p>
<strong><u>Index</u></strong>
</p>
<p>
<a href="#introduction">Introduction</a><br>
<a href="#configuration_options">Configuration options</a><br>
<a href="#initial_scheduling">Initial scheduling</a><br>
<a href="#inter_check_delay">Inter-check delay</a><br>
<a href="#service_interleaving">Service interleaving</a><br>
<a href="#max_concurrent_checks">Max concurrent service checks</a><br>
<a href="#time_restraints">Time restraints</a><br>
<a href="#normal_scheduling">Normal scheduling</a><br>
<a href="#problem_scheduling">Scheduling during problems</a><br>
<a href="#host_checks">Host checks</a><br>
<a href="#scheduling_delays">Scheduling delays</a><br>
<a href="#scheduling_example">Scheduling example</a><br>
<a href="#service_options">Service definition options that affect scheduling</a><br>
</p>
<p>
<a name="introduction"></a>
<strong><u>Introduction</u></strong>
</p>
<p>
I've gotten a lot of questions regarding how service checks are scheduled in certain situations, along with how the scheduling differs from when the checks are actually executed and their results are processed. I'll try to go into a little more detail on how this all works...
</p>
<p>
<a name="configuration_options"></a>
<strong><u>Configuration Options</u></strong>
</p>
<p>
Before we begin, there are several configuration options that affect how service checks are scheduled, executed, and processed. For starters, each service definition contains three options that determine when and how each specific service check is scheduled and executed. Those three options include:
</p>
<p>
<ul>
<li><i>normal_check_interval</i>
<li><i>retry_check_interval</i>
<li><i>check_period</i>
</ul>
</p>
<p>
There are also four configuration options in the <a href="configmain.html">main configuration file</a> that affect service checks. These include:
</p>
<p>
<ul>
<li><a href="configmain.html#inter_check_delay_method"><i>inter_check_delay_method</i></a>
<li><a href="configmain.html#service_interleave_factor"><i>service_interleave_factor</i></a>
<li><a href="configmain.html#max_concurrent_checks"><i>max_concurrent_checks</i></a>
<li><a href="configmain.html#service_reaper_frequency"><i>service_reaper_frequency</i></a>
</ul>
</p>
<p>
We'll go into more detail on how all these options affect service check scheduling as we progress. First off, let's see how services are initially scheduled when Nagios first starts or restarts...
</p>
<p>
<a name="initial_scheduling"></a>
<strong><u>Initial Scheduling</u></strong>
</p>
<p>
When Nagios (re)starts, it will attempt to schedule the initial check of all services in a manner that will minimize the load imposed on the local and remote hosts. This is done by spacing the initial service checks out, as well as interleaving them. The spacing of service checks (also known as the inter-check delay) is used to minimize/equalize the load on the local host running Nagios and the interleaving is used to minimize/equalize load imposed on remote hosts. Both the inter-check delay and interleave functions are discussed below.
</p>
<p>
Even though service checks are initially scheduled to balance the load on both the local and remote hosts, things will eventually give in to the ensuing chaos and be a bit random. Reasons for this include the fact that services are not all checked at the same interval, some services take longer to execute than others, host and/or service problems can alter the timing of one or more service checks, etc. At least we try to get things off to a good start. Hopefully the initial scheduling will keep the load on the local and remote hosts fairly balanced as time goes by...
</p>
<p>
<strong>Note:</strong> If you want to view the initial service check scheduling information, start Nagios using the <b>-s</b> command line option. Doing so will display basic scheduling information (inter-check delay, interleave factor, first and last service check time, etc) and will create a new status log that shows the exact time that all services are initially scheduled. Because this option will overwrite the status log, you should not use it when another copy of Nagios is running. Nagios does <i>not</i> start monitoring anything when this argument is used.
</p>
<p>
<a name="inter_check_delay"></a>
<strong><u>Inter-Check Delay</u></strong>
</p>
<p>
As mentioned before, Nagios attempts to equalize the load placed on the machine that is running Nagios by equally spacing out initial service checks. The spacing between consecutive service checks is called the inter-check delay. By giving a value to the <a href="configmain.html#inter_check_delay_method">inter_check_delay_method</a> variable in the main config file, you can modify how this delay is calculated. I will discuss how the "smart" calculation works, as this is the setting you will want to use for normal operation.
</p>
<p>
When using the "smart" setting of the <i>inter_check_delay_method</i> variable, Nagios will calculate an inter-check delay value by using the following calculation:
</p>
<p>
<i>inter-check delay = (average check interval for all services) / (total number of services)
</p>
<p>
Let's take an example. Say you have 1,000 services that each have a normal check interval of 5 minutes (obviously some services are going to be checked at different intervals, but let's look at an easy case...). The total check interal time for all services is 5,000 (1,000 * 5). That means that the average check interval for each service is 5 minutes (5,000 / 1,000). Give that information, we realize that (on average) we need to re-check 1,000 services every 5 minutes. This means that we should use an inter-check delay of 0.005 minutes (0.3 seconds) when spacing out the initial service checks. By spacing each service check out by 0.3 seconds, we can somewhat guarantee that Nagios is scheduling and/or executing 3 new service checks every second. By spacing the checks out evenly over time like this, we can hope that the load on the local server that is running Nagios remains somewhat balanced.
</p>
<p>
<a name="service_interleaving"></a>
<strong><u>Service Interleaving</u></strong>
</p>
<p>
As discussed above, the inter-check delay helps to equalize the load that Nagios imposes on the local host. What about remote hosts? Is it necessary to equalize load on remote hosts? Why? Yes, it is important and yes, Nagios can help out with this. Equalizing load on remote hosts is especially important with the advent of <a href="parallelization.html">service check parallelization</a>. If you monitor a large number of services on a remote host and the checks were not spread out, the remote host might think that it was the victim of a SYN attack if there were a lot of open connections on the same port. Plus, attempting to equalize the load on hosts is just a nice thing to do...
</p>
<p>
By giving a value to the <a href="configmain.html#service_interleave_factor">service_interleave_factor</a> variable in the main config file, you can modify how the interleave factor is calculated. I will discuss how the "smart" calculation works, as this will probably be the setting you will want to use for normal operation. You can, however, use a pre-set interleave factor instead of having Nagios calculate one for you. Also of note, if you use an interleave factor of 1, service check interleaving is basically disabled.
</p>
<p>
When using the "smart" setting of the <i>service_interleave_factor</i> variable, Nagios will calculate an interleave factor by using the following calculation:
</p>
<p>
<i>interleave factor = ceil ( total number of services / total number of hosts )</i>
</p>
<p>
Let's take an example. Say you have a total of 1,000 services and 150 hosts that you monitor. Nagios would calculate the interleave factor to be 7. This means that when Nagios schedules initial service checks it will schedule the first one it finds, skip the next 6, schedule the next one, and so on... This process will keep repeating until all service checks have been scheduled. Since services are sorted (and thus scheduled) by the name of the host they are associated with, this will help with minimizing/equalizing the load placed upon remote hosts.
</p>
<p>
The images below depict how service checks are scheduled when they are not interleaved (<i>service_interleave_factor</i>=1) and when they are interleaved with the <i>service_interleave_factor</i> variable equal to 4.
</p>
<p>
<table border=0>
<tr><td><strong>Non-Interleaved Checks:</strong></td><td><strong>Interleaved Checks:</strong></td></tr>
<tr>
<td><a href="images/noninterleaved1.png"><img src="images/noninterleaved1.png" border=1 alt="Non-Interleaved Checks" width=300 height=125></a></td>
<td><a href="images/interleaved1.png"><img src="images/interleaved1.png" border=1 alt="Interleaved Checks" width=300 height=125></a></td>
</tr>
<tr>
<td><a href="images/noninterleaved2.png"><img src="images/noninterleaved2.png" border=1 alt="Non-Interleaved Checks" width=300 height=125></a></td>
<td><a href="images/interleaved2.png"><img src="images/interleaved2.png" border=1 alt="Interleaved Checks" width=300 height=125></a></td>
</tr>
<tr>
<td></td>
<td><a href="images/interleaved3.png"><img src="images/interleaved3.png" border=1 alt="Interleaved Checks" width=300 height=125></a></td>
</tr>
</table>
</p>
<p>
<a name="max_concurrent_checks"></a>
<strong><u>Maximum Concurrent Service Checks</u></strong>
</p>
<p>
In order to prevent Nagios from consuming all of your CPU resources, you can restrict the maximum number of concurrent service checks that can be running at any given time. This is controlled by using the <a href="configmain.html#max_concurrent_checks">max_concurrent_checks</a> option in the main config file.
</p>
<p>
The good thing about this setting is that you can regulate Nagios' CPU usage. The down side is that service checks may fall behind if this value is set too low. When it comes time to execute a service check, Nagios will make sure that no more than <i>x</i> service checks are either being executed or waiting to have their results processed (where <i>x</i> is the number of checks you specified for the <i>max_concurrent_checks</i> option). If that limit has been reached, Nagios will postpone the execution of any pending checks until some of the previous checks have completed. So how does one determine a reasonable value for the <i>max_concurrent_checks</i> option?
</p>
<p>
First off, you need to know the following things...
</p>
<p>
<ul>
<li>The inter-check delay that Nagios uses to initially schedule service checks (use the <b>-s</b> command line argument to check this)
<li>The frequency (in seconds) of service reaper events, as specified by the <a href="configmain.html#service_reaper_frequency">service_reaper_frequency</a> variable in the main config file.
<li>A general idea of the average time that service checks actually take to execute (most plugins timeout after 10 seconds, so the average is probably going to be lower)
</ul>
</p>
<p>
Next, use the following calculation to determine a reasonable value for the maximum number of concurrent checks that are allowed...
</p>
<p>
<i>max. concurrent checks = ceil( max( service reaper frequency , average check execution time ) / inter-check delay )</i>
</p>
<p>
The calculated number should provide a reasonable starting point for the <i>max_concurrent_checks</i> variable. You may have to increase this value a bit if service checks are still falling behind schedule or decrease it if Nagios is hogging too much CPU time.
</p>
<p>
Let's say you are monitoring 875 services, each with an average check interval of 2 minutes. That means that your inter-check delay is going to be 0.137 seconds. If you set the service reaper frequency to be 10 seconds, you can calculate a rough value for the max. number of concurrent checks as follows (I'll assume that the average execution time for service checks is less than 10 seconds) ...
</p>
<p>
<i>max. concurrent checks = ceil( 10 / 0.137 )</i>
</p>
<p>
In this case, the calculated value is going to be 73. This makes sense because (on average) Nagios are going to be executing just over 7 new service checks per second and it only processes service check results every 10 seconds. That means at given time there will be a just over 70 service checks that are either being executed or waiting to have their results processed. In this case, I would probably recommend bumping the max. concurrent checks value up to 80, since there will be delays when Nagios processes service check results and does its other work. Obviously, you're going to have test and tweak things a bit to get everything running smoothly on your system, but hopefully this provided some general guidelines...
</p>
<p>
<a name="time_restraints"></a>
<strong><u>Time Restraints</u></strong>
</p>
<p>
The <i>check_period</i> option determines the <a href="timeperiods.html">time period</a> during which Nagios can run checks of the service. Regardless of what status a particular service is in, if the time that it is actually executed is not a vaid time within the time period that has been specified, the check will <i>not</i> be executed. Instead, Nagios will reschedule the service check for the next valid time in the time period. If the check can be run (e.g. the time is valid within the time period), the service check is executed.
</p>
<p>
<strong>Note:</strong> Even though a service check may not be able to be executed at a given time, Nagios may still <i>schedule</i> it to be run at that time. This is most likely to happen during the initial scheduling of services, although it may happen in other instances as well. This does <i>not</i> mean that Nagios will execute the check! When it comes time to actually <i>execute</i> a service check, Nagios will verify that the check can be run at the current time. If it cannot, Nagios will not execute the service check, but will instead just reschedule it for a later time. Don't let this one throw you confuse you! The scheduling and execution of service checks are two distinctly different (although related) things.
</p>
<p>
<a name="normal_scheduling"></a>
<strong><u>Normal Scheduling</u></strong>
</p>
<p>
In an ideal world you wouldn't have network problems. But if that were the case, you wouldn't need a network monitoring tool. Anyway, when things are running smoothly and a service is in an OK state, we'll call that "normal". Service checks are normally scheduled at the frequency specified by the <i>check_interval</i> option. That's it. Simple, huh?
</p>
<p>
<a name="problem_scheduling"></a>
<strong><u>Scheduling During Problems</u></strong>
</p>
<p>
So what happens when there are problems with a service? Well, one of the things that happens is the service check scheduling changes. If you've configured the <i>max_attempts</i> option of the service definition to be something greater than 1, Nagios will recheck the service before deciding that a real problem exists. While the service is being rechecked (up to <i>max_attempts</i> times) it is considered to be in a "soft" state (as described <a href="statetypes.html">here</a>) and the service checks are rescheduled at a frequency determined by the <i>retry_interval</i> option.
</p>
<p>
If Nagios rechecks the service <i>max_attempts</i> times and it is still in a non-OK state, Nagios will put the service into a "hard" state, send out notifications to contacts (if applicable), and start rescheduling future checks of the service at a frequency determined by the <i>check_interval</i> option.
</p>
<p>
As always, there are exceptions to the rules. When a service check results in a non-OK state, Nagios will check the host that the service is associated with to determine whether or not is up (see the note <a href="#host_checks">below</a> for info on how this is done). If the host is not up (i.e. it is either down or unreachable), Nagios will immediately put the service into a hard non-OK state and it will reset the current attempt number to 1. Since the service is in a hard non-OK state, the service check will be rescheduled at the normal frequency specified by the <i>check_interval</i> option instead of the <i>retry_interval</i> option.
</p>
<p>
<a name="host_checks"></a>
<strong><u>Host Checks</u></strong>
</p>
<p>
Unlike service checks, host checks are <i>not</i> scheduled on a regular basis. Instead they are run on demand, as Nagios sees a need. This is a common question asked by users, so it needs to be clarified.
</p>
<p>
One instance where Nagios checks the status of a host is when a service check results in a non-OK status. Nagios checks the host to decide whether or not the host is up, down, or unreachable. If the first host check returns a non-OK state, Nagios will keep pounding out checks of the host until either (a) the maximum number of host checks (specified by the <i>max_attempts</i> option in the host definition) is reached or (b) a host check results in an OK state.
</p>
<p>
Also of note - when Nagios is check the status of a host, it holds off on doing anything else (executing new service checks, processing other service check results, etc). This can slow things down a bit and cause pending service checks to be delayed for a while, but it is necessary to determine the status of the host before Nagios can take any further action on the service(s) that are having problems.
</p>
<p>
<a name="scheduling_delays"></a>
<strong><u>Scheduling Delays</u></strong>
</p>
<p>
It should be noted that service check scheduling and execution is done on a best effort basis. Individual service checks are considered to be low priority events in Nagios, so they can get delayed if high priority events need to be executed. Examples of high priority events include log file rotations, external command checks, and service reaper events. Additionally, host checks will slow down the execution and processing of service checks.
</p>
<p>
<a name="scheduling_example"></a>
<strong><u>Scheduling Example</u></strong>
</p>
<p>
The scheduling of service checks, their execution, and the processing of their results can be a bit difficult to understand, so let's look at a simple example. Look at the diagram below - I'll refer to it as I explain how things are done.
</p>
<p>
<table border=0 cellspacing=10>
<tr><td><strong>Image 5.</strong></td></tr>
<tr><td><img src="images/checktiming.png" border=0></td></tr>
</table>
</p>
<p>
First off, the <b>X</b><sub>n</sub> events are service reaper events that are scheduled at a frequency specified by the <a href="configmain.html#service_reaper_frequency">service_reaper_frequency</a> option in the main config file. Service reaper events do the work of gathering and processing service check results. They serve as the core logic for Nagios, kicking off host checks, event handlers and notifications as necessary.
</p>
<p>
For the example here, a service has been scheduled to be executed at time <b>A</b>. However, Nagios got behind in its event queue, so the check was not actually executed until time <b>B</b>. The service check finished executing at time <b>C</b>, so the difference between points <b>C</b> and <b>B</b> is the actual amount of time that the check was running.
</p>
<p>
The results of the service check are not processed immediately after the check is done executing. Instead, the results are saved for later processing by a service reaper event. The next service reaper event occurs at time <b>D</b>, so that is approximately the time that the results are processed (the actual time may be later than <b>D</b> since other service check results may be processed before this one).
</p>
<p>
At the time that the service reaper event processes the service check results, it will reschedule the next service check and place it into Nagios' event queue. We'll assume that the service check resulted in an OK status, so the next check at time <b>E</b> is scheduled after the originally scheduled check time by a length of time specified by the <i>check_interval</i> option. Note that the service is <i>not</i> rescheduled based off the time that it was actually executed! There is one exception to this (isn't there always?) - if the time that the service check is actually executed (point <b>B</b>) occurs after the next service check time (point <b>E</b>), Nagios will compensate by adjusting the next check time. This is done to ensure that Nagios doesn't go nuts trying to keep up with service checks if it comes under heavy load. Besides, what's the point of scheduling something in the past...?
</p>
<p>
<a name="service_options"></a>
<strong><u>Service Definition Options That Affect Scheduling</u></strong>
</p>
<p>
Each service definition contains a <i>normal_check_interval</i> and <i>retry_check_interval</i> option. Hopefully this will clarify what these two options do, how they relate to the <i>max_check_attempts</i> option in the service definition, and how they affect the scheduling of the service.
</p>
<p>
First off, the <i>normal_check_interval</i> option is the interval at which the service is checked under "normal" circumstances. "Normal" circumstances mean whenever the service is in an OK state or when its in a <a href="statetypes.html">hard</a> non-OK state.
</p>
<p>
When a service first changes from an OK state to a non-OK state, Nagios gives you the ability to temporarily slow down or speed up the interval at which subsequent checks of that service will occur. When the service first changes state, Nagios will perform up to <i>max_check_attempts</i>-1 retries of the service check before it decides its a real problem. While the service is being retried, it is scheduled according to the <i>retry_check_interval</i> option, which might be faster or slower than the normal <i>normal_check_interval</i> option. While the service is being rechecked (up to <i>max_check_attempts</i>-1 times), the service is in a <a href="statetypes.html">soft state</a>. If the service is rechecked <i>max_check_attempts</i>-1 times and it is still in a non-OK state, the service turns into a <a href="statetypes.html">hard state</a> and is subsequently rescheduled at the normal rate specified by the <i>check_interval</i> option.
</p>
<p>
On a side note, it you specify a value of 1 for the <i>max_check_attempts</i> option, the service will not ever be checked at the interval specified by the <i>retry_check_interval</i> option. Instead, it immediately turns into a <a href="statetypes.html">hard state</a> and is subsequently rescheduled at the rate specified by the <i>normal_check_interval</i> option.
</p>
<hr>
</body>
</html>
|
