1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
|
<HTML>
<BODY BGCOLOR=white>
<PRE>
<!-- Manpage converted by man2html 3.0.1 -->
NAME
reporting - Sun Grid Engine reporting file format
DESCRIPTION
A Sun Grid Engine system writes a reporting file to
$SGE_ROOT/default/common/reporting. The reporting file con-
tains data that can be used for accounting, monitoring and
analysis purposes. It contains information about the clus-
ter (hosts, queues, load values, consumables, etc.), about
the jobs running in the cluster and about sharetree confi-
guration and usage. All information is time related, events
are dumped to the reporting file in a configurable interval.
It allows to monitor a "real time" status of the cluster as
well as historical analysis.
FORMAT
The reporting file is an ASCII file. Each line contains one
record, and the fields of a record are separated by a delim-
iter (:). The reporting file contains records of different
type. Each record type has a specific record structure.
The first two fields are common to all reporting records:
time Time (GMT unix timestamp) when the record was created.
record type
Type of the accounting record. The different types of
records and their structure are described in the fol-
lowing text.
new_job
The new_job record is written whenever a new job enters the
system (usually by a submitting command). It has the follow-
ing fields:
submission_time
Time (GMT unix time stamp) when the job was submitted.
job_number
The job number.
task_number
The array task id. Always has the value -1 for new_job
records (as we don't have array tasks yet).
pe_taskid
The task id of parallel tasks. Always has the value
"none" for new_job records.
job_name
The job name (from -N submission option)
owner
The job owner.
group
The unix group of the job owner.
project
The project the job is running in.
department
The department the job owner is in.
account
The account string specified for the job (from -A sub-
mission option).
priority
The job priority (from -p submission option).
job_log
The job_log record is written whenever a job, an array task
or a pe tasks is changing status. A status change can be the
transition from pending to running, but can also be trig-
gered by user actions like suspension of a job. It has the
following fields:
event_time
Time (GMT unix time stamp) when the event was gen-
erated.
event
A one word description of the event.
job_number
The job number.
task_number
The array task id. Always has the value -1 for new_job
records (as we don't have array tasks yet).
pe_taskid
The task id of parallel tasks. Always has the value
"none" for new_job records.
state
The state of the job after the event was processed.
user The user who initiated the event (or special usernames
"qmaster", "scheduler" and "execd" for actions of the
system itself like scheduling jobs, executing jobs
etc.).
host The host from which the action was initiated (e.g. the
submit host, the qmaster host, etc.).
state_time
Reserved field for later use.
submission_time
Time (GMT unix time stamp) when the job was submitted.
job_name
The job name (from -N submission option)
owner
The job owner.
group
The unix group of the job owner.
project
The project the job is running in.
department
The department the job owner is in.
account
The account string specified for the job (from -A sub-
mission option).
priority
The job priority (from -p submission option).
message
A message describing the reported action.
acct
Records of type acct are accounting records. Normally, they
are written whenever a job, a task of an array job, or the
task of a parallel job terminates. However, for long running
jobs an intermediate acct record is created once a day after
a midnight. This results in multiple accounting records for
a particular job and allows for a fine-grained resource
usage monitoring over time. Accounting records comprise the
following fields:
qname
Name of the cluster queue in which the job has run.
hostname
Name of the execution host.
group
The effective group id of the job owner when executing
the job.
owner
Owner of the Sun Grid Engine job.
job_name
Job name.
job_number
Job identifier - job number.
account
An account string as specified by the <B><A HREF="../htmlman1/qsub.html?pathrev=V62u5_TAG">qsub(1)</A></B> or
<B><A HREF="../htmlman1/qalter.html?pathrev=V62u5_TAG">qalter(1)</A></B> -A option.
priority
Priority value assigned to the job corresponding to the
priority parameter in the queue configuration (see
<B><A HREF="../htmlman5/queue_conf.html?pathrev=V62u5_TAG">queue_conf(5)</A></B>).
submission_time
Submission time (GMT unix time stamp).
start_time
Start time (GMT unix time stamp).
end_time
End time (GMT unix time stamp).
failed
Indicates the problem which occurred in case a job
could not be started on the execution host (e.g.
because the owner of the job did not have a valid
account on that machine). If Sun Grid Engine tries to
start a job multiple times, this may lead to multiple
entries in the accounting file corresponding to the
same job ID.
exit_status
Exit status of the job script (or Sun Grid Engine
specific status in case of certain error conditions).
ru_wallclock
Difference between end_time and start_time (see above).
The remainder of the accounting entries follows the contents
of the standard UNIX rusage structure as described in
<B><A HREF="../htmlman2/getrusage.html?pathrev=V62u5_TAG">getrusage(2)</A></B>. Depending on the operating system where the
job was executed some of the fields may be 0. The following
entries are provided:
ru_utime
ru_stime
ru_maxrss
ru_ixrss
ru_ismrss
ru_idrss
ru_isrss
ru_minflt
ru_majflt
ru_nswap
ru_inblock
ru_oublock
ru_msgsnd
ru_msgrcv
ru_nsignals
ru_nvcsw
ru_nivcsw
project
The project which was assigned to the job.
department
The department which was assigned to the job.
granted_pe
The parallel environment which was selected for that
job.
slots
The number of slots which were dispatched to the job by
the scheduler.
task_number
Array job task index number.
cpu The cpu time usage in seconds.
mem The integral memory usage in Gbytes seconds.
io The amount of data transferred in input/output opera-
tions.
category
A string specifying the job category.
iow The io wait time in seconds.
pe_taskid
If this identifier is set the task was part of a paral-
lel job and was passed to Sun Grid Engine via the qrsh
-inherit interface.
maxvmem
The maximum vmem size in bytes.
arid Advance reservation identifier. If the job used
resources of an advance reservation then this field
contains a positive integer identifier otherwise the
value is "0" .
queue
Records of type queue contain state information for queues
(queue instances). A queue record has the following fields:
qname
The cluster queue name.
hostname
The hostname of a specific queue instance.
report_time
The time (GMT unix time stamp) when a state change was
triggered.
state
The new queue state.
queue_consumable
A queue_consumable record contains information about queue
consumable values in addition to queue state information:
qname
The cluster queue name.
hostname
The hostname of a specific queue instance.
report_time
The time (GMT unix time stamp) when a state change was
triggered.
state
The new queue state.
consumables
Description of consumable values. Information about
multiple consumables is separated by space. A consum-
able description has the format
<name>=<actual_value>=<configured value>.
host
A host record contains information about hosts and host load
values. It contains the following information:
hostname
The name of the host.
report_time
The time (GMT unix time stamp) when the reported infor-
mation was generated.
state
The new host state. Currently, Sun Grid Engine doesn't
track a host state, the field is reserved for future
use. Always contains the value X.
load values
Description of load values. Information about multiple
load values is separated by space. A load value
description has the format <name>=<actual_value>.
host_consumable
A host_consumable record contains information about hosts
and host consumables. Host consumables can for example be
licenses. It contains the following information:
hostname
The name of the host.
report_time
The time (GMT unix time stamp) when the reported infor-
mation was generated.
state
The new host state. Currently, Sun Grid Engine doesn't
track a host state, the field is reserved for future
use. Always contains the value X.
consumables
Description of consumable values. Information about
multiple consumables is separated by space. A consum-
able description has the format
<name>=<actual_value>=<configured value>.
sharelog
The Sun Grid Engine qmaster can dump information about
sharetree configuration and use to the reporting file. The
parameter <I>sharelog</I> sets an interval in which sharetree
information will be dumped. It is set in the format
HH:MM:SS. A value of 00:00:00 configures qmaster not to dump
sharetree information. Intervals of several minutes up to
hours are sensible values for this parameter. The record
contains the following fields
current time
The present time
usage time
The time used so far
node name
The node name
user name
The user name
project name
The project name
shares
The total shares
job count
The job count
level
The percentage of shares used
total
The adjusted percentage of shares used
long target share
The long target percentage of resource shares used
short target share
The short target percentage of resource shares used
actual share
The actual percentage of resource shares used
usage
The combined shares used
cpu The cpu used
mem The memory used
io The IO used
long target cpu
The long target cpu used
long target mem
The long target memory used
long target io
The long target IO used
new_ar
A new_ar record contains information about advance reserva-
tion objects. Entries of this type will be added if an
advance reservation is created. It contains the following
information:
submission_time
The time (GMT unix time stamp) when the advance reser-
vation was created.
ar_number
The advance reservation number identifying the reserva-
tion.
ar_owner
The owner of the advance reservation.
ar_attribute
The ar_attribute record is written whenever a new advance
reservation was added or the attribute of an existing
advance reservation has changed. It has following fields.
event_time
The time (GMT unix time stamp) when the event was gen-
erated.
submission_time
The time (GMT unix time stamp) when the advance reser-
vation was created.
ar_number
The advance reservation number identifying the reserva-
tion.
ar_name
Name of the advance reservation.
ar_account
An account string which was specified during the crea-
tion of the advance reservation.
ar_start_time
Start time.
ar_end_time
End time.
ar_granted_pe
The parallel environment which was selected for an
advance reservation.
ar_granted_resources
The granted resources which were selected for an
advance reservation.
ar_log
The ar_log record is written whenever a advance reservation
is changing status. A status change can be from pending to
active, but can also be triggered by system events like host
outage. It has following fields.
ar_state_change_time
The time (GMT unix time stamp) when the event occurred
which caused a state change.
submission_time
The time (GMT unix time stamp) when the advance reser-
vation was created.
ar_number
The advance reservation number identifying the reserva-
tion.
ar_state
The new state.
ar_event
An event id identifying the event which caused the
state change.
ar_message
A message describing the event which caused the state
change.
ar_acct
The ar_acct records are accounting records which are written
for every queue instance whenever a advance reservation ter-
minates. Advance reservation accounting records comprise
following fields.
ar_termination_time
The time (GMT unix time stamp) when the advance reser-
vation terminated.
submission_time
The time (GMT unix time stamp) when the advance reser-
vation was created.
ar_number
The advance reservation number identifying the reserva-
tion.
ar_qname
Cluster queue name which the advance reservation
reserved.
ar_hostname
The name of the execution host.
ar_slots
The number of slots which were reserved.
SEE ALSO
<B><A HREF="../htmlman5/sge_conf.html?pathrev=V62u5_TAG">sge_conf(5)</A></B>. <B><A HREF="../htmlman5/host_conf.html?pathrev=V62u5_TAG">host_conf(5)</A></B>.
COPYRIGHT
See <B><A HREF="../htmlman1/sge_intro.html?pathrev=V62u5_TAG">sge_intro(1)</A></B> for a full statement of rights and permis-
sions.
</PRE>
<HR>
<ADDRESS>
Man(1) output converted with
<a href="http://www.oac.uci.edu/indiv/ehood/man2html.html">man2html</a>
</ADDRESS>
</BODY>
</HTML>
|
