1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
|
<HTML>
<BODY BGCOLOR=white>
<PRE>
<!-- Manpage converted by man2html 3.0.1 -->
NAME
JSV - Sun Grid Engine Job Submission Verifier
DESCRIPTION
JSV is an abbreviation for Job Submission Verifier. A JSV is
a script or binary that can be used to verify, modify or
reject a job during the time of job submission.
JSVs will be triggered by submit clients like <I>qsub</I>, <I>qrsh</I>,
<I>qsh</I> and <I>qmon</I> on submit hosts (Client JSV) or they verify
incoming jobs on the master host (Server JSV) or both.
CONFIGURATION
JSVs can be configured on various locations. Either a
jsv_url can be provided with the -jsv submit parameter dur-
ing job submission, a corresponding switch can be added to
one of the sge_request files or a jsv_url can be configured
in the global cluster configuration of the Sun Grid Engine
installation.
All defined JSV instances will be executed in following
order:
1) qsub -jsv ...
2) $cwd/.sge_request
3) $HOME/.sge_request
4) $SGE_ROOT/$SGE_CELL/common/sge_request
5) Global configuration
The Client JSVs (1-3) can be defined by Sun Grid Engine end
users whereas the client JSV defined in the global
sge_request file (4) and the server JSV (5) can only be
defined by the Sun Grid Engine administrators.
Due to the fact that (4) and (5) are defined and configured
by Sun Grid Engine administrators and because they are exe-
cuted as last JSV instances in the sequence of JSV scripts,
an administrator has an additional way to define certain
policies for a cluster.
As soon as one JSV instance rejects a job the whole process
of verification is stopped and the end user will get a
corresponding error message that the submission of the job
has failed.
If a JSV accepts a job or accepts a job after it applied
several modifications then the following JSV instance will
get the job parameters including all modifications as input
for the verification process. This is done as long as either
the job is accepted or rejected.
Find more information how to use Client JSVs in <B><A HREF="../htmlman1/qsub.html?pathrev=V62u5_TAG">qsub(1)</A></B> and
for Server JSVs in <B><A HREF="../htmlman5/sge_conf.html?pathrev=V62u5_TAG">sge_conf(5)</A></B>
LIFETIME
A Client or Server JSV is started as own UNIX process. This
process communicates either with a Sun Grid Engine client
process or the master daemon by exchanging commands, job
parameters and other data via stdin/stdout channels.
Client JSV instances are started by client applications
before a job is sent to qmaster. This instance does the job
verification for the job to be submitted. After that verifi-
cation the JSV process is stopped.
Server JSV instances are started for each worker thread part
of the qmaster process (for version 6.2 of Sun Grid Engine
this means that two processes are started). Each of those
processes have to verify job parameters for multiple jobs as
long as the master is running, the underlying JSV configura-
tion is not changed and no error occurs.
TIMEOUT
The timeout is a modifiable value that will measure the
response time of either the client or server JSV. In the
event that the response time of the JSV is longer than
timeout value specified, this will result in the JSV being
re-started. The server JSV timeout value is specified
through the qmaster parameter jsv_timeout. The client JSV
timeout value is set through the environment variable
SGE_JSV_TIMEOUT. The default value is 10 seconds, and this
value must be greater than 0. If the timeout has been reach,
the JSV will only try to re-start once, if the timeout is
reached again an error will occur.
THRESHOLD
The threshold value is defined as a qmaster parameter
jsv_threshold. This value measures the time for a server job
verification. If this time exceeds the defined threshold
then additional logging will appear in the master message
file at the INFO level. This value is specified in mil-
liseconds and has a default value of 5000. If a value of 0
is defined then this means all jobs will be logged in the
message file.
PROTOCOL
After a JSV script or binary is started it will get commands
through its stdin stream and it has to respond with certain
commands on the stdout stream. Data which is send via the
stderr stream of a JSV instance is ignored. Each command
which is send to/by a JSV script has to be terminated by a
new line character ('\n') whereas new line characters are
not allowed in the whole command string itself.
In general commands which are exchanged between a JSV and
client/qmaster have following format. Commands and arguments
are case sensitive. Find the EBNF command description below.
command := command_name ' ' { argument ' ' } ;
A command starts with a command_name followed by a space
character and a space separated list of arguments.
PROTOCOL (JSV side)
Following commands have to be implemented by an JSV script
so that it conforms to version 1.0 of the JSV protocol which
was first implemented in Sun Grid Engine 6.2u2:
begin_command := 'BEGIN' ;
After a JSV instance has received all env_commands and
param_commands of a job which should be verified, the
client/qmaster will trigger the verification process by
sending one begin_command. After that it will wait for
param_commands and env_commands which are sent back
from the JSV instance to modify the job specification.
As part of the verification process a JSV script or
binary has to use the result_command to indicate that
the verification process is finished for a job.
env_command := ENV ' ' modifier ' ' name ' ' value ;
modifier := 'ADD' | 'MOD' | 'DEL' ;
The env_command is an optional command which has only
to be implemented by a JSV instance if the
send_data_command is sent by this JSV before a the
started_command was sent. Only in that case the client
or master will use one or multiple env_commands to pass
the environment variables (name and value) to the JSV
instance which would be exported to the job environment
when the job would be started. Client and qmaster will
only sent env_commands with the modifier 'ADD'.
JSV instances modify the set of environment variables
by sending back env_commands and by using the modifiers
ADD, MOD and DEL.
param_command := 'PARAM' ' ' param_parameter ' ' value ;
param_parameter := submit_parameter | pseudo_parameter ;
The param_command has two additional arguments which
are separated by space characters. The first argument
is either a submit_parameter as it is specified in
<B><A HREF="../htmlman1/qsub.html?pathrev=V62u5_TAG">qsub(1)</A></B> or it is a pseudo_parameters as documented
below. The second parameter is the value of the
corresponding param_parameter.
Multiple param_commands will be sent to a JSV instance
after the JSV has sent a started_command. The sum of
all param_commands which is sent represents a job
specification of that job which should be verified.
submit_parameters are for example b (similar to the
qsub -b switch) or masterq (similar to qsub -masterq
switch). Find a complete list of submit_parameters in
the <B><A HREF="../htmlman1/qsub.html?pathrev=V62u5_TAG">qsub(1)</A></B> man page. Please note that not in all cases
the param_parameter name and the corresponding value
format is equivalent with the qsub switch name and its
argument format. E.g. the qsub -pe parameters will by
available as a set of parameters with the name pe_name,
pe_min, pe_max or the switch combination -soft -l will
be passed to JSV scripts as l_soft parameter. For
details concerning this differences consult also the
<B><A HREF="../htmlman1/qsub.html?pathrev=V62u5_TAG">qsub(1)</A></B> man page.
start_command := 'START' ;
The start_command has no additional arguments. This
command indicates that a new job verification should be
started. It is the first command which will be sent to
JSV script after it has been started and it will ini-
tiate each new job verification. A JSV instance might
trash cached values which are still stored due to a
previous job verification. The application which send
the start_command will wait for a started_command
before it continues.
quit_command := 'QUIT' ;
The quit_command has no additional arguments. If this
command is sent to a JSV instance then it should ter-
minate itself immediately.
PROTOCOL (client/qmaster side)
A JSV script or binary can send a set of commands to a
client/qmaster process to indicate its state in the communi-
cation process, to change the job specification of a job
which should be verified and to report messages or errors.
Below you can find the commands which are understood by the
client/qmaster which will implement version 1.0 of the com-
munication protocol which was first implemented in Sun Grid
Engine 6.2u2:
error_command := 'ERROR' message ;
Any time a JSV script encounters an error it might
report it to the client/qmaster. If the error happens
during a job verification the job which is currently
verified will be rejected. The JSV binary or script
will also be restarted before it gets a new verifica-
tion task.
log_command := 'LOG' log_level ;
log_level := 'INFO' | 'WARNING' | 'ERROR'
log_commands can be used whenever the client or qmaster
expects input from a JSV instance. This command can be
used in client JSVs to send information to the user
submitting the job. In client JSVs all messages,
independent of the log_level will be printed to the
stdout stream of the used submit client. If a server
JSV receives a log_command it will add the received
message to the message file respecting the specified
log_level. Please note that message might contain
spaces but no new line characters.
param_command (find definition above)
By sending param_commands a JSV script can change the
job specification of the job which should be verified.
If a JSV instance later on sends a result_command which
indicates that a JSV instance should be accepted with
correction then the values provided with these
param_commands will be used to modify the job before it
is accepted by the Sun Grid Engine system.
result_command := 'RESULT' result_type [ message ] ;
result_type := 'ACCEPT' | 'CORRECT' | 'REJECT' | 'REJECT_WAIT' ;
After the verification of a job is done a JSV script or
binary has to send a result_command which indicates
what should happen with the job. If the result_type is
ACCEPTED the job will be accepted as it was initially
submitted by the end user. All param_commands and
env_commands which might have been sent before the
result_command are ignored in this case. The
result_type CORRECT indicates that the job should be
accepted after all modifications sent via
param_commands and env_commands are applied to the job.
REJECT and REJECT_WAIT cause the client or qmaster
instance to reject the job.
send_data_command := 'SEND' data_name ;
data_name := 'ENV';
If a client/qmaster receives a send_env_command from a
JSV instance before a started_command is sent, then it
will not only pass job parameters with param_commands
but also env_commands which provide the JSV with the
information which environment variables would be
exported to the job environment if the job is accepted
and started later on.
The job environment is not passed to JSV instances as
default because the job environment of the end user
might contain data which might be interpreted wrong in
the JSV context and might therefore cause errors or
security issues.
started_command := 'STARTED' ;
By sending the started_command a JSV instance indicates
that it is ready to receive param_commands and
env_commands for a new job verification. It will only
receive env_commands if it sends a send_data_command
before the started_command.
PSEUDO PARAMETERS
CLIENT
The corresponding value for the CLIENT parameters is
either 'qmaster' or the name of a submit client like
'qsub',
'qsh', 'qrsh', 'qlogin' and so on This parameter value
can't be changed by JSV instances. It will always be
sent as part of a job verification.
CMDARGS
Number of arguments which will be passed to the job
script or command when the job execution is started.
It will always be sent as part of a job verification.
If no arguments should be passed to the job script or
command it will have the value 0. This parameter can be
changed by JSV instances. If the value of CMDARGS is
bigger than the number of available CMDARG<id> parame-
ters then the missing parameters will be automatically
passed as empty parameters to the job script.
CMDNAME
Either the path to the script or the command name in
case of binary submission. It will always be sent as
part of a job verification.
CONTEXT
Either 'client' if the JSV which receives this
param_command was started by a commandline client like
<I>qsub</I>, <I>qsh</I>, ... or 'master' if it was started by the
<I>sge</I>_<I>qmaster</I> process. It will always be sent as part of
a job verification. Changing the value of this parame-
ters is not possible within JSV instances.
GROUP
Defines Primary group of the user which tries to submit
the job which should be verified. This parameter cannot
be changed but is always sent as part of the verifica-
tion process. The user name is passed as parameters
with the name USER.
JOB_ID
Not available in the client context (see CONTEXT). Oth-
erwise it contains the job number of the job which will
be submitted to Grid Engine when the verification pro-
cess is successful. JOB_ID is an optional parameter
which can't be changed by JSV instances.
USER Username of the user which tries to submit the job
which should be verified. Cannot be changed but is
always sent as part of the verification process. The
group name is passed as parameter with the name GROUP
VERSION
VERSION will always be sent as part of a job verifica-
tion process and it will always be the first parameter
which is sent. It will contain a version number of the
format <major>.<minor>. In version 6.2u2 and higher the
value will be '1.0'. The value of this parameter can't
be changed.
EXAMPLE
Here is an example for the communication of a client with a
JSV instance when following job is submitted:
> qsub -pe p 3 -hard -l a=1,b=5 -soft -l q=all.q $SGE_ROOT/examples/jobs/sleeper.sh
Data in the first column is sent from the client/qmaster to
the JSV instance. That data contained in the second column
is sent from the JSV script to the client/qmaster. New line
characters which terminate each line in the communication
protocol are omitted.
START
SEND ENV
STARTED
PARAM VERSION 1.0
PARAM CONTEXT client
PARAM CLIENT qsub
PARAM USER ernst
PARAM GROUP staff
PARAM CMDNAME /sge_root/examples/jobs/sleeper.sh
PARAM CMDARGS 1
PARAM CMDARG0 12
PARAM l_hard a=1,b=5
PARAM l_soft q=all.q
PARAM M user@hostname
PARAM N Sleeper
PARAM o /dev/null
PARAM pe_name pe1
PARAM pe_min 3
PARAM pe_max 3
PARAM S /bin/sh
BEGIN
RESULT STATE ACCEPT
SEE ALSO
<B><A HREF="../htmlman1/sge_intro.html?pathrev=V62u5_TAG">sge_intro(1)</A></B>, <B><A HREF="../htmlman1/qalter.html?pathrev=V62u5_TAG">qalter(1)</A></B>, <B><A HREF="../htmlman1/qlogin.html?pathrev=V62u5_TAG">qlogin(1)</A></B>, <B><A HREF="../htmlman1/qmake.html?pathrev=V62u5_TAG">qmake(1)</A></B>, <B><A HREF="../htmlman1/qrsh.html?pathrev=V62u5_TAG">qrsh(1)</A></B>,
<B><A HREF="../htmlman1/qsh.html?pathrev=V62u5_TAG">qsh(1)</A></B>, <B><A HREF="../htmlman1/qsub.html?pathrev=V62u5_TAG">qsub(1)</A></B>, <B><A HREF="../htmlman1/qtcsh.html?pathrev=V62u5_TAG">qtcsh(1)</A></B>,
COPYRIGHT
See <B><A HREF="../htmlman1/sge_intro.html?pathrev=V62u5_TAG">sge_intro(1)</A></B> for a full statement of rights and permis-
sions.
</PRE>
<HR>
<ADDRESS>
Man(1) output converted with
<a href="http://www.oac.uci.edu/indiv/ehood/man2html.html">man2html</a>
</ADDRESS>
</BODY>
</HTML>
|
