Slurm supports job preemption, the act of "stopping" one or more "low-priority" jobs to let a "high-priority" job run. Job preemption is implemented as a variation of Slurm's Gang Scheduling logic. When a job that can preempt others is allocated resources that are already allocated to one or more jobs that could be preempted by the first job, the preemptable job(s) are preempted. Based on the configuration the preempted job(s) can be cancelled, or can be requeued and started using other resources, or suspended and resumed once the preemptor job completes, or can even share resources with the preemptor using Gang Scheduling.
The PriorityTier of the Partition of the job or its Quality Of Service (QOS) can be used to identify which jobs can preempt or be preempted by other jobs. Slurm offers the ability to configure the preemption mechanism used on a per partition or per QOS basis. For example, jobs in a low priority queue may get requeued, while jobs in a medium priority queue may get suspended.
There are several important configuration parameters relating to preemption:
enforced,select, MCS labels will
be set by default on jobs, causing this restriction to be universal.
To enable preemption after making the configuration changes described above, restart Slurm if it is already running. Any change to the plugin settings in Slurm requires a full restart of the daemons. If you just change the partition PriorityTier or OverSubscribe setting, this can be updated with scontrol reconfig.
If a job request restricts Slurm's ability to run jobs from multiple users or accounts on a node by using the "--exclusive=user" or "--exclusive=mcs" job options, the job will be prevented from preempting or being preempted by any job that does not match the user or MCS. The one exception is that these exclusive=user jobs will be able to preempt (but not be preempted by) fully "--exclusive" jobs from other users. If preemption is used, it is generally advisable to disable the "--exclusive=user" and "--exclusive=mcs" job options by using a job_submit plugin (set the value of "job_desc.shared" to "NO_VAL16").
For heterogeneous job to be considered for preemption all components must be eligible for preemption. When a heterogeneous job is to be preempted the first identified component of the job with the highest order PreemptMode (SUSPEND (highest), REQUEUE, CANCEL (lowest)) will be used to set the PreemptMode for all components. The GraceTime and user warning signal for each component of the heterogeneous job remain unique.
Because licenses are not freed when jobs are suspended, jobs using licenses requested by higher priority jobs will only be prempted when PreemptMode is either REQUEUE or CANCEL and PreemptParameters=reclaim_licenses is set.
The SelectType plugin will identify resources where a pending job can begin execution. When PreemptMode is configured to CANCEL, SUSPEND or REQUEUE, the select plugin will also preempt running jobs as needed to initiate the pending job. When PreemptMode=SUSPEND,GANG the select plugin will initiate the pending job and rely upon the gang scheduling logic to perform job suspend and resume, as described below.
The select plugin is passed an ordered list of preemptable jobs to consider for each pending job which is a candidate to start. This list is sorted by either:
The select plugin will determine if the pending job can start without preempting any jobs and if so, starts the job using available resources. Otherwise, the select plugin will simulate the preemption of each job in the priority ordered list and test if the job can be started after each preemption. Once the job can be started, the higher priority jobs in the preemption queue will not be considered, but the jobs to be preempted in the original list may be sub-optimal. For example, to start an 8 node job, the ordered preemption candidates may be 2 node, 4 node and 8 node. Preempting all three jobs would allow the pending job to start, but by reordering the preemption candidates it is possible to start the pending job after preempting only one job. To address this issue, the preemption candidates are re-ordered with the final job requiring preemption placed first in the list and all of the other jobs to be preempted ordered by the number of nodes in their allocation which overlap the resources selected for the pending job. In the example above, the 8 node job would be moved to the first position in the list. The process of simulating the preemption of each job in the priority ordered list will then be repeated for the final decision of which jobs to preempt. This two stage process may preempt jobs which are not strictly in preemption priority order, but fewer jobs will be preempted than otherwise required. See the PreemptParameters configuration parameter options of reorder_count and strict_order for preemption tuning parameters.
When enabled, the gang scheduling logic (which is also supports job preemption) keeps track of the resources allocated to all jobs. For each partition an "active bitmap" is maintained that tracks all concurrently running jobs in the Slurm cluster. Each partition also maintains a job list for that partition, and a list of "shadow" jobs. The "shadow" jobs are high priority job allocations that "cast shadows" on the active bitmaps of the low priority jobs. Jobs caught in these "shadows" will be preempted.
Each time a new job is allocated to resources in a partition and begins running, the gang scheduler adds a "shadow" of this job to all lower priority partitions. The active bitmap of these lower priority partitions are then rebuilt, with the shadow jobs added first. Any existing jobs that were replaced by one or more "shadow" jobs are suspended (preempted). Conversely, when a high priority running job completes, its "shadow" goes away and the active bitmaps of the lower priority partitions are rebuilt to see if any suspended jobs can be resumed.
The gang scheduler plugin is designed to be reactive to the resource allocation decisions made by the "select" plugins. The "select" plugins have been enhanced to recognize when job preemption has been configured, and to factor in the priority of each partition when selecting resources for a job. When choosing resources for each job, the selector avoids resources that are in use by other jobs (unless sharing has been configured, in which case it does some load-balancing). However, when job preemption is enabled, the select plugins may choose resources that are already in use by jobs from partitions with a lower priority setting, even when sharing is disabled in those partitions.
This leaves the gang scheduler in charge of controlling which jobs should run on the over-allocated resources. If PreemptMode=SUSPEND, jobs are suspended using the same internal functions that support scontrol suspend and scontrol resume. A good way to observe the operation of the gang scheduler is by running squeue -i<time> in a terminal window.
For performance reasons, the backfill scheduler reserves whole nodes for jobs, not partial nodes. If during backfill scheduling a job preempts one or more other jobs, the whole nodes for those preempted jobs are reserved for the preemptor job, even if the preemptor job requested fewer resources than that. These reserved nodes aren't available to other jobs during that backfill cycle, even if the other jobs could fit on the nodes. Therefore, jobs may preempt more resources during a single backfill iteration than they requested.
The following example is configured with select/linear and PreemptMode=SUSPEND,GANG. This example takes place on a cluster of 5 nodes:
[user@n16 ~]$ sinfo PARTITION AVAIL TIMELIMIT NODES STATE NODELIST active* up infinite 5 idle n[12-16] hipri up infinite 5 idle n[12-16]
Here are the Partition settings:
[user@n16 ~]$ grep PartitionName /shared/slurm/slurm.conf PartitionName=DEFAULT OverSubscribe=FORCE:1 Nodes=n[12-16] PartitionName=active PriorityTier=1 Default=YES PartitionName=hipri PriorityTier=2
The runit.pl script launches a simple load-generating app that runs for the given number of seconds. Submit 5 single-node runit.pl jobs to run on all nodes:
[user@n16 ~]$ sbatch -N1 ./runit.pl 300 sbatch: Submitted batch job 485 [user@n16 ~]$ sbatch -N1 ./runit.pl 300 sbatch: Submitted batch job 486 [user@n16 ~]$ sbatch -N1 ./runit.pl 300 sbatch: Submitted batch job 487 [user@n16 ~]$ sbatch -N1 ./runit.pl 300 sbatch: Submitted batch job 488 [user@n16 ~]$ sbatch -N1 ./runit.pl 300 sbatch: Submitted batch job 489 [user@n16 ~]$ squeue -Si JOBID PARTITION NAME USER ST TIME NODES NODELIST 485 active runit.pl user R 0:06 1 n12 486 active runit.pl user R 0:06 1 n13 487 active runit.pl user R 0:05 1 n14 488 active runit.pl user R 0:05 1 n15 489 active runit.pl user R 0:04 1 n16
Now submit a short-running 3-node job to the hipri partition:
[user@n16 ~]$ sbatch -N3 -p hipri ./runit.pl 30 sbatch: Submitted batch job 490 [user@n16 ~]$ squeue -Si JOBID PARTITION NAME USER ST TIME NODES NODELIST 485 active runit.pl user S 0:27 1 n12 486 active runit.pl user S 0:27 1 n13 487 active runit.pl user S 0:26 1 n14 488 active runit.pl user R 0:29 1 n15 489 active runit.pl user R 0:28 1 n16 490 hipri runit.pl user R 0:03 3 n[12-14]
Job 490 in the hipri partition preempted jobs 485, 486, and 487 from the active partition. Jobs 488 and 489 in the active partition remained running.
This state persisted until job 490 completed, at which point the preempted jobs were resumed:
[user@n16 ~]$ squeue JOBID PARTITION NAME USER ST TIME NODES NODELIST 485 active runit.pl user R 0:30 1 n12 486 active runit.pl user R 0:30 1 n13 487 active runit.pl user R 0:29 1 n14 488 active runit.pl user R 0:59 1 n15 489 active runit.pl user R 0:58 1 n16
In this example we have three different partitions using three different job preemption mechanisms.
# Excerpt from slurm.conf PartitionName=low Nodes=linux Default=YES OverSubscribe=NO PriorityTier=10 PreemptMode=requeue PartitionName=med Nodes=linux Default=NO OverSubscribe=FORCE:1 PriorityTier=20 PreemptMode=suspend PartitionName=hi Nodes=linux Default=NO OverSubscribe=FORCE:1 PriorityTier=30 PreemptMode=off
$ sbatch tmp
Submitted batch job 94
$ sbatch -p med tmp
Submitted batch job 95
$ sbatch -p hi tmp
Submitted batch job 96
$ squeue
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
96 hi tmp moe R 0:04 1 linux
94 low tmp moe PD 0:00 1 (Resources)
95 med tmp moe S 0:02 1 linux
(after job 96 completes)
$ squeue
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
94 low tmp moe PD 0:00 1 (Resources)
95 med tmp moe R 0:24 1 linux
In this example we have one partition on which we want to execute only one job per resource (e.g. core) at a time except when a job submitted to the partition from a high priority Quality Of Service (QOS) is submitted. In that case, we want that second high priority job to be started and be gang scheduled with the other jobs on overlapping resources.
# Excerpt from slurm.conf PreemptMode=Suspend,Gang PreemptType=preempt/qos PartitionName=normal Nodes=linux Default=NO OverSubscribe=FORCE:1
More intelligence in the select plugins: This implementation of preemption relies on intelligent job placement by the select plugins.
Take the following example:
[user@n8 ~]$ sinfo
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST
active* up infinite 5 idle n[1-5]
hipri up infinite 5 idle n[1-5]
[user@n8 ~]$ sbatch -N1 -n2 ./sleepme 60
sbatch: Submitted batch job 17
[user@n8 ~]$ sbatch -N1 -n2 ./sleepme 60
sbatch: Submitted batch job 18
[user@n8 ~]$ sbatch -N1 -n2 ./sleepme 60
sbatch: Submitted batch job 19
[user@n8 ~]$ squeue
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
17 active sleepme cholmes R 0:03 1 n1
18 active sleepme cholmes R 0:03 1 n2
19 active sleepme cholmes R 0:02 1 n3
[user@n8 ~]$ sbatch -N3 -n6 -p hipri ./sleepme 20
sbatch: Submitted batch job 20
[user@n8 ~]$ squeue -Si
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
17 active sleepme cholmes S 0:16 1 n1
18 active sleepme cholmes S 0:16 1 n2
19 active sleepme cholmes S 0:15 1 n3
20 hipri sleepme cholmes R 0:03 3 n[1-3]
[user@n8 ~]$ sinfo
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST
active* up infinite 3 alloc n[1-3]
active* up infinite 2 idle n[4-5]
hipri up infinite 3 alloc n[1-3]
hipri up infinite 2 idle n[4-5]
It would be more ideal if the "hipri" job were placed on nodes n[3-5], which would allow jobs 17 and 18 to continue running. However, a new "intelligent" algorithm would have to include factors such as job size and required nodes in order to support ideal placements such as this, which can quickly complicate the design. Any and all help is welcome here!
Last modified 21 May 2024