Slurm Burst Buffer Guide

Overview

This guide explains how to use Slurm burst buffer plugins. Where appropriate, it explains how these plugins work in order to give guidance about how to best use these plugins.

The Slurm burst buffer plugins call a script at different points during the lifetime of a job:

  1. At job submission
  2. While the job is pending after an estimated start time is established. This is called "stage-in."
  3. Once the job has been scheduled but has not started running yet. This is called "pre-run."
  4. Once the job has completed or been cancelled, but Slurm has not released resources for the job yet. This is called "stage-out."
  5. Once the job has completed, and Slurm has released resources for the job. This is called "teardown."

This script runs on the slurmctld node. These are the supported plugins:

Datawarp

This plugin provides hooks to Cray's Datawarp APIs. Datawarp implements burst buffers, which are a shared high-speed storage resource. Slurm provides support for allocating these resources, staging files in, scheduling compute nodes for jobs using these resources, and staging files out. Burst buffers can also be used as temporary storage during a job's lifetime, without file staging. Another typical use case is for persistent storage, not associated with any specific job.

Lua

This plugin provides hooks to an API that is defined by a Lua script. This plugin was developed to provide system administrators with a way to do any task (not only file staging) at different points in a job's life cycle. These tasks might include file staging, node maintenance, or any other task that is desired to run during one or more of the five job states listed above.

The burst buffer APIs will only be called for a job that specifically requests using them. The Job Submission Commands section explains how a job can request using the burst buffer APIs.

Configuration (for system administrators)

Common Configuration

Datawarp

slurm.conf:

BurstBufferType=burst_buffer/datawarp

The datawarp plugin calls two scripts:

Lua

slurm.conf:

BurstBufferType=burst_buffer/lua

The lua plugin calls a single script which must be named burst_buffer.lua. This script needs to exist in the same directory as slurm.conf. The following functions are required to exist, although they may do nothing but return success:

A template of burst_buffer.lua is provided with Slurm: etc/burst_buffer.lua.example

This template documents many more details about the functions such as required parameters, when each function is called, return values for each function, and some simple examples.

Lua Implementation

This purpose of this section is to provide additional information about the Lua plugin to help system administrators who desire to implement the Lua API. The most important points in this section are:

How does burst_buffer.lua run?

Lua scripts may either be run by themselves in a separate process via the fork() and exec() system calls, or they may be called via Lua's C API from within an existing process. One of the goals of the lua plugin was to avoid calling fork() from within slurmctld because it can severely harm performance of the slurmctld. The datawarp plugin calls fork() and exec() from slurmctld for every burst buffer API call, and this has been shown to severely harm slurmctld performance. Therefore, slurmctld calls burst_buffer.lua using Lua's C API instead of using fork().

Some functions in burst_buffer.lua are allowed to run for a long time, but they may need to be killed if the job is cancelled, if slurmctld is restarted, or if they run for longer than the configured timeout in burst_buffer.conf. However, a call to a Lua script via Lua's C API cannot be killed from within the same process; only killing the entire process that called the Lua script can kill the Lua script.

To address this situation, burst_buffer.lua is called in two different ways:

The way in which each function is called is also documented in the burst_buffer.lua.example file.

Warnings

Do not install a signal handler in burst_buffer.lua because it is called directly from slurmctld. If slurmctld receives a signal, it could attempt to run the signal handler from burst_buffer.lua, even after a call to burst_buffer.lua is completed, which results in a crash.

Burst Buffer Resources

The burst buffer API may define burst buffer resource "pools" from which a job may request a certain amount of pool space. If a pool does not have sufficient space to fulfill a job's request, that job will remain pending until the pool does have enough space. Once the pool has enough space, Slurm may begin stage-in for the job. When stage-in begins, Slurm subtracts the job's requested space from the pool's available space. When teardown completes, Slurm adds the job's requested space back into the pool's available space. The Job Submission Commands section explains how a job may request space from a pool. Pool space is a scalar quantity.

Datawarp

Lua

Job Submission Commands

The normal mode of operation is for batch jobs to specify burst buffer requirements within the batch script. Commented batch script lines containing a specific directive (depending on which plugin is being used) will inform Slurm that it should run the burst buffer stages for that job. These lines will also describe the burst buffer requirements for the job.

The salloc and srun commands can specify burst buffer requirements with the --bb and --bbf options. This is described in the Command-line Job Options section.

All burst buffer directives should be specified in comments at the top of the batch script. They may be placed before, after, or interspersed with any #SBATCH directives. All burst buffer stages happen at specific points in the job's life cycle, as described in the Overview section; they do not happen during the job's execution. For example, all of the persistent burst buffer (used only by the datawarp plugin) creations and deletions happen before the job's compute portion happens. In a similar fashion, you can't run stage-in at various points in the script execution; burst buffer stage-in is performed before the job begins and stage-out is performed after the job completes.

For both plugins, a job may request a certain amount of space (size or capacity) from a burst buffer resource pool.

At job submission, Slurm performs basic directive validation and also runs a function in the burst buffer script. This function can perform validation of the directives used in the job script. If Slurm determines options are invalid, or if the burst buffer script returns an error, the job will be rejected and an error message will be returned directly to the user.

Note that unrecognized options may be ignored in order to support backward compatibility (i.e. a job submission would not fail in the case of an option recognized by some versions of Slurm, but not recognized by other versions). If the job is accepted, but later fails (e.g. some problem staging files), the job will be held and its "Reason" field will be set to an error message provided by the underlying infrastructure.

Users may also request to be notified by email upon completion of burst buffer stage out using the --mail-type=stage_out or --mail-type=all option. The subject line of the email will be of this form:

SLURM Job_id=12 Name=my_app Staged Out, StageOut time 00:05:07

The following plugin subsections give additional information that is specific to each plugin and provide example job scripts. Command-line examples are given in the Command-line Job Options section.

Datawarp

The directive of #DW (for "DataWarp") is used for burst buffer directives when using the burst_buffer/datawarp plugin. Please reference Cray documentation for details about the DataWarp options. For DataWarp systems, the directive of #BB can be used to create or delete persistent burst buffer storage.
NOTE: The #BB directive is used since the command is interpreted by Slurm and not by the Cray Datawarp software. This is discussed more in the Persistent Burst Buffer section.

For job-specific burst buffers, it is required to specify a burst buffer capacity. If the job does not specify capacity then the job will be rejected. A job may also specify the pool from which it wants resources; if the job does not specify a pool, then the pool specified by DefaultPool in burst_buffer.conf will be used (if configured).

The following job script requests burst buffer resources from the default pool and requests files to be staged in and staged out:

#!/bin/bash
#DW jobdw type=scratch capacity=1GB access_mode=striped,private pfs=/scratch
#DW stage_in type=file source=/tmp/a destination=/ss/file1
#DW stage_out type=file destination=/tmp/b source=/ss/file1
srun application.sh

Lua

The default directive for this plugin is #BB_LUA. The directive used by this plugin may be changed by setting the Directive option in burst_buffer.conf. Since the directive must always begin with a # sign (which starts a comment in a shell script) this option should specify only the string following the # sign. For example, if burst_buffer.conf contains the following:

Directive=BB_EXAMPLE

then the burst buffer directive will be #BB_EXAMPLE.

If the Directive option is not specified in burst_buffer.conf, then the default directive for this plugin (#BB_LUA) will be used.

Since this plugin was designed to be generic and flexible, this plugin only requires the directive to be given. If the directive is given, Slurm will run all burst buffer stages for the job.

Example of the minimum information required for all burst buffer stages to run for the job:

#!/bin/bash
#BB_LUA
srun application.sh

Because burst buffer pools are optional for this plugin (see the Burst Buffer Resources section), a job is not required to specify a pool or capacity. If pools are provided by the burst buffer API, then a job may request a pool and capacity:

#!/bin/bash
#BB_LUA pool=pool1 capacity=1K
srun application.sh

A job may choose whether or not to specify a pool. If a job does not specify a pool, then the job is still allowed to run and the burst buffer stages will still run for this job (as long as the burst buffer directive was given). If the job specifies a pool but that pool is not found, then the job is rejected.

The system administrator may validate burst buffer options in the slurm_bb_job_process function in burst_buffer.lua. This might include requiring a job to specify a pool or validating any additional options that the system administrator decides to implement.

Persistent Burst Buffer Creation and Deletion Directives

This section only applies to the datawarp plugin, since persistent burst buffers are not used in any other burst buffer plugin.

These options are used to create and delete persistent burst buffers:

Options for creating and deleting persistent burst buffers:

Multiple persistent burst buffers may be created or deleted within a single job.

Example - Creating two persistent burst buffers:

#!/bin/bash
#BB create_persistent name=alpha capacity=32GB access=striped type=scratch
#BB create_persistent name=beta capacity=16GB access=striped type=scratch
srun application.sh

Example - Destroying two persistent burst buffers:

#!/bin/bash
#BB destroy_persistent name=alpha
#BB destroy_persistent name=beta
srun application.sh

Persistent burst buffers can be created and deleted by a job requiring no compute resources. Submit a job with the desired burst buffer directives and specify a node count of zero (e.g. sbatch -N0 setup_buffers.bash). Attempts to submit a zero size job without burst buffer directives or with job-specific burst buffer directives will generate an error. Note that zero size jobs are not supported for job arrays or heterogeneous job allocations.

NOTE: The ability to create and destroy persistent burst buffers may be limited by the Flags option in the burst_buffer.conf file. See the burst_buffer.conf man page for more information. By default only privileged users (i.e. Slurm operators and administrators) can create or destroy persistent burst buffers.

Heterogeneous Job Support

Heterogeneous jobs may request burst buffers. Burst buffer hooks will run once for each component that has burst buffer directives. For example, if a heterogeneous job has three components and two of them have burst buffer directives, the burst buffer hooks will run once for each of the two components with burst buffer directives, but not for the third component without burst buffer directives. Further information and examples can be found in the heterogeneous jobs page.

Command-line Job Options

In addition to putting burst buffer directives in the batch script, the command-line options --bb and --bbf may also include burst buffer directives. These command-line options are available for salloc, sbatch, and srun. Note that the --bb option cannot create or destroy persistent burst buffers.

The --bbf option takes as an argument a filename and that file should contain a collection of burst buffer operations identical to those used for batch jobs.

Alternatively, the --bb option may be used to specify burst buffer directives as the option argument. The behavior of this option depends on which burst buffer plugin is used. When the --bb option is used, Slurm parses this option and creates a temporary burst buffer script file that is used internally by the burst buffer plugins.

Datawarp

When using the --bb option, the format of the directives can either be identical to those used in a batch script OR a very limited set of options can be used, which are translated to the equivalent script for later processing. The following options are allowed:

Multiple options should be space separated. If a swap option is specified, the job must also specify the required node count.

Example:

# Sample execute line:
srun --bb="capacity=1G access=striped type=scratch" a.out

# Equivalent script as generated by Slurm's burst_buffer/datawarp plugin
#DW jobdw capacity=1GiB access_mode=striped type=scratch

Lua

This plugin does not do any special parsing or translating of burst buffer directives given by the --bb option. When using the --bb option, the format is identical to the batch script: Slurm only enforces that the burst buffer directive must be specified. See additional information in the Lua subsection of Job Submission Commands.

Example:

# Sample execute line:
srun --bb="#BB_LUA pool=pool1 capacity=1K"

# Equivalent script as generated by Slurm's burst_buffer/lua plugin
#BB_LUA pool=pool1 capacity=1K

Symbol Replacement

Slurm supports a number of symbols that can be used to automatically fill in certain job details, e.g. to make stage-in or stage-out directory paths vary with each job submission.

Supported symbols include:
%%%
%AArray Master Job Id
%aArray Task Id
%dWorkdir
%jJob Id
%uUser Name
%xJob Name
\\Stop further processing of the line

Status Commands

Burst buffer information that Slurm tracks is available by using the scontrol show burst command or by using the sview command's Burst Buffer tab. Examples follow.

Datawarp plugin example:

$ scontrol show burst
Name=datawarp DefaultPool=wlm_pool Granularity=200GiB TotalSpace=5800GiB FreeSpace=4600GiB UsedSpace=1600GiB
  Flags=EmulateCray
  StageInTimeout=86400 StageOutTimeout=86400 ValidateTimeout=5 OtherTimeout=300
  GetSysState=/home/marshall/slurm/master/install/c1/sbin/dw_wlm_cli
  GetSysStatus=/home/marshall/slurm/master/install/c1/sbin/dwstat
  Allocated Buffers:
    JobID=169509 CreateTime=2021-08-11T10:19:06 Pool=wlm_pool Size=1200GiB State=allocated UserID=marshall(1017)
    JobID=169508 CreateTime=2021-08-11T10:18:46 Pool=wlm_pool Size=400GiB State=staged-in UserID=marshall(1017)
  Per User Buffer Use:
    UserID=marshall(1017) Used=1600GiB

Lua plugin example:

$ scontrol show burst
Name=lua DefaultPool=(null) Granularity=1 TotalSpace=0 FreeSpace=0 UsedSpace=0
  PoolName[0]=pool1 Granularity=1KiB TotalSpace=10000KiB FreeSpace=9750KiB UsedSpace=250KiB
  PoolName[1]=pool2 Granularity=2 TotalSpace=10 FreeSpace=10 UsedSpace=0
  PoolName[2]=pool3 Granularity=1 TotalSpace=4 FreeSpace=4 UsedSpace=0
  PoolName[3]=pool4 Granularity=1 TotalSpace=5GB FreeSpace=4GB UsedSpace=1GB
  Flags=DisablePersistent
  StageInTimeout=86400 StageOutTimeout=86400 ValidateTimeout=5 OtherTimeout=300
  GetSysState=(null)
  GetSysStatus=(null)
  Allocated Buffers:
    JobID=169504 CreateTime=2021-08-11T10:13:38 Pool=pool1 Size=250KiB State=allocated UserID=marshall(1017)
    JobID=169502 CreateTime=2021-08-11T10:12:06 Pool=pool4 Size=1GB State=allocated UserID=marshall(1017)
  Per User Buffer Use:
    UserID=marshall(1017) Used=1000256KB

Access to a burst buffer status API is available from scontrol using the scontrol show bbstat ... or scontrol show dwstat ... commands. Options following bbstat or dwstat on the scontrol execute line are passed directly to the bbstat or dwstat commands, as shown below. In the datawarp plugin, this command calls Cray's dwstat script. See Cray Datawarp documentation for details about dwstat options and output. In the lua plugin, this command calls the slurm_bb_get_status function in burst_buffer.lua.

Datawarp plugin example:

/opt/cray/dws/default/bin/dwstat
$ scontrol show dwstat
    pool units quantity    free gran'
wlm_pool bytes  7.28TiB 7.28TiB 1GiB'

$ scontrol show dwstat sessions
 sess state      token creator owner             created expiration nodes
  832 CA---  783000000  tester 12345 2015-09-08T16:20:36      never    20
  833 CA---  784100000  tester 12345 2015-09-08T16:21:36      never     1
  903 D---- 1875700000  tester 12345 2015-09-08T17:26:05      never     0

$ scontrol show dwstat configurations
 conf state inst    type access_type activs
  715 CA---  753 scratch      stripe      1
  716 CA---  754 scratch      stripe      1
  759 D--T-  807 scratch      stripe      0
  760 CA---  808 scratch      stripe      1

A Lua plugin example can be found in the slurm_bb_get_status function in the etc/burst_buffer.lua.example file provided with Slurm.

Advanced Reservations

Burst buffer resources can be placed in an advanced reservation using the BurstBuffer option. The argument consists of four elements:
[plugin:][pool:]#[units]