.\" Automatically generated by Pandoc 2.9.2.1
.\"
.TH "fspl" "1" "May 2022" "John Goerzen" "fspl Manual"
.hy
.SH NAME
.PP
fspl - sequential, distributed job queue processing
.SH SYNOPSIS
.PP
\f[B]fspl\f[R] [ \f[I]OPTIONS\f[R] ] \f[B]COMMAND\f[R] [
\f[I]command_options\f[R] ]
.SH OVERVIEW
.PP
\f[B]fspl\f[R] is the CLI part of the
Filespooler (https://www.complete.org/filespooler) package.
.PP
\f[B]fspl\f[R] is a Unix-style tool that facilitates local or remote
command execution, complete with stdin capture, with easy integration
with various tools.
Here\[aq]s a brief Filespooler feature list:
.IP \[bu] 2
It can easily use tools such as S3, Dropbox, Syncthing, NNCP, ssh, UUCP,
USB drives, CDs, etc.
as transport.
.RS 2
.IP \[bu] 2
Translation: you can use basically anything that is a filesystem as a
transport
.RE
.IP \[bu] 2
It can use arbitrary decoder command pipelines (eg, zcat, stdcat, gpg,
age, etc) to pre-process stored packets.
.IP \[bu] 2
It can send and receive packets by pipes.
.IP \[bu] 2
Its storage format is simple on-disk files with locking.
.IP \[bu] 2
It supports one-to-one and one-to-many configurations.
.IP \[bu] 2
Locking is unnecessary when writing new jobs to the queue, and many
arbitrary tools (eg, Syncthing, Dropbox, etc) can safely write directly
to the queue without any assistance.
.IP \[bu] 2
Queue processing is strictly ordered based on the order on the creation
machine, even if job files are delivered out of order to the
destination.
.IP \[bu] 2
stdin can be piped into the job creation tool, and piped to a later
executor at process time on a remote machine.
.IP \[bu] 2
The file format is lightweight; less than 100 bytes overhead unless
large extra parameters are given.
.IP \[bu] 2
The queue format is lightweight; having 1000 different queues on a
Raspberry Pi would be easy.
.IP \[bu] 2
Processing is stream-based throughout; arbitrarily-large packets are
fine and sizes in the TB range are no problem.
.IP \[bu] 2
The Filespooler command, fspl, is extremely lightweight, consuming less
than 10MB of RAM on x86_64.
.IP \[bu] 2
Filespooler has extensive documentation.
.PP
Filespooler consists of a command-line tool (fspl) for interacting with
queues.
It also consists of a Rust library that is used by fspl.
main.rs for fspl is just a few lines long.
.SH A WORD ABOUT DOCUMENTATION
.PP
This manual is the reference for fspl.
The filespooler homepage, <https://www.complete.org/filespooler/>
contains many examples, instructions on how to integrate with everything
from file syncers to encryption tools, and so forth.
Please refer to it for further information.
.SH BASIC OPERATION
.PP
The basic idea is this:
.IP \[bu] 2
Before starting, on the receiving end, you run \f[B]fspl queue-init\f[R]
to prepare a queue directory.
.IP \[bu] 2
On the sending end, you use \f[B]fspl prepare\f[R] to prepare a job file
(packet).
This packet is written to stdout.
From there, you can pipe it to \f[B]fspl write\f[R] to inject it into a
local queue, or use various kinds of transport to get it to a remote
machine.
.IP \[bu] 2
You use \f[B]fspl queue-process\f[R] to execute packets.
.RS 2
.IP \[bu] 2
Alternatively, the \f[B]fspl stdin-\f[R] series of commands let you have
more manual control over queue processing, accepting job packets in
stdin.
They can let you completely ignore the built-in queue mechanism if you
so desire.
.RE
.SH ON-DISK FORMATS
.PP
The key way to ensure the ordered processing of the job queue is with a
sequence number.
This is a 64-bit unsigned integer.
It is stored in a \f[I]seqfile\f[R] on both the sending and the
receiving side.
On the sending side, the seqfile is standalone; there is only an
accompanying \f[C].lock\f[R] file for it.
On the receiving side, the seqfile and its accompanying lock file live
within the queue directory.
.PP
When the seqfile is referenced on the sending side, it will be created
and initialized with the value \f[B]1\f[R] if it does not already exist.
On the receiving side, it is created as part of \f[B]fspl
queue-init\f[R].
.PP
In either case, the seqfile consists of one newline-terminated line,
containing the next number to process.
On the sending side, this is used by \f[B]fspl prepare\f[R] as the
sequence number for the next generated packet.
On the receiving side, it is used by \f[B]fspl queue-process\f[R] to
determine which job to process next (unless changed by
\f[B]--order-by\f[R]).
.SS THE QUEUE
.PP
The queue has this general layout:
.IP
.nf
\f[C]
queuedir/           Top-level queue directory
  nextseq           Sequence file
  nextseq.lock      Lock file
  jobs/             Job files stored here
 
\f[R]
.fi
.PP
When passing the \f[B]--queuedir\f[R] to one of the \f[B]fspl
queue-\f[R] commands, you give it the path to the top-level queuedir as
shown here.
.PP
You are free to create additional directories within the
\f[B]queuedir\f[R] so long as they don\[aq]t use one of the names listed
above.
This can be helpful for receiving queue contents in certain situations.
.SS Append-Only Queues
.PP
You can specify \f[B]--append-only\f[R] to \f[B]fspl queue-init\f[R],
which will cause the \f[B]nextseq\f[R] and \f[B]nextseq.lock\f[R] files
to be omitted.
This has the effect of making the queue write-only.
This can be useful if you are synchronizing the \f[B]jobs\f[R]
subdirectory between machines, but still want to be able to use
\f[B]fspl queue-write\f[R] to add jobs to that folder.
It will prevent \f[B]fspl queue-process\f[R] from running.
You can still inspect an append-only queue with commands like \f[B]fspl
queue-ls\f[R] and \f[B]fspl queue-info\f[R].
.SS JOB FILES
.PP
Job files live within \f[B]queuedir/jobs\f[R].
They all must follow this naming pattern:
.IP
.nf
\f[C]
fspl-*.fspl
\f[R]
.fi
.PP
This pattern is specifically designed to facilitate safe injection of
job files into the queue by other tools.
Many other tools prepend or append a temporary string to a filename to
signify that it has not yet been fully transferred.
The Filespooler assumption is that once a file appears in
\f[B]jobs/\f[R] with a name matching this pattern, than it has been
fully transferred and can be processed at any time.
.PP
So long as the filename begins with \f[B]fspl-\f[R] and ends with
\f[B].fspl\f[R], you are free to put whatever string you like in the
middle.
The only other requirement, of course, is that each job must have a
unique filename within the directory.
To simplify things, you can pipe a job file to \f[B]fspl
queue-write\f[R] and let that command take care of naming.
Or, you can generate a random (or non-random) string yourself in a shell
script.
.PP
The job file itself consists of a small binary header, which is
CRC32-checked.
This header is normally less than 100 bytes and the length of it is
encoded within the file.
Following the header, if \f[B]--input\f[R] was given to \f[B]fspl
prepare\f[R], whatever was piped to \f[B]prepare\f[R] is included as the
\[dq]payload\[dq].
This will be piped to the executor command when run by \f[B]fspl
queue-process\f[R] or \f[B]fspl stdin-process\f[R].
The payload is not validated by CRC or length by Filespooler, since this
is assumed to be the role of the transport layer.
The website contains examples of using GPG or other tools to ensure
integrity.
.PP
There are three types of job files:
.IP \[bu] 2
Command, created by \f[B]fspl prepare\f[R].
This is the typical kind of job file, and is used to request the
execution of a command by the processor.
.IP \[bu] 2
NOP, created by \f[B]fspl prepare-nop\f[R].
This is a \[dq]no-op\[dq] job file, which does not run a command but is
considered to always succeed.
.IP \[bu] 2
Fail, created by \f[B]fspl prepare-fail\f[R].
This is a \[dq]fail\[dq] job file, which does not run a command but is
considered to always fail.
This could be usedful, for instance, to create a \[dq]barrier\[dq] to
prevent a queue processor from continuing to execute commands past there
without human intervention.
.SS ADDING FILES TO THE QUEUE
.PP
To expand slightly on the discussion above about adding files to the
queue:
.PP
A common way to do this if your transport tool doesn\[aq]t use a nice
temporary name is to transport the file to an adjacent directory, and
then use \f[B]mv(1)\f[R] or, better, make a hard link with
\f[B]ln(1)\f[R] to get the file into the jobs/ directory.
Note that in both cases, you must take care that you are not crossing a
filesystem boundary; on some platforms such as Linux, mv will revert to
copy instead of rename if you cross the boundary and then the
assumptions about completeness are violated.
.SS JOB FILE ENCODING AND DECODING
.PP
Job files are, by default, stored exactly as laid out above.
However, in many cases, it may be desirable to store them
\[dq]encoded\[dq] - compressed or encrypted.
In this case, the output from \f[B]fspl prepare\f[R] can be piped
through, say, \f[B]gzip\f[R] and the resulting packet can still be
stored in \f[B]jobs/\f[R] by \f[B]fspl queue-write\f[R] or any related
tool.
.PP
Now, however, we arrive at the question: how can Filespooler process a
queue containing files that have been compressed, encrypted, or so
forth?
.PP
Every \f[B]fspl queue\f[R] command takes an optional \f[B]--decoder\f[R]
(or \f[B]-d\f[R]) parameter, which is a command string that will be
executed by the shell.
This decoder command will receive the entire job file (not just the
payload) piped to it on stdin, and is expected to write the decoded file
to stdout.
.PP
The \f[B]fspl stdin\f[R] pairs to the queue commands do not accept a
decoder parameter, since it is assumed you would do that in the pipeline
on the way to the stdin command.
.PP
For instance:
.IP
.nf
\f[C]
date | fspl prepare -s \[ti]/state -i - | gzip | fspl queue-write -q \[ti]/queue

fspl queue-ls -q \[ti]/queue -d zcat
ID                   creation timestamp          filename
48                   2022-05-07T21:07:02-05:00   fspl-48aa52ad-c65c-478a-9d37-123d4bebcb30.fspl
\f[R]
.fi
.PP
Normally, \f[B]fspl\f[R] ignores files that fail to decode the header.
If you omit the \f[B]--decoder\f[R], it may just look like your queue is
empty.
(Using \f[B]--log-level=debug\f[R] will illuminate what is happening.)
.SH DISTRIBUTED NATURE OF FILESPOOLER
.PP
As mentioned, Filespooler is designed to be used as a distributed,
asynchronous, ordered command queue.
The homepage contains many more examples.
Here is one simple example of using ssh as a transport to get commands
to a remote queue:
.IP
.nf
\f[C]
tar -cpf - /usr/local | fspl prepare -s \[ti]/state -i - | ssh remote queue-write -q \[ti]/queue
\f[R]
.fi
.SH INSTALLATION
.PP
\f[B]fspl\f[R] is a Rust program.
If you don\[aq]t already have Rust installed, it can be easily installed
from <https://www.rust-lang.org/>.
.PP
Once Rust is installed, Filespooler can be installed with this command:
.IP
.nf
\f[C]
cargo install filespooler
\f[R]
.fi
.PP
From a checked-out source tree, it can be built by running
\f[B]\f[CB]cargo build --release\f[B]\f[R].
The executable will then be placed in \f[B]target/release/xbnet\f[R].
.PP
You can also obtain pre-built binaries for x86_64 Linux from
<https://salsa.debian.org/jgoerzen/filespooler/-/releases> .
.SH ENVIRONMENT
.PP
\f[B]fspl prepare\f[R] will save certain environment variables to the
packet, which will be set later at process time.
\f[B]fspl {queue,stdin}-process\f[R] will set a number of useful
environment variables in the execution environment.
\f[B]fspl {queue,stdin}-info\f[R] will show the environment that will be
passed to the commands.
See each of these for further discussion.
.SH EXIT CODE
.PP
In general, the commands exit with 0 on success and nonzero on failure.
The concept of success and failure can be complicated in some
situations; see the discussion of the process command.
.PP
These situations explicitly cause a nonzero (error) exit code:
.IP \[bu] 2
Failure to obtain a lock (see \[dq]locking and concurrency\[dq] below),
but only if a lock is required; for many commands, no lock is needed.
.IP \[bu] 2
An I/O error
.IP \[bu] 2
For commands that require a specific job ID (eg, \f[B]fspl
queue-info\f[R]), no job with that ID can be located
.IP \[bu] 2
While processing, the executed command returns a nonzero exit status and
\f[B]--on-error\f[R] is set to \f[B]Retry\f[R] (the default)
.IP \[bu] 2
In some cases, the presence of multiple files in the queuedir with the
same sequence number.
The presence of this condition with commands that take a \f[B]-j ID\f[R]
option, or with \f[B]queue-process\f[R] in its standard configuration,
will cause an error.
.RS 2
.IP \[bu] 2
However, this condition is acceptable for \f[B]queue-ls\f[R] and
\f[B]queue-process --order-by=Timestamp\f[R].
.RE
.PP
These situations explicitly terminate with success (0):
.IP \[bu] 2
While processing, the \f[B]--maxjobs\f[R] limit is reached before some
other error causes an abnormal exit
.IP \[bu] 2
An error while running a command while \f[B]--on-error\f[R] is set to
\f[B]Delete\f[R] or \f[B]Leave\f[R]
.IP \[bu] 2
Files are encountered in the queuedir/jobs directory with unparsable
headers.
\f[B]fspl\f[R] detects and logs (subject to \f[B]--log-level\f[R]) this
condition, but does not consider it an error, on the grounds that the
presence of extra data should not prevent the proper functioning of the
queue.
This may manifest itself in the queue appearing to have nothing to do,
\f[B]queue-ls\f[R] showing fewer jobs than there are files, etc.
A common cause of this may be an incorrect \f[B]--decoder\f[R].
.IP \[bu] 2
Zero jobs in the queue, or zero jobs available to process.
.SH LOCKING AND CONCURRENCY
.PP
Next to every \f[B]seqfile\f[R] on both the sender and within the queue
on the recipient is a file named \f[I]seqfile\f[R].lock.
An exclusive lock is held on this file during the following conditions:
.IP \[bu] 2
On the sender with \f[B]fspl prepare\f[R] and related functions, briefly
while obtaining the next sequence number.
Once this is done, the lock is released, even if the process of
consuming stdin takes a long time.
.IP \[bu] 2
On the recipient, when processing the queue with \f[B]fspl
queue-process\f[R] or other commands that access the seqfile (eg,
\f[B]fspl queue-set-next\f[R]).
.PP
fspl will exit with an error code if it cannot obtain the lock when it
needs it.
.PP
These are situations that explicitly do \f[I]NOT\f[R] obtain a lock:
.IP \[bu] 2
\f[B]fspl queue-write\f[R] or other non-fspl method of injecting packets
into the queue
.IP \[bu] 2
The \f[B]fspl stdin-\f[R] series of commands
.IP \[bu] 2
Commands that scan the queue without accessing the state of the seqfile.
Examples include \f[B]queue-ls\f[R], \f[B]queue-info\f[R], and
\f[B]queue-payload\f[R].
.PP
Note that if the queue is being actively processed while a
\f[B]queue-ls\f[R] is in process, a race condition is possible if a file
disappears between the readdir() call and the time the file is opened
for reading, which could potentially cause queue-ls to fail.
queue-ls intentionally does not attempt to acquire the lock, however,
because it would \f[I]always\f[R] fail while the queue is being
processed in that case, preventing one from being able to list the queue
at all while long-running jobs are in process.
.PP
Note that \f[B]fspl queue-write\f[R] does not need to obtain a lock.
The \f[B]fspl stdin-\f[R] series of commands also do not obtain a lock.
.PP
Taken together, this means that any given queue is intended to be
processed sequentially, not in parallel.
However, if parallel processing is desired, it is trivial to iterate
over the jobs and use \f[B]fspl stdin-process\f[R] in whatever custom
manner you would like.
Also, since queues are so lightweight, there is no problem with creating
thousands of them.
.SH INVOCATION: GLOBAL OPTIONS
.PP
These options may be specified for any command, and must be given before
the command on the command line.
.TP
\f[B]-l\f[R], \f[B]--log-level\f[R] \f[I]LEVEL\f[R]
Information about the progress of \f[B]fspl\f[R] is written to stderr.
This parameter controls how much information is written.
In order from most to least information, the options are: trace, debug,
info, warn, error.
The default is info.
.TP
\f[B]-V\f[R], \f[B]--version\f[R]
Print version information and exit
.TP
\f[B]-h\f[R], \f[B]--help\f[R]
Print help information and exit.
Can also be given after a subcommand, in which case it displays more
detailed information about that subcommand.
.TP
\f[I]COMMAND\f[R]
The subcommand which will be executed.
Required unless using \f[B]--version\f[R] or \f[B]--help\f[R].
.SH INVOCATION: SUBCOMMANDS
.PP
Every subcomand accepts \f[B]--help\f[R] to display a brief summary of
options, invoked as: \f[B]fspl\f[R] \f[I]SUBCOMMAND\f[R]
\f[B]--help\f[R] .
.SS fspl ... prepare
.PP
Generates a packet (job file data) and writes it to stdout.
This file can be piped to other programs (particularly \f[B]fspl
queue-write\f[R]) or saved directly to disk.
.PP
Usage:
.PP
\f[B]fspl\f[R] \f[B]prepare\f[R] [ \f[I]OPTIONS\f[R] ] \f[B]-s\f[R]
\f[I]FILE\f[R] [ \f[B]-- PARAMS...\f[R] ]
.TP
\f[B]-s\f[R], \f[B]--seqfile\f[R] \f[I]FILE\f[R]
Path to the local seqfile.
If it does not already exist, it will be created.
If set to \[dq]-\[dq], then no sequence file is used and the sequence
emitted will always be 1.
.TP
\f[B]-i\f[R], \f[B]--input\f[R] \f[I]INPUT\f[R]
By default, prepare will not read anything as payload.
If \f[I]INPUT\f[R] is set to \[dq]-\[dq], then prepare will read
standard input (stdin) and use it as input.
Otherwise, if \f[I]INPUT\f[R] is anything other than \[dq]-\[dq], it is
assumed to be a filename, which is opened and read for input.
.TP
\f[B]-- \f[BI]PARAMS\f[B]...\f[R]
If a \[dq]--\[dq] is present on the command line, everything after it is
taken as parameters to be added to the generated job packet.
When the packet is later processed, if \f[B]--allow-job-params\f[R] is
given to \f[B]queue-process\f[R] or \f[B]stdin-process\f[R], then these
parameters will be appended to the command line of the executed command.
.PP
In addition to these options, any environment variable beginning with
\f[B]FSPL_SET_\f[R] will be saved in the packet and will be set in the
execution environment at processing time.
.SS fspl ... prepare-fail, prepare-nop
.PP
These commands create a non-command packet, one which is either
considered to always fail or to always succeed (nop).
These two commands take only one option, which is required:
.TP
\f[B]-s\f[R], \f[B]--seqfile\f[R] \f[I]FILE\f[R]
Path to the local seqfile.
Required.
If set to \[dq]-\[dq], then no sequence file is used and the sequence
emitted will always be 1.
.SS fspl ... prepare-get-next
.PP
Prints the sequence number that will be used by the next prepare
command.
.PP
Usage:
.PP
\f[B]fspl\f[R] \f[B]prepare-get-next\f[R] \f[B]-s\f[R] \f[I]FILE\f[R]
.TP
\f[B]-s\f[R], \f[B]--seqfile\f[R] \f[I]FILE\f[R]
Path to the local seqfile.
Required.
If set to \[dq]-\[dq], then no sequence file is used and the sequence
emitted will always be 1.
.SS fspl ... prepare-set-next
.PP
Changes the sequence number that will be used by the next prepare
command.
.PP
Usage:
.PP
\f[B]fspl\f[R] \f[B]prepare-set-next\f[R] \f[B]-s\f[R] \f[I]FILE\f[R]
\f[I]ID\f[R]
.TP
\f[B]-s\f[R], \f[B]--seqfile\f[R] \f[I]FILE\f[R]
Path to the local seqfile.
Required.
\f[I]ID\f[R]
The numeric ID to set the seqfile to.
.SS fspl ... stdin-info, queue-info
.PP
These two commands display information about a given packet.
This information is printed to stdout in a style that is similar to how
the shell sets environment variables.
In fact, it shows precisely the environment variables that will be set
by a corresponding \f[B]process\f[R] command.
.PP
stdin-info expects the packet to be piped in to stdin; queue-info will
find it in the given queue.
.PP
This command will not attempt to read the payload of the file; it will
only read the header.
(Note that this is not a guarantee that some layer of the system may not
try to read a few KB past the header, merely a note that running this
command will not try to read all of a 1TB packet.)
.PP
Usage:
.PP
\f[B]fspl\f[R] \f[B]queue-info\f[R] [ \f[I]OPTIONS\f[R] ] \f[B]-q\f[R]
\f[I]DIR\f[R] \f[B]-j\f[R] \f[I]ID\f[R]
.PP
\f[B]fspl\f[R] \f[B]stdin-info\f[R]
.PP
Options (valid for \f[B]queue-info\f[R] only):
.TP
\f[B]-q\f[R], \f[B]--queuedir\f[R] \f[I]DIR\f[R]
Path to the local queue directory.
Required.
.TP
\f[B]-j\f[R], \f[B]--job\f[R] \f[I]ID\f[R]
Numeric job ID to process.
See \f[B]fspl queue-ls\f[R] to determine this.
Required.
.TP
\f[B]-d\f[R], \f[B]--decoder\f[R] \f[I]DECODECMD\f[R]
Decoder command to run.
This string is passed to \f[B]$SHELL -c\f[R].
See the above conversation about decoders.
Optional.
.PP
Example:
.IP
.nf
\f[C]
fspl queue-info -q /tmp/queue -j 45 -d zcat
FSPL_SEQ=45
FSPL_CTIME_SECS=1651970311
FSPL_CTIME_NANOS=425412511
FSPL_CTIME_RFC3339_UTC=2022-05-08T00:38:31Z
FSPL_CTIME_RFC3339_LOCAL=2022-05-07T19:38:31-05:00
FSPL_JOB_FILENAME=fspl-29342606-02a0-438c-81f2-efdfb80afbe9.fspl
FSPL_JOB_QUEUEDIR=/tmp/bar
FSPL_JOB_FULLPATH=/tmp/bar/jobs/fspl-29342606-02a0-438c-81f2-efdfb80afbe9.fspl
FSPL_PARAM_1=hithere
FSPL_SET_FOO=bar
\f[R]
.fi
.PP
Some notes on these variables:
.IP \[bu] 2
The \f[B]FSPL_JOB_FILENAME\f[R] is relative to the jobs subdirectory of
the queue directory.
.IP \[bu] 2
The \f[B]FSPL_JOB_FULLPATH\f[R] is relative to the current working
directory; that is, it is what was given by \f[B]-q\f[R] plus the path
within that directory to the filename.
It is not guaranteed to be absolute.
.IP \[bu] 2
\f[B]FSPL_PARAM_n\f[R] will be set to the optional parameters passed to
\f[B]fspl prepare\f[R], with n starting at 1.
.IP \[bu] 2
\f[B]FSPL_SET_x\f[R] will reflect any \f[B]FSPL_SET_x\f[R] parameters
that were in the environment when \f[B]fspl prepare\f[R] was run.
.IP \[bu] 2
Filespooler does not enforce limits to environment variable content.
If you want to do something like embed newlines in variable content,
Filespooler will happily accept this (since it is valid POSIX) and
handle it properly - but your shell scripts may not be so lucky.
It is advisable that you avoid this and other weird constructions for
your sanity in working with things outside Filespooler - though
Filespooler won\[aq]t prevent you from doing it.
.SS fspl ... stdin-payload, queue-payload
.PP
These two commands extract the payload (if any) from the given packet.
This is written to stdout.
No header or other information is written to stdout.
.PP
stdin-payload expect the packet to be piped in to stdin; queue-stdout
will find it in the given queue.
.PP
The payload will be piped to the command started by the process
commands.
The payload will be 0-bytes if \f[B]-i\f[R] was not passed to \f[B]fspl
prepare\f[R], or if an empty payload was given to it.
.PP
Usage:
.PP
\f[B]fspl\f[R] \f[B]queue-payload\f[R] [ \f[I]OPTIONS\f[R] ]
\f[B]-q\f[R] \f[I]DIR\f[R] \f[B]-j\f[R] \f[I]ID\f[R]
.PP
\f[B]fspl\f[R] \f[B]stdin-payload\f[R]
.PP
Options (valid for \f[B]queue-payload\f[R] only):
.TP
\f[B]-q\f[R], \f[B]--queuedir\f[R] \f[I]DIR\f[R]
Path to the local queue directory.
Required.
.TP
\f[B]-j\f[R], \f[B]--job\f[R] \f[I]ID\f[R]
Numeric job ID to process.
See \f[B]fspl queue-ls\f[R] to determine this.
Required.
.TP
\f[B]-d\f[R], \f[B]--decoder\f[R] \f[I]DECODECMD\f[R]
Decoder command to run.
This string is passed to \f[B]$SHELL -c\f[R].
See the above conversation about decoders.
Optional.
.SS fspl ... stdin-process, queue-process
.PP
Process packet(s).
stdin-process will process exactly one packet on stdin.
queue-process will process zero or more packets, depending on the
content of the queue and options given.
.PP
Usage:
.PP
\f[B]fspl\f[R] \f[B]queue-process\f[R] [ \f[I]OPTIONS\f[R] ]
\f[B]-q\f[R] \f[I]DIR\f[R] \f[I]COMMAND\f[R] [ \f[B]-- PARAMS...\f[R] ]
.PP
\f[B]fspl\f[R] \f[B]stdin-process\f[R] [ \f[I]OPTIONS\f[R] ]
\f[I]COMMAND\f[R] [ \f[B]-- PARAMS...\f[R] ]
.PP
Common options:
.TP
\f[B]--allow-job-params\f[R]
Specifies that optional parameters given to \f[B]fspl prepare\f[R] will
be passed on the command line to this command
.TP
\f[B]--ignore-payload\f[R]
Ignores the payload; does not pipe it to the command.
.TP
\f[B]--timeout\f[R] \f[I]SECONDS\f[R]
Specifies a timeout, in seconds, for the command.
If the command has not exited within that timeframe, SIGKILL is sent to
the process.
Failing to exit within the timeout is considered an error for
Filespooler\[aq]s purposes.
.TP
\f[I]COMMAND\f[R]
The command to run.
This is \f[I]not\f[R] passed to the shell, so it must point to an
executable.
This command will not be run for NOP or Fail packets.
.TP
\f[B]-- \f[BI]PARAMS\f[B]...\f[R]
If a \[dq]--\[dq] is present on the command line, everything after it is
taken as parameters to be sent to the given command.
If \f[B]--allow-job-params\f[R] is given, then those parameters will be
sent after these.
.PP
Options valid only for \f[B]queue-process\f[R]:
.TP
\f[B]-q\f[R], \f[B]--queuedir\f[R] \f[I]DIR\f[R]
Path to the local queue directory.
Required.
.TP
\f[B]-d\f[R], \f[B]--decoder\f[R] \f[I]DECODECMD\f[R]
Decoder command to run.
This string is passed to \f[B]$SHELL -c\f[R].
See the above conversation about decoders.
Optional.
.TP
\f[B]-n\f[R], \f[B]--maxjobs\f[R] \f[I]JOBS\f[R]
The maximum number of jobs to process.
There is no limit by default.
.TP
\f[B]--never-delete\f[R]
Never delete the job file after processing in any circumstance,
regardless of other options.
.TP
\f[B]--order-by\f[R] \f[I]ORDERING\f[R]
In what order to process the queue.
When \f[B]Sequence\f[R], which is the default, process the queue in
order of sequence number.
When set to \f[B]Timestamp\f[R], process the queue in order of the
creation timestamp as it appears in the job header.
Note that when set to \f[B]Timestamp\f[R], the seqfile within the queue
is neither used nor changed.
\f[B]Timestamp\f[R] implies that you do not care about a strict
sequential ordering of items in cases where items arrive out of order.
.TP
\f[B]--on-error\f[R] \f[I]ONERROR\f[R]
What to do when the supplied command fails (is a fail packet or a
command exits with a nonzero status).
If set to \f[B]Retry\f[R], abort processing with a nonzero error code
and leave the packet in the queue to be tried again by a later
invocation of \f[B]queue-process\f[R].
If set to \f[B]Delete\f[R], delete the packet from the queue (unless
\f[B]--never-delete\f[R] is given), increment the next job counter, and
continue processing the queue normally.
If set to \f[B]Leave\f[R], then leave the packet on disk, increment the
next job counter, and continue processing the rest of the queue
normally.
\f[B]Retry\f[R] is the only option that will cause a failure to not
increment the next job counter.
\f[B]Retry\f[R] is the default.
.TP
\f[B]--output-to\f[R] \f[I]DEST\f[R]
What to do with the stdout and stderr of the invoked command.
If set to \f[B]PassBoth\f[R], then they are simply written to the
stdout/stderr of \f[B]fspl queue-process\f[R].
If set to \f[B]SaveBoth\f[R], then both are added to a file in the
queue\[aq]s jobs directory named \f[B]filename.out\f[R].
This file is up to you to process whenever you wish.
The default is \f[B]PassBoth\f[R].
.PP
The environment is set as described above.
Note that since no queue directory or filename is relevant with the
\f[B]stdin-process\f[R] flavor, those variables are unset under
\f[B]stdin-process\f[R].
.PP
To skip a failing job at the head of the queue, you can use \f[B]fspl
queue-set-next\f[R], or alternatively, \f[B]fspl queue-process
--on-error Delete --maxjobs 1\f[R] to cause it to be deleted.
You would probably not wish to combine this with timestamp ordering.
.SS fspl ... queue-set-next
.PP
Changes the sequence number that will be used by the next \f[B]fspl
queue-process\f[R] command.
.PP
Usage:
.PP
\f[B]fspl\f[R] \f[B]queue-set-next\f[R] \f[B]-q\f[R] \f[I]DIR\f[R]
\f[I]ID\f[R]
.TP
\f[B]-q\f[R], \f[B]--queuedir\f[R] \f[I]DIR\f[R]
Path to the local queue directory.
Required.
.TP
\f[B]--append-only\f[R]
Creates an append-only queue.
\f[I]ID\f[R]
The numeric ID to set the seqfile to.
.SS fspl ... queue-write
.PP
Receives a packet on stdin and writes it to the queue.
This command does not bother to decode, process, or validate the packet
in any way.
It simply writes it to the queue safely, using a temporary filename
until completely written, at which point it is renamed to a
**fspl-*.fspl** file with a random middle part.
.PP
Usage:
.PP
\f[B]fspl\f[R] \f[B]queue-write\f[R] \f[B]-q\f[R] \f[I]DIR\f[R]
.TP
\f[B]-q\f[R], \f[B]--queuedir\f[R] \f[I]DIR\f[R]
Path to the local queue directory.
Required.
.SS fspl ... queue-init
.PP
Creates the queue directory and the needed files and subdirectories
within it.
.PP
Usage:
.PP
\f[B]fspl\f[R] \f[B]queue-init\f[R] \f[B]-q\f[R] \f[I]DIR\f[R]
.TP
\f[B]-q\f[R], \f[B]--queuedir\f[R] \f[I]DIR\f[R]
Path to the local queue directory.
Required.
.SS fspl ... gen-filename
.PP
Generates a filename matching the \f[C]fspl-*.fspl\f[R] pattern, which
will be valid for a job file in a Filespooler queue.
This is often useful when generating a filename that will be used by a
tool other than \f[B]fspl queue-write\f[R].
.PP
Usage:
.PP
\f[B]fspl gen-filename\f[R]
.PP
Example:
.IP
.nf
\f[C]
fspl gen-filename
fspl-b3bd6e63-f62c-49ee-8c46-6677069d2c58.fspl
\f[R]
.fi
.SS fspl ... gen-uuid
.PP
Generates a random UUID and prints it to stdout.
This is generated using the same algorithm as \f[B]fspl queue-write\f[R]
uses.
It can be used in scripts for making your own unique filenames.
.PP
Usage:
.PP
\f[B]fspl gen-uuid\f[R]
.PP
Example:
.IP
.nf
\f[C]
fspl gen-uuid
2896c849-37c5-4a6d-8b90-0cf63e3e9daa
\f[R]
.fi
.SS fspl show-license
.PP
Displays the copyright and license information for fspl.
.SH AUTHOR
.PP
John Goerzen <jgoerzen@complete.org>
.SH HOMEPAGE
.PP
<https://www.complete.org/filespooler/>
.SH COPYRIGHT AND LICENSE
.PP
Copyright (C) 2022 John Goerzen <jgoerzen@complete.org>
.PP
This program is free software: you can redistribute it and/or modify it
under the terms of the GNU General Public License as published by the
Free Software Foundation, either version 3 of the License, or (at your
option) any later version.
.PP
This program is distributed in the hope that it will be useful, but
WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
See the GNU General Public License for more details.
.PP
You should have received a copy of the GNU General Public License along
with this program.
If not, see <http://www.gnu.org/licenses/>.
.SH AUTHORS
John Goerzen.