.\" Man page generated from reStructuredText.
.
.
.nr rst2man-indent-level 0
.
.de1 rstReportMargin
\\$1 \\n[an-margin]
level \\n[rst2man-indent-level]
level margin: \\n[rst2man-indent\\n[rst2man-indent-level]]
-
\\n[rst2man-indent0]
\\n[rst2man-indent1]
\\n[rst2man-indent2]
..
.de1 INDENT
.\" .rstReportMargin pre:
. RS \\$1
. nr rst2man-indent\\n[rst2man-indent-level] \\n[an-margin]
. nr rst2man-indent-level +1
.\" .rstReportMargin post:
..
.de UNINDENT
. RE
.\" indent \\n[an-margin]
.\" old: \\n[rst2man-indent\\n[rst2man-indent-level]]
.nr rst2man-indent-level -1
.\" new: \\n[rst2man-indent\\n[rst2man-indent-level]]
.in \\n[rst2man-indent\\n[rst2man-indent-level]]u
..
.TH "CLUSH" "1" "2025-01-23" "1.9.3" "ClusterShell User Manual"
.SH NAME
clush \- execute shell commands on a cluster
.SH SYNOPSIS
.sp
\fBclush\fP \fB\-a\fP | \fB\-g\fP \fIgroup\fP | \fB\-w\fP \fInodes\fP  [OPTIONS]
.sp
\fBclush\fP \fB\-a\fP | \fB\-g\fP \fIgroup\fP | \fB\-w\fP \fInodes\fP  [OPTIONS] \fIcommand\fP
.sp
\fBclush\fP \fB\-a\fP | \fB\-g\fP \fIgroup\fP | \fB\-w\fP \fInodes\fP  [OPTIONS] \-\-copy
\fIfile\fP | \fIdir\fP [ \fIfile\fP | \fIdir\fP ...] [ \-\-dest \fIpath\fP ]
.sp
\fBclush\fP \fB\-a\fP | \fB\-g\fP \fIgroup\fP | \fB\-w\fP \fInodes\fP  [OPTIONS] \-\-rcopy
\fIfile\fP | \fIdir\fP [ \fIfile\fP | \fIdir\fP ...] [ \-\-dest \fIpath\fP ]
.SH DESCRIPTION
.sp
\fBclush\fP is a program for executing commands in parallel on a cluster and for
gathering their results. \fBclush\fP executes commands interactively or can be
used within shell scripts and other applications.  It is a partial front\-end
to the ClusterShell library that ensures a light, unified and robust parallel
command execution framework. Thus, it allows traditional shell scripts to
benefit from some of the library features. \fBclush\fP currently makes use of
the Ssh worker of ClusterShell, by default, that only requires \fBssh\fP(1)
(OpenSSH SSH client).
.SH INVOCATION
.sp
\fBclush\fP can be started non\-interactively to run a shell \fIcommand\fP, or can
be invoked as an interactive shell. To start a \fBclush\fP interactive session,
invoke the \fBclush\fP command without providing \fIcommand\fP\&.
.INDENT 0.0
.TP
.B Non\-interactive mode
When \fBclush\fP is started non\-interactively, the \fIcommand\fP is executed on
the specified remote hosts in parallel. If option \fB\-b\fP or \fB\-\-dshbak\fP
is specified, \fBclush\fP waits for command completion and then displays
gathered output results.
.sp
The \fB\-w\fP option allows you to specify remote hosts by using ClusterShell
NodeSet syntax, including the node groups \fB@group\fP special syntax and the
\fBExtended Patterns\fP syntax to benefits from NodeSet basic arithmetic
(like \fB@Agroup\e&@Bgroup\fP). See EXTENDED PATTERNS in \fBnodeset\fP(1) and
also \fBgroups.conf\fP(5) for more information.
.sp
Unless the option \fB\-\-nostdin\fP (or \fB\-n\fP) is specified, \fBclush\fP detects
when its standard input is connected to a terminal (as determined by
\fBisatty\fP(3)).  If actually connected to a terminal, \fBclush\fP listens to
standard input when commands are running, waiting for an \fIEnter\fP key press.
Doing so will display the status of current nodes.  If standard input is not
connected to a terminal, and unless the option \fB\-\-nostdin\fP is specified,
\fBclush\fP binds the standard input of the remote commands to its own standard
input, allowing scripting methods like:
.INDENT 7.0
.INDENT 3.5
.nf
# echo foo | clush \-w node[40\-42] \-b cat
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
node[40\-42]
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
foo
.fi
.sp
.UNINDENT
.UNINDENT
.sp
Please see some other great examples in the EXAMPLES section below.
.TP
.B Interactive session
If a \fIcommand\fP is not specified, and its standard input is connected to a
terminal, \fBclush\fP runs interactively. In this mode, \fBclush\fP uses the GNU
\fBreadline\fP library to read command lines. Readline provides commands for
searching through the command history for lines containing a specified
string. For instance, type Control\-R to search in the history for the next
entry matching the search string typed so far.  \fBclush\fP also recognizes
special single\-character prefixes that allows the user to see and modify
the current nodeset (the nodes where the commands are executed).
.INDENT 7.0
.TP
.B Single\-character interactive commands are:
.INDENT 7.0
.TP
.B clush> ?
show current nodeset
.TP
.B clush> @<NODESET>
set current nodeset
.TP
.B clush> +<NODESET>
add nodes to current nodeset
.TP
.B clush> \-<NODESET>
remove nodes from current nodeset
.TP
.B clush> !COMMAND
execute COMMAND on the local system
.TP
.B clush> =
toggle the output format (gathered or standard mode)
.UNINDENT
.UNINDENT
.sp
To leave an interactive session, type \fBquit\fP or Control\-D.
.TP
.B Local execution ( \fB\-\-worker=exec\fP or \fB\-R exec\fP )
Instead of running provided command on remote nodes, \fBclush\fP can use the
dedicated \fIexec\fP worker to launch the command \fIlocally\fP, for each node.
Some parameters could be used in the command line to make a different
command for each node. \fB%h\fP or \fB%host\fP will be replaced by node name and
\fB%n\fP or \fB%rank\fP by the remote rank [0\-N] (to get a literal % use %%)
.TP
.B File copying mode ( \fB\-\-copy\fP )
When \fBclush\fP is started with the \fB\-c\fP or \fB\-\-copy\fP option, it will
attempt to copy specified \fIfiles\fP and/or \fIdirectories\fP to the provided
cluster nodes.  The \fB\-\-dest\fP option can be used to specify a single
path where all the file(s) should be copied to on the target nodes.
In the absence of \fB\-\-dest\fP, \fBclush\fP will attempt to copy each file or
directory found in the command line to their same location on the
target nodes.
.TP
.B Reverse file copying mode ( \fB\-\-rcopy\fP )
When \fBclush\fP is started with the \fB\-\-rcopy\fP option, it will attempt to
retrieve specified \fIfile\fP and/or \fIdir\fP from provided cluster nodes. If the
\fB\-\-dest\fP option is specified, it must be a directory path where the files
will be stored with their hostname appended. If the destination path is not
specified, it will take each \fIfile\fP or \fIdirectory\fP\(aqs parent directory as the
local destination.
.UNINDENT
.SH OPTIONS
.INDENT 0.0
.TP
.B  \-\-version
show \fBclush\fP version number and exit
.TP
.BI \-s \ GROUPSOURCE\fR,\fB \ \-\-groupsource\fB= GROUPSOURCE
optional \fBgroups.conf\fP(5) group source to use
.TP
.B  \-n\fP,\fB  \-\-nostdin
do not watch for possible input from stdin; this should be used when \fBclush\fP is run in the background (or in scripts).
.TP
.BI \-\-groupsconf\fB= FILE
use alternate config file for groups.conf(5)
.TP
.BI \-\-conf\fB= FILE
use alternate config file for clush.conf(5)
.TP
.BI \-O \ <KEY=VALUE>\fR,\fB \ \-\-option\fB= <KEY=VALUE>
override any key=value \fBclush.conf\fP(5) options (repeat as needed)
.UNINDENT
.INDENT 0.0
.TP
.B Selecting target nodes:
.INDENT 7.0
.TP
.BI \-w \ NODES
nodes where to run the command
.TP
.BI \-x \ NODES
exclude nodes from the node list
.TP
.B  \-a\fP,\fB  \-\-all
run command on all nodes
.TP
.BI \-g \ GROUP\fR,\fB \ \-\-group\fB= GROUP
run command on a group of nodes
.TP
.BI \-X \ GROUP
exclude nodes from this group
.TP
.BI \-\-hostfile\fB= FILE\fR,\fB \ \-\-machinefile\fB= FILE
path to a file containing a list of single hosts, node sets or node groups, separated by spaces and lines (may be specified multiple times, one per file)
.TP
.BI \-\-topology\fB= FILE
topology configuration file to use for tree mode
.TP
.BI \-\-pick\fB= N
pick N node(s) at random in nodeset
.UNINDENT
.TP
.B Output behaviour:
.INDENT 7.0
.TP
.B  \-q\fP,\fB  \-\-quiet
be quiet, print essential output only
.TP
.B  \-v\fP,\fB  \-\-verbose
be verbose, print informative messages
.TP
.B  \-d\fP,\fB  \-\-debug
output more messages for debugging purpose
.TP
.B  \-G\fP,\fB  \-\-groupbase
do not display group source prefix
.TP
.B  \-L
disable header block and order output by nodes; if \-b/\-B is not specified, \fBclush\fP will wait for all commands to finish and then display aggregated output of commands with same return codes, ordered by node name; alternatively, when used in conjunction with \-b/\-B (eg. \-bL), \fBclush\fP will enable a \(dqlife gathering\(dq of results by line, such as the next line is displayed as soon as possible (eg. when all nodes have sent the line)
.TP
.B  \-N
disable labeling of command line
.TP
.B  \-P\fP,\fB  \-\-progress
show progress during command execution; if writing is performed to standard input, the live progress indicator will display the global bandwidth of data written to the target nodes
.TP
.B  \-b\fP,\fB  \-\-dshbak
display gathered results in a dshbak\-like way (note: it will only try to aggregate the output of commands with same return codes)
.TP
.B  \-B
like \-b but including standard error
.TP
.B  \-r\fP,\fB  \-\-regroup
fold nodeset using node groups
.TP
.B  \-S\fP,\fB  \-\-maxrc
return the largest of command return codes
.TP
.BI \-\-color\fB= WHENCOLOR
\fBclush\fP can use NO_COLOR, CLICOLOR and CLICOLOR_FORCE environment variables. NO_COLOR takes precedence over CLICOLOR_FORCE which takes precedence over CLICOLOR.  When \fB\-\-color\fP option is used these environment variables are not taken into account. \fB\-\-color\fP tells whether to use ANSI colors to surround node or nodeset prefix/header with escape sequences to display them in color on the terminal. \fIWHENCOLOR\fP is \fBnever\fP, \fBalways\fP or \fBauto\fP (which use color if standard output/error refer to a terminal). Colors are set to [34m (blue foreground text) for stdout and [31m (red foreground text) for stderr, and cannot be modified.
.TP
.B  \-\-diff
show diff between common outputs (find the best reference output by focusing on largest nodeset and also smaller command return code)
.TP
.BI \-\-outdir\fB= OUTDIR
output directory for stdout files (OPTIONAL)
.TP
.BI \-\-errdir\fB= ERRDIR
output directory for stderr files (OPTIONAL)
.UNINDENT
.TP
.B File copying:
.INDENT 7.0
.TP
.B  \-c\fP,\fB  \-\-copy
copy local file or directory to remote nodes
.TP
.B  \-\-rcopy
copy file or directory from remote nodes
.TP
.BI \-\-dest\fB= DEST_PATH
destination file or directory on the nodes
(optional: use the first source directory
path when not specified)
.TP
.B  \-p
preserve modification times and modes
.UNINDENT
.TP
.B Connection options:
.INDENT 7.0
.TP
.BI \-f \ FANOUT\fR,\fB \ \-\-fanout\fB= FANOUT
do not execute more than FANOUT commands at the same time, useful to limit resource usage. In tree mode, the same \fIfanout\fP value is used on the head node and on each gateway (the \fIfanout\fP value is propagated). That is, if the \fIfanout\fP is \fB16\fP, each gateway will initiate up to \fB16\fP connections to their target nodes at the same time. Default \fIfanout\fP value is defined in \fBclush.conf\fP(5).
.TP
.BI \-l \ USER\fR,\fB \ \-\-user\fB= USER
execute remote command as user
.TP
.BI \-o \ OPTIONS\fR,\fB \ \-\-options\fB= OPTIONS
can be used to give ssh options, eg. \fB\-o \(dq\-p 2022 \-i ~/.ssh/myidrsa\(dq\fP; these options are added first to ssh and override default ones
.TP
.BI \-t \ CONNECT_TIMEOUT\fR,\fB \ \-\-connect_timeout\fB= CONNECT_TIMEOUT
limit time to connect to a node
.TP
.BI \-u \ COMMAND_TIMEOUT\fR,\fB \ \-\-command_timeout\fB= COMMAND_TIMEOUT
limit time for command to run on the node
.TP
.BI \-m \ MODE\fR,\fB \ \-\-mode\fB= MODE
run mode; define MODEs in \fB<confdir>/*.conf\fP
.TP
.BI \-R \ WORKER\fR,\fB \ \-\-worker\fB= WORKER
worker name to use for connection (\fBexec\fP, \fBssh\fP, \fBrsh\fP, \fBpdsh\fP, or the name of a Python worker module), default is \fBssh\fP
.TP
.BI \-\-remote\fB= REMOTE
whether to enable remote execution: in tree mode, \(aqyes\(aq forces connections to the leaf nodes for execution, \(aqno\(aq establishes connections up to the leaf parent nodes for execution (default is \(aqyes\(aq)
.UNINDENT
.UNINDENT
.sp
For a short explanation of these options, see \fB\-h, \-\-help\fP\&.
.SH EXIT STATUS
.sp
By default, an exit status of zero indicates success of the \fBclush\fP command
but gives no information about the remote commands exit status. However, when
the \fB\-S\fP option is specified, the exit status of \fBclush\fP is the largest
value of the remote commands return codes.
.sp
For failed remote commands whose exit status is non\-zero, and unless the
combination of options \fB\-qS\fP is specified, \fBclush\fP displays messages
similar to:
.INDENT 0.0
.TP
.B clush: node[40\-42]: exited with exit code 1
.UNINDENT
.SH EXAMPLES
.SS Remote parallel execution
.INDENT 0.0
.TP
.B # clush \-w node[3\-5,62] uname \-r
Run command \fIuname \-r\fP in parallel on nodes: node3, node4, node5 and node62
.UNINDENT
.SS Local parallel execution
.INDENT 0.0
.TP
.B # clush \-w node[1\-3] \-\-worker=exec ping \-c1 %host
Run locally, in parallel, a ping command for nodes: node1, node2 and node3.
You may also use \fB\-R exec\fP as the shorter and pdsh compatible option.
.UNINDENT
.SS Display features
.INDENT 0.0
.TP
.B # clush \-w node[3\-5,62] \-b uname \-r
Run command \fIuname \-r\fP on nodes[3\-5,62] and display gathered output results (integrated \fBdshbak\fP\-like).
.TP
.B # clush \-w node[3\-5,62] \-bL uname \-r
Line mode: run command \fIuname \-r\fP on nodes[3\-5,62] and display gathered output results without default header block.
.TP
.B # ssh node32 find /etc/yum.repos.d \-type f | clush \-w node[40\-42] \-b xargs ls \-l
Search some files on node32 in /etc/yum.repos.d and use clush to list the matching ones on node[40\-42], and use \fB\-b\fP to display gathered results.
.TP
.B # clush \-w node[3\-5,62] \-\-diff dmidecode \-s bios\-version
Run this Linux command to get BIOS version on nodes[3\-5,62] and show version differences (if any).
.UNINDENT
.SS All nodes
.INDENT 0.0
.TP
.B # clush \-a uname \-r
Run command \fIuname \-r\fP on all cluster nodes, see \fBgroups.conf\fP(5) to setup all cluster nodes (\fIall:\fP field).
.TP
.B # clush \-a \-x node[5,7] uname \-r
Run command \fIuname \-r\fP on all cluster nodes except on nodes node5 and node7.
.TP
.B # clush \-a \-\-diff cat /some/file
Run command \fIcat /some/file\fP on all cluster nodes and show differences (if any), line by line, between common outputs.
.UNINDENT
.SS Node groups
.INDENT 0.0
.TP
.B # clush \-w @oss modprobe lustre
Run command \fImodprobe lustre\fP on nodes from node group named \fIoss\fP, see \fBgroups.conf\fP(5) to setup node groups (\fImap:\fP field).
.TP
.B # clush \-g oss modprobe lustre
Same as previous example but using \fB\-g\fP to avoid \fI@\fP group prefix.
.TP
.B # clush \-w @mds,@oss modprobe lustre
You may specify several node groups by separating them with commas (please see EXTENDED PATTERNS in \fBnodeset\fP(1) and also \fBgroups.conf\fP(5) for more information).
.UNINDENT
.SS Copy files
.INDENT 0.0
.TP
.B # clush \-w node[3\-5,62] \-\-copy /etc/motd
Copy local file \fI/etc/motd\fP to remote nodes node[3\-5,62].
.TP
.B # clush \-w node[3\-5,62] \-\-copy /etc/motd \-\-dest /tmp/motd2
Copy local file \fI/etc/motd\fP to remote nodes node[3\-5,62] at path \fI/tmp/motd2\fP\&.
.TP
.B # clush \-w node[3\-5,62] \-c /usr/share/doc/clustershell
Recursively copy local directory \fI/usr/share/doc/clustershell\fP to the same
path on remote nodes node[3\-5,62].
.TP
.B # clush \-w node[3\-5,62] \-\-rcopy /etc/motd \-\-dest /tmp
Copy \fI/etc/motd\fP from remote nodes node[3\-5,62] to local \fI/tmp\fP directory, each file having their remote hostname appended, eg. \fI/tmp/motd.node3\fP\&.
.UNINDENT
.SH FILES
.INDENT 0.0
.TP
.B \fI$CLUSTERSHELL_CFGDIR/clush.conf\fP
Global clush configuration file. If $CLUSTERSHELL_CFGDIR is not defined,
\fI/etc/clustershell/clush.conf\fP is used instead.
.TP
.B \fI$XDG_CONFIG_HOME/clustershell/clush.conf\fP
User configuration file for clush. If $XDG_CONFIG_HOME is not defined,
\fI$HOME/.config/clustershell/clush.conf\fP is used instead.
.TP
.B \fI$HOME/.local/etc/clustershell/clush.conf\fP
Local user configuration file for clush (default installation for pip \-\-user)
.TP
.B \fI~/.clush.conf\fP
Deprecated per\-user clush configuration file.
.TP
.B \fI~/.clush_history\fP
File in which interactive \fBclush\fP command history is saved.
.UNINDENT
.SH SEE ALSO
.sp
\fBclubak\fP(1), \fBcluset\fP(1), \fBnodeset\fP(1), \fBreadline\fP(3), \fBclush.conf\fP(5), \fBgroups.conf\fP(5).
.sp
 <http://clustershell.readthedocs.org/> 
.SH BUG REPORTS
.INDENT 0.0
.TP
.B Use the following URL to submit a bug report or feedback:
 <https://github.com/cea\-hpc/clustershell/issues> 
.UNINDENT
.SH AUTHOR
Stephane Thiell <sthiell@stanford.edu>
.SH COPYRIGHT
GNU Lesser General Public License version 2.1 or later (LGPLv2.1+)
.\" Generated by docutils manpage writer.
.