General documentation for the parallel package for Octave.

Copyright © 2016-2023 Olaf Till <i7tiol@t-online.de>

You can redistribute this documentation and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 3 of the License, or (at your option) any later version.

This documentation is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this documentation; if not, see <http://www.gnu.org/licenses/>.

     [file macros.texi not included]

General documentation for the parallel package for Octave
*********************************************************

This documentation applies to version of the parallel package.

The package contains functions for explicit local parallel execution and functions for parallel execution over a cluster of machines, possibly combined with local parallel execution at each machine of the cluster.

Earlier versions of the package used Octave sessions directly generated by the fork system call. Since this causes problems, the current version uses newly started Octave sessions, even for local parallel execution. This causes some limitations, see Limitations.

The server for cluster execution is installed at GNU/Linux, but not at some other operating systems, e.g. not at Windows or macOS. The client functions for cluster execution should be installed at all systems, though.

Installation hints
******************

Note: For using a cluster of machines, identical versions of the package are required to be installed at each machine.

Maybe your distribution provides the parallel package.

If not, or if you want a newer version, Octaves pkg command allows building the package from source and installing it. Note that you have to load any Octave package before you can use it. See Octaves documentation of pkg.

Parallel execution over a cluster by default is protected by encryption with authentication. This needs a version of the gnutls library in which TLS-SRP is not disabled. For building the parallel package from source, this library must be present during the build; some operating systems provide development files of libraries in exta ...-dev packages, which are also needed during the build.

Building or using the package for cluster execution without gnutls will work, but encryption and authentication are not available in this case.

Functions for local parallel execution
**************************************

Explicit local parallel execution, with the intent to exploit more than one local processor(-core), is performed by calling a user-defined function in parallel with several different arguments. This is done in parallel processes. Note that there is no way to pass global variables to or from these processes.

The interface of the functions for local parallel execution is similar to Octaves cellfun and arrayfun functions.

Note that some operations in Octave, particularly some matrix operations, may already be performed in parallel threads. This may limit the advantage yielded by explicit local parallel execution. Also, RAM access can be a bottleneck which limits computation speed of multicore computing.

For limitations, see Limitations.

Function parcellfun
===================

(parcellfun)

Function pararrayfun
====================

(pararrayfun)

See also parcellfun, arrayfun (octave).

Function parcellfun_set_nproc
=============================

(parcellfun_set_nproc)

Functions for parallel execution over a cluster of machines
***********************************************************

There are high level functions, similar to parcellfun and pararrayfun, which only require the user to provide a function to be called and a series of argument sets. Parallel function calls are then internally distributed over the cluster, and over the local processors (or processor cores) within the single machines of the cluster.

Medium and low level functions are also provided. These require the user to program the scheduling of parallel execution.

Data transfer is possible between the client machine and each server machine, but also directly between server machines. The latter feature is exploited e.g. by the install_vars function.

For limitations, see Limitations.

Security considerations
=======================

Over a connection to a server of the parallel package arbitrary Octave commands can be executed at the server machine. This means that any change can be done to the server system and the resident data, only limited by the rights of the system account under which the server runs.

By default, the server is started with authentication and encryption enabled, to avoid unauthorized access. If the server is started with authentication disabled (maybe to avoid the encryption overhead), it must be cared for that no TCP connection by unauthorized persons is possible to the server ports, possibly by running the client and all server machines behind a firewall and assuring that only trusted persons have access to any machine behind the firewall. This scenario might be achievable in home-nets.

The server currently uses port 12502 for receiving commands and port 12501 for data exchange.

The client and the servers used by the client with pconnect must agree on using authentication or not.

Do not start the server as root.

Generating authentication keys
==============================

(parallel_generate_srp_data)

Starting servers
================

(pserver)

Establishing cluster connections
================================

(pconnect)

Closing cluster connections
===========================

(sclose)

Get network information
=======================

(network_get_info)

Set network properties
======================

(network_set)

Function netcellfun
===================

(netcellfun)

Function netarrayfun
====================

(netarrayfun)

Distribute Octave variables
===========================

(install_vars)

Single remote function evaluation
=================================

(rfeval)

Sending Octave variables
========================

(psend)

Receiving Octave variables
==========================

(precv)

Remote command evaluation
=========================

(reval)

Call Unix select for data connections
=====================================

(select_sockets)

Example
=======

     # From Octave prompt, generate authentication files, set user name to
     # 'test'. When prompted for a password, press <enter>.
     parallel_generate_srp_data ('test')

     # From Octave prompt, get location of the generated files.
     authpath = fullfile (a = pkg ("prefix"), "parallel-srp-data")

     Copy server files to servers, authpath is assumed to be
     "/home/test/octave/parallel-srp-data/", the same directory is assumed to
     exist on the servers. From the system shell, do e.g.:

     scp -r /home/test/octave/parallel-srp-data/server server1:/home/test/octave/parallel-srp-data/
     scp -r /home/test/octave/parallel-srp-data/server server2:/home/test/octave/parallel-srp-data/

     Start server at remote machines. From the system shell, do e.g.:

     ssh server1 'octave --eval "pserver"'
     ssh server2 'octave --eval "pserver"'

     # From Octave prompt, connect the cluster.
     conns = pconnect ({"server1", "server2"})

     # And perform some parallel execution. Single function calls take 1
     # second each.
     results = netcellfun (conns,  (x) {x, pause(1)}{:}, num2cell (1:30))

     # Close network.
     sclose (conns)

Functions possibly helpful in parallel execution
************************************************

Transfer variable within the local machine over an Octave stream
================================================================

(fsave)

(fload)

Call Unix select on Octave streams
==================================

Note that for cluster connections select_sockets should be used instead.

(select)

Limitations of high- and medium-level functions
***********************************************

Note that netcellfun, netarrayfun, and rfeval, but even parcellfun and pararrayfun, execute code in separate Octave processes. This causes some limitations.

For the function handles passed as arguments to parcellfun, pararrayfun, netcellfun, netarrayfun, and rfeval, the following limitations may apply:

  * In Octave versions before 5.1, anonymous functions may only be passed if they don't use varargin and don't contain nested anonymous functions.

  * If a handle to a named function, written in a function file, is passed, this function must exist on the server machine. This is usually no problem for parcellfun and pararrayfun.

  * Named functions defined in the command line interface or by scripts are not exported to the working sessions used by parcellfun or pararrayfun, and not automatically to the server for remote parallel execution.

  * For handles to subfunctions or to private functions limitations depending on the used Octave version may apply.

Anonymous functions should be usable as usual, since the function specification sent to the server will include the anonymous functions context.

If parcellfun or pararrayfun (and of course the cluster-related functions, too) are called from within a function with subfunctions, the subfunctions won't be visible in parallel execution.

There is no way to set global variables for the function called by parcellun or pararrayfun. For cluster execution, setting global variables is possible, e.g. with install_vars; however, netcellfun and netarrayfun won't 'see' global variables if they use parcellfun internally (which is the default at remote machines with more than one available processor core).

Function parallel_doc to view documentation
*******************************************

(parallel_doc)