.. -*- rst -*-

   Copyright (c) 2022-2023 Nanook Consulting.  All rights reserved.
   Copyright (c) 2023      Jeffrey M. Squyres.  All rights reserved.

   $COPYRIGHT$

   Additional copyrights may follow

   $HEADER$

.. The following line is included so that Sphinx won't complain
   about this file not being directly included in some toctree

Overloading and Oversubscribing
===============================

This section explores the difference between the terms "overloading"
and "oversubscribing". Users are often confused by the difference
between these two scenarios. As such, this section provides a number
of scenarios to help illustrate the differences.

* ``--map-by :OVERSUBSCRIBE`` allow more processes on a node than
  allocated

* ``--bind-to <object>:overload-allowed`` allows for binding more than
  one process in relation to a CPU

The important thing to remember with *oversubscribing* is that it can
be defined separately from the actual number of CPUs on a node. This
allows the mapper to place more or fewer processes per node than
CPUs. By default, PRRTE uses cores to determine slots in the absence
of such information provided in the hostfile or by the resource
manager (except in the case of the ``--host`` as described in
the section on that command line option.

The important thing to remember with *overloading* is that it is
defined as binding more processes than CPUs. By default, PRRTE uses
cores as a means of counting the number of CPUs. However, the user can
adjust this. For example when using the ``:HWTCPUS`` qualifier to the
``--map-by`` option PRRTE will use hardware threads as a means of
counting the number of CPUs.

For the following examples consider a node with:

* 2 processor packages,
* 10 cores per package, and
* 8 hardware threads per core.

Consider the node from above with the hostfile below:

.. code::

   $ cat myhostfile
   node01 slots=32
   node02 slots=32

The ``slots`` token tells PRRTE that it can place up to 32 processes
before *oversubscribing* the node.

If we run the following:

.. code::

   prun --np 34 --hostfile myhostfile --map-by core --bind-to core hostname

It will return an error at the binding time indicating an
*overloading* scenario.

The mapping mechanism assigns 32 processes to ``node01`` matching the
``slots`` specification in the hostfile. The binding mechanism will bind
the first 20 processes to unique cores leaving it with 12 processes
that it cannot bind without overloading one of the cores (putting more
than one process on the core).

Using the ``overload-allowed`` qualifier to the ``--bind-to core``
option tells PRRTE that it may assign more than one process to a core.

If we run the following:

.. code::

   prun --np 34 --hostfile myhostfile --map-by core --bind-to core:overload-allowed hostname

This will run correctly placing 32 processes on ``node01``, and 2
processes on ``node02``. On ``node01`` two processes are bound to
cores 0-11 accounting for the overloading of those cores.

Alternatively, we could use hardware threads to give binding a lower
level CPU to bind to without overloading.

If we run the following:

.. code::

   prun --np 34 --hostfile myhostfile --map-by core:HWTCPUS --bind-to hwthread hostname

This will run correctly placing 32 processes on ``node01``, and 2
processes on ``node02``. On ``node01`` two processes are mapped to
cores 0-11 but bound to different hardware threads on those cores (the
logical first and second hardware thread). Thus no hardware threads
are overloaded at binding time.

In both of the examples above the node is not oversubscribed at
mapping time because the hostfile set the oversubscription limit to
``slots=32`` for each node. It is only after we exceed that limit that
PRRTE will throw an oversubscription error.

Consider next if we ran the following:

.. code::

   prun --np 66 --hostfile myhostfile --map-by core:HWTCPUS --bind-to hwthread hostname

This will return an error at mapping time indicating an
oversubscription scenario. The mapping mechanism will assign all of
the available slots (64 across 2 nodes) and be left two processes to
map. The only way to map those processes is to exceed the number of
available slots putting the job into an oversubscription scenario.

You can force PRRTE to oversubscribe the nodes by using the
``:OVERSUBSCRIBE`` qualifier to the ``--map-by`` option as seen in the
example below:

.. code::

   prun --np 66 --hostfile myhostfile \
       --map-by core:HWTCPUS:OVERSUBSCRIBE --bind-to hwthread hostname

This will run correctly placing 34 processes on ``node01`` and 32 on
``node02``.  Each process is bound to a unique hardware thread.

Overloading vs. Oversubscription: Package Example
-------------------------------------------------

Let's extend these examples by considering the package level.
Consider the same node as before, but with the hostfile below:

.. code::

   $ cat myhostfile
   node01 slots=22
   node02 slots=22

The lowest level CPUs are "cores" and we have 20 total (10 per
package).

If we run:

.. code::

   prun --np 20 --hostfile myhostfile --map-by package \
       --bind-to package:REPORT hostname

Then 10 processes are mapped to each package, and bound at the package
level.  This is not overloading since we have 10 CPUs (cores)
available in the package at the hardware level.

However, if we run:

.. code::

   prun --np 21 --hostfile myhostfile --map-by package \
       --bind-to package:REPORT hostname

Then 11 processes are mapped to the first package and 10 to the second
package.  At binding time we have an overloading scenario because
there are only 10 CPUs (cores) available in the package at the
hardware level. So the first package is overloaded.

Overloading vs. Oversubscription: Hardware Threads Example
----------------------------------------------------------

Similarly, if we consider hardware threads.

Consider the same node as before, but with the hostfile below:

.. code::

   $ cat myhostfile
   node01 slots=165
   node02 slots=165

The lowest level CPUs are "hwthreads" (because we are going to use the
``:HWTCPUS`` qualifier) and we have 160 total (80 per package).

If we re-run (from the package example) and add the ``:HWTCPUS``
qualifier:

.. code::

   prun --np 21 --hostfile myhostfile --map-by package:HWTCPUS \
       --bind-to package:REPORT hostname

Without the ``:HWTCPUS`` qualifier this would be overloading (as we
saw previously). The mapper places 11 processes on the first package
and 10 to the second package. The processes are still bound to the
package level. However, with the ``:HWTCPUS`` qualifier, it is not
overloading since we have 80 CPUs (hwthreads) available in the package
at the hardware level.

Alternatively, if we run:

.. code::

   prun --np 161 --hostfile myhostfile --map-by package:HWTCPUS \
       --bind-to package:REPORT hostname

Then 81 processes are mapped to the first package and 80 to the second
package.  At binding time we have an overloading scenario because
there are only 80 CPUs (hwthreads) available in the package at the
hardware level.  So the first package is overloaded.