Source code
===========

Code style
----------

We intentionally do not have too many code conventions in the Open MPI
code base.

All languages
^^^^^^^^^^^^^

* 4 space tabs.  No more, no less.
* **NEVER** use actual tab characters; always use spaces.  Both emacs
  and vim have secret mojo that can automatically use spaces when you
  hit the ``<TAB>`` key.  This makes the code look the same in every
  browser, regardless of individual tab display settings.

C / C++
^^^^^^^

* When comparing constants for equality or inequality, always put the
  constant on the left.  This is defensive programming: if you have a
  typo in the test and miss a ``!`` or ``=``, you'll get a compiler error.
  For example:

  .. code-block:: c

     /* Do this */
     if (NULL == foo) { ... }

     /* Because if you have a typo (i.e., = instead of ==), this will
        be a compile error rather than a subtle bug */
     if (NULL = foo) { ... }

* More defensive programming: *always* include blocks in curly braces
  ``{ }``, even if they are only one line long.  For example:

  .. code-block:: c

     /* Do this */
     if (whatever) {
         return OMPI_SUCCESS;
     }

     /* Not this */
     if (whatever)
        return OMPI_SUCCESS;

* Starting with Open MPI 1.7, Open MPI requires a C99-compliant
  compiler.

  * C++-style comments are now allowed (and preferred).
  * C99-style mixing declarations are allow allowable (and preferred).

* **ALWAYS** include ``<level>_config.h`` as your first #include file,
  where ``<level>`` is one of ``ompi``, ``oshmem``, or ``opal`` |mdash| the
  level that you're writing in.  There are very, very few cases where
  this is not true (E.g., some bizarre Windows scenarios).  But in
  99.9999% of cases, this file should be included **first** so that it
  can affect system-level #include files if necessary.
* Filenames and symbols must follow the **prefix rule** (see [e-mail
  thread](http://www.open-mpi.org/community/lists/devel/2009/07/6389.php)):

  * Filenames must be prefixed with ``<framework>_<component>``.
  * Public symbols must be prefixed in components with
    ``<project>_<framework>_<component>``, where ``<project>`` is one
    of ``mca``, ``ompi``, ``oshmem``, or ``opal``.  Note that `mca`
    used to be the most common, but it has fallen out of favor
    compared to the other ``<project>`` prefixes.  When in doubt about
    whether a symbol is public, be safe and add the prefix.
  * Non-public symbols must be declared ``static`` or otherwise made to
    not appear in the global scope.

* **ALWAYS** #define macros, even for logical values.

  * The GNU Way is to ``#define`` a macro when it is "true" and to
    ``#undef`` it when it is "false".
  * In Open MPI, we **always** ``#define`` a logical macro to be
    either 0 or 1 |mdash| we never ``#undef`` it.
  * The reason for this is defensive programming: if you are only
    checking if a preprocessor macro is defined (via ``#ifdef FOO`` or
    ``#if defined(FOO)``), you will get no warning when compiling if
    you accidentally misspell the macro name.  However, if you use the
    logic test ``#if FOO`` with an undefined macro (e.g., because you
    misspelled it), you'll get a compiler warning or error.

    .. admonition:: Rationale
       :class: tip

       Misspelled macro names can be tremendously difficult to find
       when they are buried in thousands of lines of code; we will
       take all the help from the preprocessor/compiler that we can
       get!

  .. code-block:: c

     /* GNU Way - you will get no warning from the compiler if you
        misspell "FOO"; the test will simply be false */
     #ifdef FOO
     ...
     #else
     ...
     #endif

     /* Open MPI Way - you will get a warning from the compiler if you
        misspell "FOO"; the result of the test is a different value
        than whether you spelled the macro name right or not */
     #if FOO
     ...
     #else
     ...
     #endif

Fortran
^^^^^^^

We do not have specific coding style guidelines for Fortran.  Please
read some of the existing Fortran code in the source code tree and try
to use a similar style.

Shell scripting
^^^^^^^^^^^^^^^

Please read some of the existing shell code in the source code tree
and try to use a similar style.

* Always enclose evaluated shell variables in quotes to ensure that
  multi-token values are handled properly.

  .. code-block:: sh

     # This is bad
     if test $foo = bar; then

     # This is good
     if test "$foo" = "bar"; then

  * The one exception to this is that when doing an assignment to a
    shell variable from another shell variable, it is not necessary to
    use quotes on the right hand side:

    .. code-block:: sh

       # This is harmless, but unnecessary
       foo="$bar"

       # This is actually sufficient, even for multi-token values of $bar
       foo=$bar

* Do not use the ``==`` operator for ``test`` |mdash| this is a GNU
  extension and can cause portability problems on BSD systems.
  Instead, use the single ``=`` operator.

  .. code-block:: sh

     # This is bad
     if test "$foo" == "bar"; then

     # This is good
     if test "$foo" = "bar"; then

m4
^^^

We do not have specific coding style guidelines for m4 (the language
used to create the ``configure`` script).  Please read some of the
existing m4 code in the source code tree and try to use a similar
style.

Tree layout
-----------

There are a few notable top-level directories in the source
tree:

* The main sub-projects:

    * ``oshmem``: Top-level OpenSHMEM code base
    * ``ompi``: The Open MPI code base
    * ``opal``: The OPAL code base

* ``config``: M4 scripts supporting the top-level ``configure`` script
  ``mpi.h``
* ``etc``: Some miscellaneous text files
* ``docs``: Source code for Open MPI documentation
* ``examples``: Trivial MPI / OpenSHMEM example programs
* ``3rd-party``: Included copies of required core libraries (either
  via Git submodules in Git clones or via binary tarballs).

  .. note:: While it may be considered unusual, we include binary
     tarballs (instead of Git submodules) for 3rd party projects that
     are:

     #. Needed by Open MPI for correct operation, and
     #. Not universally included in OS distributions, and
     #. Rarely updated.

Each of the three main source directories (``oshmem``, ``ompi``, and
``opal``) generate at least a top-level library named ``liboshmem``,
``libmpi``, and ``libopen-pal``, respectively.  They can be built as
either static or shared libraries.  Executables are also produced in
subdirectories of some of the trees.

The ``libopen-pal`` top-level library is built internally in two parts:

* ``libopen-pal_core`` Internal "core" portion of OPAL containing the essential source and MCA needed for tools like mpicc/mpirun to link against. The "core" library is not installed.

   * Includes the following MCA frameworks: ``backtrace``, ``dl``, ``installdirs``,  ``threads``,  ``timer``
   * Includes all of the source under ``opal/class`` and most of ``opal/util``
   * Includes the files suffixed with ``_core`` in ``opal/runtime``

* ``libopen-pal``  Includes "core" plus all of the other OPAL project sources. This is installed.

Each of the sub-project source directories have similar (but not
identical) directory structures under them:

* ``class``: C++-like "classes" (using the OPAL class system)
  specific to this project
* ``include``: Top-level include files specific to this project
* ``mca``: MCA frameworks and components specific to this project
* ``runtime``: Startup and shutdown of this project at runtime
* ``tools``: Executables specific to this project
* ``util``: Random utility code

There are other top-level directories in each of the sub-projects,
each having to do with specific logic and code for that project.  For
example, the MPI API implementations can be found under
``ompi/mpi/LANGUAGE``, where ``LANGUAGE`` is ``c``, ``fortran``, or
``java``.

The layout of the ``mca`` trees are strictly defined.  They are of the
form:

.. code-block:: text

    PROJECT/mca/FRAMEWORK/COMPONENT

To be explicit: it is forbidden to have a directory under the ``mca``
trees that does not meet this template (with the exception of ``base``
directories, explained below).  Hence, only framework and component
code can be in the ``mca`` trees.

That is, framework and component names must be valid directory names
(and C variables; more on that later).  For example, the TCP BTL
component is located in ``opal/mca/btl/tcp/``.

The name ``base`` is reserved; there cannot be a framework or component
named ``base``. Directories named ``base`` are reserved for the
implementation of the MCA and frameworks.  Here are a few examples (as
of the |ompi_series| source tree):

.. code-block:: sh

    # Main implementation of the MCA
    opal/mca/base

    # Implementation of the btl framework
    opal/mca/btl/base

    # Implementation of the sysv framework
    oshmem/mcs/sshmem/sysv

    # Implementation of the pml framework
    ompi/mca/pml/base

Under these mandated directories, frameworks and/or components may have
arbitrary directory structures, however.

Symbol Visibility
-----------------

The ``*_DECLSPEC`` macros provide a method to annotate symbols to indicate
their intended visibility when compiling dynamically shared object files
(e.g., ``libmpi.so``).  The macros are defined on a per project basis:

* Open MPI: ``OMPI_DECLSPEC``
* Open PAL: ``OPAL_DECLSPEC``
* OpenSHMEM: ``OSHMEM_DECLSPEC``

The macros expand to the appropriate compiler and platform flags for marking
whether a symbol should be explicitly made public in the target project's
library namespace.
The ``*_DECLSPEC`` attributes are used to declare that a symbol is to be
visible outside of that library/DSO's scope.  For example, ``OMPI_DECLSPEC``
is used to control what symbols are visible in the ``libmpi.so`` scope.

.. note:: This is entirely related to dynamic library compilation and does not
   apply to static compilation.

.. note:: The macros were originally introduced when Open MPI supported
   Windows (circa Open MPI v1.0.0) and are motivated by the Windows
   `__declspec <https://docs.microsoft.com/en-us/cpp/cpp/declspec?view=msvc-170>`_.
   While support for Windows has been dropped from Open MPI, the symbol
   visibility macros remain.