..  Copyright (c) 2013-2020, SIB - Swiss Institute of Bioinformatics and
..                           Biozentrum - University of Basel
..  
..  Licensed under the Apache License, Version 2.0 (the "License");
..  you may not use this file except in compliance with the License.
..  You may obtain a copy of the License at
..  
..    http://www.apache.org/licenses/LICENSE-2.0
..  
..  Unless required by applicable law or agreed to in writing, software
..  distributed under the License is distributed on an "AS IS" BASIS,
..  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
..  See the License for the specific language governing permissions and
..  limitations under the License.


|project| Setup
================================================================================

The following should give an overview of how this project is set up. Anyone
planning to develop parts of |project| should read this! Important topics are
|git| branches, the directory structure and tightly linked with this also
|cmake|.

.. _git-branches:

--------------------------------------------------------------------------------
|git| Branches
--------------------------------------------------------------------------------
Basically we have two, sometimes three major branches. ``master``, ``develop``
and in front of a new release a dedicated release branch. For bugs, hotfix
branches of a rather short life are used.

``master`` is the stable branch, corresponding to a released version. It is
solely fed by a release or hotfix branch.

Release branches, usually labelled ``release-<VERSION>``, are branched of
``develop`` to fix features and thoroughly test them before a new major
release. Once everything looks trustworthy, such a branch is merged into
``master`` and since there should be a few bug fixes in, ``master`` is merged
into ``develop``. Bugs are fixed in dedicated hotfix branches, which should
only exist for the fix and testing itself. Those are forged from release
branches or ``master``. If created for ``master``, they are also merged back
into ``develop``.

The ``develop`` branch exists to introduce new features up to the level of
whole projects extending |project| and see that they work seamlessly together
with the rest of the system. There do exist a couple of rather strict rules for
what goes into this branch:

* Your code must have been (briefly) reviewed by others
* There have to be unit tests
* It needs to pass ``make check`` **including** ``doctest`` & ``linkcheck``
* Your project needs documentation
* It must not break the ability of out-of-source builds

The reason to be a bit restrictive on branches which end up in actual releases,
should be mostly obvious: |project| is used by productive services as a third
party toolbox. There it is not an item of active development and people
probably have no insight in its internals. So messing up a new release creates
a lot of extra work for a lot of people. First for the developer of a service
to find out that |project| has turned malicious, then for the maintainer of
this package to figure out that its your contribution messing things up and in
the end for you, fixing the problems.

The place where you may get messy is your own |git| branch within the |project|
repository. This is basically where you should develop your project. Once you
created something that could go into a release, tidy things up according to the
rules from above and merge it into ``develop``. From there it will
automatically find its way into the next release.

To set up your own branch, start from a current ``develop`` branch:

.. code-block:: console

   $ git checkout develop         # switch to branch develop
   $ git pull --rebase            # update branch develop
   $ git checkout -b <BRANCHNAME> # create branch <BRANCHNAME> and switch to it

Over time, ``develop`` may recognise some changes, e.g. new features, which you
want to make use of in your project. Keeping your branch up to date is a three
step process. |git| does not allow updates on top of changed code, so either
changes have to be committed, or if in the middle of implementing something,
stored away temporarily. Making commits is straight forward:

.. code-block:: console

   $ git commit -m '<DESCRIPTION>' # commit changes including a comment

Hiding your changes away from |git| just for updating files is a bit more
involved. Everything is easily stored on an internal stack and needs to be
fetched from there, once the branch was updated. One major problem in the past
was a possible loss of code by those operations. If the update changes a file
you have changed, too, and stashed away, this may end up in a non-resolvable
merge conflict and your changes are lost. Usually the log tells you, which
files were recently modified. Moving all current changes to the stack is
achieved by:

.. code-block:: console

   $ git stash save

To revive them, use:

.. code-block:: console

   $ git stash pop

After cleaning up your branch, switch to ``develop``, update it and switch back:

.. code-block:: console

   $ git checkout develop
   $ git pull --rebase
   $ git checkout <BRANCHNAME>

Now for actually updating your branch, there are two different ways: merging
and rebasing. A rebase may only be done, if you **never** pushed your branch to
the origin of the repository (otherwise you will mess up history, in the worst
case ``develop`` may be unusable once you merge):

.. code-block:: console

   $ git rebase develop

For branches which are available to others, do a proper merge:

.. code-block:: console

   $ git merge develop

This may require some manual conflict solving and will end up in a merge commit.

--------------------------------------------------------------------------------
|git| Hooks
--------------------------------------------------------------------------------
|git| hooks are scripts invoked by |git| in connection to certain commands.
|project| currently provides one for :command:`commit`. It is installed by

.. code-block:: console

   $ cp extras/pre_commit/pre-commit .git/hooks/

Its task is applying coding standards and doing a bunch of other checks on the
files involved in a commit. Everything around the script is hosted in 
:file:`extras/pre_commit/`. The checks can be manually executed with

.. code-block:: console

   $ python .git/hooks/pre-commit

If you ever have to skip the hook,

.. code-block:: console

   $ git commit --no-verify

does the trick. **But** checks are always run on the complete file containing
changes, not only on the lines changed. This means if you opt out of an issue,
it will reappear next time that very file changes.

For checking |python| code, the pre-commit hook employs |pylint|_, to make sure
we stay close to |pep8|_. If you feel the need to make changes to the |pylint|
call, please make sure you fully understand what the complaints are. Sometimes
|pep8| sounds overly restrictive but it may help with performance and
compatibility with |python| 3. For |project| it is also important that the code
looks similar throughout the various modules. So do not disable a check because
it just seems inconvenient or you do not understand why |pylint| is croaking at
what looks like 'working' code. But then there are also cases where |pylint| is
not smart enough to cope with valid |pep8| code. For changes with valid cause,
the configuration flushed into |pylint| may be found at
:file:`extras/pre_commit/pm3_csc/filecheck/pylintrc` and
:file:`extras/pre_commit/pm3_csc/filecheck/pylint-unittest-rc`. The latter one
is invoked on unit test code, where we may go a little bit less restrictive.

--------------------------------------------------------------------------------
Directory Structure
--------------------------------------------------------------------------------
The directory structure of the |project| repository is supposed to 'keep
everything together that belongs together'. That is, code, documentation and
extra data should be gathered on a per-module basis immediately in the
repository root. The directory structure of your module should look like this:

.. code-block:: text

   promod3.git/                            Project folder
         your_module/                      Module directory
               CMakeLists.txt              CMake configuration
               data/                       Extra data (if needed)
                     CMakeLists.txt        CMake configuration
                     ...
               doc/                        Documentation
                     CMakeLists.txt        CMake configuration
                     your_module.rst       Overview/frame of your module
                     ...
               pymod/                      Python code
                     CMakeLists.txt        CMake configuration
                     __init__.py           Init file needed for import
                     submodule1.py         Code
                     ...
               src/                        C/ C++ code
                     CMakeLists.txt        CMake configuration
                     source1.cc            C++ code
                     source2.hh            Header
                     ...
               tests/                      Unit tests
                     CMakeLists.txt        CMake configuration
                     data/                 Test data (if needed)
                           ...
                     test_your_module.py   Unit tests for your_module
                     test_submodule1.py    Unit tests for submodule1
                     ...

Additionally to the module directories there are a few extra folders:

- :file:`actions`: Scripts callable as ``pm <ACTION_NAME>``.
  See :ref:`here <how-to-start-your-own-action>` for details.
- :file:`cmake_support`: Helper functions for |cmake|.
  See :ref:`here <pm3-cmake-doc>` for details.
- :file:`doc`: High-level documentation, test scripts (:file:`doc/tests`) and a
  copy of the generated html documentation (:file:`doc/html`). The latter must
  be kept up-to-date at least on the ``master`` branch.
  See :ref:`here <writing-documentation>` for details.
- :file:`extras`: Extra data and information that doesn't fit anywhere
  else (e.g. |git| hooks or scripts to recreate the binary files).
- :file:`scripts`: Input for scripts that end up in :file:`stage/bin`

--------------------------------------------------------------------------------
|cmake|
--------------------------------------------------------------------------------
The attentive reader may have noticed all the :file:`CMakeLists.txt` files in
the directory structure. Those are needed to configure the build system, e.g.
tell it which files have to be considered packaging, compiling, etc.. Also
|python| modules are declared there as well as which files belong to the
documentation. |cmake| is a rather complex topic (unfortunately all usable
build systems seem to be) so we skip a detailed view, here, and just advice you
to go by example. There is a tiny bit of documentation on our additions to
|cmake| :ref:`here <pm3-cmake-doc>`. If you really need to make changes to the
build system, other than adding new files and modules, you have to dive into
|cmake| documentation all by yourself and on your own responsibility. You have
been warned.

--------------------------------------------------------------------------------
The :file:`stage` Directory
--------------------------------------------------------------------------------
Once you hit :command:`make` in your build directory, a directory :file:`stage`
in this path will be populated. It just resembles a directory structure as of a
usual Unix file system filled with the build products of |project|. The
:file:`stage` directory tree can already be utilised. You may import Python
modules from there, use the binaries from :file:`stage/bin`, etc..


.. |pylint| replace:: Pylint
.. _pylint: https://pylint.readthedocs.io