/*! 
@page inside_tests Testing SimGrid

This page will teach you how to run the tests, selecting the ones you
want, and how to add new tests to the archive.

@tableofcontents

SimGrid code coverage is usually between 70% and 80%, which is much
more than most projects out there. This is because we consider SimGrid
to be a rather complex project, and we want to modify it with less fear.

We have two sets of tests in SimGrid: Each of the 10,000+ unit tests
check one specific case for one specific function, while the 500+
integration tests run a given simulation specifically intended to
exercise a larger amount of functions together. Every example provided
in examples/ is used as an integration test, while some other torture
tests and corner cases integration tests are located in teshsuite/.
For each integration test, we ensure that the output exactly matches
the defined expectations. Since SimGrid displays the timestamp of
every logged line, this ensures that every change of the models'
prediction will be noticed. All these tests should ensure that SimGrid
is safe to use and to depend on.

@section inside_tests_runintegration Running the tests

Running the tests is done using the ctest binary that comes with
cmake. These tests are run for every commit and the result is publicly
<a href="https://ci.inria.fr/simgrid/">available</a>.

@verbatim
ctest		 	  # Launch all tests
ctest -R msg              # Launch only the tests which name match the string "msg"
ctest -j4                 # Launch all tests in parallel, at most 4 at the same time
ctest --verbose           # Display all details on what's going on
ctest --output-on-failure # Only get verbose for the tests that fail

ctest -R msg- -j5 --output-on-failure # You changed MSG and want to check that you didn't break anything, huh?
                                      # That's fine, I do so all the time myself.
@endverbatim

@section inside_tests_rununit Running the unit tests

All unit tests are packed into the unit-tests binary, that lives at the
source root. These tests are run when you launch ctest, don't worry.

@verbatim
make unit-tests                 # Rebuild the test runner on need
./unit-tests                    # Launch all tests
./unit-tests --help             # revise how it goes if you forgot
@endverbatim


@section inside_tests_add_units Adding unit tests

Our unit tests are written using the Catch2 library, that is included
in the source tree. Please check for examples, listed at the end of
tools/cmake/Tests.cmake.

It is important to keep your tests fast. We run them very very often,
and you should strive to make them as fast as possible, to not bother
the other developers. Do not hesitate to stress test your code, but
make sure that it runs reasonably fast, or nobody will run "ctest"
before commiting code.

@section inside_tests_add_integration Adding integration tests

TESH (the TEsting SHell) is the test runner that we wrote for our
integration tests. It is distributed with the SimGrid source file, and
even comes with a man page. TESH ensures that the output produced by a
command perfectly matches the expected output. This is very precious
to ensure that no change modifies the timings computed by the models
without notice. 

To add a new integration test, you thus have 3 things to do:

 - <b>Write the code exercising the feature you target</b>. You should
   strive to make this code clear, well documented and informative for
   the users. If you manage to do so, put this somewhere under
   examples/ and modify the cmake files as explained on this page:
   @ref inside_cmake_examples. If you feel like you should write a
   torture test that is not interesting to the users (because nobody
   would sanely write something similar in user code), then put it under 
   teshsuite/ somewhere.
   
 - <b>Write the tesh file</b>, containing the command to run, the
   provided input (if any, but almost no SimGrid test provide such an
   input) and the expected output. Check the tesh man page for more
   details.@n
   Tesh is sometimes annoying as you have to ensure that the expected
   output will always be exactly the same. In particular, your should
   not output machine dependent informations such as absolute data
   path, nor memory adresses as they would change on each run. Several
   steps can be used here, such as the obfucation of the memory
   adresses unless the verbose logs are displayed (using the
   #XBT_LOG_ISENABLED() macro), or the modification of the log formats
   to hide the timings when they depend on the host machine.@n
   The script located in <project/directory>/tools/tesh/generate_tesh can
   help you a lot in particular if the output is large (though a smaller output is preferable). 
   There are also example tesh files in the <project/directory>/tools/tesh/ directory, that can be useful to understand the tesh syntax.
   
 - <b>Add your test in the cmake infrastructure</b>. For that, modify
   the following file:
   @verbatim
   <project/directory>/teshsuite/<interface eg msg>/CMakeLists.txt
   @endverbatim   
   Make sure to pick a wise name for your test. It is often useful to
   check a category of tests together. The only way to do so in ctest
   is to use the -R argument that specifies a regular expression that
   the test names must match. For example, you can run all MSG test
   with "ctest -R msg". That explains the importance of the test
   names.

Once the name is chosen, create a new test by adding a line similar to
the following (assuming that you use tesh as expected).

@verbatim
# Usage: ADD_TEST(test-name ${CMAKE_BINARY_DIR}/bin/tesh <options> <tesh-file>)
#  option --setenv bindir set the directory containing the binary
#         --setenv srcdir set the directory containing the source file
#         --cd set the working directory
ADD_TEST(my-test-name ${CMAKE_BINARY_DIR}/bin/tesh 
         --setenv bindir=${CMAKE_BINARY_DIR}/examples/my-test/
         --setenv srcdir=${CMAKE_HOME_DIRECTORY}/examples/my-test/
         --cd ${CMAKE_HOME_DIRECTORY}/examples/my-test/
         ${CMAKE_HOME_DIRECTORY}/examples/deprecated/msg/io/io.tesh
)
@endverbatim             

As usual, you must run "make distcheck" after modifying the cmake files,
to ensure that you did not forget any files in the distributed archive.

@section inside_tests_ci Continous Integration

We use several systems to automatically test SimGrid with a large set
of parameters, across as many platforms as possible. 
We use <a href="https://ci.inria.fr/simgrid/">Jenkins on Inria
servers</a> as a workhorse: it runs all of our tests for many
configurations. It takes a long time to answer, and it often reports
issues but when it's green, then you know that SimGrid is very fit!
We use <a href="https://travis-ci.org/simgrid/simgrid">Travis</a> to
quickly run some tests on Linux and Mac. It answers quickly but may
miss issues. And we use <a href="https://ci.appveyor.com/project/mquinson/simgrid">AppVeyor</a>
to build and somehow test SimGrid on windows. 

@subsection inside_tests_jenkins Jenkins on the Inria CI servers

You should not have to change the configuration of the Jenkins tool
yourself, although you could have to change the slaves' configuration
using the <a href="https://ci.inria.fr">CI interface of INRIA</a> --
refer to the <a href="https://wiki.inria.fr/ciportal/">CI documentation</a>.

The result can be seen here: https://ci.inria.fr/simgrid/

We have 2 interesting projects on Jenkins:
@li <a href="https://ci.inria.fr/simgrid/job/SimGrid/">SimGrid</a>
    is the main project, running the tests that we spoke about.@n It is
    configured (on Jenkins) to run the script <tt>tools/jenkins/build.sh</tt>
@li <a href="https://ci.inria.fr/simgrid/job/SimGrid-DynamicAnalysis/">SimGrid-DynamicAnalysis</a>
    should be called "nightly" because it does not only run dynamic
    tests, but a whole bunch of long lasting tests: valgrind (memory
    errors), gcovr (coverage), Sanitizers (bad pointer usage, threading
    errors, use of unspecified C constructs) and the clang static analyzer.@n It is configured
    (on Jenkins) to run the script <tt>tools/jenkins/DynamicAnalysis.sh</tt>

In each case, SimGrid gets built in
/builds/workspace/$PROJECT/build_mode/$CONFIG/label/$SERVER/build 
with $PROJECT being for instance "SimGrid", $CONFIG "DEBUG" or
"ModelChecker" and $SERVER for instance "simgrid-fedora20-64-clang".

If some configurations are known to fail on some systems (such as
model-checking on non-linux systems), go to your Project and click on
"Configuration". There, find the field "combination filter" (if your
interface language is English) and tick the checkbox; then add a
groovy-expression to disable a specific configuration. For example, in
order to disable the "ModelChecker" build on host
"small-netbsd-64-clang", use:

@verbatim
(label=="small-netbsd-64-clang").implies(build_mode!="ModelChecker")
@endverbatim

Just for the record, the slaves were created from the available
template with the following commands:
@verbatim
#debian/ubuntu
apt-get install gcc g++ gfortran automake cmake libboost-dev openjdk-8-jdk openjdk-8-jre libxslt-dev libxml2-dev libevent-dev libunwind-dev libdw-dev htop git python3 xsltproc libboost-context-dev
#for dynamicanalysis: 
apt-get install jacoco libjacoco-java libns3-dev pcregrep gcovr ant lua5.3-dev sloccount

#fedora
dnf install libboost-devel openjdk-8-jdk openjdk-8-jre libxslt-devel libxml2-devel xsltproc git python3 libdw-devel libevent-devel libunwind-devel htop lua5.3-devel

#netbsd
pkg_add cmake gcc7 boost boost-headers automake openjdk8 libxslt libxml2 libunwind git htop python36

#opensuse
zypper install cmake automake clang boost-devel java-1_8_0-openjdk-devel libxslt-devel libxml2-devel xsltproc git python3 libdw-devel libevent-devel libunwind-devel htop binutils ggc7-fortran

#freebsd
pkg install boost-libs cmake openjdk8 automake libxslt libxml2 libunwind git htop python3  automake gcc6 flang elfutils libevent
#+ clang-devel from ports

#osx
brew install cmake boost libunwind-headers libxslt git python3 
@endverbatim

@subsection inside_tests_travis Travis

Travis is a free (as in free beer) Continuous Integration system that
open-sourced project can use freely. It is very well integrated in the
GitHub ecosystem. There is a plenty of documentation out there. Our
configuration is in the file .travis.yml as it should be, and the
result is here: https://travis-ci.org/simgrid/simgrid

The .travis.yml configuration file can be useful if you fail to get
SimGrid to compile on modern mac systems. We use the @c brew package
manager there, and it works like a charm.

@subsection inside_tests_appveyor AppVeyor

AppVeyor aims at becoming the Travis of Windows. It is maybe less
mature than Travis, or maybe it is just that I'm less trained in
Windows. Our configuration is in the file appveyor.yml as it should
be, and the result is here: https://ci.appveyor.com/project/mquinson/simgrid

We use @c Choco as a package manager on AppVeyor, and it is sufficient
for us. In the future, we will probably move to the ubuntu subsystem
of Windows 10: SimGrid performs very well under these settings, as
tested on Inria's CI servers. For the time being having a native
library is still useful for the Java users that don't want to install
anything beyond Java on their windows.

@subsection inside_tests_debian Debian builders

Since SimGrid is packaged in Debian, we benefit from their huge
testing infrastructure. That's an interesting torture test for our
code base. The downside is that it's only for the released versions of
SimGrid. That is why the Debian build does not stop when the tests
fail: post-releases fixes do not fit well in our workflow and we fix
only the most important breakages.

The build results are here:
https://buildd.debian.org/status/package.php?p=simgrid

@subsection inside_tests_sonarqube SonarQube

SonarQube is an open-source code quality analysis solution. Their nice
code scanners are provided as plugin. The one for C++ is not free, but
open-source project can use it at no cost. That is what we are doing.

Don't miss the great looking dashboard here: 
https://sonarcloud.io/dashboard?id=simgrid_simgrid

This tool is enriched by the script @c tools/internal/travis-sonarqube.sh 
that is run from @c .travis.yml

*/