/*
 * Robot Testing Framework
 *
 * Copyright (C) 2015-2019 Istituto Italiano di Tecnologia (IIT)
 *
 * This library is free software; you can redistribute it and/or
 * modify it under the terms of the GNU Lesser General Public
 * License as published by the Free Software Foundation; either
 * version 2.1 of the License, or (at your option) any later version.
 *
 * This library is distributed in the hope that it will be useful,
 * but WITHOUT ANY WARRANTY; without even the implied warranty of
 * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
 * Lesser General Public License for more details.
 *
 * You should have received a copy of the GNU Lesser General Public
 * License along with this library; if not, write to the Free Software
 * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301  USA
 */

/**

@page robottestingframework-testrunner Running test case plug-ins using robottestingframework-testrunner

\section content Contents
\li \ref desc
\li \ref single
\li \ref multiple
\li \ref suite
\li \ref single-suite
\li \ref multiple-suite
\li \ref weblistener
\li \ref stress-test
\li \ref options

<br>
\section desc Description
The \c robottestingframework-testrunner is a command line tool to load and run
multiple tests which has been developed as Robot Testing Framework plug-ins.
The test case plug-ins are built as dynamic loadable libraries and depending on
the target platform can be found as \c *.so, \c *.dll or *.dylib or using Lua,
Python, Ruby and other scripting languages (e.g., .lua, .py, .rb).
The \c robottestingframework-testrunner can be used to run a single test case,
multiple test cases from a given path, a \ref suite or multiple test suites from
a given path.

<br>
\section single Running a single test case
To run a single test case plug-in, it is enough to give the name of the
corresponding plug-ing file to the \c robottestingframework-testrunner using
\c --test option.
For example on a Linux machine:

\verbatim
 $ robottestingframework-testrunner --verbose --test libmytest.so
\endverbatim

The \c robottestingframework-testrunner follows the platform policy to look for
the plug-in file.
On a Linux machine, the default folders for the shared libraries and the paths
indicated in the \c `LD_LIBRARY_PATH` are searched for the corresponding
plug-in.
Alternatively, the complete path to the plug-in file can be given to load the
corresponding plug-in:

\verbatim
 $ robottestingframework-testrunner --verbose --test ~/my-plugins/libmytest.so
\endverbatim

In the similar way you can run a test case which has been developed using Lua,
Python, Ruby or other languages:

\verbatim
 $ robottestingframework-testrunner --verbose --test ~/my-plugins/mytest.lua
 $ robottestingframework-testrunner --verbose --test ~/my-plugins/mytest.py
 $ robottestingframework-testrunner --verbose --test ~/my-plugins/mytest.rb
\endverbatim

The \c `--verbose` enables the verbose mode which allows that the result of the
tests to be also written to the standard out put.
By default the result of the test are stored in the \c `result.txt` which can be
changed by using \c `--output` option.

<br>
\section multiple Running all test cases from a given path
Multiple test cases can be easily loaded and run using \c `--tests` option.
The expected value for this option is a directory where test case plug-ins are
stored.

\verbatim
 $ robottestingframework-testrunner --verbose --tests ~/my-plugins
\endverbatim

If the \c `--recursive` switch is given with the \c `--tests` option, the
sub-folders are also searched for plug-ins.

<br>
\section suite Test suite
By definition, a test suite is a set of test cases which share the same test
fixture.
In Robot Testing Framework a set of test cases (plug-ins) can be grouped in a
test suite using a XML file.
Here is an example of test suite:

\htmlonly
<!-- HTML generated using hilite.me --><div style="background: #ffffff; overflow:auto;width:auto;border:solid gray;border-width:.1em .1em .1em .8em;padding:.2em .6em;"><pre style="margin: 0; line-height: 125%"><span style="color: #557799">&lt;?xml version=&quot;1.0&quot; encoding=&quot;UTF-8&quot;?&gt;</span>

<span style="color: #007700">&lt;suite</span> <span style="color: #0000CC">name=</span><span style="background-color: #fff0f0">&quot;my suite&quot;</span><span style="color: #007700">&gt;</span>
    <span style="color: #007700">&lt;description&gt;</span> a simple example <span style="color: #007700">&lt;/description&gt;</span>
    <span style="color: #007700">&lt;environment&gt;</span> --robot simulator <span style="color: #007700">&lt;/environment&gt;</span>
    <span style="color: #007700">&lt;fixture</span> <span style="color: #0000CC">param=</span><span style="background-color: #fff0f0">&quot;--from myfixture.ini&quot;</span><span style="color: #007700">&gt;</span> libfixture.so <span style="color: #007700">&lt;/fixture&gt;</span>
    <span style="color: #007700">&lt;fixture</span> <span style="color: #0000CC">param=</span><span style="background-color: #fff0f0">&quot;--from myfixture2.ini&quot;</span><span style="color: #007700">&gt;</span> libfixture.so <span style="color: #007700">&lt;/fixture&gt;</span>

    <span style="color: #007700">&lt;test</span> <span style="color: #0000CC">param=</span><span style="background-color: #fff0f0">&quot;--from myconf.ini&quot;</span><span style="color: #007700">&gt;</span> libtest1 <span style="color: #007700">&lt;/test&gt;</span>
    <span style="color: #007700">&lt;test</span> <span style="color: #0000CC">type=</span><span style="background-color: #fff0f0">&quot;dll&quot;</span><span style="color: #007700">&gt;</span> libtest1 <span style="color: #007700">&lt;/test&gt;</span>
    <span style="color: #007700">&lt;test</span> <span style="color: #0000CC">param=</span><span style="background-color: #fff0f0">&quot;&quot;</span><span style="color: #007700">&gt;</span> ~/mytest/libtest2.so <span style="color: #007700">&lt;/test&gt;</span>
    <span style="color: #007700">&lt;test</span> <span style="color: #0000CC">repetition=</span><span style="background-color: #fff0f0">&quot;5&quot;</span><span style="color: #007700">&gt;</span> ~/mytest/test3.lua <span style="color: #007700">&lt;/test&gt;</span>
    <span style="color: #007700">&lt;test</span> <span style="color: #0000CC">param=</span><span style="background-color: #fff0f0">&quot;&quot;</span><span style="color: #007700">&gt;</span> ~/mytest/test4.py <span style="color: #007700">&lt;/test&gt;</span>
    <span style="color: #007700">&lt;test</span> <span style="color: #0000CC">param=</span><span style="background-color: #fff0f0">&quot;&quot;</span><span style="color: #007700">&gt;</span> ~/mytest/test5.rb <span style="color: #007700">&lt;/test&gt;</span>
    ...
<span style="color: #007700">&lt;/suite&gt;</span>
</pre></div>
\endhtmlonly

The example demonstrates a test suite which includes two test cases
(\c libmytest1.so and \c libmytest2.so).
An optional string parameter can be also specified for each test using \c param
attribute in the \c test tag.
​The parameter string is parsed into (argc/argv) format and will be passed to the
test case using robottestingframework::TestCase::setup(int argc, char** argv).
Alternatively, the raw value of the parameter can be access in the test plug-in
implementation using robottestingframework::TestCase::getParam method.

The environment can be used in the implementation of the test cases to know in
which setup (e.g., simulation, real world environment) the test case is
employed.
The environment value can also be accessed in the test plug-in implementation
using robottestingframework::TestCase::getEnvironment method.

If a fixture manager plug-in is specified using \c <fixture> tag, it will be
loaded by the \c robottestingframework-testrunner and is set for the current
test suite.
When suite is executed, the fixture manager will be called automatically to
setup and monitor the test fixture which is implemented in the fixture plug-in.
(Please see the \c examples/fixture-plugin folder for an example of implementing
fixture plug-ins.)

Notice that, for \c libtest2.so,  a complete path to the plug-in file is given.
Thus, the \c robottestingframework-testrunner does not look into system default
paths (e.g., \c LD_LIBRARY_PATH) for the plug-in file and it expects to find the
plug-in file using the given path.
However, this is not the case for \c libtest1.so.

The `robottestingframework-testrunner` usually checks for the extension of the
plug-in file name to recognize the type of the plug-in.
There is also possibility to explicitly indicate the type of the plug-in using
the `type` attribute.
For example if `type="dll"`, then the plug-in test should be given without the
extension.
Depending on which platform (Windows, Linux or Mac) the test is running, the
`robottestingframework-testrunner` looks for the correct (shared library)
plug-in name (.dll, .so or .dylib).

<br>
\section single-suite Running a single test suite
A single test suite (XML file) can be run using \c `--suite` option.
For example:
\verbatim
 $ robottestingframework-testrunner --verbose --suite ~/my-suites/suite.xml
\endverbatim

<br>
\section multiple-suite Running multiple test suites
Multiple test suites can be loaded and rune using \c `--suites` option.
The expected value for this option is a directory where test suites (XML files)
are stored.

\verbatim
 $ robottestingframework-testrunner --verbose --suites ~/my-suites
\endverbatim

If the \c `--recursive` switch is given with the \c `--suites` option, the
sub-folders are also searched for test suites.

<br>
\section weblistener Monitoring tests from your browser
The `robottestingframework-testrunner` uses
`robottestingframework::WebProgressListener` to provide a simple web server
which allows you to monitor the test's running progress from a browser.
To do that, you need to run a test/suite by the
`robottestingframework-testrunner` using `--web-reporter` option.

\verbatim
 $ robottestingframework-testrunner --web-reporter --verbose --suites ~/my-suites
\endverbatim

Then open a web browser and type `http://127.0.0.1:8080` or the remote address
where the `robottestingframework-testrunner` has been launched.
You can also change the `robottestingframework-testrunner` default port by using
`--web-port <port number>` option.

Notice that, the web reporting functionality may not be available by default and
you need to enable it when building the Robot Testing Framework by turning on
the `ENABLE_WEB_LISTENER` from the cmake.

<br>
\section stress-test Using Robot Testing Framework for stress testing
For stress testing, it is possible to repeat a test case for desired iterations.
This can be easily done by providing the number of the test runs as the
parameter (e.g., `--repetition 5`) for the `robottestingframework-testrunner`
or setting as the attribute of the test case within a test suite XML file.
The `robottestingframework-testrunner` repeats running the test for the desired
iterations if the test does not fail.
Notice that, only the `robottestingframework::TestCase::run()` will be called
repeatedly.

\verbatim
 $ robottestingframework-testrunner --verbose --test ~/my-plugins/mytest.lua --repetition 5
\endverbatim



<br>
\section options The robottestingframework-testrunner options
Here is the full list of \c robottestingframework-testrunner options:

\verbatim
usage: robottestingframework-testrunner [options] ...
options:
  -t, --test            Runs a single test from the given plugin file. (string [=])
      --tests           Runs multiple tests from the given folder which contains plugins. (string [=])
  -s, --suite           Runs a single test suite from the given XML file. (string [=])
      --suites          Runs multiple test suites from the given folder which contains XML files. (string [=])
  -o, --output          The output file to save the result. Default is result.txt (string [=])
      --output-type     The output file type (text, junit) (string [=text])
  -p, --param           Sets the test case parameters. (Can be used only with --test option.) (string [=])
  -e, --environment     Sets the test case environment. (Can be used only with --test option.) (string [=])
      --repetition      Sets the test run repetition. (Can be used only with --test option.) (j [=0])
  -w, --web-reporter    Enables web reporter
      --web-port        Sets the web reporter server port. (The default port number is 8080.) (i [=8080])
  -r, --recursive       Searches into subfolders for plugins or XML files. (Can be used with --tests or --suites options.)
  -d, --detail          Enables verbose mode of test assertions.
  -v, --verbose         Enables verbose mode.
      --version         Shows version information.
  -?, --help            print this message
\endverbatim

\author Ali Paikan

*/