= OpenSCAP Developer Manual
:oscap_git: https://github.com/OpenSCAP/openscap
:toc: preamble
:numbered:

This part of documentation is meant to serve mainly to developers who want to
contribute to OpenSCAP, help to fix bugs, or take an advantage of
the OpenSCAP library and create own projects on top of it.


== Building OpenSCAP on Linux
If you want to build the `libopenscap` library and the `oscap` tool from
the {oscap_git}[source code] then follow these instructions:

. *Get the source code*
+
Choose *1a* or *1b* depending on whether you want sources from a release tarball or the git repository.

.. Use a release tarball:
+
----
# replace ${version} with the desired version
wget https://github.com/OpenSCAP/openscap/releases/download/${version}/openscap-${version}.tar.gz
tar -xzpf openscap-${version}.tar.gz
cd openscap-${version}
mkdir -p build
----
+
**OR**

.. Use fresh sources from git repository.
+
----
$ git clone --recurse-submodules https://github.com/OpenSCAP/openscap.git
$ cd openscap
----
+

NOTE: We include https://github.com/OpenSCAP/yaml-filter[yaml-filter] library
as a git submodule. To get more information about using git submodules, read
https://git-scm.com/book/en/v2/Git-Tools-Submodules[Git Tools - Submodules].

. *Get the build dependencies*
+
--
To build the library you will also need to install the build dependencies.

Build dependencies may vary depending on enabled features (by the `cmake` command).
Some of the dependencies are optional, if they are not detected, openscap will be compiled
without respective optional features.

On Fedora 24+, the command to install the build dependencies is:

----
sudo dnf install \
cmake dbus-devel GConf2-devel libacl-devel libblkid-devel libcap-devel libcurl-devel \
libgcrypt-devel libselinux-devel libxml2-devel libxslt-devel libattr-devel make openldap-devel \
pcre2-devel perl-XML-Parser perl-XML-XPath perl-devel python3-devel python3-dbusmock rpm-devel swig \
bzip2-devel gcc-c++ libyaml-devel xmlsec1-devel xmlsec1-openssl-devel
----

On RHEL 8+ / CentOS 8+, the command to install the build dependencies is:

----
sudo dnf install \
cmake dbus-devel libacl-devel libblkid-devel libcap-devel libcurl-devel \
libgcrypt-devel libselinux-devel libxml2-devel libxslt-devel libattr-devel make openldap-devel \
pcre2-devel perl-XML-Parser perl-XML-XPath perl-devel python36-devel rpm-devel swig \
bzip2-devel gcc-c++ libyaml-devel xmlsec1-devel xmlsec1-openssl-devel
----

On Ubuntu 16.04, Debian 8 or Debian 9, the command to install the build dependencies is:

----
sudo apt-get install -y cmake libdbus-1-dev libdbus-glib-1-dev libcurl4-openssl-dev \
libgcrypt20-dev libselinux1-dev libxslt1-dev libgconf2-dev libacl1-dev libblkid-dev \
libcap-dev libxml2-dev libldap2-dev libpcre3-dev python-dev swig libxml-parser-perl \
libxml-xpath-perl libperl-dev libbz2-dev librpm-dev g++ libyaml-dev \
libxmlsec1-dev libxmlsec1-openssl
----

Since version 1.4.0 OpenSCAP uses PCRE2 library to handle regular
expressions. Dependencies: Debian/Ubuntu — pcre2-dev,
Fedora/RHEL — pcre2-devel.

When you have all the build dependencies installed you can build the library.
--

. *Build the library*
+
--
Run the following commands to build the library:

----
$ cd build/
$ cmake ../
$ make
----

On Ubuntu 18.04 and potentially other distro, the python3 dist-packages path is wrong.
If the following command:

----
$ python3 -c "import sysconfig; print(sysconfig.get_path('platlib'))"
----

returns "/usr/local/lib/python3/dist-packages" instead of a path like
"/usr/local/lib/python3.6/dist-packages", you must override this path,
otherwise you will not be able to import openscap_api.

----
$ cmake ../ -DPYTHON_SITE_PACKAGES_INSTALL_DIR=/usr/local/lib/python3.6/dist-packages
----

--

. *Build the HTML documentation*
+
--
It is possible to generate a complete API documentation, User manual,
Developer manual and contribute documents in HTML format.

If you want to build the HTML documentation you will need to install Doxygen
and AsciiDoc.
To install AsciiDoc, you can run `dnf install asciidoc`.
To install Doxygen, you can run `dnf install doxygen`.

Run the following command to build the documentation:

----
$ cmake -DENABLE_DOCS=TRUE ../
$ make docs
----

The resulting documentation is located in `build/docs` directory in these
subdirectories:
--
* `html` - contains the API documentation generated by Doxygen
* `manual` - contains OpenSCAP User Manual
* `developer` - contains OpenSCAP Developer Manual
* `contribute` - contains contribute documents

. *Run the tests*
+
--
After building the library you might want to run library self-checks. To do
that you need to have these additional packages installed:

----
wget lua which procps-ng initscripts chkconfig sendmail bzip2 rpm-build strace
----

On Ubuntu 18.04, also install:

----
rpm-common
----

It is also required to have `sendmail` service running on the system:

----
$ systemctl start sendmail.service
----

Now you can execute the following command to run library self-checks:

----
$ ctest
----

Note that using the `--jobs/-j` flag is currently not supported.
It will cause unexpected test failures.
See link:https://github.com/OpenSCAP/openscap/issues/2057[#2057] for more details.

It's also possible to use `ctest` to test any other oscap binary present in the system. You just have to set the path of the binary to the CUSTOM_OSCAP variable:

----
$ export CUSTOM_OSCAP=/usr/bin/oscap; ctest
----

Some tests that use the so-called offline mode of probes need to chroot during the test execution.
Some of those probes use the chroot syscall, which an unprivileged process is not allowed to do.
This is not a problem during the scanning itself, as oscap is usually scanning as root.
However, we don't want to run oscap as root during tests, as the whole test suite would have to use root privileges to clean up.

Instead, build the `oscap-chrootable` target as superuser, or build `oscap-chrootable-nocap` first and then grant the capability manually.
This target creates the binary that the test suite will use for some of those offline tests.
In offline tests, use the `set_offline_test_mode [chroot directory]` and `unset_offline_test_mode` functions from the common test module - those will set variables in such way that the unquoted `$OSCAP` invocation will use the chroot-capable binary, or it will exit with an error code, aborting the test.
Therefore, it is recommended to run

----
$ sudo make oscap-chrootable
----

Not every check tests the oscap tool, however, when the `CUSTOM_OSCAP` variable is set, only the checks which do are executed.

To enable the MITRE tests, use the `ENABLE_MITRE` flag:

----
$ cmake -DENABLE_MITRE=TRUE ..
----

These test require specific features of the environment to function properly; most notably, a MTA needs to be listening on port 25. We suggest using our container `mitre_tests` to test MITRE functionality if possible:

----
$ docker build --tag openscap_mitre_tests:latest -f Dockerfiles/mitre_tests . && docker run openscap_mitre_tests:latest
----

--

. *Install*
+
--
Run the installation procedure by executing the following command:

----
$ make install
----
--

== Running oscap
It is important to use your compiled `libopenscap.so` library with your `oscap` tool.
The easiest way how to achieve that without need to install `libopenscap.so` to the system path, is to use a shell script called *oscap_wrapper* or *run* in the OpenSCAP build directory.

-------------------------------------------------
$ cd build/
$ ./oscap_wrapper xccdf eval ... whatever
$ ./run valgrind utils/oscap xccdf eval ... whatever
-------------------------------------------------

The *run* script is generated at configure time by CMake and it sets the following environment variables:

* *LD_LIBRARY_PATH* - path to `libopenscap.so`
* *OSCAP_SCHEMA_PATH* - path to XCCDF, OVAL, CPE, ... XSD schemas and schematrons
(required for correct SCAP content validation)
* *OSCAP_XSLT_PATH* - path to XSLT transformations. (required if you want
to generate html documents from xml)
* *OSCAP_CPE_PATH* - path to the OpenSCAP CPE dictionary.

The *oscap_wrapper* script is a convenience shortcut for `run utils/oscap` call.


== Debugging
Developers and users who intend to help find and fix possible bugs in OpenSCAP
or possible bugs in their security policies have these possibilities:

=== Verbose mode
The verbose mode provides user additional information about process of system
scanning. The mode is useful for diagnostics of SCAP content evaluation
and also for debugging. It produces a detailed report log with various messages.
There is no need to special compilation, the feature is available for all
OpenSCAP users.

To turn the verbose mode on, run `oscap` with this option:

* `--verbose VERBOSITY_LEVEL` - Turn on verbose mode at specified
verbosity level.

The `VERBOSITY_LEVEL` can be one of:

1. *DEVEL* - the most detailed information for developers and bug hunters
2. *INFO* - reports content processing and system scanning
3. *WARNING* - possible failures which OpenSCAP can recover from
4. *ERROR* - shows only serious errors

The verbose messages will be written on standard error output (stderr).
Optionally, you can write the log into a file using
 `--verbose-log-file FILE`.

This is an example describing how to run OpenSCAP in verbose mode:

----
$ oscap oval eval --results results.xml --verbose INFO --verbose-log-file log.txt oval.xml
----

Then see the log using eg.:

----
$ less log.txt
----

=== Debug mode
Debug mode is useful for programmers. You need to build OpenSCAP from source code
with a custom configuration to enable the debug mode. Use this command:

------------------------------------
$ cmake -DCMAKE_BUILD_TYPE=Debug .. && make
------------------------------------

Debug mode provides:

* debug symbols on and optimization off - you can use `gdb`,
every process that was run.
* http://www.gnu.org/software/gawk/manual/html_node/Assert-Function.html[assertions]
are evaluated.


==== Example

----
$ bash ./run gdb --args utils/oscap xccdf eval \
--profile hard --results xccdf-results.xml \
--oval-results my-favourite-xccdf-checklist.xml
----


The `--oval-results` option force `oscap` tool to generate OVAL Result file
for each OVAL session used for evaluation. It's also very useful for
debugging!

=== Environment variables
There are few more environment variables that control `oscap` tool
behaviour.

* *OSCAP_FULL_VALIDATION=1* - validate all exported documents (slower)
* *SEXP_VALIDATE_DISABLE=1* - do not validate SEXP expressions (faster)
* *OSCAP_PCRE_EXEC_RECURSION_LIMIT* - override default recursion limit
  for match in pcre_exec/pcre2_match calls in textfilecontent(54) probes.



== Generating of code coverage
Code coverage can be useful during writing of test or performance profiling.
We could separate the process into five phases.

1) *Get dependencies*

----
# dnf install lcov
----

2) *Run CMake & make*

To allow code to generate statistics, we need to compile it with specific flags.

----
$ CFLAGS="--coverage -ftest-coverage -fprofile-arcs" LDFLAGS=-lgcov cmake -DCMAKE_BUILD_TYPE=Debug ../
$ make
----

3) *Run code*

In this phase we should run code. We can run it directly or via test suite.

----
$ bash ./run utils/oscap
----

4) *Generate and browse results*

----
$ lcov -t "OpenSCAP coverage" -o ./coverage.info -c -d .
$ genhtml -o ./coverage ./coverage.info
$ xdg-open ./coverage/index.html # open results in browser
----

5) *Clean stats*

Every run only modify our current statistics and not rewrite them completely.
If we want to generate new statistics, we should remove the old ones.

----
$ lcov --directory ./ --zerocounters ; find ./ -name "*.gcno" | xargs rm
$ rm -rf ./coverage
----

== OpenSCAP Reference Manual
For more information about OpenSCAP library, you can refer to this online
reference manual: http://static.open-scap.org/openscap-1.2/[OpenSCAP
reference manual]. This manual is included in a release tarball and can be
regenerated from project sources by Doxygen documentation system.