%\VignetteIndexEntry{Using tinytest}
\documentclass[11pt]{article}
\usepackage{enumitem}
\usepackage{xcolor}  % for color definitions
\usepackage{sectsty} % to modify heading colors
\usepackage{fancyhdr}
\setlist{nosep}

% simpler, but issue with your margin notes
\usepackage[left=1cm,right=3cm, bottom=2cm, top=1cm]{geometry}
%\usepackage{vmargin}
%\setpapersize{USletter} % or a4 for you
% Left Top Right Bottom headheight?? headsep footheight footsep
%\setmarginsrb{0.75in}{0.25in}{1.1in}{0.25in}{15pt}{0pt}{10pt}{20pt}

\usepackage{hyperref}

\definecolor{bluetext}{RGB}{0,101,165}
\definecolor{graytext}{RGB}{80,80,80}

\hypersetup{
  pdfborder={0 0 0}
 , colorlinks=true 
 , urlcolor=blue
 , linkcolor=bluetext
 , linktoc=all
 , citecolor=blue
}

\sectionfont{\color{bluetext}}
\subsectionfont{\color{bluetext}}
\subsubsectionfont{\color{bluetext}}

% no serif=better reading from screen.
\renewcommand{\familydefault}{\sfdefault}

% header and footers
\pagestyle{fancy}
\fancyhf{}                          % empty header and footer
\renewcommand{\headrulewidth}{0pt}  % remove line on top
\rfoot{\color{bluetext} tinytest \Sexpr{packageVersion("tinytest")}}
\lfoot{\color{black}\thepage}  % side-effect of \color{}: lowers the printed text a little(?)

\usepackage{fancyvrb}


% custom commands make life easier.
\newcommand{\code}[1]{\texttt{#1}}
\newcommand{\pkg}[1]{\textbf{#1}}
\let\oldmarginpar\marginpar
\renewcommand{\marginpar}[1]{\oldmarginpar{\color{bluetext}\raggedleft\scriptsize #1}}

% skip line at start of new paragraph
\setlength{\parindent}{0pt}
\setlength{\parskip}{1ex}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\title{Using tinytest}
\author{Mark van der Loo}
\date{\today{} | Package version \Sexpr{packageVersion("tinytest")}}

\begin{document}
\DefineVerbatimEnvironment{Sinput}{Verbatim}{fontshape=n,formatcom=\color{graytext}}
\DefineVerbatimEnvironment{Soutput}{Verbatim}{fontshape=sl,formatcom=\color{graytext}}
\newlength{\fancyvrbtopsep}
\newlength{\fancyvrbpartopsep}
\makeatletter
\FV@AddToHook{\FV@ListParameterHook}{\topsep=\fancyvrbtopsep\partopsep=\fancyvrbpartopsep}
\makeatother


\setlength{\fancyvrbtopsep}{0pt}
\setlength{\fancyvrbpartopsep}{0pt}
\maketitle{}
\thispagestyle{empty}

\tableofcontents{}
<<echo=FALSE>>=
options(prompt="R>  ",
        continue = "    ",
        width=75,
        tt.pr.color=FALSE)
@


\subsection*{Reading guide}
Readers of this document are expected to know how to write R functions
and have a basic understanding of a package source directory structure.



\newpage{}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Purpose of this package: unit testing}

The purpose of \emph{unit testing} is to check whether a function gives the
output you expect, when it is provided with certain input. So unit testing is
all about comparing \emph{desired} outputs with \emph{realized} outputs. The
purpose of this package is to facilitate writing, executing and analyzing unit
tests.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Expressing tests}
Suppose we define a function translating pounds (lbs) to kilograms inaccurately.
<<>>=
lbs2kg <- function(x){
  if ( x < 0 ){
    stop(sprintf("Expected nonnegative weight, got %g",x))
  }
  x/2.20
}
@
We like to check a few things before we trust it.
<<>>=
library(tinytest)
expect_equal(lbs2kg(1), 1/2.2046)
expect_error(lbs2kg(-3))
@

The value of an \code{expect\_*} function is a \code{logical}, with some
attributes that record differences, if there are any. These attributes are used
to pretty-print the results.
<<>>=
isTRUE( expect_true(2 == 1 + 1) )
@



\subsection{Test functions}
Currently, the following expectations are implemented.
\begin{center}
\begin{tabular}{ll}
\textbf{Function} & \textbf{what it expects}\\
\code{expect\_equal(current, target)}      & equality (using \code{all.equal})\\
\code{expect\_equivalent(current, target)} & equality, ignoring attributes\\
\code{expect\_identical(current, target)}  & equality, (using, \code{identical})\\
\code{expect\_length(current, length)}     & check lenght of object\\
\code{expect\_true(current)}               & \code{current} evaluates to \code{TRUE}\\
\code{expect\_false(current)}              & \code{current} evaluates to \code{FALSE}\\
\code{expect\_match(current, pattern)}     & All strings in \code{current} match \code{pattern}\\
\code{expect\_inherits(current, class)}    & \code{current} inherits from \code{class}\\
\code{expect\_null(current)}               & \code{current} evaluates to \code{NULL}\\
\code{expect\_error(current, pattern)}     & error message matching \code{pattern}\\
\code{expect\_warning(current, pattern)}   & warning message matching \code{pattern}\\
\code{expect\_message(current, pattern)}   & message matching \code{pattern}\\
\code{expect\_silent(current)}             & expect no warnings or errors (just run)\\
\code{expect\_stdout(current, pattern)}    & expect output to \code{stdout} matching \code{pattern}\\
\code{expect\_equal\_to\_reference(current, file)} & expect object equal to object stored in RDS file\\
\code{expect\_equivalent\_to\_reference(current, file)} & expect object equivalent to object stored in RDS file\\

\end{tabular}
\end{center}
Here, \code{target} is the intended outcome and \code{current} is the observed
outcome. Also, \code{pattern} is interpreted as a regular expression. 
<<>>=
expect_error(lbs2kg(-3), pattern="nonnegative")
expect_error(lbs2kg(-3), pattern="foo")
@


\subsection{Alternative syntax}
The syntax of the test functions should be familiar to users of the
\code{testthat} package\cite{wickham2016testthat}. In test files only, you can
use equivalent functions in the style of \code{RUnit}\cite{burger2016RUnit}. To
be precise, for each function of the form \code{expect\_lol} there is a
function of the form \code{checkLol}.


\subsection{Interpreting the output and print options}
Let's have a look at an example again.
<<>>=
expect_false( 1 + 1 == 2, info="My personal message to the tester" )
@

The output of these functions is pretty self-explanatory, nevertheless we see
that the output of these expect-functions consist of
\begin{itemize}
\item The result: \code{FAILED}, \code{PASSED} or \code{SIDEFX}. The latter only occurs
  when side effects are monitored (see \S\ref{sect:side})
  \item The type of failure (if any) between square brackets. Current options
       are as follows.
  \begin{itemize}
    \item \code{[data]} there are differences between observed and expected values.
    \item \code{[attr]} there are differences between observed and expected attributes, such as column names.
    \item \code{[xcpt]} an exception (warning, error) was expected but not observed.
  \end{itemize}
    When side effects are monitored, and the result is \code{SIDEFX}, a side
    effect was observed. The type of side effect is reported between square brackets.
  \begin{itemize}
    \item \code{[envv]} An environmental variable was created, changed, or deleted.
    \item \code{[wdir]} The working directory has changed.
    \item \code{[file]} A file operation occurred in the test directory or one of its subdirectories.
   \end{itemize}
  \item When relevant (see \S\ref{sect:testfiles}), the location of the test file and the relevant line numbers.
  \item The test call.
  \item When relevant, a summary of the differences between observed and expected
    values or attributes, or a summary of the observed side effect.
  \item When present, a user-defined information message.
\end{itemize}

The result of an \code{expect\_} function is a \code{tinytest} object. You can
print them in long format (default) or in short, one-line format like so.
<<>>=
print(expect_equal(1+1, 3), type="short")
@
\marginpar{\code{print} method}
Functions that run multiple tests return an object of class \code{tinytests}
(notice the plural). Since there may be a lot of test results, \pkg{tinytest}
tries to be smart about printing them. The user has ultimate control over this
behaviour. See \code{?print.tinytests} for a full specification of the options.







\section{Test files}
\label{sect:testfiles}
In \pkg{tinytest}, tests are scripts, interspersed with statements that perform
checks. An example test file in tinytest can look like this.


\begin{verbatim}
    # contents of test_addOne.R

    addOne <- function(x) x + 2

    expect_true(addOne(0) > 0)

    hihi <- 1
    expect_equal(addOne(hihi), 2)
\end{verbatim}

A particular file can be run using\marginpar{\code{run\_test\_file}}
<<eval>>=
run_test_file("test_addOne.R", verbose=0)
@
We use \code{verbose=0} to avoid cluttering the output in this vignette.  By
default, verbosity is turned on, and a counter is shown while tests are run.
The counter is colorized on terminals supporting ANSI color extensions. If you
are uncomfortable reading these colors or prefer colorless output, use
\code{color=FALSE} or set \code{options(tt.pr.color=FALSE)}.


The numbers between \code{<-->} indicate at what lines in the file the failing
test can be found.
By default only failing tests are  printed. You can store the output and print 
all of them.
<<>>=
test_results <- run_test_file("test_addOne.R", verbose=0)
print(test_results, passes=TRUE)
@
Or you can set
<<eval=FALSE>>=
options(tt.pr.passes=TRUE)
@
to print all results during the active R session.

To run all test files in a certain directory, we can
use\marginpar{\code{run\_test\_dir}}
<<eval=FALSE>>=
run_test_dir("/path/to/your/test/directory")
@
By default, this will run all files of which the name starts with
\code{test\_}, but this is customizable.

\subsection{Summarizing test results, getting the data}
To create some results, run the tests in this package.
<<>>=
out <- run_test_dir(system.file("tinytest", package="tinytest")
       , verbose=0)
@
The results can be turned into data using \code{as.data.frame}.
\marginpar{\code{as.data.frame}}
<<>>=
head(as.data.frame(out), 3)
@
The last two columns indicate the line numbers where the test was defined.

A `summary` of the output gives a table with passes and fails per file.
\marginpar{\code{summary}}
<<>>=
summary(out)
@



\subsection{Programming over tests, ignoring test results, exiting early}
Test scripts are just R scripts interspersed with tests. The test runners make
sure that all test results are caught, unless you tell them not to. For
example, since the result of a test is a \code{logical} you can use them as a
condition.
<<eval=FALSE>>=
if ( expect_equal(1 + 1, 2) ){
    expect_true( 2 > 0)
}
@
Here, the second test (\code{expect\_true(2 > 0)}) is only executed if the
first test results in \code{TRUE}. In any case the result of the first test
will be caught in the test output, when this is run with \code{run\_test\_file}
\code{run\_test\_dir}, \code{test\_all}, \code{build\_install\_test} or through
\code{R CMD check} using \code{test\_package}.

If you want to perform the test, but not record the test result you can do the
following \marginpar{\code{ignore}} (note the placement of the brackets).
<<eval=TRUE>>=
if ( ignore(expect_equal)(1+1, 2) ){
  expect_true(2>0)
}
@
Other cases where this may be useful is to perform tests in a loop, e.g. when
there is a systematic set of cases to test.

It is possible to exit a test file prematurely. For example when there
are a number of tests that are not relevant or possible on some OS,
you can do the following. \marginpar{\code{exit\_file}}
<<>>=
if ( Sys.info()[['sysname']] == "Windows"){
  exit_file("Cannot test this on Windows")
}
@
This will cause \code{run\_test\_file} to stop file execution, print the
message,  and report the information gathered up to where \code{exit} was
called. A function like \code{test\_all} will then continue with the next file,
so testing is not aborted completely.

There is also a convenience function called \code{exit\_if\_not} that functions
similarly to base R's \code{stopifnot}. It accepts a comma-separated list
of expressions, and if any one of them does not result in a single \code{TRUE}
the execution of the test file is stopped with a message. So you can
do things like this.
<<eval=FALSE>>=
exit_if_not(requireNamespace("slartibartfast", quietly=TRUE)
           , packageVersion("slartibartfast") >= "1.0.0")
@



%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Running order and side effects}
\label{sect:running}
It is a generally a good idea to write test files that are independent from
each other.  This means that the order of running them is unimportant for the
test results and test files can be maintained independently. The function
\code{run\_test\_file} and by extension \code{run\_test\_dir},
\code{test\_all}, and \code{test\_package} encourage this by resetting
\begin{itemize}
\item options, set with \code{options()};
\item environment variables, set with \code{Sys.setenv()}
\end{itemize}
after a test file is executed. 


To escape this behavior, use \code{base::Sys.setenv()} respectively
\code{base::options()}. Alternatively use 
<<eval=FALSE>>=
run_test_dir("/path/to/my/testdir"
           , remove_side_effects = FALSE)
test_all("/path/to/my/testdir"
           , remove_side_effects = FALSE)
# Only in tests/tinytest.R:
test_package("PACKAGENAME", remove_side_effects=FALSE)
@

Test files are sorted and run based on the current locale. This means
that the order of execution is in general not platform-independent.
You can control the sorting behavior interactively or by setting
\code{options(tt.collate)}. To be precise, adding 
<<eval=FALSE>>=
options(tt.collate="C")
@
to \code{/tests/tinytest.R} before running \code{test\_package} will
ensure bytewise sorting on most systems. See also
\code{help("run\_test\_dir")}.

\subsection{Monitoring side effects}
\label{sect:side}
The term 'side effect' is the technical expression for the situation where a
function or expression changes something outside of its scope. Examples include
creating, removing, or changing variables in R's global work space, R options,
or environment variables of your operating system.  We will call such variables
or options \emph{external variables}.

To test for side-effects once, use the \code{side\_effects} argument
to any of the test runners. For example
<<eval=FALSE>>=
test_package("pkg", side_effects=TRUE)
@
There is control over which side-effects to track. For example to prevent
tracking changes in the working directory, do the following.
<<eval=FALSE>>=
test_package("pkg", side_effects=list(pwd=FALSE))
@


If you add \code{report\_side\_effects()} anywhere in a test file, certain
external variables are  monitored from that point on, and for that file only.
It can be switched off again by calling \code{report\_side\_effects(FALSE)}
anywhere in the file.  The reporting functionality will compare the external
state before and after every expression in the test file is run and report any
changes. 

At the moment, effects that can be monitored include environment
variables, locale settings, the present working directory, and file operations
in the test directory.


Below is an example of a test file where side effects are recorded. The third
line creates an explicit side effect by creating a new environment variable
called \code{hihi} with the value \code{"lol"}.
\begin{verbatim}
  # contents of test_se.R
  report_side_effects()
  expect_equal(1+1, 2)
  Sys.setenv(hihi="lol")
  expect_equal(1+1, 3)
  Sys.setenv(hihi="lulz ftw")
\end{verbatim}
Running the test file yields an object of class \code{tinytests} as usual, only
now changes in environment variables are reported.
<<>>=
run_test_file("test_se.R", verbose=1)
@
Note that as discussed in Section~\S\ref{sect:running}, \pkg{tinytest} will
unset the environment variable \code{hihi} automatically after running the file
because it was set directly by the author of the test file using
\code{Sys.setenv}. The real value of the reporting functionality is that it
also reports on external variables that are touched by other functions than
those you call explicitly in the file.

Reading and comparing versions of external variables takes some time.
Especially when it requires a call to the operating service such as a request
for values of environment variables. We therefore recommend this to be used
only when you suspect a side effect. Or, for example to execute
\code{report\_side\_effects()} conditional on \code{at\_home()}.

It is not possible to catch all types of side effects, even in principle, using
the \pkg{tinytest} reporting functionality. Examples include: packages that
keep a global variable or environment within their namespace to store some
state, and packages that rely on compiled code where there are global objects
within the shared object.

Side effects are to be avoided as a general and strong principle, but sometimes
there is little or no choice. In Section~\ref{sect:devil} we give some tips on
how to properly handle such situations.






\section{Testing packages}

Using \pkg{tinytest} for your package is pretty easy.
\begin{enumerate}
\item Testfiles are placed in \code{/inst/tinytest}. The testfiles all have
names starting with \code{test} (for example \code{test\_haha.R}).
\item In the file \code{/tests/tinytest.R} you place the code
\begin{verbatim}
    if ( requireNamespace("tinytest", quietly=TRUE) ){
      tinytest::test_package("PACKAGENAME")
    }
\end{verbatim}
\item In your \code{DESCRIPTION} file, add \pkg{tinytest} to \code{Suggests:}.
\end{enumerate}
You can automatically create a minimal running test infrastructure
with the \code{setup\_tinytest} function. \marginpar{\code{setup\_tinytest}}
<<eval=FALSE>>=
setup_tinytest("/path/to/your/package")
@

In a terminal, you can now do
\begin{verbatim}
    R CMD build /path/to/your/package
    R CMD check PACKAGENAME_X.Y.Z.tar.gz
\end{verbatim}
and all tests will run. 


To run all the tests interactively, make sure that all functions of your 
new package are loaded. After that, run\marginpar{\code{test\_all}}
<<eval=FALSE>>=
test_all("/path/to/your/package")
@
where the default package directory is the current working directory.




%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Build--install--test interactively}
The most realistic way to unit-test your package is to build it, install it
and then run all the tests. The function
<<eval=FALSE>>=
  build_install_test("/path/to/your/package")
@
does exactly that. It builds and installs the package in a temporary directory,
starts a fresh R session, loads the newly installed package and runs all tests.
The return value is a \code{tinytests} object.

The package is built without manual or vignettes to speed up the
whole process.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Testing functions that are not exported: use `:::`}
In Section~\ref{sect:spherical} it is argued that unit tests should as a rule
of thumb focus on the functions that are visible to the user. However, there
are cases where it may be preferred to test an internal function. For example
when there are two user-visible functions that call the same underlying,
unexported function with different arguments. Or when one of the internal
functions implements a numerical algorithm that requires thorough testing.

To test functions in your package that are not visible to users that load
your package, use the triple-colon operator like so.
<<eval=FALSE>>=
output = pkg:::some_internal_function(1)
expect_equal(output, 2)
@
This is perfectly ok, and is also accepted by \code{R CMD check --as-cran}.


\pkg{Tinytest} does not make those internal functions directly callable like some 
other unit testing packages do. Making internal functions callable means that 
\begin{enumerate}
\item \pkg{tinytest} needs to simulate loading a package, except for the namespace
restrictions;
\item there may be significant differences between the environment in which you
test the functions, and the environment which a user sees when loading the
package;
\item the way the package is loaded during testing may differ from the way it is loaded
      when a user loads it;
\item exported functions are not clearly distinguished from internal functions in
the test code.
\end{enumerate}
accurately simulating how R loads a package is no small matter. It would
require a significant epansion of \code{tinytest}'s code base that would have
to be kept synchronised with the way R loads packages (possibly with backport
options when R would change in that area).

So in short: let's keep things simple and let R do what it knows how to do.




%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Using data stored in files}
\label{sect:comparefiles}
When your package is tested with \code{test\_package}, \pkg{tinytest} ensures
that your working directory is the testing directory (by default
\code{tinytest}).  This means you can read files that are stored in your folder
directly.

Suppose that your package directory structure looks like this (default):
\begin{itemize}
\item[]\code{/inst}
  \begin{itemize}
    \item[]\code{/tinytest}
    \begin{itemize}
       \item[]\code{/test.R}
       \item[]\code{/women.csv}
    \end{itemize}
  \end{itemize}
\end{itemize}

Then, to check whether the contents of \code{women.csv} is equal to the 
built-in \code{women} dataset,  the content of \code{test.R} looks as follows.
<<eval=FALSE>>=
dat <- read.csv("women.csv")
expect_equal(dat, women)
@



%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Skipping tests on \code{CRAN}}
It is not possible to detect whether a test is running on CRAN. This means we
are forced to detect that we are running tests in our own environment. 

In the following example we use the host name to detect if we are running on
our own machine and explicitly pass this information to \code{test\_package}.
<<echo=FALSE>>=
options(prompt="  ", continue="  ")
@
<<eval=FALSE>>=
# contents of pkgdir/tests/tinytest.R
if ( requireNamespace("tinytest", quietly=TRUE) ){
   home <- identical( Sys.info()["nodename"], "YOURHOSTNAME" )
   tinytest::test_package("PKGNAME", at_home = home)
}
@
Other ways to detect whether you are running `at home' include
\begin{itemize}
\item Set a custom environment variable (from your OS) and detect it with \code{Sys.getenv}.
<<>>=
home <- identical( Sys.getenv("HONEYIMHOME"), "TRUE" )
@
\item Use 4-number package versioning while developing and 3-number versioning
for CRAN releases\footnote{As \href{https://stackoverflow.com/questions/36166288/skip-tests-on-cran-but-run-locally}{recommended here} by Dirk Eddelbuettel.}.
<<eval=FALSE>>=
home <- length(unclass(packageVersion("PKGNAME"))[[1]]) == 4
@
\end{itemize}

<<echo=FALSE>>=
options(prompt="R> ", continue="   ")
@


\subsubsection*{When tests are run interactively}
All the interactive test runners have \code{at\_home=TRUE} by default, so while
you are developing all tests are run, unless you exclude them explicitly.
<<>>=
run_test_file("test_hehe.R", verbose=0)
run_test_file("test_hehe.R", verbose=0, at_home=FALSE)
@

Here is an overview of test runners and their default setting for \code{at\_home}.

\begin{center}
\begin{tabular}{rll}
Function               & Default \code{at\_home}  & Intended use \\
\hline
\code{run\_test\_file}      & \code{TRUE}   & Interactive by developer\\
\code{run\_test\_dir}       & \code{TRUE}   & Interactive by developer\\
\code{test\_all}            & \code{TRUE}   & Interactive by developer\\
\code{build\_install\_test} & \code{TRUE}   & Interactive by developer\\
\code{test\_package}        & \code{FALSE}  & \code{R CMD check}, or after installation by user.
\end{tabular}
\end{center}



%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Testing your package after installation}
Supposing your package is called \pkg{hehe} and the \pkg{tinytest}
infrastructure is used. If the package is installed, the following command runs
\pkg{hehe}'s tests. 
<<eval=FALSE>>=
tinytest::test_package("hehe")
@
This can come in handy when a user of \pkg{hehe} reports a bug and you want to
make sure all tested functionality works on the user's system.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Using extension packages}

It is possible for other packages to add custom assertions
(\code{expect}-functions).  To use such a package:
\begin{enumerate}
\item Add the extension package to the \code{Suggests:} field in the \code{DESCRIPTION} file.
\item Add \code{using(pkg)} to \emph{each} test file that use the extensions (see \code{?using}). \marginpar{\code{using}}
\end{enumerate}

When multiple extension packages are loaded, and when there are name
collisions, the packages loaded later takes precedence over the ones loaded
earlier (as usual in R). This includes assertions exported by \pkg{tinytest}.

\textbf{Note.} Other than in regular R, it is not possible to disambiguate
functions using namespace resolution as in \code{pkg::expect\_something},
because in that case the test result will not be caught by \pkg{tinytest}.

The API for building extension packages is described in \code{?register\_tinytest\_extension}.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Mocking databases}

The \pkg{dittodb} package\cite{keane2020dittodb} is capable of mocking
pre-recorded database requests. 
To use extensions like \code{dittodb}, put the package in \code{Suggests:} in
the \code{DESCRIPTION} and load it in the test file. The package captures
responses from SQL connections and saves them to R files. Therefore, capture
the requested response prior to testing by wrapping your request in
\code{start\_db\_capturing()} and \code{stop\_db\_capturing()}. Optionally, specify
the file path you want your mocks to be saved. For examples using
\code{dittodb}, see \href{https://dittodb.jonkeane.com/}{here}.

In the testing scripts, load the package and wrap your tests in
\code{with\_mock\_path(path, with\_mock\_db())} like
<<eval=FALSE>>=
require(dittodb)
with_mock_path(
    system.file("<path-to-your-mocks>", package = "myPackage"),
    with_mock_db({
    # <unit tests which rely on database connections>
    })
)
@



%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Testing with environmental variables}
\label{sect:envvar}
In \pkg{tinytest} you can run tests with custom environment variable settings
easily. Just add the \code{set\_env} argument to any of \code{run\_test\_file()},
\code{run\_test\_dir()} or \code{test\_package()}. Here is an example.
<<eval=FALSE>>=
test_package("tinytest", set_env = list(WA_BABALOOBA="BA_LA_BAMBOO"))
@
With this option, the environment variable will be set during testing and unset
afterwards. Setting and unsetting environment variables like this will not be
recorded as a side effect. If there is code in the test file that changes this
variable, then it is recorded.

Do note that R uses some environment variables that are read during startup,
such as \code{\_R\_OPTIONS\_STRINGS\_AS\_FACTORS\_}. Setting these at runtime
has no effect.



%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Running tests in parallel}
In \pkg{tinytest}, a file should be considered a closed unit: no information
created in one test file should be used in another.  Under this condition,
tests can automatically run in parallel by running different files in different
R sessions.

Running code in parallel takes some careful consideration around setting up a
cluster, running the tests, and closing the cluster of preparing it for the
next run. Depending on the test runner used, there are different levels of
control and responsibility for the user to prepare the program for
parallelization. Below we describe them from less easier to more control.

\subsubsection*{\code{build\_install\_test}}
This function creates and installs a package in a temporary
location. By setting the \code{ncpu} parameter, the number of
cores used at the testing phase can be increased.
<<eval=FALSE>>=
build_install_test("/path/to/your/package", ncpu=2)
@
We already mentioned that the order in which files are run is in principle
system-dependent and it is a good practice not to rely on it. Under parallel
situations, all bets on file order are off.


\subsubsection*{\code{test\_package}.} 
This function assumes that a package is installed. It can gather any
information necessary to parallelize a test run. The simplest way to
parallelize is to specify the number of CPUs used.
<<eval=FALSE>>=
test_package("PACKAGENAME", ncpu=2)
@
Here, \code{test\_package} will
\begin{enumerate}
\item Set up a local cluster using \code{parallel::makeCluster}.
\item Load the package on each R instance of the cluster.
\item Run test files in parallel over the cluster.
\item Collect the results and close the cluster.
\end{enumerate}
In stead of just passing the number of CPUs it is possible to pass a
\code{cluster} object. In that case \code{test\_package} will still load the
package on each node. However, note that if the package gets updated and
reinstalled, it should also be reloaded.  It is in general hard to completely
unload a package in R (see \code{?detach} and \code{?unloadNamespace} for some
details on artifacts that will not be removed). So our advice is
to restart a cluster for each test run.


\subsubsection*{\code{run\_test\_dir}, \code{test\_all}}
These function assumes that all functionality needed to run the 
tests is loaded. They accept an object of type \code{cluster}.
The user is responsible for setting up the nodes. 
<<eval=FALSE>>=
cl <- parallel::makeCluster(4, outfile="")
parallel::clusterCall(cl, source, "R/myfunctions.R")
run_test_dir("inst/tinytest", cluster=cl)
@
where the argument \code{outfile=""} ensures that messages from each node
are forwarded to the master node.
It is possible to keep the cluster `alive', so modifications
can be made to \code{"R/myfunctions.R"} and then run for
example the following.
<<eval=FALSE>>=
parallel::clusterCall(cl, source, "R/myfunctions.R")
test_all(cluster=cl)
stopCluster(cl)
@
For heavy test routines it is thus possible to keep a test cluster
up to offload computations.

For more complex situations, including packages that use \code{S4} classes, or
compiled code, (re)loading takes more effort than sourcing a few R files.  In
this cases it is often easier to restart a clean cluster for each test round.















%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{A few tips on packages and unit testing}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Make your package spherical}
\label{sect:spherical}

Larger packages typically consist of functions that are visible to the users
(exported functions) and a number of functions that are only used by the
exported functions. For example:

<<>>=
# exported, user-visible function
inch2cm <- function(x){
  x*conversion_factor("inch")
}
# not exported function, package-internal
conversion_factor <- function(unit){
  confac <- c(inch=2.54, pound=1/2.2056)
  confac[unit]
}
@

We can think of the exported functions as the \emph{surface} of the package and
all the other functions as the \emph{volume}. The surface is what a user sees,
the volume is what the developer sees. The surface is how a user interacts with
a package. 


If the surface is small (few functions exported), users are limited in the ways
they can interact with your package and that means there is less to test. So as
a rule of thumb, it is a good idea to keep the surface small. Since a sphere
has the smallest surface-to-volume ratio possible, I refer to this rule as
\emph{keep your package spherical}.

By the way, the technical term for the surface of a package is API (application
program interface).




%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Test the surface, not the volume}

Unexpected behavior (a bug) is often discovered when someone who is not the
developer starts using code. Bugfixing implies altering code and it may even
require you to refactor large chunks of code that is internal to a package.  If
you defined extensive tests on non-exported functions, this means you need to
rewrite the tests as well. As a rule of thumb, it is a good idea to test only
the behaviour at the surface, so as a developer you have more freedom to change
the internals. This includes rewriting and renaming internal functions completely.


By the way, it is bad practice to change the surface, since that means you are
going to break other people's code. Nobody likes to program against an API that
changes frequently, and everybody hates to program against an API that changes
unexpectedly.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{How many tests do I need?}
When you call a function, you can think of its arguments flowing through a
certain path from input to output. As an example, let's take a look again at a
new, slightly safer unit conversion function.
<<>>=
pound2kg <- function(x){
  stopifnot( is.numeric(x) )
  if ( any(x < 0) ){
    warning("Found negative input, converting to positive")
    x <- abs(x)
  }
  x/2.2046
}
@
If we call \code{lbs2kg} with argument \code{2}, we can write:
\begin{verbatim}
    2 -> /2.2046 -> output
\end{verbatim}
If we call \code{lbs2kg} with argument \code{-3} we can write
\begin{verbatim}
   -3 -> abs() -> /2.2046 -> output
\end{verbatim}
Finally, if we call \code{pound2kg} with \code{"foo"} we can write
\begin{verbatim}
   "foo" -> stop() -> Exception
\end{verbatim}

So we have three possible paths. In fact, we see that every nonnegative number
will follow the first path, every negative number will follow the second path
and anything nonnumeric follows the third path. So the following test suite
fully tests the behaviour of our function.
<<eval=FALSE>>=
    expect_equal(pound2kg(1), 1/2.2046  )
    # test for expected warning, store output
    expect_warning( out <- pound2kg(-1) )
    # test the output
    expect_equal( out, 1/2.2046)
    expect_error(pound2kg("foo"))
@

The number of paths of a function is called its \emph{cyclomatic complexity}.
For larger functions, with multiple arguments, the number of paths typically
grows extremely fast, and it quickly becomes impossible to define a test for
each and every one of them. If you want to get an impression of how many tests
one of your functions in needs in principle, you can have a look at the
\pkg{cyclocomp} package of G\'abor Cs\'ardi\cite{csardi2016cyclocomp}.

Since full path coverage is out of range in most cases, developers often strive
for something simpler, namely \emph{full code coverage}. This simply means that
each line of code is run in at least one test. Full code coverage is no
guarantee for bugfree code. Besides code coverage it is therefore a good idea
to think about the various ways a user might use your code and include tests
for that.

To measure code coverage, I recommend using the \pkg{covr} package by Jim
Hester\cite{hester2018covr}. Since \pkg{covr} is independent of the tools or
packages used for testing, it also works fine with \pkg{tinytest}.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{It's not a bug, it's a test!}
If users of your code are friendly enough to submit a bug report when they find
one, it is a good idea to start by writing a small test that reproduces the
error and add that to your test suite. That way, whenever you work on your
code, you can be sure to be alarmed when a bug reappears.

Tests that represent earlier bugs are sometimes called \emph{regression tests}.
If a bug reappears during development, software engineers sometimes refer
to this as a \emph{regression}.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Side effects are the Devil}
\label{sect:devil}
Since side-effects manipulate variables outside of the scope of a function, or
even outside of R, they can cause bugs that are hard to reproduce.  R offers a
mechanism to ensure that a function leaves the outside world as it was, once
your code stops running. 

Suppose you need to change working directory within a function, \code{source} a
file and return to the working directory. A naive way to do this is like so.
\marginpar{\code{on.exit}}
<<>>=
bad_function <- function(file){
  oldwd <- getwd()
  setwd(dirname(file))
  source(basename(file))
  setwd(oldwd)
}
@
This all works fine, untill \code{file} contains faulty code and throws an
error.  As result, the execution of \code{bad\_function} will stop and leave
the user in a changed working directory. With \code{on.exit} you can define
code that will be carried out before the function exits, either normally
or with an error.
<<>>=
good_function <- function(file){
  oldwd <- getwd()
  on.exit(setwd(oldwd))
  setwd(dirname(file))
  source(basename(file))
}
@




\newpage

\begin{thebibliography}{5}
  \bibitem{wickham2016testthat}
    \href{https://cran.r-project.org/package=testthat}{Unit Testing for R}
    Hadley Wickham (2016). testthat: Get Started with Testing. The R Journal,
    vol. 3, no. 1, pp. 5--10, 2011

  \bibitem{burger2016RUnit}
  Matthias Burger, Klaus Juenemann and Thomas Koenig (2018). 
  \href{https://CRAN.R-project.org/package=RUnit}{RUnit: R Unit Test Framework}
  R package version 0.4.32.



  \bibitem{csardi2016cyclocomp}
    \href{https://cran.r-project.org/package=cyclocomp}{cyclocomp: cyclomatic complexity of R code}
    G\'abor Cs\'ardi (2016).
    R package version 1.1.0

  \bibitem{hester2018covr}
    \href{https://CRAN.R-project.org/package=covr}{covr: Test Coverage for Packages}
    Jim Hester (2018).
    R package version 3.2.1

  \bibitem{keane2020dittodb}
    \href{https://CRAN.R-project.org/package=dittodb}{dittodb: A Test Environment for Database Requests}
    Jonathan Keane and Mauricio Vargas (2020).
    R package version 0.1.1

\end{thebibliography}

\end{document}