1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
|
<%@meta language="R-vignette" content="--------------------------------
%\VignetteIndexEntry{A Future for R: How the Future Framework is Validated}
%\VignetteAuthor{Henrik Bengtsson}
%\VignetteKeyword{R}
%\VignetteKeyword{package}
%\VignetteKeyword{vignette}
%\VignetteKeyword{future}
%\VignetteKeyword{promise}
%\VignetteEngine{R.rsp::rsp}
%\VignetteTangle{FALSE}
--------------------------------------------------------------------"%>
<%
library("R.utils")
`%<-%` <- future::`%<-%`
options("withCapture/newline" = FALSE)
%>
# <%@meta name="title"%>
Since correctness and reproducibility is essential to all data processing, validation is a top priority and part of the design and implementation throughout the future ecosystem. Several types of testing are performed.
First, all the essential core packages part of the future framework,
**[future]**, **[globals]**, and **[listenv]**, implement a rich set of package tests.
These are validated regularly across the wide-range of operating
systems (Linux, Solaris, macOS, and MS Windows) and R versions available
on CRAN, on continuous integration (CI) services ([GitHub Actions], [Travis CI],
and [AppVeyor CI]), an on [R-hub].
Second, for each new release, these packages undergo full reverse-package
dependency checks using **[revdepcheck]**.
As of October 2020, the **future** package is tested against 140+
direct reverse-package dependencies available on CRAN and Bioconductor.
These checks are performed on Linux with both the default settings and
when forcing tests to use multisession workers (SOCK clusters), which further
validates that globals and packages are identified correctly.
Third, a suite of _Future API conformance tests_ available in the
**[future.tests]** package validates the
correctness of all future backends. Any new future backend developed, must
pass these tests to comply with the _Future API_.
By conforming to this API, the end-user can trust that the backend will
produce the same correct and reproducible results as any other backend,
including the ones that the developer have tested on.
Also, by making it the responsibility of the developer to assert that their
new future backend conforms to the _Future API_, we relieve other
developers from having to test that their future-based software works on all
backends.
It would be a daunting task for a developer to validate the correctness of
their software with all existing backends. Even if they would achieve that,
there may be additional third-party future backends that they are not aware
of, that they do not have the possibility to test with, or that yet have not
been developed.
Fourth, since **[foreach]** is used by a large number of essential
CRAN packages, it provides an excellent opportunity for supplementary
validation. Specifically, we dynamically tweak the examples of
**[foreach]** and popular CRAN packages **[caret]**, **[glmnet]**, **[NMF]**,
**[plyr]**, and **[TSP]** to use the **[doFuture]** adaptor.
This allows us to run these examples with a variety of future backends to
validate that the examples produce no run-time errors, which indirectly
validates the backends as well as the _Future API_.
In the past, these types of tests helped to identify and resolve corner cases
where automatic identification of global variables would fail.
As a side note, several of these foreach-based examples fail when using a
parallel foreach adaptor because they do not properly export globals or
declare package dependencies. The exception is when using the sequential
_doSEQ_ adaptor (default), fork-based ones such as **[doMC]**, or
the generic **[doFuture]**, which supports any future backend and
relies on the future framework for handling globals and packages(*)
Lastly, analogously to above reverse-dependency checks of each new release,
CRAN and Bioconductor continuously run checks on all these direct, but
also indirect, reverse dependencies, which further increases the validation
of the _Future API_ and the future ecosystem at large.
(\*) There is a plan to update **[foreach]** to use the exact same static-code-analysis method as the **[future]** package use for identifying globals. As the maintainer of the future framework, I collaborate with the maintainer of the **[foreach]** package to implement this.
[caret]: https://CRAN.R-Project.org/package=caret
[doFuture]: https://CRAN.R-Project.org/package=doFuture
[doMC]: https://CRAN.R-Project.org/package=doMC
[foreach]: https://CRAN.R-Project.org/package=foreach
[future]: https://CRAN.R-Project.org/package=future
[future.tests]: https://CRAN.R-Project.org/package=future
[glmnet]: https://CRAN.R-Project.org/package=glmnet
[globals]: https://CRAN.R-Project.org/package=globals
[listenv]: https://CRAN.R-Project.org/package=listenv
[NMF]: https://CRAN.R-Project.org/package=NMF
[plyr]: https://CRAN.R-Project.org/package=plyr
[TSP]: https://CRAN.R-Project.org/package=TSP
[revdepcheck]: https://github.com/r-lib/revdepcheck
[GitHub Actions]: https://github.com/features/actions
[Travis CI]: https://travis-ci.org/
[AppVeyor CI]: https://www.appveyor.com/
[R-hub]: https://builder.r-hub.io/
|
