1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
|
# Testing
The Remote Protocol has unit- and functional tests located under different folders:
* Marionette: `remote/marionette/`.
* Shared Modules: `remote/shared/`
* WebDriver BiDi: `remote/webdriver-bidi/`
You may want to run all the tests under a particular subfolder locally like this:
```shell
% ./mach test remote
```
## Unit tests
Because tests are run in parallel and [xpcshell] itself is quite
chatty, it can sometimes be useful to run the tests in sequence:
```shell
% ./mach xpcshell-test --sequential remote/shared/test/xpcshell
```
The unit tests will appear as part of the `X` (for _xpcshell_) jobs
on Treeherder.
[xpcshell]: /testing/xpcshell/index.rst
## Browser Chrome Mochitests
We also have a set of functional browser-chrome mochitests located
under several components, ie. _remote/shared/messagehandler/test/browser_:
```shell
% ./mach mochitest remote/shared/messagehandler/test/browser/browser_*
```
The functional tests will appear under the `M` (for _mochitest_)
category in the `remote` jobs on Treeherder.
As the functional tests will sporadically pop up new Firefox
application windows, a helpful tip is to run them in headless
mode:
```shell
% ./mach mochitest --headless remote/shared/messagehandler/test/browser
```
The `--headless` flag is equivalent to setting the `MOZ_HEADLESS`
environment variable. You can additionally use `MOZ_HEADLESS_WIDTH`
and `MOZ_HEADLESS_HEIGHT` to control the dimensions of the virtual
display.
The `add_task()` function used for writing asynchronous tests is
replaced to provide some additional test setup and teardown useful
for writing tests against the Remote Agent and the targets.
## Puppeteer tests
In addition to our own Firefox-specific tests, we run the upstream
[Puppeteer test suite] against our implementation to [track progress]
towards achieving full [Puppeteer support] in Firefox when using the
WebDriver BiDi protocol. The tests are written in the behavior-driven
testing framework [Mocha].
Puppeteer tests are vendored under _remote/test/puppeteer/_ and are
run locally like this:
```shell
% ./mach puppeteer-test
```
You can also run them against Chrome as:
```shell
% ./mach puppeteer-test --product=chrome
```
By default the mach command will automatically install Puppeteer but that's
only needed for the very first time, or when a new Puppeteer release has been
vendored in. To skip the install step use the `--no-install` option.
To run only some specific tests from the whole test suite the appropriate
test files have to be updated first. To select specific tests or test
groups within a file define [exclusive tests] by adding the `.only` suffix
like `it.only()` or `describe.only()`.
It is also possible, similar to how it works in CI, to run tests in chunks:
```shell
% ./mach puppeteer-test --this-chunk=1 --total-chunks=2
```
More customizations for [Mocha] can be found in its own documentation.
Test expectation metadata is collected in [TestExpectations.json]
via log parsing and a custom Mocha reporter under
_remote/test/puppeteer/json-mocha-reporter.js_. For Nightly, overrides to these expectations
are stored in [CanaryTestExpectations.json].
Check the upstream [Puppeteer test suite] documentation for instructions on
how to skip tests, run only one test or a subsuite of tests.
## Testing on Try
To schedule all the Remote Protocol tests on try, you can use the
`remote-protocol` [try preset]:
```shell
% ./mach try --preset remote-protocol
```
But you can also schedule tests by selecting relevant jobs yourself:
```shell
% ./mach try fuzzy
```
[Puppeteer test suite]: https://github.com/puppeteer/puppeteer/blob/master/test/README.md
[Puppeteer support]: https://bugzilla.mozilla.org/show_bug.cgi?id=puppeteer
[Mocha]: https://mochajs.org/
[exclusive tests]: https://mochajs.org/#exclusive-tests
[track progress]: https://puppeteer.github.io/ispuppeteerfirefoxready/
[try preset]: /tools/try/presets
[TestExpectations.json]: https://searchfox.org/mozilla-central/source/remote/test/puppeteer/test/TestExpectations.json
[CanaryTestExpectations.json]: https://searchfox.org/mozilla-central/source/remote/test/puppeteer/test/CanaryTestExpectations.json
|
