1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
|
# parallel_tests
[](https://rubygems.org/gems/parallel_tests)
[](https://github.com/grosser/parallel_tests/actions?query=workflow%3Atest)
Speedup Test::Unit + RSpec + Cucumber + Spinach by running parallel on multiple CPU cores.<br/>
ParallelTests splits tests into even groups (by number of lines or runtime) and runs each group in a single process with its own database.
Setup for Rails
===============
[RailsCasts episode #413 Fast Tests](http://railscasts.com/episodes/413-fast-tests)
### Install
`Gemfile`:
```ruby
gem 'parallel_tests', group: [:development, :test]
```
### Add to `config/database.yml`
ParallelTests uses 1 database per test-process.
<table>
<tr><td>Process number</td><td>1</td><td>2</td><td>3</td></tr>
<tr><td>ENV['TEST_ENV_NUMBER']</td><td>''</td><td>'2'</td><td>'3'</td></tr>
</table>
```yaml
test:
database: yourproject_test<%= ENV['TEST_ENV_NUMBER'] %>
```
### Create additional database(s)
rake parallel:create
### Copy development schema (repeat after migrations)
rake parallel:prepare
### Run migrations in additional database(s) (repeat after migrations)
rake parallel:migrate
### Setup environment from scratch (create db and loads schema, useful for CI)
rake parallel:setup
### Drop all test databases
rake parallel:drop
### Run!
rake parallel:test # Test::Unit
rake parallel:spec # RSpec
rake parallel:features # Cucumber
rake parallel:features-spinach # Spinach
rake parallel:test[1] --> force 1 CPU --> 86 seconds
rake parallel:test --> got 2 CPUs? --> 47 seconds
rake parallel:test --> got 4 CPUs? --> 26 seconds
...
Test by pattern with Regex (e.g. use one integration server per subfolder / see if you broke any 'user'-related tests)
rake parallel:test[^test/unit] # every test file in test/unit folder
rake parallel:test[user] # run users_controller + user_helper + user tests
rake parallel:test['user|product'] # run user and product related tests
rake parallel:spec['spec\/(?!features)'] # run RSpec tests except the tests in spec/features
### Example output
2 processes for 210 specs, ~ 105 specs per process
... test output ...
843 examples, 0 failures, 1 pending
Took 29.925333 seconds
### Run an arbitrary task in parallel
```Bash
RAILS_ENV=test parallel_test -e "rake my:custom:task"
# or
rake parallel:rake[my:custom:task]
# limited parallelism
rake parallel:rake[my:custom:task,2]
```
Running things once
===================
```Ruby
require "parallel_tests"
# preparation:
# affected by race-condition: first process may boot slower than the second
# either sleep a bit or use a lock for example File.lock
ParallelTests.first_process? ? do_something : sleep(1)
# cleanup:
# last_process? does NOT mean last finished process, just last started
ParallelTests.last_process? ? do_something : sleep(1)
at_exit do
if ParallelTests.first_process?
ParallelTests.wait_for_other_processes_to_finish
undo_something
end
end
```
Even test group run-times
=========================
Test groups are often not balanced and will run for different times, making everything wait for the slowest group.
Use these loggers to record test runtime and then use the recorded runtime to balance test groups more evenly.
### RSpec
Rspec: Add to your `.rspec_parallel` (or `.rspec`) :
--format progress
--format ParallelTests::RSpec::RuntimeLogger --out tmp/parallel_runtime_rspec.log
To use a custom logfile location (default: `tmp/parallel_runtime_rspec.log`), use the CLI: `parallel_test spec -t rspec --runtime-log my.log`
### Minitest
Add to your `test_helper.rb`:
```ruby
require 'parallel_tests/test/runtime_logger' if ENV['RECORD_RUNTIME']
```
results will be logged to tmp/parallel_runtime_test.log when `RECORD_RUNTIME` is set,
so it is not always required or overwritten.
Loggers
=======
RSpec: SummaryLogger
--------------------
Log the test output without the different processes overwriting each other.
Add the following to your `.rspec_parallel` (or `.rspec`) :
--format progress
--format ParallelTests::RSpec::SummaryLogger --out tmp/spec_summary.log
RSpec: FailuresLogger
-----------------------
Produce pastable command-line snippets for each failed example. For example:
```bash
rspec /path/to/my_spec.rb:123 # should do something
```
Add to `.rspec_parallel` or use as CLI flag:
--format progress
--format ParallelTests::RSpec::FailuresLogger --out tmp/failing_specs.log
(Not needed to retry failures, for that pass [--only-failures](https://relishapp.com/rspec/rspec-core/docs/command-line/only-failures) to rspec)
Cucumber: FailuresLogger
-----------------------
Log failed cucumber scenarios to the specified file. The filename can be passed to cucumber, prefixed with '@' to rerun failures.
Usage:
cucumber --format ParallelTests::Cucumber::FailuresLogger --out tmp/cucumber_failures.log
Or add the formatter to the `parallel:` profile of your `cucumber.yml`:
parallel: --format progress --format ParallelTests::Cucumber::FailuresLogger --out tmp/cucumber_failures.log
Note if your `cucumber.yml` default profile uses `<%= std_opts %>` you may need to insert this as follows `parallel: <%= std_opts %> --format progress...`
To rerun failures:
cucumber @tmp/cucumber_failures.log
Setup for non-rails
===================
gem install parallel_tests
# go to your project dir
parallel_test
parallel_rspec
parallel_cucumber
parallel_spinach
- use `ENV['TEST_ENV_NUMBER']` inside your tests to select separate db/memcache/etc. (docker compose: expose it)
- Only run a subset of files / folders:
`parallel_test test/bar test/baz/foo_text.rb`
- Pass test-options and files via `--`:
`parallel_rspec -- -t acceptance -f progress -- spec/foo_spec.rb spec/acceptance`
- Pass in test options, by using the -o flag (wrap everything in quotes):
`parallel_cucumber -n 2 -o '-p foo_profile --tags @only_this_tag or @only_that_tag --format summary'`
Options are:
<!-- copy output from bundle exec ./bin/parallel_test -h -->
-n [PROCESSES] How many processes to use, default: available CPUs
-p, --pattern [PATTERN] run tests matching this regex pattern
--exclude-pattern [PATTERN] exclude tests matching this regex pattern
--group-by [TYPE] group tests by:
found - order of finding files
steps - number of cucumber/spinach steps
scenarios - individual cucumber scenarios
filesize - by size of the file
runtime - info from runtime log
default - runtime when runtime log is filled otherwise filesize
-m, --multiply-processes [FLOAT] use given number as a multiplier of processes to run
-s, --single [PATTERN] Run all matching files in the same process
-i, --isolate Do not run any other tests in the group used by --single(-s)
--isolate-n [PROCESSES] Use 'isolate' singles with number of processes, default: 1.
--highest-exit-status Exit with the highest exit status provided by test run(s)
--specify-groups [SPECS] Use 'specify-groups' if you want to specify multiple specs running in multiple
processes in a specific formation. Commas indicate specs in the same process,
pipes indicate specs in a new process. Cannot use with --single, --isolate, or
--isolate-n. Ex.
$ parallel_tests -n 3 . --specify-groups '1_spec.rb,2_spec.rb|3_spec.rb'
Process 1 will contain 1_spec.rb and 2_spec.rb
Process 2 will contain 3_spec.rb
Process 3 will contain all other specs
--only-group INT[,INT]
-e, --exec [COMMAND] execute this code parallel and with ENV['TEST_ENV_NUMBER']
-o, --test-options '[OPTIONS]' execute test commands with those options
-t, --type [TYPE] test(default) / rspec / cucumber / spinach
--suffix [PATTERN] override built in test file pattern (should match suffix):
'_spec.rb$' - matches rspec files
'_(test|spec).rb$' - matches test or spec files
--serialize-stdout Serialize stdout output, nothing will be written until everything is done
--prefix-output-with-test-env-number
Prefixes test env number to the output when not using --serialize-stdout
--combine-stderr Combine stderr into stdout, useful in conjunction with --serialize-stdout
--non-parallel execute same commands but do not in parallel, needs --exec
--no-symlinks Do not traverse symbolic links to find test files
--ignore-tags [PATTERN] When counting steps ignore scenarios with tags that match this pattern
--nice execute test commands with low priority.
--runtime-log [PATH] Location of previously recorded test runtimes
--allowed-missing [INT] Allowed percentage of missing runtimes (default = 50)
--unknown-runtime [FLOAT] Use given number as unknown runtime (otherwise use average time)
--first-is-1 Use "1" as TEST_ENV_NUMBER to not reuse the default test environment
--fail-fast Stop all groups when one group fails (best used with --test-options '--fail-fast' if supported
--verbose Print debug output
--verbose-process-command Displays only the command that will be executed by each process
--verbose-rerun-command When there are failures, displays the command executed by each process that failed
--quiet Print only tests output
-v, --version Show Version
-h, --help Show this.
You can run any kind of code in parallel with -e / --exec
parallel_test -n 5 -e 'ruby -e "puts %[hello from process #{ENV[:TEST_ENV_NUMBER.to_s].inspect}]"'
hello from process "2"
hello from process ""
hello from process "3"
hello from process "5"
hello from process "4"
<table>
<tr><td></td><td>1 Process</td><td>2 Processes</td><td>4 Processes</td></tr>
<tr><td>RSpec spec-suite</td><td>18s</td><td>14s</td><td>10s</td></tr>
<tr><td>Rails-ActionPack</td><td>88s</td><td>53s</td><td>44s</td></tr>
</table>
TIPS
====
### RSpec
- Add a `.rspec_parallel` to use different options, e.g. **no --drb**
- Remove `--loadby` from `.rspec`
- Instantly see failures (instead of just a red F) with [rspec-instafail](https://github.com/grosser/rspec-instafail)
- Use [rspec-retry](https://github.com/NoRedInk/rspec-retry) (not rspec-rerun) to rerun failed tests.
- [JUnit formatter configuration](https://github.com/grosser/parallel_tests/wiki#with-rspec_junit_formatter----by-jgarber)
- Use [parallel_split_test](https://github.com/grosser/parallel_split_test) to run multiple scenarios in a single spec file, concurrently. (`parallel_tests` [works at the file-level and intends to stay that way](https://github.com/grosser/parallel_tests/issues/747#issuecomment-580216980))
### Cucumber
- Add a `parallel: foo` profile to your `config/cucumber.yml` and it will be used to run parallel tests
- [ReportBuilder](https://github.com/rajatthareja/ReportBuilder) can help with combining parallel test results
- Supports Cucumber 2.0+ and is actively maintained
- Combines many JSON files into a single file
- Builds a HTML report from JSON with support for debug msgs & embedded Base64 images.
### General
- [ZSH] use quotes to use rake arguments `rake "parallel:prepare[3]"`
- [Memcached] use different namespaces<br/>
e.g. `config.cache_store = ..., namespace: "test_#{ENV['TEST_ENV_NUMBER']}"`
- Debug errors that only happen with multiple files using `--verbose` and [cleanser](https://github.com/grosser/cleanser)
- `export PARALLEL_TEST_PROCESSORS=13` to override default processor count
- Shell alias: `alias prspec='parallel_rspec -m 2 --'`
- [Spring] Add the [spring-commands-parallel-tests](https://github.com/DocSpring/spring-commands-parallel-tests) gem to your `Gemfile` to get `parallel_tests` working with Spring.
- `--first-is-1` will make the first environment be `1`, so you can test while running your full suite.<br/>
`export PARALLEL_TEST_FIRST_IS_1=true` will provide the same result
- [email_spec and/or action_mailer_cache_delivery](https://github.com/grosser/parallel_tests/wiki)
- [zeus-parallel_tests](https://github.com/sevos/zeus-parallel_tests)
- [Distributed Parallel Tests on CI systems)](https://github.com/grosser/parallel_tests/wiki/Distributed-Parallel-Tests-on-CI-systems) learn how `parallel_tests` can run on distributed servers such as Travis and GitLab-CI. Also shows you how to use parallel_tests without adding `TEST_ENV_NUMBER`-backends
- [Capybara setup](https://github.com/grosser/parallel_tests/wiki)
- [Sphinx setup](https://github.com/grosser/parallel_tests/wiki)
- [Capistrano setup](https://github.com/grosser/parallel_tests/wiki/Remotely-with-capistrano) let your tests run on a big box instead of your laptop
Contribute your own gotchas to the [Wiki](https://github.com/grosser/parallel_tests/wiki) or even better open a PR :)
Authors
====
inspired by [pivotal labs](https://blog.pivotal.io/labs/labs/parallelize-your-rspec-suite)
### [Contributors](https://github.com/grosser/parallel_tests/contributors)
- [Charles Finkel](http://charlesfinkel.com/)
- [Indrek Juhkam](http://urgas.eu)
- [Jason Morrison](http://jayunit.net)
- [jinzhu](http://github.com/jinzhu)
- [Joakim Kolsjö](http://www.rubyblocks.se)
- [Kevin Scaldeferri](http://kevin.scaldeferri.com/blog/)
- [Kpumuk](http://kpumuk.info/)
- [Maksim Horbul](http://github.com/mhorbul)
- [Pivotal Labs](http://www.pivotallabs.com)
- [Rohan Deshpande](http://github.com/rdeshpande)
- [Tchandy](http://thiagopradi.net/)
- [Terence Lee](http://hone.heroku.com/)
- [Will Bryant](http://willbryant.net/)
- [Fred Wu](http://fredwu.me)
- [xxx](https://github.com/xxx)
- [Levent Ali](http://purebreeze.com/)
- [Michael Kintzer](https://github.com/rockrep)
- [nathansobo](https://github.com/nathansobo)
- [Joe Yates](http://titusd.co.uk)
- [asmega](http://www.ph-lee.com)
- [Doug Barth](https://github.com/dougbarth)
- [Geoffrey Hichborn](https://github.com/phene)
- [Trae Robrock](https://github.com/trobrock)
- [Lawrence Wang](https://github.com/levity)
- [Sean Walbran](https://github.com/seanwalbran)
- [Lawrence Wang](https://github.com/levity)
- [Potapov Sergey](https://github.com/greyblake)
- [Łukasz Tackowiak](https://github.com/lukasztackowiak)
- [Pedro Carriço](https://github.com/pedrocarrico)
- [Pablo Manrubia Díez](https://github.com/pmanrubia)
- [Slawomir Smiechura](https://github.com/ssmiech)
- [Georg Friedrich](https://github.com/georg)
- [R. Tyler Croy](https://github.com/rtyler)
- [Ulrich Berkmüller](https://github.com/ulrich-berkmueller)
- [Grzegorz Derebecki](https://github.com/madmax)
- [Florian Motlik](https://github.com/flomotlik)
- [Artem Kuzko](https://github.com/akuzko)
- [Zeke Fast](https://github.com/zekefast)
- [Joseph Shraibman](https://github.com/jshraibman-mdsol)
- [David Davis](https://github.com/daviddavis)
- [Ari Pollak](https://github.com/aripollak)
- [Aaron Jensen](https://github.com/aaronjensen)
- [Artur Roszczyk](https://github.com/sevos)
- [Caleb Tomlinson](https://github.com/calebTomlinson)
- [Jawwad Ahmad](https://github.com/jawwad)
- [Iain Beeston](https://github.com/iainbeeston)
- [Alejandro Pulver](https://github.com/alepulver)
- [Felix Clack](https://github.com/felixclack)
- [Izaak Alpert](https://github.com/karlhungus)
- [Micah Geisel](https://github.com/botandrose)
- [Exoth](https://github.com/Exoth)
- [sidfarkus](https://github.com/sidfarkus)
- [Colin Harris](https://github.com/aberant)
- [Wataru MIYAGUNI](https://github.com/gongo)
- [Brandon Turner](https://github.com/blt04)
- [Matt Hodgson](https://github.com/mhodgson)
- [bicarbon8](https://github.com/bicarbon8)
- [seichner](https://github.com/seichner)
- [Matt Southerden](https://github.com/mattsoutherden)
- [Stanislaw Wozniak](https://github.com/sponte)
- [Dmitry Polushkin](https://github.com/dmitry)
- [Samer Masry](https://github.com/smasry)
- [Volodymyr Mykhailyk](https:/github.com/volodymyr-mykhailyk)
- [Mike Mueller](https://github.com/mmueller)
- [Aaron Jensen](https://github.com/aaronjensen)
- [Ed Slocomb](https://github.com/edslocomb)
- [Cezary Baginski](https://github.com/e2)
- [Marius Ioana](https://github.com/mariusioana)
- [Lukas Oberhuber](https://github.com/lukaso)
- [Ryan Zhang](https://github.com/ryanus)
- [Rhett Sutphin](https://github.com/rsutphin)
- [Doc Ritezel](https://github.com/ohrite)
- [Alexandre Wilhelm](https://github.com/dogild)
- [Jerry](https://github.com/boblington)
- [Aleksei Gusev](https://github.com/hron)
- [Scott Olsen](https://github.com/scottolsen)
- [Andrei Botalov](https://github.com/abotalov)
- [Zachary Attas](https://github.com/snackattas)
- [David Rodríguez](https://github.com/deivid-rodriguez)
- [Justin Doody](https://github.com/justindoody)
- [Sandeep Singh](https://github.com/sandeepnagra)
- [Calaway](https://github.com/calaway)
- [alboyadjian](https://github.com/alboyadjian)
- [Nathan Broadbent](https://github.com/ndbroadbent)
- [Vikram B Kumar](https://github.com/v-kumar)
- [Joshua Pinter](https://github.com/joshuapinter)
- [Zach Dennis](https://github.com/zdennis)
- [Jon Dufresne](https://github.com/jdufresne)
- [Eric Kessler](https://github.com/enkessler)
- [Adis Osmonov](https://github.com/adis-io)
[Michael Grosser](http://grosser.it)<br/>
michael@grosser.it<br/>
License: MIT
|
