1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
|
@with-clean-spec-opts
Feature: Bisect
RSpec's `--order random` and `--seed` options help surface flickering examples that only fail when one or more other examples are executed first. It can be very difficult to isolate the exact combination of examples that triggers the failure. The `--bisect` flag helps solve that problem.
Pass the `--bisect` option (in addition to `--seed` and any other options) and RSpec will repeatedly run subsets of your suite in order to isolate the minimal set of examples that reproduce the same failures.
At any point during the bisect run, you can hit ctrl-c to abort and it will provide you with the most minimal reproduction command it has discovered so far.
To get more detailed output (particularly useful if you want to report a bug with bisect), use `--bisect=verbose`.
Background:
Given a file named "lib/calculator.rb" with:
"""ruby
class Calculator
def self.add(x, y)
x + y
end
end
"""
And a file named "spec/calculator_1_spec.rb" with:
"""ruby
require 'calculator'
RSpec.describe "Calculator" do
it 'adds numbers' do
expect(Calculator.add(1, 2)).to eq(3)
end
end
"""
And files "spec/calculator_2_spec.rb" through "spec/calculator_9_spec.rb" with an unrelated passing spec in each file
And a file named "spec/calculator_10_spec.rb" with:
"""ruby
require 'calculator'
RSpec.describe "Monkey patched Calculator" do
it 'does screwy math' do
# monkey patching `Calculator` affects examples that are
# executed after this one!
def Calculator.add(x, y)
x - y
end
expect(Calculator.add(5, 10)).to eq(-5)
end
end
"""
Scenario: Use `--bisect` flag to create a minimal repro case for the ordering dependency
When I run `rspec --seed 1234`
Then the output should contain "10 examples, 1 failure"
When I run `rspec --seed 1234 --bisect`
Then bisect should succeed with output like:
"""
Bisect started using options: "--seed 1234"
Running suite to find failures... (0.16755 seconds)
Starting bisect with 1 failing example and 9 non-failing examples.
Checking that failure(s) are order-dependent... failure appears to be order-dependent
Round 1: bisecting over non-failing examples 1-9 .. ignoring examples 6-9 (0.30166 seconds)
Round 2: bisecting over non-failing examples 1-5 .. ignoring examples 4-5 (0.30306 seconds)
Round 3: bisecting over non-failing examples 1-3 .. ignoring example 3 (0.33292 seconds)
Round 4: bisecting over non-failing examples 1-2 . ignoring example 1 (0.16476 seconds)
Bisect complete! Reduced necessary non-failing examples from 9 to 1 in 1.26 seconds.
The minimal reproduction command is:
rspec ./spec/calculator_10_spec.rb[1:1] ./spec/calculator_1_spec.rb[1:1] --seed 1234
"""
When I run `rspec ./spec/calculator_10_spec.rb[1:1] ./spec/calculator_1_spec.rb[1:1] --seed 1234`
Then the output should contain "2 examples, 1 failure"
Scenario: Ctrl-C can be used to abort the bisect early and get the most minimal command it has discovered so far
When I run `rspec --seed 1234 --bisect` and abort in the middle with ctrl-c
Then bisect should fail with output like:
"""
Bisect started using options: "--seed 1234"
Running suite to find failures... (0.17102 seconds)
Starting bisect with 1 failing example and 9 non-failing examples.
Checking that failure(s) are order-dependent... failure appears to be order-dependent
Round 1: bisecting over non-failing examples 1-9 .. ignoring examples 6-9 (0.32943 seconds)
Round 2: bisecting over non-failing examples 1-5 .. ignoring examples 4-5 (0.3154 seconds)
Round 3: bisecting over non-failing examples 1-3 .. ignoring example 3 (0.2175 seconds)
Bisect aborted!
The most minimal reproduction command discovered so far is:
rspec ./spec/calculator_10_spec.rb[1:1] ./spec/calculator_1_spec.rb[1:1] ./spec/calculator_3_spec.rb[1:1] --seed 1234
"""
When I run `rspec ./spec/calculator_10_spec.rb[1:1] ./spec/calculator_1_spec.rb[1:1] ./spec/calculator_3_spec.rb[1:1] --seed 1234`
Then the output should contain "3 examples, 1 failure"
Scenario: Use `--bisect=verbose` to enable verbose debug mode for more detail
When I run `rspec --seed 1234 --bisect=verbose`
Then bisect should succeed with output like:
"""
Bisect started using options: "--seed 1234" and bisect runner: :fork
Running suite to find failures... (0.16528 seconds)
- Failing examples (1):
- ./spec/calculator_1_spec.rb[1:1]
- Non-failing examples (9):
- ./spec/calculator_10_spec.rb[1:1]
- ./spec/calculator_2_spec.rb[1:1]
- ./spec/calculator_3_spec.rb[1:1]
- ./spec/calculator_4_spec.rb[1:1]
- ./spec/calculator_5_spec.rb[1:1]
- ./spec/calculator_6_spec.rb[1:1]
- ./spec/calculator_7_spec.rb[1:1]
- ./spec/calculator_8_spec.rb[1:1]
- ./spec/calculator_9_spec.rb[1:1]
Checking that failure(s) are order-dependent..
- Running: rspec ./spec/calculator_1_spec.rb[1:1] --seed 1234 (n.nnnn seconds)
- Failure appears to be order-dependent
Round 1: bisecting over non-failing examples 1-9
- Running: rspec ./spec/calculator_1_spec.rb[1:1] ./spec/calculator_6_spec.rb[1:1] ./spec/calculator_7_spec.rb[1:1] ./spec/calculator_8_spec.rb[1:1] ./spec/calculator_9_spec.rb[1:1] --seed 1234 (0.15302 seconds)
- Running: rspec ./spec/calculator_10_spec.rb[1:1] ./spec/calculator_1_spec.rb[1:1] ./spec/calculator_2_spec.rb[1:1] ./spec/calculator_3_spec.rb[1:1] ./spec/calculator_4_spec.rb[1:1] ./spec/calculator_5_spec.rb[1:1] --seed 1234 (0.19708 seconds)
- Examples we can safely ignore (4):
- ./spec/calculator_6_spec.rb[1:1]
- ./spec/calculator_7_spec.rb[1:1]
- ./spec/calculator_8_spec.rb[1:1]
- ./spec/calculator_9_spec.rb[1:1]
- Remaining non-failing examples (5):
- ./spec/calculator_10_spec.rb[1:1]
- ./spec/calculator_2_spec.rb[1:1]
- ./spec/calculator_3_spec.rb[1:1]
- ./spec/calculator_4_spec.rb[1:1]
- ./spec/calculator_5_spec.rb[1:1]
Round 2: bisecting over non-failing examples 1-5
- Running: rspec ./spec/calculator_1_spec.rb[1:1] ./spec/calculator_4_spec.rb[1:1] ./spec/calculator_5_spec.rb[1:1] --seed 1234 (0.15836 seconds)
- Running: rspec ./spec/calculator_10_spec.rb[1:1] ./spec/calculator_1_spec.rb[1:1] ./spec/calculator_2_spec.rb[1:1] ./spec/calculator_3_spec.rb[1:1] --seed 1234 (0.19065 seconds)
- Examples we can safely ignore (2):
- ./spec/calculator_4_spec.rb[1:1]
- ./spec/calculator_5_spec.rb[1:1]
- Remaining non-failing examples (3):
- ./spec/calculator_10_spec.rb[1:1]
- ./spec/calculator_2_spec.rb[1:1]
- ./spec/calculator_3_spec.rb[1:1]
Round 3: bisecting over non-failing examples 1-3
- Running: rspec ./spec/calculator_1_spec.rb[1:1] ./spec/calculator_2_spec.rb[1:1] --seed 1234 (0.21028 seconds)
- Running: rspec ./spec/calculator_10_spec.rb[1:1] ./spec/calculator_1_spec.rb[1:1] ./spec/calculator_3_spec.rb[1:1] --seed 1234 (0.1975 seconds)
- Examples we can safely ignore (1):
- ./spec/calculator_2_spec.rb[1:1]
- Remaining non-failing examples (2):
- ./spec/calculator_10_spec.rb[1:1]
- ./spec/calculator_3_spec.rb[1:1]
Round 4: bisecting over non-failing examples 1-2
- Running: rspec ./spec/calculator_10_spec.rb[1:1] ./spec/calculator_1_spec.rb[1:1] --seed 1234 (0.17173 seconds)
- Examples we can safely ignore (1):
- ./spec/calculator_3_spec.rb[1:1]
- Remaining non-failing examples (1):
- ./spec/calculator_10_spec.rb[1:1]
Bisect complete! Reduced necessary non-failing examples from 9 to 1 in 1.47 seconds.
The minimal reproduction command is:
rspec ./spec/calculator_10_spec.rb[1:1] ./spec/calculator_1_spec.rb[1:1] --seed 1234
"""
When I run `rspec ./spec/calculator_10_spec.rb[1:1] ./spec/calculator_1_spec.rb[1:1] --seed 1234`
Then the output should contain "2 examples, 1 failure"
Scenario: Pick a bisect runner via a config option
Given a file named "spec/spec_helper.rb" with:
"""
RSpec.configure do |c|
c.bisect_runner = :shell
end
"""
And a file named ".rspec" with:
"""
--require spec_helper
"""
When I run `rspec --seed 1234 --bisect=verbose`
Then bisect should succeed with output like:
"""
Bisect started using options: "--seed 1234" and bisect runner: :shell
# ...
The minimal reproduction command is:
rspec ./spec/calculator_10_spec.rb[1:1] ./spec/calculator_1_spec.rb[1:1] --seed 1234
"""
|
