1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
|
Feature: Hooks processing in case of errors (exceptions)
. SPECIFICATION:
. * Hook errors in before_all/after_all hook aborts the test run (and fail).
. * Hook errors in before_feature/after_feature causes the feature to fail.
. * Hook errors in before_feature causes feature scenarios to be skipped (untested).
. * Hook errors in before_scenario/after_scenario causes the scenario to fail.
. * Hook errors in before_scenario causes scenario steps to be skipped (untested).
. * Hook errors in before_step/after_step causes step to fail.
. * Hook errors in before_tag/after_tag of a feature causes feature to fail.
. * Hook errors in before_tag/after_tag of a scenario causes scenario to fail.
. * If a hook error occurs in a "before_xxx" hook,
. then the "after_xxx" hook will also be called (to cleanup some stuff).
.
. NOTE:
. The --verbose flag can be used to show the exception traceback.
@setup
Scenario: Feature Setup
Given a new working directory
And a file named "features/steps/steps.py" with:
"""
from behave import step
@step(u'{word:w} step passes')
def step(context, word):
pass
"""
And a file named "features/passing.feature" with:
"""
@foo
Feature: Alice
@soo
Scenario: A1
Given a step passes
"""
And a file named "features/environment.py" with:
"""
from __future__ import print_function
def cause_hook_to_fail():
raise RuntimeError("FAIL")
def before_all(context):
userdata = context.config.userdata
if userdata.get("HOOK_ERROR_LOC") == "before_all":
cause_hook_to_fail()
def after_all(context):
userdata = context.config.userdata
if userdata.get("HOOK_ERROR_LOC") == "before_all":
print("called_hook:after_all")
if userdata.get("HOOK_ERROR_LOC") == "after_all":
cause_hook_to_fail()
def before_feature(context, feature):
userdata = context.config.userdata
if userdata.get("HOOK_ERROR_LOC") == "before_feature":
cause_hook_to_fail()
def after_feature(context, feature):
userdata = context.config.userdata
if userdata.get("HOOK_ERROR_LOC") == "before_feature":
print("called_hook:after_feature")
if userdata.get("HOOK_ERROR_LOC") == "after_feature":
cause_hook_to_fail()
def before_scenario(context, scenario):
userdata = context.config.userdata
if userdata.get("HOOK_ERROR_LOC") == "before_scenario":
cause_hook_to_fail()
def after_scenario(context, scenario):
userdata = context.config.userdata
if userdata.get("HOOK_ERROR_LOC") == "before_scenario":
print("called_hook:after_scenario")
if userdata.get("HOOK_ERROR_LOC") == "after_scenario":
cause_hook_to_fail()
def before_step(context, step):
userdata = context.config.userdata
if userdata.get("HOOK_ERROR_LOC") == "before_step":
cause_hook_to_fail()
def after_step(context, step):
userdata = context.config.userdata
if userdata.get("HOOK_ERROR_LOC") == "before_step":
print("called_hook:after_step")
if userdata.get("HOOK_ERROR_LOC") == "after_step":
cause_hook_to_fail()
def before_tag(context, tag):
userdata = context.config.userdata
if userdata.get("HOOK_ERROR_LOC") == "before_tag":
error_tag = userdata.get("HOOK_ERROR_TAG")
if not error_tag or tag == error_tag:
cause_hook_to_fail()
def after_tag(context, tag):
userdata = context.config.userdata
if userdata.get("HOOK_ERROR_LOC") == "before_tag":
error_tag = userdata.get("HOOK_ERROR_TAG")
if tag == error_tag:
print("called_hook:after_tag: tag=%s" % tag)
if userdata.get("HOOK_ERROR_LOC") == "after_tag":
error_tag = userdata.get("HOOK_ERROR_TAG")
if not error_tag or tag == error_tag:
cause_hook_to_fail()
"""
And a file named "behave.ini" with:
"""
[behave]
show_skipped = false
show_timings = false
"""
@hook.before_all
Scenario: Hook error in before_all
When I run "behave -f plain -D HOOK_ERROR_LOC=before_all features/passing.feature"
Then it should fail with
"""
HOOK-ERROR in before_all: RuntimeError: FAIL
called_hook:after_all
ABORTED: By user.
0 features passed, 0 failed, 0 skipped, 1 untested
0 scenarios passed, 0 failed, 0 skipped, 1 untested
0 steps passed, 0 failed, 0 skipped, 0 undefined, 1 untested
"""
But note that "the after_all hook is called, too"
@hook.after_all
Scenario: Hook error in after_all
When I run "behave -f plain -D HOOK_ERROR_LOC=after_all features/passing.feature"
Then it should fail with:
"""
Feature: Alice
Scenario: A1
Given a step passes ... passed
HOOK-ERROR in after_all: RuntimeError: FAIL
ABORTED: By user.
1 feature passed, 0 failed, 0 skipped
1 scenario passed, 0 failed, 0 skipped
1 step passed, 0 failed, 0 skipped, 0 undefined
"""
@hook.before_feature
Scenario: Hook error in before_feature
When I run "behave -f plain -D HOOK_ERROR_LOC=before_feature features/passing.feature"
Then it should fail with:
"""
HOOK-ERROR in before_feature: RuntimeError: FAIL
Feature: Alice
called_hook:after_feature
0 features passed, 1 failed, 0 skipped
0 scenarios passed, 0 failed, 0 skipped, 1 untested
0 steps passed, 0 failed, 0 skipped, 0 undefined, 1 untested
"""
But note that "the after_feature hook is called, too."
@hook.after_feature
Scenario: Hook error in after_feature
When I run "behave -f plain -D HOOK_ERROR_LOC=after_feature features/passing.feature"
Then it should fail with:
"""
Feature: Alice
Scenario: A1
Given a step passes ... passed
HOOK-ERROR in after_feature: RuntimeError: FAIL
0 features passed, 1 failed, 0 skipped
1 scenario passed, 0 failed, 0 skipped
1 step passed, 0 failed, 0 skipped, 0 undefined
"""
@hook.before_scenario
Scenario: Hook error in before_scenario
When I run "behave -f plain -D HOOK_ERROR_LOC=before_scenario features/passing.feature"
Then it should fail with:
"""
Feature: Alice
HOOK-ERROR in before_scenario: RuntimeError: FAIL
Scenario: A1
called_hook:after_scenario
Failing scenarios:
features/passing.feature:4 A1
0 features passed, 1 failed, 0 skipped
0 scenarios passed, 1 failed, 0 skipped
0 steps passed, 0 failed, 0 skipped, 0 undefined, 1 untested
"""
But note that "the after_scenario hook is called, too."
@hook.after_scenario
Scenario: Hook error in after_scenario
When I run "behave -f plain -D HOOK_ERROR_LOC=after_scenario features/passing.feature"
Then it should fail with:
"""
Feature: Alice
Scenario: A1
Given a step passes ... passed
HOOK-ERROR in after_scenario: RuntimeError: FAIL
Failing scenarios:
features/passing.feature:4 A1
0 features passed, 1 failed, 0 skipped
0 scenarios passed, 1 failed, 0 skipped
1 step passed, 0 failed, 0 skipped, 0 undefined
"""
@hook.before_step
Scenario: Hook error in before_step
When I run "behave -f plain -D HOOK_ERROR_LOC=before_step features/passing.feature"
Then it should fail with:
"""
Feature: Alice
Scenario: A1
Given a step passes ... failed
Captured stdout:
HOOK-ERROR in before_step: RuntimeError: FAIL
called_hook:after_step
Failing scenarios:
features/passing.feature:4 A1
0 features passed, 1 failed, 0 skipped
0 scenarios passed, 1 failed, 0 skipped
0 steps passed, 1 failed, 0 skipped, 0 undefined
"""
But note that "the after_step hook is called, too."
@hook.after_step
Scenario: Hook error in after_step
When I run "behave -f plain -D HOOK_ERROR_LOC=after_step features/passing.feature"
Then it should fail with:
"""
Feature: Alice
Scenario: A1
Given a step passes ... failed
Captured stdout:
HOOK-ERROR in after_step: RuntimeError: FAIL
Failing scenarios:
features/passing.feature:4 A1
0 features passed, 1 failed, 0 skipped
0 scenarios passed, 1 failed, 0 skipped
0 steps passed, 1 failed, 0 skipped, 0 undefined
"""
@hook.before_tag
Scenario: Hook error in before_tag for feature
When I run "behave -f plain -D HOOK_ERROR_LOC=before_tag -D HOOK_ERROR_TAG=foo features/passing.feature"
Then it should fail with:
"""
HOOK-ERROR in before_tag(tag=foo): RuntimeError: FAIL
Feature: Alice
called_hook:after_tag: tag=foo
0 features passed, 1 failed, 0 skipped
0 scenarios passed, 0 failed, 0 skipped, 1 untested
0 steps passed, 0 failed, 0 skipped, 0 undefined, 1 untested
"""
And the command output should contain:
"""
HOOK-ERROR in before_tag(tag=foo): RuntimeError: FAIL
Feature: Alice
"""
But note that "the hook-error in before_tag of the feature causes it to fail and be skipped (untested)"
And note that "the after_tag hook is still called"
@hook.after_tag
Scenario: Hook error in after_tag for feature
When I run "behave -f plain -D HOOK_ERROR_LOC=after_tag -D HOOK_ERROR_TAG=foo features/passing.feature"
Then it should fail with:
"""
Feature: Alice
Scenario: A1
Given a step passes ... passed
HOOK-ERROR in after_tag(tag=foo): RuntimeError: FAIL
0 features passed, 1 failed, 0 skipped
1 scenario passed, 0 failed, 0 skipped
1 step passed, 0 failed, 0 skipped, 0 undefined
"""
But note that "the hook-error in after_tag of the scenario causes it to fail"
@hook.before_tag
Scenario: Hook error in before_tag for scenario
When I run "behave -f plain -D HOOK_ERROR_LOC=before_tag -D HOOK_ERROR_TAG=soo features/passing.feature"
Then it should fail with:
"""
Feature: Alice
HOOK-ERROR in before_tag(tag=soo): RuntimeError: FAIL
Scenario: A1
called_hook:after_tag: tag=soo
Failing scenarios:
features/passing.feature:4 A1
0 features passed, 1 failed, 0 skipped
0 scenarios passed, 1 failed, 0 skipped
0 steps passed, 0 failed, 0 skipped, 0 undefined, 1 untested
"""
But note that "the hook-error in before_tag of the scenario causes it to fail and be skipped (untested)"
And note that "the after_tag hook is still called"
@hook.after_tag
Scenario: Hook error in after_tag for scenario
When I run "behave -f plain -D HOOK_ERROR_LOC=after_tag -D HOOK_ERROR_TAG=soo features/passing.feature"
Then it should fail with:
"""
Feature: Alice
Scenario: A1
Given a step passes ... passed
HOOK-ERROR in after_tag(tag=soo): RuntimeError: FAIL
Failing scenarios:
features/passing.feature:4 A1
0 features passed, 1 failed, 0 skipped
0 scenarios passed, 1 failed, 0 skipped
1 step passed, 0 failed, 0 skipped, 0 undefined
"""
But note that "the hook-error in after_tag of the scenario causes it to fail"
@skipped.hook.after_feature
Scenario: Skipped feature with potential hook error (hooks are not run)
This goes unnoticed because hooks are not run for a skipped feature/scenario.
NOTE: Except if before_feature(), before_scenario() hook skips the feature/scenario.
When I run "behave -f plain -D HOOK_ERROR_LOC=after_feature -t ~@foo features/passing.feature"
Then it should pass with:
"""
0 features passed, 0 failed, 1 skipped
0 scenarios passed, 0 failed, 1 skipped
0 steps passed, 0 failed, 1 skipped, 0 undefined
"""
But note that "hooks are not executed for skipped features/scenarios"
Scenario: Show hook error details (traceback)
When I run "behave -f plain -D HOOK_ERROR_LOC=before_feature --verbose features/passing.feature"
Then it should fail with:
"""
HOOK-ERROR in before_feature: RuntimeError: FAIL
"""
And the command output should contain:
"""
self.hooks[name](context, *args)
File "features/environment.py", line 21, in before_feature
cause_hook_to_fail()
File "features/environment.py", line 4, in cause_hook_to_fail
raise RuntimeError("FAIL")
"""
But note that "the traceback caused by the hook-error is shown"
|
