1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
|
# Known issues
<!--
This source file is part of the Swift.org open source project
Copyright © 2023–2024 Apple Inc. and the Swift project authors
Licensed under Apache License v2.0 with Runtime Library Exception
See https://swift.org/LICENSE.txt for license information
See https://swift.org/CONTRIBUTORS.txt for Swift project authors
-->
Mark issues as known when running tests.
## Overview
The testing library provides several functions named `withKnownIssue()` that
you can use to mark issues as known. Use them to inform the testing library that
a test should not be marked as failing if only known issues are recorded.
### Mark an expectation failure as known
Consider a test function with a single expectation:
```swift
@Test func grillHeating() throws {
var foodTruck = FoodTruck()
try foodTruck.startGrill()
#expect(foodTruck.grill.isHeating) // ❌ Expectation failed
}
```
If the value of the `isHeating` property is `false`, `#expect` will record an
issue. If you cannot fix the underlying problem, you can surround the failing
code in a closure passed to `withKnownIssue()`:
```swift
@Test func grillHeating() throws {
var foodTruck = FoodTruck()
try foodTruck.startGrill()
withKnownIssue("Propane tank is empty") {
#expect(foodTruck.grill.isHeating) // Known issue
}
}
```
The issue recorded by `#expect` will then be considered "known" and the test
will not be marked as a failure. You may include an optional comment to explain
the problem or provide context.
### Mark a thrown error as known
If an `Error` is caught by the closure passed to `withKnownIssue()`, the issue
representing that caught error will be marked as known. Continuing the previous
example, suppose the problem is that the `startGrill()` function is throwing an
error. You can apply `withKnownIssue()` to this situation as well:
```swift
@Test func grillHeating() {
var foodTruck = FoodTruck()
withKnownIssue {
try foodTruck.startGrill() // Known issue
#expect(foodTruck.grill.isHeating)
}
}
```
Because all errors thrown from the closure are caught and interpreted as known
issues, the `withKnownIssue()` function is not throwing. Consequently, any
subsequent code which depends on the throwing call having succeeded (such as the
`#expect` after `startGrill()`) must be included in the closure to avoid
additional issues.
- Note: `withKnownIssue()` is recommended instead of `#expect(throws:)` for any
error which is considered a known issue so that the test status and results
will reflect the situation more accurately.
### Match a specific issue
By default, `withKnownIssue()` considers all issues recorded while invoking the
body closure known. If multiple issues may be recorded, you can pass a trailing
closure labeled `matching:` which will be called once for each recorded issue
to determine whether it should be treated as known:
```swift
@Test func batteryLevel() throws {
var foodTruck = FoodTruck()
try withKnownIssue {
let batteryLevel = try #require(foodTruck.batteryLevel) // Known
#expect(batteryLevel >= 0.8) // Not considered known
} matching: { issue in
guard case .expectationFailed(let expectation) = issue.kind else {
return false
}
return expectation.isRequired
}
}
```
### Resolve a known issue
If there are no issues recorded while calling `function`, `withKnownIssue()`
will record a distinct issue about the lack of any issues having been recorded.
This notifies you that the underlying problem may have been resolved so that you
can investigate and consider removing `withKnownIssue()` if it's no longer
necessary.
### Handle a nondeterministic failure
If `withKnownIssue()` sometimes succeeds but other times records an issue
indicating there were no known issues, this may indicate a nondeterministic
failure or a "flaky" test.
The first step in resolving a nondeterministic test failure is to analyze the
code being tested and determine the source of the unpredictable behavior. If
you discover a bug such as a race condition, the ideal resolution is to fix
the underlying problem so that the code always behaves consistently even if
it continues to exhibit the known issue.
If the underlying problem only occurs in certain circumstances, consider
including a precondition. For example, if the grill only fails to heat when
there's no propane, you can pass a trailing closure labeled `when:` which
determines whether issues recorded in the body closure should be considered
known:
```swift
@Test func grillHeating() throws {
var foodTruck = FoodTruck()
try foodTruck.startGrill()
withKnownIssue {
// Only considered known when hasPropane == false
#expect(foodTruck.grill.isHeating)
} when: {
!hasPropane
}
}
```
If the underlying problem is unpredictable and fails at random, you can pass
`isIntermittent: true` to let the testing library know that it will not always
occur. Then, the testing library will not record an issue when zero known issues
are recorded:
```swift
@Test func grillHeating() throws {
var foodTruck = FoodTruck()
try foodTruck.startGrill()
withKnownIssue(isIntermittent: true) {
#expect(foodTruck.grill.isHeating)
}
}
```
## Topics
### Recording known issues in tests
- ``withKnownIssue(_:isIntermittent:sourceLocation:_:)``
- ``withKnownIssue(_:isIntermittent:isolation:sourceLocation:_:)``
- ``withKnownIssue(_:isIntermittent:sourceLocation:_:when:matching:)``
- ``withKnownIssue(_:isIntermittent:isolation:sourceLocation:_:when:matching:)``
- ``KnownIssueMatcher``
### Describing a failure or warning
- ``Issue``
|
