1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
|
# DExTer (Debugging Experience Tester)
## Introduction
DExTer is a suite of tools used to evaluate the "User Debugging Experience". DExTer drives an external debugger, running on small test programs, and collects information on the behavior at each debugger step to provide quantitative values that indicate the quality of the debugging experience.
## Supported Debuggers
DExTer currently supports the Visual Studio 2015 and Visual Studio 2017 debuggers via the [DTE interface](https://docs.microsoft.com/en-us/dotnet/api/envdte.dte), and LLDB via its [Python interface](https://lldb.llvm.org/python-reference.html). GDB is not currently supported.
The following command evaluates your environment, listing the available and compatible debuggers:
dexter.py list-debuggers
## Dependencies
[TODO] Add a requirements.txt or an install.py and document it here.
### Python 3.6
DExTer requires python version 3.6 or greater.
### pywin32 python package
This is required to access the DTE interface for the Visual Studio debuggers.
<python-executable> -m pip install pywin32
### clang
DExTer is current compatible with 'clang' and 'clang-cl' compiler drivers. The compiler must be available for DExTer, for example the following command should successfully build a runnable executable.
<compiler-executable> tests/nostdlib/fibonacci/test.cpp
## Running a test case
The following commands build fibonacci.cpp from the tests/nostdlib directory and run it in LLDB, reporting the debug experience heuristic. The first pair of commands build with no optimizations (-O0) and score 1.0000. The second pair of commands build with optimizations (-O2) and score 0.2832 which suggests a worse debugging experience.
clang -O0 -g tests/nostdlib/fibonacci.cpp -o tests/nostdlib/fibonacci/test
dexter.py test --binary tests/nostdlib/fibonacci/test --debugger lldb -- tests/nostdlib/fibonacci/test.cpp
test.cpp = (1.0000)
clang -O2 -g tests/nostdlib/fibonacci/test.cpp -o tests/nostdlib/fibonacci/test
dexter.py test --binary tests/nostdlib/fibonacci/test --debugger lldb -- tests/nostdlib/fibonacci/test.cpp
test.cpp = (0.2832)
## An example test case
The sample test case (tests/nostdlib/fibonacci) looks like this:
1. #ifdef _MSC_VER
2. # define DEX_NOINLINE __declspec(noinline)
3. #else
4. # define DEX_NOINLINE __attribute__((__noinline__))
5. #endif
6.
7. DEX_NOINLINE
8. void Fibonacci(int terms, int& total)
9. {
0. int first = 0;
11. int second = 1;
12. for (int i = 0; i < terms; ++i)
13. {
14. int next = first + second; // DexLabel('start')
15. total += first;
16. first = second;
17. second = next; // DexLabel('end')
18. }
19. }
20.
21. int main()
22. {
23. int total = 0;
24. Fibonacci(5, total);
25. return total;
26. }
27.
28. /*
29. DexExpectWatchValue('i', '0', '1', '2', '3', '4',
30. from_line='start', to_line='end')
31. DexExpectWatchValue('first', '0', '1', '2', '3', '5',
32. from_line='start', to_line='end')
33. DexExpectWatchValue('second', '1', '2', '3', '5',
34 from_line='start', to_line='end')
35. DexExpectWatchValue('total', '0', '1', '2', '4', '7',
36. from_line='start', to_line='end')
37. DexExpectWatchValue('next', '1', '2', '3', '5', '8',
38. from_line='start', to_line='end')
39. DexExpectWatchValue('total', '7', on_line=25)
40. DexExpectStepKind('FUNC_EXTERNAL', 0)
41. */
[DexLabel][1] is used to give a name to a line number.
The [DexExpectWatchValue][2] command states that an expression, e.g. `i`, should
have particular values, `'0', '1', '2', '3','4'`, sequentially over the program
lifetime on particular lines. You can refer to a named line or simply the line
number (See line 39).
At the end of the test is the following line:
DexExpectStepKind('FUNC_EXTERNAL', 0)
This [DexExpectStepKind][3] command indicates that we do not expect the debugger
to step into a file outside of the test directory.
[1]: Commands.md#DexLabel
[2]: Commands.md#DexExpectWatchValue
[3]: Commands.md#DexExpectStepKind
## Detailed DExTer reports
Running the command below launches the tests/nostdlib/fibonacci test case in DExTer, using LLDB as the debugger and producing a detailed report:
$ dexter.py test --vs-solution clang-cl_vs2015 --debugger vs2017 --cflags="/Ox /Zi" --ldflags="/Zi" -v -- tests/nostdlib/fibonacci
The detailed report is enabled by `-v` and shows a breakdown of the information from each debugger step. For example:
fibonacci = (0.2832)
## BEGIN ##
[1, "main", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 23, 1, "BREAKPOINT", "FUNC", {}]
[2, "main", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 24, 1, "BREAKPOINT", "VERTICAL_FORWARD", {}]
[3, "main", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 25, 1, "BREAKPOINT", "VERTICAL_FORWARD", {}]
. [4, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 13, 1, "BREAKPOINT", "FUNC", {}]
. [5, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 16, 1, "BREAKPOINT", "VERTICAL_FORWARD", {"i": "Variable is optimized away and not available.", "next": "Variable is optimized away and not available.", "second": "Variable is optimized away and not available.", "total": "0", "first": "Variable is optimized away and not available."}]
. [6, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 13, 1, "BREAKPOINT", "VERTICAL_BACKWARD", {}]
. [7, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 15, 1, "BREAKPOINT", "VERTICAL_FORWARD", {"i": "Variable is optimized away and not available.", "second": "Variable is optimized away and not available.", "total": "0", "first": "Variable is optimized away and not available."}]
. [8, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 16, 1, "BREAKPOINT", "VERTICAL_FORWARD", {"i": "Variable is optimized away and not available.", "next": "Variable is optimized away and not available.", "second": "Variable is optimized away and not available.", "total": "0", "first": "Variable is optimized away and not available."}]
. [9, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 15, 1, "BREAKPOINT", "VERTICAL_BACKWARD", {"i": "Variable is optimized away and not available.", "second": "1", "total": "0", "first": "0"}]
. [10, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 13, 1, "BREAKPOINT", "VERTICAL_BACKWARD", {}]
. [11, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 16, 1, "BREAKPOINT", "VERTICAL_FORWARD", {"i": "Variable is optimized away and not available.", "next": "Variable is optimized away and not available.", "second": "Variable is optimized away and not available.", "total": "0", "first": "Variable is optimized away and not available."}]
. [12, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 15, 1, "BREAKPOINT", "VERTICAL_BACKWARD", {"i": "Variable is optimized away and not available.", "second": "1", "total": "0", "first": "1"}]
. [13, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 13, 1, "BREAKPOINT", "VERTICAL_BACKWARD", {}]
. [14, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 16, 1, "BREAKPOINT", "VERTICAL_FORWARD", {"i": "Variable is optimized away and not available.", "next": "Variable is optimized away and not available.", "second": "Variable is optimized away and not available.", "total": "0", "first": "Variable is optimized away and not available."}]
. [15, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 15, 1, "BREAKPOINT", "VERTICAL_BACKWARD", {"i": "Variable is optimized away and not available.", "second": "2", "total": "0", "first": "1"}]
. [16, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 13, 1, "BREAKPOINT", "VERTICAL_BACKWARD", {}]
. [17, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 16, 1, "BREAKPOINT", "VERTICAL_FORWARD", {"i": "Variable is optimized away and not available.", "next": "Variable is optimized away and not available.", "second": "Variable is optimized away and not available.", "total": "0", "first": "Variable is optimized away and not available."}]
. [18, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 15, 1, "BREAKPOINT", "VERTICAL_BACKWARD", {"i": "Variable is optimized away and not available.", "second": "3", "total": "0", "first": "2"}]
. [19, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 13, 1, "BREAKPOINT", "VERTICAL_BACKWARD", {}]
. [20, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 16, 1, "BREAKPOINT", "VERTICAL_FORWARD", {"i": "Variable is optimized away and not available.", "next": "Variable is optimized away and not available.", "second": "Variable is optimized away and not available.", "total": "0", "first": "Variable is optimized away and not available."}]
. [21, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 15, 1, "BREAKPOINT", "VERTICAL_BACKWARD", {"i": "Variable is optimized away and not available.", "second": "5", "total": "0", "first": "3"}]
. [22, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 13, 1, "BREAKPOINT", "VERTICAL_BACKWARD", {}]
. [23, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 16, 1, "BREAKPOINT", "VERTICAL_FORWARD", {"i": "Variable is optimized away and not available.", "next": "Variable is optimized away and not available.", "second": "Variable is optimized away and not available.", "total": "0", "first": "Variable is optimized away and not available."}]
. [24, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 20, 1, "BREAKPOINT", "VERTICAL_FORWARD", {}]
[25, "main", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 26, 1, "BREAKPOINT", "FUNC", {"total": "7"}]
## END (25 steps) ##
step kind differences [0/1]
FUNC_EXTERNAL:
0
test.cpp:15-18 [first] [9/21]
expected encountered values:
0
1
2
3
missing values:
5 [-6]
result optimized away:
step 5 (Variable is optimized away and not available.) [-3]
step 7 (Variable is optimized away and not available.)
step 8 (Variable is optimized away and not available.)
step 11 (Variable is optimized away and not available.)
step 14 (Variable is optimized away and not available.)
step 17 (Variable is optimized away and not available.)
step 20 (Variable is optimized away and not available.)
step 23 (Variable is optimized away and not available.)
test.cpp:15-18 [i] [15/21]
result optimized away:
step 5 (Variable is optimized away and not available.) [-3]
step 7 (Variable is optimized away and not available.) [-3]
step 8 (Variable is optimized away and not available.) [-3]
step 9 (Variable is optimized away and not available.) [-3]
step 11 (Variable is optimized away and not available.) [-3]
step 12 (Variable is optimized away and not available.)
step 14 (Variable is optimized away and not available.)
step 15 (Variable is optimized away and not available.)
step 17 (Variable is optimized away and not available.)
step 18 (Variable is optimized away and not available.)
step 20 (Variable is optimized away and not available.)
step 21 (Variable is optimized away and not available.)
step 23 (Variable is optimized away and not available.)
test.cpp:15-18 [second] [21/21]
expected encountered values:
1
2
3
5
result optimized away:
step 5 (Variable is optimized away and not available.) [-3]
step 7 (Variable is optimized away and not available.) [-3]
step 8 (Variable is optimized away and not available.) [-3]
step 11 (Variable is optimized away and not available.) [-3]
step 14 (Variable is optimized away and not available.) [-3]
step 17 (Variable is optimized away and not available.) [-3]
step 20 (Variable is optimized away and not available.) [-3]
step 23 (Variable is optimized away and not available.)
test.cpp:15-18 [total] [21/21]
expected encountered values:
0
missing values:
1 [-6]
2 [-6]
4 [-6]
7 [-3]
test.cpp:16-18 [next] [15/21]
result optimized away:
step 5 (Variable is optimized away and not available.) [-3]
step 8 (Variable is optimized away and not available.) [-3]
step 11 (Variable is optimized away and not available.) [-3]
step 14 (Variable is optimized away and not available.) [-3]
step 17 (Variable is optimized away and not available.) [-3]
step 20 (Variable is optimized away and not available.)
step 23 (Variable is optimized away and not available.)
test.cpp:26 [total] [0/7]
expected encountered values:
7
The first line
fibonacci = (0.2832)
shows a score of 0.2832 suggesting that unexpected behavior has been seen. This score is on scale of 0.0000 to 1.000, with 0.000 being the worst score possible and 1.000 being the best score possible. The verbose output shows the reason for any scoring. For example:
test.cpp:15-18 [first] [9/21]
expected encountered values:
0
1
2
3
missing values:
5 [-6]
result optimized away:
step 5 (Variable is optimized away and not available.) [-3]
step 7 (Variable is optimized away and not available.)
step 8 (Variable is optimized away and not available.)
step 11 (Variable is optimized away and not available.)
step 14 (Variable is optimized away and not available.)
step 17 (Variable is optimized away and not available.)
step 20 (Variable is optimized away and not available.)
step 23 (Variable is optimized away and not available.)
shows that for `first` the expected values 0, 1, 2 and 3 were seen, 5 was not. On some steps the variable was reported as being optimized away.
## Writing new test cases
Each test can be either embedded within the source file using comments or included as a separate file with the .dex extension. Dexter does not include support for building test cases, although if a Visual Studio Solution (.sln) is used as the test file, VS will build the program as part of launching a debugger session if it has not already been built.
|
