1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
|
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
<html>
<body>
<h2>Unit Testing</h2>
<p>
GSF makes unit testing your specific features easy.
By extending the <code>GsfTestBase</code> unit test class
(which in turn extends <code>NbTestCase</code> which in turn
extends JUnit's <code>TestCase</code>), you inherit a number
of additional unit testing methods.
</p>
<p>
Unfortunately, you have to write a few configuration steps
for your language, once - this is described at the end of
this document in the <a href="#configuration">Configuration</a> section.
</p>
<h3>Unit Test Workflow</h3>
<p>
All GSF unit tests are based on "golden files".
<ul>
<li>You have a source file in your language.</li>
<li>The source file is parsed and then your language feature
is run on the given parse result</li>
<li> The result from your feature is pretty printed</li>
<li> The pretty-printed output is compared to a previously
saved golden file result. If there is a difference, the test fails.
</li>
<li>Crucially, the FIRST time you run the test, there is no golden file,
so one is created instead! This makes it trivial to create new
tests. Just add more test files, manually check that your feature
is producing the correct result for these test files, and then
run them as unit tests. You now have golden files which ensure
that for these testcases you get the same feature output every time.
</li>
</ul>
</p>
<h3>Example: Semantic Highlighting</h3>
<p>
To illustrate how this works, I'll use an example. To create
a semantic highlighting unit test, create a new test method like this
in your SemanticHighlighterTest:
<pre style="background: #ffffcc; color: black; border: solid 1px black; padding: 5px">
public void testSemantic1() throws Exception {
checkSemantic("testfiles/test1.js");
}
</pre>
Here, the <code>checkSemantic</code> method is inherited for you from
<code>GsfTestBase</code>, and handles testing semantic highlighting code
by parsing your testfile (which is specified by a path relative to your
<code>test/unit/data/</code> directory in your plugin), then hands it
to your semantic highlighter, and then prints and diffs the output.
The <code>checkSemantic</code> method will pretty print your source
file by inserting markers for the logical attributes you've added superimposed
on top of the code.
For example, here is <code>test1.js</code>, a simple JavaScript file:
<pre style="background: #ffffcc; color: black; border: solid 1px black; padding: 5px">
Draggable.prototype = {
scroll: function() {
var wind, window2, y = 1, win3 = 50;
var windo
if(this.scroll == window) {
}
}
}
</pre>
When I run the above test the first time, it fails, along with this error
message: "<code>Please re-run the test</code>".
If you now run the test again, it will pass, and will continue to pass
until you the output of your semantic highlighter changes for this input.
If we look in the unit test directory, there is a new file right next
to the testfile with the extra extension <code>.semantic</code>:
<code>test1.js.semantic</code>.
Here's how that file looks:
<pre style="background: #ffffcc; color: black; border: solid 1px black; padding: 5px">
|>CLASS:Draggable<|.prototype = {
|>METHOD:scroll<|: function() {
var |>UNUSED:wind<|, |>UNUSED:window2<|, |>UNUSED:y<| = 1, |>UNUSED:win3<| = 50;
var |>UNUSED:windo<|
if(this.scroll == |>GLOBAL:window<|) {
}
}
}
</pre>
Notice how this is identical to the input file, but with |>MARKERS:<| inserted around
sections of code that indicate what kind of highlighting attribute is associated with
that span. It should be trivial to look at this golden file and convince yourself
that the semantic highlighting worked correctly here.
</p>
<p>
Repeat this process for all the different scenarios you want to test your semantic
highlighter for.
</p>
<p>
Occasionally, you get a unit test failure. Perhaps you've accidentally
broken your code - or perhaps you've actually fixed a bug, which changes the
output of your semantic highlighter in some way, so your golden file diff
fails and the test files. There is a textual diff printed to the output when
you run the test in the IDE, but personally I find it hard to read.
Your tests, and your golden files, should all be checked in with your code,
and therefore version controlled. So what I do in this scenario is just
remove all the golden files (<code>rm *.semantic</code> for example),
and run the tests again. I then use the builtin IDE diff tool to diff
all the golden files and see what the changes are. Sometimes, the changes
are an improvement, and I want the goldenfile updated (which it now has been!)
to reflect the new behavior. And in some cases, the change is a bad one,
and I have a regression on my hands, so I revert the file change,
and start debugging to track down the problem.
Here's an example of diffing golden files, where in this case there is
definitely a problem - the highlighter is no longer identifying regular
expressions in JavaScript (which has to be done at the parse level rather
than at the lexical level in JavaScript):
<br/>
<img src="test-diff.png" />
<br/>
<br/>
</p>
<h3>Caret-based Tests</h3>
<p>
Many GSF features are associated with a caret position. For example,
code completion applies at a particular offset, and ditto for
mark occurrences.
</p>
<p>
The convention in GSF is to tell unit tests about a caret position
by passing in a substring of the testfile, with an extra <code>^</code>
(arrow up) indicating the caret position. For example, in the
above semantic highlighting code fragment, if I wanted to identify
a caret position between the "f" and the "u" of the function keyword
on the second line, I could pass in this string:
<pre style="background: #ffffcc; color: black; border: solid 1px black; padding: 5px">
"scroll: f<b>^</b>unction"
</pre>
Note that this isn't a complete line of the source - it can be just a
portion of a line. The key part is that it tells GSF where the caret
offset should be in the file, without having to tell it something
awkward like an integer offset. Unit tests will assert if you're
passing in a caret offset and it cannot find the corresponding position
in the source file.
</p>
<p>
Here's an example of a unit test which uses this:
<pre style="background: #ffffcc; color: black; border: solid 1px black; padding: 5px">
checkOccurrences("testfiles/semantic4.js", "function myfunc(param1, pa^ram2) {", true);
</pre>
This is the test for the Mark Occurrences feature. We're telling it
which source file we want to test, <b>and</b> where the caret is. <b>Your</b>
occurrences marker will be asked to compute the occurrences corresponding
to the given offset. If we run this test, it will compute all the occurrences
and pretty print them by listing ONLY the lines in the file that contain
an occurrence, and showing that occurrence in context (e.g. the source
code with the region containing the marker). Here's the occurrences output
for another test file:
<pre style="background: #ffffcc; color: black; border: solid 1px black; padding: 5px">
Spry.Effect.Registry.prototype.getRegisteredEffect = function(|>MARK_OCCURRENCES:elem^ent<|, effect)
var eleIdx = this.getIndexOfElement(|>MARK_OCCURRENCES:element<|);
var addedElement = new Spry.Effect.AnimatedElement(|>MARK_OCCURRENCES:element<|);
</pre>
</p>
<h3>Code Completion Tests</h3>
<p>
Here's an example of a code completion test. Again, we point to some test
file, as well as a caret position - in this case, it's at the beginning
of the symbol "escape", so there's no prefix:
<pre style="background: #ffffcc; color: black; border: solid 1px black; padding: 5px">
public void testExpression1() throws Exception {
checkCompletion("testfiles/completion/lib/expressions.js", "^escape", false);
}
</pre>
This will generate (and then diff as usual) the following code completion output:
<pre style="background: #ffffcc; color: black; border: solid 1px black; padding: 5px">
Results for escape with queryType=COMPLETION and nameKind=PREFIX
METHOD exec(str) : void RegExp
METHOD test(str) : Boolean RegExp
METHOD toSource() : void RegExp
METHOD toString() : String RegExp
METHOD watch(prop, handler) : void RegExp
PROPERTY constructor RegExp
PROPERTY global : Boolean RegExp
PROPERTY ignoreCase : Boolean RegExp
PROPERTY lastIndex : Number RegExp
PROPERTY multiline RegExp
PROPERTY prototype RegExp
PROPERTY source : String RegExp
------------------------------------
METHOD eval(string) : void Object
METHOD hasOwnProperty(prop) : Boolean Object
METHOD isPrototypeOf(object) : Boolea Object
METHOD propertyIsEnumerable(prop) : B Object
METHOD toLocaleString() : String Object
</pre>
The divider line here is showing the difference between the smart
items and the non-smart items (just like the dividing line you see
in the IDE).
</p>
<p>
As you can see, writing code completion tests is pretty trivial.
</p>
<h3>Other Features</h3>
<p>
There are many other feature tests like this, and all are very
similar to the above: a single line call with test data which
gets pretty printed and diffed. For example,
the <b>insertChar</b> test (and removeChar and insertBreak etc) let
you test what happens when you type keys in the editor. You provide
the document contents and caret position before, the character to be
typed, and the contents and caret position after the insert, like this:
<pre>
insertChar("x = ^", '\'', "x = '^'");
</pre>
This test ensures that bracket matching works correctly for the apostrophy
for example. You should obviously write lots of other tests for this to
make sure that bracket matching doesn't happen inside strings and
comments, when there is already a bracket there, etc:
<pre>
insertChar("x = '^'", '\'', "x = ''^");
insertChar("// Hello^", '\'', "// Hello'^");
<i>etc.</i>
</pre>
There are unit tests for all kinds of features. Formatting. Keystroke
handling. Quickfixes. Instant Rename. Indexing. Code Folding. Navigation.
</p>
<h3>AST Offset Tests</h3>
<p>
Unlike the feature tests, the AST offset test support involves a bit
more work. That's because GSF does not have any clue about
<b>your</b> AST. GSF doesn't even know about it; you typically stash
it in your own subclass of the <code>ParserResult</code> class.
However, GSF makes it easier to make sure that your AST offsets
are correct, which is typically vital for correct IDE operation.
To test your own ASTs, you need to override a couple of protected
methods in the GsfTestBase:
<pre style="background: #ffffcc; color: black; border: solid 1px black; padding: 5px">
protected String <b>describeNode</b>(CompilationInfo info, Object node, boolean includePath);
protected void <b>initializeNodes</b>(CompilationInfo info, List validNodes,
Map<Object,OffsetRange> positions, List invalidNodes);
</pre>
</p>
<p>
The first method just lets you create a description of the node
(which will show up in the AST golden file). Typically there will be some
AST specific way to identify node types - for example, STRING, EXPRESSION,
FUNCTION, etc.
</p>
<p>
In the second method, you need to go and pull ALL nodes out of the
AST for your parser result, and put all of them into a map, where
each node maps to a given OffsetRange (the start/end offsets) of your
nodes that you are trying to test. You can ignore the other offsets
for now. Once you do that, you can write tests like this:
<pre style="background: #ffffcc; color: black; border: solid 1px black; padding: 5px">
public void testOffsets1() throws Exception {
checkOffsets("testfiles/test1.js");
}
</pre>
This is the <code>test1.js</code> file we saw for semantic highlighting above.
Here the golden file will be named <code>test1.js.offsets</code> (as opposed
to the <code>.semantic</code> golden file we get from the semantic test).
The golden file will use XML-like syntax to show the node descriptions around
the begin/end offsets for your nested AST hierarchy that corresponds to
your source:
<pre style="background: #ffffcc; color: black; border: solid 1px black; padding: 5px">
<SCRIPT><EXPR_RESULT><SETPROP><NAME><b>Draggable</b></NAME><b>.</b><STRING><b>prototype</b></STRING><b> = </b><OBJECTLIT><b>{</b>
<OBJLITNAME><b>scroll</b></OBJLITNAME><b>: </b><FUNCTION><b>function() </b><BLOCK><b>{</b>
<VAR>var <NAME>wind</NAME>, <NAME>window2</NAME>, <NAME>y = <NUMBER>1</NUMBER></NAME>, <NAME>win3 = <NUMBER>50</NUMBER></NAME></VAR>;
<VAR>var <NAME>windo</NAME></VAR>
<BLOCK>if(<IFNE><EQ><GETPROP><THIS>this</THIS>.<STRING>scroll</STRING></GETPROP> == <NAME>window</NAME></EQ></IFNE>) <BLOCK>{
}<TARGET></TARGET></BLOCK></BLOCK> <RETURN></RETURN></BLOCK>}</FUNCTION>
}</OBJECTLIT></SETPROP></EXPR_RESULT></SCRIPT>
</pre>
I've bolded the beginning of the embedded code here so you can see the code
through the AST node markup.
</p>
<p>
I have made a lot of diffs to the Rhino parser for the JavaScript support to add in
AST offset support (Rhino doesn't record source offsets, we've added that for NetBeans).
Having tests like these makes it a lot easier to feel confident when I make any changes to the
parser. As shown earlier, if there are diffs, I delete the offset files, run the tests, and
then diff the golden files through the IDE.
</p>
<a name="Configuration"/>
<h3>Configuration</h3>
<p>
Typically, you'll want to create your own <code>TestBase</code> class,
which extends <code>GsfTestBase</code>, and then make all your unit tests
extend your own <code>TestBase</code>. In your superclass, you need to
do a few configuration steps.
</p>
<p>
<blockquote style="background-color: #ffdddd; color: black; padding: 20px; border: solid 1px black">
<b>These steps should not be necessary</b>. Unfortunately, I haven't
been able to get the GSF services working without them.
What I'm observing is that from a unit test, services that are
registered via the layer file systems don't work. I've been playing
around with module dependencies, Mock services etc. but I haven't
gotten it to work. So, for example, for some reason from a unit test,
the GSF Data Loader isn't used, and attempting to open documents give
me simple PlainDocuments instead of BaseDocuments.
I would <b>really</b> appreciate some help from people who know more
about unit testing and netbeans service loading to help me track
this down so that these manual steps aren't necessary.
</blockquote>
</p>
<p>
Here's the code you have to add to your own test class such that all
your unit tests inherit it.
<pre style="background: #ffffcc; color: black; border: solid 1px black; padding: 5px">
TODO - document this. For now, anyone curious, go look at
<code>javascript.editing/test/unit/src/org/netbeans/modules/javascript/editing/JsTestBase.java</code>
</pre>
</p>
<br/>
<span style="color: #cccccc">Tor Norbye <tor@netbeans.org></span>
</body>
</html>
|
