1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
|
<html>
<head>
<title>Eclipse Testing Instructions</title>
</head>
<body>
<h1>The Eclipse Test Framework</h1><p>Last Modified: Oct. 27, 2005</p><p><font size="+1"><br>
Introduction</font></p><p>The testing framework is comprised of the org.eclipse.test plugin and the org.eclipse.ant.optional.junit
fragment. </p><p>These two projects are available from the dev.eclipse.org repository and are
included in the eclipse-Automated-Tests.<buildid>.zip from the eclipse.org
downloads page.</p><p><font size="+1">Building and Installation</font></p><p>Since the org.eclipse.test plugin is stored in the repository in source-code
form, it needs to be compiled before it can be used. The org.eclipse.ant.optional.junit
fragment does not contain any source and can be used as is from the repository.</p><ol>
<li>Turn of automatic builds. Window->Preferences->Workbench and uncheck "Perform
build automatically on resource modification"</li><li>Load org.eclipse.test into your workspace.</li><li>Right-click on the org.eclipse.test project in either the Navigator or Packages
view. Select 'Rebuild Project' from the context menu. This will compile the
entire org.eclipse.test plugin.</li><li>Finally, copy the org.eclipse.test plugin into your target Eclipse.</li><li>The org.eclipse.ant.optional.junit fragment only needs to be present in
the environment of the Eclipse that is overseeing the test process. If you
are running the test script from within the Workbench, this means that the
fragments need to be present withinn your development Eclipse. If you are
running the tests from the command line, then the fragments will need to be
present in your target Eclipse.</li></ol><p><font size="+1">Setup</font></p><p>Follow the steps given above to build and install all of the neccessary plugins
and fragments. Please note that the current version of the test framework is
not compatible with the PDE notion of self-hosting. If you want to run the tests,
you will need to setup a full target Eclipse so that the testing framework can
detect everything that is needed.</p><p>If you are writing tests for one or more Eclipse plugins, then you should
create a separate plugin for tests. The test plugin will contain all of the
tests that will be run on your development plugins, as well as defining how
those tests get run.</p><p>If you are not writing tests for an Eclipse plugin, then you should look into
using JUnit on its own to test your work. JUnit is designed for testing Java
code, and by default has no knowledge of Eclipse. There are separate mechanisms
for using JUnit on Java code in Eclipse. See the documentation provided here:</p><p> <a href="http://dev.eclipse.org/viewcvs/index.cgi/%7Echeckout%7E/jdt-ui-home/plugins/org.eclipse.jdt.junit/index.html">JUnit
Support in Eclipse</a><br>
<a href="http://dev.eclipse.org:8080/help/content/help:/org.eclipse.jdt.doc.user/tasks/tasks-207.htm?resultof=%6a%75%6e%69%74%20&toc=%2Forg.eclipse.jdt.doc.user%2Ftoc.xml">Using
JUnit</a></p><p>Once a test plugin has been created, you must then create an Ant script that
will run all of the tests. Create a file called 'test.xml', which is a sibling
of 'plugin.xml'. This file defines how each test is going to be run. The 'test.xml'
file is like a DOS batch file that scripts the entire testing process. Typically,
the 'test.xml' file will contain targets for setting up the test run, executing
the tests, cleaning up afterwards, and running the entire process.</p><p> </p><p><font size="+1">Converting existing Test Suites</font></p><p>If you are converting an existing set of tests to use the new framework, the
actual tests that have been written should not need much change.</p><p>If you have tests in multiple plugins, it is helpful to designate one plugin
as the main plugin for your component. This can be one of your existing
plugins, or it can be a new plugin, created specifically for this purpose. Your
main plugin is the only plugin that needs to contain a 'test.xml' script. This
script can invoke the tests in each of your other plugins, so there is no need
for multiple scripts. Your main plugin does not need to depend on your other
plugins at all.</p><p>Make sure that the tests are defined in a plugin. This is probably the most
common cause of confusion in the entire test process. Your tests need to be
in a plugin so that Eclipse can find them when it tries to load them.</p><p> </p><p><font size="+1">Creating new Test Suites</font></p><p>Creating new JUnit tests for an Eclipse plugin should be no more difficult
than writing standard JUnit tests. Since the framework allows tests to be run
inside of a working Eclipse, any tests that you write have available to them
any of the methods supplied by the Eclipse platform, provided that you add the
appropriate dependencies to your tests' 'plugin.xml'.</p><p> </p><p><font size="+1">Performance Issues</font></p><p>You should keep in mind the number of times that Eclipse needs to be
started. Launching Eclipse has a substantial cost in terms of runtime. To
minimize the number of times the platform is started, you should consider
writing a TestSuite called AllTests for each of your test plugins. AllTests
should invoke each of the tests that you want to run for a particular plugin.
The 'test.xml' file can then run the AllTests class, which will run all of your
tests, but the platform will only ever be started once for each of your test
plugins.</p><p>Note: Sometimes tests involve shutting down, restarting, and testing the state
of metadata that was written to disk. These session tests will require Eclipse
to be launched several times in sequence, which will increase the runtime of
your tests. This cannot be avoided.</p><p> </p><p><font size="+1">Running the Test Suite from the UI</font></p><p>Right click on the test.xml file and select 'Run Ant...' from the pull-down
menu. The Running Ant dialog box pops up. From the list of targets, select the
one that runs all of your tests. If you are using the example file provided
below, this target is called 'Run', and will be selected by default. Hit the
'Finish' button to start the test process.</p><p><font size="+1">Running the Test Suite from the command line</font></p><p>When the test suites are invoked automatically, they are run from command
line. From the ${eclipse-home} directory, the command to use is:</p><code>java -jar
startup.jar -application
org.eclipse.ant.core.antRunner -buildfile ${test-plugin-path}\test.xml
-Declipse-home=${eclipse-home}</code>
<p>Individual tests can also be invoked directly. From the ${eclipse-home}
directory, use the command:</p><code>java -jar
startup.jar -application ${launcher}
-dev bin -testlistener org.eclipse.test.XMLTestListener -testpluginname ${plugin-name}
-classname ${test-classname}</code>
<p>where ${launcher} is one of: <i>org.eclipse.test.uitestapplication</i> or <i>org.eclipse.test.coretestapplication</i>
depending on whether or not you want to run your tests within an active Workbench.</p><p><font size="+1">Output</font></p><p>By default, output from each test run is logged as XML. For each test that
is run, the output is logged into the file called ${classname}.xml. The individual
XML files are located in the ${eclipse-home} directory. When the test run is
finished, you can call the <b>"collect"</b> target in the library
file, which will collect the XML into a single file. See below for an example
of how to use this target correctly.</p><p>The JUnit integration in Eclipse supports various advanced features, such as
<b>Regression Testing</b>, <b>Email Notification</b>, and <b>HTML Output</b>.
These features are currently under development, and do not yet have a
standardized way to add them to a test run.</p><h3>Other issues</h3><h4>Headless Testing vs. UI testing</h4><p>Many plugin tests will not need the Workbench active in order to run. Indeed,
only the minimum number of plugins needed to run the plugin being tested need
to be present when testing in a target Eclipse. There are two different Ant
targets provided for running Eclipse plugin tests. One target starts the entire
Workbench. The other starts Eclipse with the minimum number of plugins needed.
It is up to you to decide which target is most appropriate. For examples, look
at the <b>"ui-test"</b> and <b>"core-test"</b> targets below.</p><h4>Disposing the Display</h4><p>Some low-level tests for the Eclipse platform take actions that are not
normally possible inside of Eclipse. An example of this behaviour would be
disposing the display. While this action can be performed while running the UI,
it will also kill the UI for the copy of Eclipse that is running, and cause
errors when the Workbench tries to shutdown. If you need to test disposing the
display, or other similar actions, your tests should be running without a UI. </p><h4>Tests that are not plugins</h4><p>It is very easy to forget to define your tests inside of a plugin. If your
tests will not load properly, make sure that a 'plugin.xml' file exists in your
test project, and also that the plugin is being loaded by the platform. Make
sure that all of the dependencies are satisfied for your test plugin.</p><p><font size="+1">Interface</font></p><p>The org.eclipse.test plugin defines many useful Ant tasks/targets to aid developers
in writing their test.xml scripts. Currently, there is only Ant targets defined,
which can be called using Ant's built-in <ant> task. To use these targets,
add the following line to the top of your script, and reference the ${library-file}
property when calling <ant>:</p><code><property name="library-file"
value="${eclipse-home}/fragments/org.eclipse.test/library.xml"/> </code>
<p>The targets that are defined are: </p><ul>
<li><b>ui-test</b> - This target runs a JUnit test suite inside of an Eclipse
Workbench. This target is mainly for testing plugins that use the Eclipse
UI and JFace. The output of the test pass is automatically logged in an XML
file called ${classname}.xml. It takes four arguments: </li><ul>
<li><i>data-dir</i> - The
data directory of the Eclipse that gets run</li><li><i>plugin-name</i> -
The name of the plugin that the test suite is defined in</li><li><i>classname</i> -
The name of the class that the test suite is defined in</li><li><i>vmargs</i> - An
optional argument string to pass to the VM running the tests</li></ul><p>For example, the following code will run the test
org.eclipse.foo.bar.MyTest in the plugin org.eclipse.foo in a new Eclipse
workbench. It passes the string "-Dbaz=true" to the VM. The Eclipse
stores its metadata in the directory "data-folder".
<code><pre>
<ant target="ui-test" antfile="${library-file}" dir="${eclipse-home}">
<property name="data-dir" value="data-folder"/>
<property name="plugin-name" value="org.eclipse.foo"/>
<property name="classname" value="org.eclipse.foo.bar.MyTest"/>
<property name="vmargs" value="-Dbaz=true"/>
</ant>
</pre></code>
<li><b>core-test</b> - This target runs a JUnit test suite inside of an IPlatformRunnable.
This target is for testing plugins that use the Eclipse platform, but do not
require a UI to be running. The output of the test pass is automatically logged
in an XML file called ${classname}.xml. It takes four arguments: </li><ul>
<li><i>data-dir</i> - The
data directory of the Eclipse that gets run</li><li><i>plugin-name</i> -
The name of the plugin that the test suite is defined in</li><li><i>classname</i> -
The name of the class that the test suite is defined in</li><li><i>vmargs</i> - An
optional argument string to pass to the VM running the tests</li></ul><p>For example, the following code will run the test
org.eclipse.foo.bar.MyTest in the plugin org.eclipse.foo in a headless Eclipse.
It passes the string "-Dbaz=true" to the VM. The Eclipse stores its
metadata in the directory "data-folder".
<code><pre>
<ant target="core-test" antfile="${library-file}" dir="${eclipse-home}">
<property name="data-dir" value="data-folder"/>
<property name="plugin-name" value="org.eclipse.foo"/>
<property name="classname" value="org.eclipse.foo.bar.MyTest"/>
<property name="vmargs" value="-Dbaz=true"/>
</ant>
</pre></code>
<li><b>collect</b> - This
target collects the XML files that are produced over the course of the
test script. It takes two arguments: </li><ul>
<li><i>includes</i> - A
pattern matching all XML files to be included in the test report. This
argument is typically "org*.xml"</li><li><i>output-file</i> -
The filename where the output of the test gets stored. For the automated
build process, this file should be ${pluginname}.xml, and be located in
the ECLIPSE_HOME directory.</li></ul><p>For example, the following code collects all of the files
matching the pattern "org*.xml" in the directory ${eclipse-home},
into the file named "logfile.xml".
<code><pre>
<ant target="collect" antfile="${library-file}" dir="${eclipse-home}">
<property name="includes" value="org*.xml"/>
<property name="output-file" value="logfile.xml"/>
</ant>
</pre></code>
<li><b>test-regressions</b> - This target takes two compound XML files (after
a collect - even if collect is only given 1 file), and produces a text file
that contains a list of all test status changes that occur between the two
files. It will report new tests added, old tests that weren't run, and tests
whose status changed (pass to fail, or fail to pass). This target takes three
arguments: </li><ul>
<li><i>oldfile</i> - the
baseline file to compare against</li><li><i>newfile</i> - the
file containing the results of the most recent test run</li><li><i>outfile</i> - the
file in which to place the results of the regression test</li></ul><p>For example, the following code compares the test runs that
were logged into the files "old.xml" and "new.xml", and
creates a file called "output.txt" that holds the differences between
the two runs.
<code><pre>
<ant target="test-regressions" antfile="${library-file}" dir="${eclipse-home}">
<property name="oldfile" value="old.xml"/>
<property name="newfile" value="new.xml"/>
<property name="outfile" value="output.txt"/>
</ant>
</pre></code>
<li><b>notify</b> - This target
sends email notification of the status of the latest test run. Typically,
this sends the results of the latest regression test to a developer so
that new test failures can be detected quickly. This target takes two
parameters: </li><ul>
<li><i>infile</i> - the
name of the file containing the message to send</li><li><i>address</i> - the
email address to send the file to</li></ul><p>For example, the following code will mail the contents of
the file "output.txt" to the address "foo@bar.com".
<code><pre>
<ant target="notify" antfile="${library-file}" dir="${eclipse-home}">
<property name="infile" value="output.txt"/>
<property name="address" value="foo@bar.com"/>
</ant>
</pre></code>
</ul><h2>Examples:</h2><p>Included is the 'test.xml' file from the org.eclipse.ui.tests plugin. This
file controls all of the automated testing that is done for the org.eclipse.ui
plugin. It can be run from inside of Eclipse or from the command line. It is
intended to serve as a template file for testing any other plugin.</p><p>Notice that the structure of the file roughly mirrors that of a JUnit test.
Targets are defined for setting up the tests, defining what needs to be done,
cleaning up after the tests, and running everything in the right order.</p><font color="red">This example is out of date.
<code><pre>
<?xml version="1.0"?>
<project name="testsuite" default="run" basedir=".">
<!-- The property ${eclipse-home} should be passed into this script -->
<!-- Set a meaningful default value for when it is not. -->
<property name="eclipse-home" value="${basedir}/../.."/>
<!-- sets the properties eclipse-home, and library-file -->
<property name="plugin-name" value="org.eclipse.ui.tests"/>
<property name="library-file"
value="${eclipse-home}/fragments/org.eclipse.test/library.xml"/>
<!-- This target holds all initialization code that needs to be done for -->
<!-- all tests that are to be run. Initialization for individual tests -->
<!-- should be done within the body of the suite target. -->
<target name="init">
<tstamp/>
<delete>
<fileset dir="${eclipse-home}" includes="org*.xml"/>
</delete>
</target>
<!-- This target defines the tests that need to be run. -->
<target name="suite">
<property name="session-folder"
value="${eclipse-home}/ui_session_sniff_folder"/>
<delete dir="${session-folder}" quiet="true"/>
<ant target="ui-test" antfile="${library-file}" dir="${eclipse-home}">
<property name="data-dir" value="${session-folder}"/>
<property name="plugin-name" value="${plugin-name}"/>
<property name="classname"
value="org.eclipse.ui.tests.api.SessionCreateTest"/>
</ant>
<ant target="ui-test" antfile="${library-file}" dir="${eclipse-home}">
<property name="data-dir" value="${session-folder}"/>
<property name="plugin-name" value="${plugin-name}"/>
<property name="classname"
value="org.eclipse.ui.tests.api.SessionRestoreTest"/>
</ant>
<property name="sniff-folder"
value="${eclipse-home}/ui_sniff_folder"/>
<delete dir="${sniff-folder}" quiet="true"/>
<ant target="ui-test" antfile="${library-file}" dir="${eclipse-home}">
<property name="data-dir" value="${sniff-folder}"/>
<property name="plugin-name" value="${plugin-name}"/>
<property name="classname"
value="org.eclipse.ui.tests.UiTestSuite"/>
</ant>
</target>
<!-- This target holds code to cleanup the testing environment after -->
<!-- after all of the tests have been run. You can use this target to -->
<!-- delete temporary files that have been created. -->
<target name="cleanup">
</target>
<!-- This target runs the test suite. Any actions that need to happen -->
<!-- after all the tests have been run should go here. -->
<target name="run" depends="init,suite,cleanup">
<ant target="collect" antfile="${library-file}" dir="${eclipse-home}">
<property name="includes" value="org*.xml"/>
<property name="output-file" value="${plugin-name}.xml"/>
</ant>
</target>
</project>
</pre></code>
</font>
<h2>Known Issues:</h2><h4>Problem 1: Issues with Ant 1.3</h4><p>Ant 1.3 has some issues when used with the <style> tag and absolute
paths. Also, any tests that use the <code>System.exit()</code> call will not
log their output properly when using Ant 1.3.</p><h4>Problem 2: ECLIPSE_HOME</h4><p>The test suites need to know where the root of the eclipse install is on the
file system (the ECLIPSE_HOME variable). However, this variable is only defined
in JDT. The ${eclipse-home} property can be set to a reasonable default inside
of the test.xml script. Then tests can be run from the standard Ant window,
without having to specify -Declipse-home=%ECLIPSE_HOME%. If a value for
${eclipse-home} does get passed in, the default (specified in test.xml) gets
overridden. The parameter is passed in by the build mechanism. For most cases,
the value "${basedir}/../.." is a reasonable default.</p><h4>Problem 3: Ugly reference to library.xml</h4><p>org.eclipse.test should provide Ant tasks, not template scripts.</p><h4>Problem 4: No console output</h4><p>When you run a TestSuite using the standard JUnit, it normally outputs a
series of dots to the console so that you can track the TestSuite's progress.
It is not possible to add this feature to the automated testing process at this
point in time.</p><h4>Problem 5: Ant java task on Linux</h4><p>Ant expects there to be a java executable on the system path. Furthermore, the executable
must be a real file, not a symbolic link. If the test framework is throwing an exception
<code>java.io.IOException: java: not found</code>, ensure that the java executable is on
your system path.</p><h4>Problem 6: PDE</h4><p>The testing framework currently has no knowledge of PDE. In order to run the automated
you must be running a self hosting environment with a full development and target Eclipse.</p>
</body>
</html>
|
