File: JUnitPerf.html

package info (click to toggle)
junitperf 1.8-1
links: PTS
area: contrib
in suites: sarge
size: 240 kB
ctags: 288
sloc: java: 1,278; xml: 121; makefile: 48
file content (1621 lines) | stat: -rw-r--r-- 44,649 bytes
<HTML>

<HEAD>
	<TITLE>JUnitPerf</TITLE>
	<LINK rel="stylesheet" type="text/css" href="stylesheet.css" title="Style">
	<META NAME="keywords" content="Clarkware, JUnitPerf, JUnit, Java performance, Java scalability">
	<META NAME="description" content="A collection of JUnit test decorators used to measure performance and scalability">
	<META NAME="Author" content="Mike Clark">
</HEAD>

<BODY>

<!--

	Title

-->

<TABLE width="100%" cellspacing="0" cellpadding="0" >
	<TR>
		<TD class="PageTitle">
			JUnitPerf	
		</TD>
	</TR>
</TABLE>

<!--

	Summary

-->
<P>
<DIV align="center">
<TABLE class="Primary" width="90%" cellspacing="0" cellpadding="1" border="0">
	<TR>
		<TD class="Primary">
			<FONT class="Reverse">
				<B>&nbsp;Summary</B>
			</FONT>
		</TD>
	</TR>
	<TR>
		<TD class="Primary">
			<TABLE class="Secondary" width="100%" cellspacing="0" cellpadding="3" border="0">
				<TR>
					<TD class="Secondary">
JUnitPerf is a collection of JUnit test decorators used to measure the
performance and scalability of functionality contained within existing JUnit 
tests.
					</TD>
				</TR>
			</TABLE>
		</TD>
	</TR>
	<TR>
		<TD>
			<BR>
			<a href="mailto:mike@clarkware.com">Mike Clark</a><BR>
			<a href="http://www.clarkware.com" target="_blank">Clarkware Consulting, Inc.</a><BR>
		</TD>
	</TR>
</TABLE>
</DIV>
</P>
<BR>


<!--

	Table Of Contents

-->
<P>
<A name="intro"></A> 
<TABLE class="Primary" width="100%" border="0" cellspacing="0" cellpadding="0">
	<TR><TD class="Border">&nbsp;</TD></TR>
	<TR> 
		<TD class="SubHeader">&nbsp;
			Table Of Contents
		</TD>
	</TR>
</TABLE>
</P>
<UL>
	<LI><A href="#overview">Overview</A>
	<LI><A href="#uses">Why Use JUnitPerf?</A>
	<LI><A href="#download">Downloading JUnitPerf</A>
	<LI><A href="#installation">Installing JUnitPerf</A>
	<LI><A href="#building">Building And Testing JUnitPerf</A> 
	<LI><A href="#howtouse">Using JUnitPerf</A>
	<LI><A href="#effectivetests">Writing Effective JUnitPerf Tests</A>
	<LI><A href="#limitations">Limitations</A>
	<LI><A href="#support">Support</A>
	<LI><A href="#consulting">Consulting Services</A>
	<LI><A href="#license">License</A>
	<LI><A href="#credits">Acknowledgments</A>
	<LI><A href="#resources">Resources</A>
	<LI><A href="#release">Release History</A>
</UL>
</P>
<br>


<!--

	Overview	

-->
<P>
<A name="overview"></A> 
<TABLE class="Primary" width="100%" border="0" cellspacing="0" cellpadding="0">
	<TR><TD class="Border">&nbsp;</TD></TR>
	<TR> 
		<TD class="SubHeader">&nbsp;
			Overview
		</TD>
	</TR>
</TABLE>
</P>
<P>
JUnitPerf is a collection of JUnit test decorators used to measure the
performance and scalability of functionality contained within existing JUnit 
tests.
</P>
<P>
JUnitPerf contains the following JUnit test decorators:
</P>
<UL>
	<LI><b>TimedTest</b>
		<P>
		A <code>TimedTest</code> is a test decorator that runs a test 
		and measures the elapsed time of the test.  
        <P>
        A <code>TimedTest</code> is constructed with a specified
        maximum elapsed time.  By default, a <code>TimedTest</code> 
        will wait for the completion of its decorated test and then 
        fail if the maximum elapsed time was exceeded.  Alternatively, 
        a <code>TimedTest</code> can be constructed to immediately 
        signal a failure when the maximum elapsed time of its decorated 
        test is exceeded.    
		<P>
	</LI>
	<LI><b>LoadTest</b>
		<P>
        A <code>LoadTest</code> is a test decorator that runs a test 
        with a simulated number of concurrent users and iterations.  
		<P>
	</LI>
	<LI><b>ThreadedTest</b>
		<P>
		A test decorator that runs a test in a separate thread.
		<P>
	</LI>
</UL>
</P>
<br>


<!--

	Uses

-->
<P>
<A name="uses"></A> 
<TABLE class="Primary" width="100%" border="0" cellspacing="0" cellpadding="0">
	<TR><TD class="Border">&nbsp;</TD></TR>
	<TR> 
		<TD class="SubHeader">&nbsp;
			Why Use JUnitPerf?	
		</TD>
	</TR>
</TABLE>
</P>
<P>
JUnitPerf tests transparently decorate existing JUnit tests.  This decoration-based 
design allows performance testing to be dynamically added to an existing JUnit test 
without affecting the use of the JUnit test independent of its performance.  By 
decorating existing JUnit tests, it's quick and easy to compose a set of performance 
tests into a performance test suite.  The performance test suite can then be run 
automatically and independent of your other JUnit tests.  In fact, you generally want 
to avoid grouping your JUnitPerf tests with your other JUnit tests so that you can run 
the test suites independently and at different frequencies.  Long-running performance 
tests will slow you down and undoubtedly tempt you to abandon unit testing altogether, 
so try to schedule them to run at times when they won't interfere with your refactoring pace.
</P>
<P>
JUnitPerf tests are intended to be used specifically in situations where you have  
quantitative performance and/or scalability requirements that you'd like to keep in 
check while refactoring code.  For example, you might write a JUnitPerf test to ensure 
that refactoring an algorithm didn't incur undesirable performance overhead in a 
performance-critical code section.  You might also write a JUnitPerf test to ensure 
that refactoring a resource pool didn't adversely affect the scalability of the pool 
under load. 
</P>
<P>
It's important to maintain a pragmatic approach when writing JUnitPerf tests to
maximize the return on your testing investment.  Traditional performance profiling 
tools and techniques should be employed first to identify which areas of code 
exhibit the highest potential for performance and scalability problems.  JUnitPerf 
tests can then be written to automatically test and check that requirements are being 
met now and in the future. 
</P>
<P>
Here's an example usage scenario:  
</P>
<P>
You've built a well-factored chunk of software, complete with the necessary suite of 
JUnit tests to validate the software.  At this point in the process you've gained as 
much knowledge about the design as possible.  
</P>
<P>
You then use a performance profiling tool to isolate where the software is spending 
most of its time.  Based on your knowledge of the design you're better equipped to make 
realistic estimates of the desired performance and scalability.  And, since your
refactorings have formed clear and succinct methods, your profiler is able to point 
you towards smaller sections of code to tune.  
</P>
<P>
You then write a JUnitPerf test with the desired performance and scalability tolerances 
for the code to be tuned.  Without making any changes to the code, the JUnitPerf test 
should fail, proving that the test is written properly.  You then make the tuning changes 
in small steps.  
</P>
<P>
After each step you compile and rerun the JUnitPerf test.  If you've improved performance 
to the expected degree, the test passes.  If you haven't improved performance to the
expected degree, the test fails and you continue the tuning process until the test passes.  
In the future, when the code is again refactored, you re-run the test.  If the test fails, 
the previously defined performance limits have been exceeded, so you back out the change 
and continue refactoring until the test passes.  
</P>
<BR>


<!--

	Download	

-->
<P>
<A name="download"></A> 
<TABLE class="Primary" width="100%" border="0" cellspacing="0" cellpadding="0">
	<TR><TD class="Border">&nbsp;</TD></TR>
	<TR> 
		<TD class="SubHeader">&nbsp;
			Downloading JUnitPerf	
		</TD>
	</TR>
</TABLE>
</P>
<P>
<a href="http://www.clarkware.com/software/junitperf1.8.zip">JUnitPerf 1.8</a> is the latest release.
</P>
<P>
This version requires Java 2 and 
<a href="http://www.xprogramming.com/ftp/TestingFramework/JUnit/junit32.zip" target="_blank">JUnit 3.2</a> (or higher).
</P>
<P>
The distribution contains a JAR file, source code, sample tests, 
API documentation, and this document.
</P>
<BR>


<!--

	Installation	

-->
<P>
<A name="installation"></A> 
<TABLE class="Primary" width="100%" border="0" cellspacing="0" cellpadding="0">
	<TR><TD class="Border">&nbsp;</TD></TR>
	<TR> 
		<TD class="SubHeader">&nbsp;
			Installing JUnitPerf
		</TD>
	</TR>
</TABLE>
</P>
<BR>
<b>Windows</b>
<P>
To install JUnitPerf, follow these steps:
<P>
<OL>
	<LI>Unzip the <code>junitperf.zip</code> distribution file to
		a directory referred to as <code>%JUNITPERF_HOME%</code>.
	</LI>
	<P>
	<LI>Add JUnitPerf to the classpath:
	<P>
		<code>set CLASSPATH=%CLASSPATH%;%JUNITPERF_HOME%\lib\junitperf.jar</code>
	<P>
	</LI>
</OL>
<BR>
<b>Unix (bash)</b>
</P>
<P>
To install JUnitPerf, follow these steps:
</P>
<P>
<OL>
	<LI>Unzip the <code>junitperf.zip</code> distribution file to
		a directory referred to as <code>$JUNITPERF_HOME</code>.
	</LI>
	<P>
	<LI>Change file permissions:
	<P>
		<code>chmod -R a+x $JUNITPERF_HOME</code>
	<P>
	</LI>
	<P>
	<LI>Add JUnitPerf to the classpath:
	<P>
		<code>export CLASSPATH=$CLASSPATH:$JUNITPERF_HOME/lib/junitperf.jar</code>
	<P>
	</LI>
</OL>
</P>
<BR>


<!--

	Building	

-->
<P>
<A name="building"></A> 
<TABLE class="Primary" width="100%" border="0" cellspacing="0" cellpadding="0">
	<TR><TD class="Border">&nbsp;</TD></TR>
	<TR> 
		<TD class="SubHeader">&nbsp;
			Building And Testing JUnitPerf
		</TD>
	</TR>
</TABLE>
</P>
<P>
The JUnitPerf distribution includes the pre-built classes in the
<code>$JUNITPERF_HOME/lib/junitperf.jar</code> file.  
<BR>
<P>
An <a href="http://jakarta.apache.org/ant" target="_blank">Ant</a>
build file is included in <code>$JUNITPERF_HOME/build.xml</code> to 
build the <code>$JUNITPERF_HOME/lib/junitperf.jar</code> file 
from the included source code.  
</P>
<P>
<b>Building</b>
<P>
To build JUnitPerf, use:
</P>
<BLOCKQUOTE>
<PRE>
cd $JUNITPERF_HOME
ant jar
</PRE>
</BLOCKQUOTE>
<P>
<b>Testing</b>
<P>
The JUnitPerf distribution includes 
<a href="http://www.junit.org" target="_blank">JUnit</a>
test cases to validate the integrity of JUnitPerf.  
</P>
<P>
To test JUnitPerf, use:
</P>
<BLOCKQUOTE>
<PRE>
cd $JUNITPERF_HOME
ant test
</PRE>
</BLOCKQUOTE>
<BR>


<!--

	How To Use	

-->
<P>
<A name="howtouse"></A> 
<TABLE class="Primary" width="100%" border="0" cellspacing="0" cellpadding="0">
	<TR><TD class="Border">&nbsp;</TD></TR>
	<TR> 
		<TD class="SubHeader">&nbsp;
			Using JUnitPerf	
		</TD>
	</TR>
</TABLE>
</P>
<P>
The easiest way to describe how to use JUnitPerf is to show examples of each
type of test decorator.
</P>
<P>
The <code>$JUNITPERF_HOME/samples</code> directory contains the set of example
JUnitPerf tests described in this section.
<P>
<BR>
<b>TimedTest</b>
<P>
A <code>TimedTest</code> test decorator is constructed with an existing
JUnit test and a maximum elapsed time in milliseconds.
</P>
<P>
For example, to create a timed test that waits for the completion of 
the <code>ExampleTestCase.testOneSecondResponse()</code> method and 
then fails if the elapsed time exceeded 1 second, use:
</P>
<P>
<BLOCKQUOTE>
<FONT CLASS="Small">
<PRE>
long maxElapsedTime = 1000;
Test testCase = new ExampleTestCase("testOneSecondResponse");
Test timedTest = new TimedTest(testCase, maxElapsedTime);
</PRE>
</FONT>
</BLOCKQUOTE>
<P>
Alternatively, to create a timed test that fails immediately when
the elapsed time of the <code>ExampleTestCase.testOneSecondResponse()</code>
test method exceeds 1 second, use:
</P>
<P>
<BLOCKQUOTE>
<FONT CLASS="Small">
<PRE>
long maxElapsedTime = 1000;
Test testCase = new ExampleTestCase("testOneSecondResponse");
Test timedTest = new TimedTest(testCase, maxElapsedTime, false);
</PRE>
</FONT>
</BLOCKQUOTE>
<P>
The following is an example test that creates a <code>TimedTest</code> 
to test the performance of the functionality being unit tested in the 
<code>ExampleTestCase.testOneSecondResponse()</code> method.  The timed 
test waits for the method under test to complete, and then fails if the 
elapsed time exceeded 1 second.
</P>
<P>
<DIV align="center">
<TABLE class="Primary" width="90%" cellspacing="0" cellpadding="1" border="0">
	<TR>
		<TD class="Primary">
			<FONT class="Reverse">
				<B>&nbsp;Example Timed Test</B>
			</FONT>
		</TD>
	</TR>
	<TR>
		<TD class="Primary">
			<TABLE class="Secondary" width="100%" cellspacing="0" cellpadding="8" border="0">
				<TR>
					<TD class="Secondary">
<FONT CLASS="Small">
<PRE>
import com.clarkware.junitperf.*;

import junit.framework.Test;
import junit.framework.TestSuite;

public class ExampleTimedTest {

    public static Test suite() {
        
        long maxElapsedTime = 1000;
        
        Test testCase = new ExampleTestCase("testOneSecondResponse");
        Test timedTest = new TimedTest(testCase, maxElapsedTime);
        
        TestSuite suite = new TestSuite();        
        suite.addTest(timedTest);
        
        return suite;
    }
    
    public static void main(String args[]) {
        junit.textui.TestRunner.run(suite());
    }
}
</PRE>
</FONT>
					</TD>
				</TR>
			</TABLE>
		</TD>
	</TR>
</TABLE>
</DIV>
</P>
<P>
The granularity of the test decoration design offered by JUnit, and 
used by JUnitPerf, imposes some limitations.  The elapsed time measured 
by a <code>TimedTest</code> decorating a single <code>testXXX()</code> 
method of a <code>TestCase</code> includes the total time of the 
<code>setUp()</code>, <code>testXXX()</code>, and <code>tearDown()</code> 
methods.  The elapsed time measured by a <code>TimedTest</code> decorating
a <code>TestSuite</code> includes the total time of all <code>setUp()</code>, 
<code>testXXX()</code>, and <code>tearDown()</code> methods for all the 
<code>Test</code> instances in the <code>TestSuite</code>.
Therefore, the expected elapsed time measurements should be adjusted 
accordingly to account for the set-up and tear-down costs of the 
decorated test.
</P>
<BR>
<b>LoadTest</b>
<P>
A <code>LoadTest</code> is a test decorator that runs a test with a 
simulated number of concurrent users and iterations.
</P>
<P>
In its simplest form, a <code>LoadTest</code> is constructed 
with a test to decorate and the number of concurrent users.
By default, each user runs one iteration of the test.
</P>
<P>
For example, to create a load test of 10 concurrent users
with each user running the <code>ExampleTestCase.testOneSecondResponse()</code> 
method once and all users starting simultaneously, use:
<BLOCKQUOTE>
<FONT CLASS="Small">
<PRE>
int users = 10;
Test testCase = new ExampleTestCase("testOneSecondResponse");
Test loadTest = new LoadTest(testCase, users);
</PRE>
</FONT>
</BLOCKQUOTE>
</P>
<P>
The load can be ramped by specifying a pluggable <code>Timer</code> 
instance that prescribes the delay between the addition of each 
concurrent user.  A <code>ConstantTimer</code> has a constant delay, 
with a zero value indicating that all users will be started 
simultaneously. A <code>RandomTimer</code> has a random delay with a 
uniformly distributed variation.
</P>
<P>
For example, to create a load test of 10 concurrent users
with each user running the <code>ExampleTestCase.testOneSecondResponse()</code> 
method once and with a 1 second delay between the addition of users, use:
<BLOCKQUOTE>
<FONT CLASS="Small">
<PRE>
int users = 10;
Timer timer = new ConstantTimer(1000);
Test testCase = new ExampleTestCase("testOneSecondResponse");
Test loadTest = new LoadTest(testCase, users, timer);
</PRE>
</FONT>
</BLOCKQUOTE>
</P>
<P>
In order to simulate each concurrent user running a test for a 
specified number of iterations, a <code>LoadTest</code> can be 
constructed to decorate a <code>RepeatedTest</code>.  Alternatively, 
a <code>LoadTest</code> convenience constructor specifying the number 
of iterations is provided which creates a <code>RepeatedTest</code>. 
</P>
<P>
For example, to create a load test of 10 concurrent users with each 
user running the <code>ExampleTestCase.testOneSecondResponse()</code> method
for 20 iterations, and with a 1 second delay between the addition of users, use:
<BLOCKQUOTE>
<FONT CLASS="Small">
<PRE>
int users = 10;
int iterations = 20;
Timer timer = new ConstantTimer(1000);
Test testCase = new ExampleTestCase("testOneSecondResponse");
Test repeatedTest = new RepeatedTest(testCase, iterations);
Test loadTest = new LoadTest(repeatedTest, users, timer);
</PRE>
</FONT>
</BLOCKQUOTE>
or, alternatively, use: 
<BLOCKQUOTE>
<FONT CLASS="Small">
<PRE>
int users = 10;
int iterations = 20;
Timer timer = new ConstantTimer(1000);
Test testCase = new ExampleTestCase("testOneSecondResponse");
Test loadTest = new LoadTest(testCase, users, iterations, timer);
</PRE>
</FONT>
</BLOCKQUOTE> 
</P>
<P>
If a test case intended to be decorated as a <code>LoadTest</code> contains
test-specific state in the <code>setUp()</code> method, then the
<code>TestFactory</code> should be used to ensure that each concurrent user
thread uses a thread-local instance of the test.  For example, to create a load 
test of 10 concurrent users with each user running a thread-local instance of 
<code>ExampleStatefulTest</code>, use:
<BLOCKQUOTE>
<FONT CLASS="Small">
<PRE>
int users = 10;
Test factory = new TestFactory(ExampleStatefulTest.class);
Test loadTest = new LoadTest(factory, users);
</PRE>
</FONT>
</BLOCKQUOTE>
or, to load test a single test method, use: 
<BLOCKQUOTE>
<FONT CLASS="Small">
<PRE>
int users = 10;
Test factory = new TestMethodFactory(ExampleStatefulTest.class, "testSomething");
Test loadTest = new LoadTest(factory, users);
</PRE>
</FONT>
</BLOCKQUOTE>
</P>
<P>
The following is an example test that creates a <code>LoadTest</code> to 
test the scalability of the functionality being unit tested in the 
<code>ExampleTestCase.testOneSecondResponse()</code> test method. 
The <code>LoadTest</code> adds 10 concurrent users without delay, 
with each user running the test method once.  The <code>LoadTest</code> 
itself is decorated with a <code>TimedTest</code> to test the throughput
of the <code>ExampleTestCase.testOneSecondResponse()</code> test method
under load.  The test will fail if the total elapsed time 
of the entire load test exceeds 1.5 seconds.
</P>
<P>
<DIV align="center">
<TABLE class="Primary" width="90%" cellspacing="0" cellpadding="1" border="0">
	<TR>
		<TD class="Primary">
			<FONT class="Reverse">
				<B>&nbsp;Example Throughput Under Load Test</B>
			</FONT>
		</TD>
	</TR>
	<TR>
		<TD class="Primary">
			<TABLE class="Secondary" width="100%" cellspacing="0" cellpadding="8" border="0">
				<TR>
					<TD class="Secondary">
<FONT CLASS="Small">
<PRE>
import com.clarkware.junitperf.*;

import junit.framework.Test;
import junit.framework.TestSuite;

public class ExampleThroughputUnderLoadTest {

    public static Test suite() {
     
        int maxUsers = 10;
        long maxElapsedTime = 1500;
        
        Test testCase = new ExampleTestCase("testOneSecondResponse");
        Test loadTest = new LoadTest(testCase, maxUsers);
        Test timedTest = new TimedTest(loadTest, maxElapsedTime);

        TestSuite suite = new TestSuite();
        suite.addTest(timedTest);
        
        return suite;
    }
    
    public static void main(String args[]) {
        junit.textui.TestRunner.run(suite());
    }
}
</PRE>
</FONT>
					</TD>
				</TR>
			</TABLE>
		</TD>
	</TR>
</TABLE>
</DIV>
</P>
<P>
In the following example, the order of test decoration is reversed.  The 
<code>TimedTest</code> measures the elapsed time of the 
<code>ExampleTestCase.testOneSecondResponse()</code> method.  
The <code>LoadTest</code> then decorates the <code>TimedTest</code> 
to simulate a 10-user load on the <code>ExampleTestCase.testOneSecondResponse()</code> 
method.  The test will fail if any user's response time exceeds 1 second.
</P>
<P>
<DIV align="center">
<TABLE class="Primary" width="90%" cellspacing="0" cellpadding="1" border="0">
	<TR>
		<TD class="Primary">
			<FONT class="Reverse">
				<B>&nbsp;Example Response Time Under Load Test</B>
			</FONT>
		</TD>
	</TR>
	<TR>
		<TD class="Primary">
			<TABLE class="Secondary" width="100%" cellspacing="0" cellpadding="8" border="0">
				<TR>
					<TD class="Secondary">
<FONT CLASS="Small">
<PRE>
import com.clarkware.junitperf.*;

import junit.framework.Test;
import junit.framework.TestSuite;

public class ExampleResponseTimeUnderLoadTest {

    public static Test suite() {
     
        int maxUsers = 10;
        long maxElapsedTime = 1000;
        
        Test testCase = new ExampleTestCase("testOneSecondResponse");
        Test timedTest = new TimedTest(testCase, maxElapsedTime);
        Test loadTest = new LoadTest(timedTest, maxUsers);

        TestSuite suite = new TestSuite();
        suite.addTest(loadTest);
        
        return suite;
    }
    
    public static void main(String args[]) {
        junit.textui.TestRunner.run(suite());
    }
}
</PRE>
</FONT>
					</TD>
				</TR>
			</TABLE>
		</TD>
	</TR>
</TABLE>
</DIV>
</P>
<BR>
<b>Performance Test Suite</b>
<P>
The following is an example <code>TestCase</code> that combines the
<code>ExampleTimedTest</code> and <code>ExampleLoadTest</code> into
a single test suite that can be run automatically to run all performance-related tests:
</P>
<P>
<DIV align="center">
<TABLE class="Primary" width="90%" cellspacing="0" cellpadding="1" border="0">
	<TR>
		<TD class="Primary">
			<FONT class="Reverse">
				<B>&nbsp;Example Performance Test Suite</B>
			</FONT>
		</TD>
	</TR>
	<TR>
		<TD class="Primary">
			<TABLE class="Secondary" width="100%" cellspacing="0" cellpadding="8" border="0">
				<TR>
					<TD class="Secondary">
<FONT CLASS="Small">
<PRE>
import junit.framework.Test;
import junit.framework.TestSuite;

public class ExamplePerfTestSuite {

    public static Test suite() {

        TestSuite suite = new TestSuite();
        suite.addTest(ExampleTimedTest.suite());
        suite.addTest(ExampleLoadTest.suite());
        
        return suite;
    }
    
    public static void main(String args[]) {
        junit.textui.TestRunner.run(suite());
    }
}
</PRE>
</FONT>
					</TD>
				</TR>
			</TABLE>
		</TD>
	</TR>
</TABLE>
</DIV>
</P>
<BR>


<!--

	Writing Effective Tests	

-->
<P>
<A name="effectivetests"></A> 
<TABLE class="Primary" width="100%" border="0" cellspacing="0" cellpadding="0">
	<TR><TD class="Border">&nbsp;</TD></TR>
	<TR> 
		<TD class="SubHeader">&nbsp;
			Writing Effective JUnitPerf Tests	
		</TD>
	</TR>
</TABLE>
</P>
<BR>
<b>Timed Tests</b>
<BR>
<P>
<i>Waiting Timed Tests</i>
</P>
<P>
By default, a <code>TimedTest</code> will wait for the completion of its 
decorated test and then fail if the maximum elapsed time was exceeded.
This type of <i>waiting</i> timed test always allows its decorated test 
to accumulate all test results until test completion and
check the accumulated test results.
</P>
<P>
If the test decorated by a waiting timed test spawns threads, either directly
or indirectly, then the decorated test must wait for those threads to run to 
completion and return control to the timed test.  Otherwise, the timed test will
wait indefinitely.  As a general rule, unit tests should always wait for spawned 
threads to run to completion, using <code>Thread.join()</code> for example, in order 
to accurately assert test results.  
</P>
<P>
<i>Non-Waiting Timed Tests</i>
</P>
<P>
<P>   
Alternatively, a <code>TimedTest</code> can be constructed to immediately 
signal a failure when the maximum elapsed time of its decorated test is 
exceeded.  This type of <i>non-waiting</i> timed test will not wait 
for its decorated test to run to completion if the maximum elapsed time is 
exceeded.  Non-waiting timed tests are more efficient than waiting timed tests 
in that non-waiting timed tests don't waste time waiting for the decorated test 
to complete only then to signal a failure, if necessary.  However, unlike 
waiting timed tests, test results from a decorated test will not be accumulated 
after the expiration of the maximum elapsed time in a non-waiting timed test.  
</P>
<BR>
<b>Load Tests</b>
<BR>
<P>
<i>Non-Atomic Load Tests</i>
</P>
<P>
By default, a <code>LoadTest</code> does not enforce test atomicity (as defined
in transaction processing) if its decorated test spawns threads, either directly 
or indirectly.  This type of <i>non-atomic</i> load test assumes that its decorated
test is transactionally complete when control is returned.  For example, if the
decorated test spawns threads and then returns control without waiting for its spawned 
threads to complete, then the decorated test is assumed to be transactionally complete. 
</P>
<P>
As a general rule, unit tests should always wait for spawned threads to run to completion, 
using <code>Thread.join()</code> for example, in order to accurately assert test results.  
However, in certain environments this isn't always possible.  For example, as a result
of a distributed lookup of an Enterprise JavaBean (EJB), an application server may
spawn a new thread to handle the request.  If the new thread belongs to the same
<code>ThreadGroup</code> as the thread running the decorated test (the default), then
a non-atomic load test will simply wait for the completion of all threads spawned 
directly by the load test and the new (rogue) thread is ignored.
</P>
<P> 
To summarize, non-atomic load tests only wait for the completion of threads spawned 
directly by the load test to simulate more than one concurrent user.
</P>
<P>
<i>Atomic Load Tests</i>
</P>
<P>
If threads are integral to the successful completion of a decorated test, meaning 
that the decorated test should not be treated as complete until all of its threads 
run to completion, then <code>setEnforceTestAtomicity(true)</code> should be invoked 
to enforce test atomicity (as defined in transaction processing).  This effectively 
causes the <i>atomic</i> load test to wait for the completion of all threads belonging 
to the same <code>ThreadGroup</code> as the thread running the decorated test.  Atomic 
load tests also treat any premature thread exit as a test failure.  If a thread dies 
abruptly, then all other threads belonging to the same <code>ThreadGroup</code> as the 
thread running the decorated test will be interrupted immediately.  
<P>
If a decorated test spawns threads belonging to the same <code>ThreadGroup</code> as 
the thread running the decorated test (the default), then the atomic load test will 
wait indefinitely for the spawned thread to complete.
</P>
<P>
To summarize, atomic load tests wait for the completion of all threads belonging to the 
same <code>ThreadGroup</code> as the threads spawned directly by the load test to simulate 
more than one concurrent user.
</P>
<BR>


<!--

	Limitations	

-->
<P>
<A name="limitations"></A> 
<TABLE class="Primary" width="100%" border="0" cellspacing="0" cellpadding="0">
	<TR><TD class="Border">&nbsp;</TD></TR>
	<TR> 
		<TD class="SubHeader">&nbsp;
			Limitations	
		</TD>
	</TR>
</TABLE>
</P>
<P>
JUnitPerf has the following known limitations:
<P>
<UL>
	<LI>The elapsed time measured by a <code>TimedTest</code> decorating
		a single <code>testXXX()</code> method of a <code>TestCase</code> 
		includes the total time of the <code>setUp()</code>, 
		<code>testXXX()</code>, and <code>tearDown()</code> methods, as 
		this is the granularity offered by decorating any <code>Test</code> 
		instance.  The expected elapsed time measurements should be adjusted 
		accordingly to account for the set-up and tear-down costs of the
		decorated test.
	</LI>
	<P>
	<LI>JUnitPerf is not intended to be a full-fledged load testing
	    or performance profiling tool, nor is it intended to replace
        the use of these tools.  JUnitPerf should be used to write
        localized performance unit tests to help developers refactor
        responsibly.  
	</LI>
    <P>
    <LI>The performance of your tests can degrade significantly if too
		many concurrent users are cooperating in a load test.  The actual 
		threshold number is JVM specific.
    </LI>
    <P>
</UL>
</P>
<BR>

<!--

	Support	

-->
<P>
<A name="support"></A> 
<TABLE class="Primary" width="100%" border="0" cellspacing="0" cellpadding="0">
	<TR><TD class="Border">&nbsp;</TD></TR>
	<TR> 
		<TD class="SubHeader">&nbsp;
			Support	
		</TD>
	</TR>
</TABLE>
</P>
<P>
If you have any questions, comments, enhancement requests, success stories,
or bug reports regarding JUnitPerf, or if you want to be notified when new
versions of JUnitPerf are available, please email 
<a href="mailto:mike@clarkware.com">mike@clarkware.com</a>.
Your information will be kept private.
</P>
<P>
A <a href="http://groups.yahoo.com/group/junitperf/" target="_blank">mailing list</a> is also available to discuss JUnitPerf or to be notified when new versions of JUnitPerf are available.
</P>
<BR>


<!--

	Consulting Services	

-->
<P>
<A name="consulting"></A> 
<TABLE class="Primary" width="100%" border="0" cellspacing="0" cellpadding="0">
	<TR><TD class="Border">&nbsp;</TD></TR>
	<TR> 
		<TD class="SubHeader">&nbsp;
			Consulting Services	
		</TD>
	</TR>
</TABLE>
</P>
<P>
As the Principal Consultant for 
<A href="http://www.clarkware.com" target="_blank">Clarkware Consulting, Inc.</A>,
I offer customized training and mentoring to help you and your team 
quickly become productive using agile design and testing tools that any
budget can afford.  Software development and performance consulting services 
are also available to help you effectively build high-quality systems based 
on today's business needs rather than speculation.  My services are always
tailored and scaled to your specific needs to deliver maximum business value.
</P>
<P>
<A href="mailto:mike@clarkware.com">Contact me</A>
for more details. 
</P>
<BR>


<!--

	License

-->
<P>
<A name="license"></A> 
<TABLE class="Primary" width="100%" border="0" cellspacing="0" cellpadding="0">
	<TR><TD class="Border">&nbsp;</TD></TR>
	<TR> 
		<TD class="SubHeader">&nbsp;
			License
		</TD>
	</TR>
</TABLE>
</P>
<P>
JUnitPerf is licensed under the 
<a href="http://www.clarkware.com/software/license.txt">BSD License</a>.
</P>
</TABLE>
<BR>


<!--

	Credits

-->
<P>
<A name="credits"></A> 
<TABLE class="Primary" width="100%" border="0" cellspacing="0" cellpadding="0">
	<TR><TD class="Border">&nbsp;</TD></TR>
	<TR> 
		<TD class="SubHeader">&nbsp;
			Acknowledgments
		</TD>
	</TR>
</TABLE>
</P>
<P>
Many thanks to Ervin Varga for improving thread safety and test atomicity by 
suggesting the use of a ThreadGroup to catch and handle thread exceptions.  
Ervin also proposed the idea and provided the implementation for the 
<code>TimedTest</code> signaling a failure immediately if the maximum time 
is exceeded and the <code>TestFactory</code>.  His review of JUnitPerf and 
his invaluable contributions are much appreciated! 
</P>
</TABLE>
<BR>


<!--

	Resources

-->
<P>
<A name="resources"></A> 
<TABLE class="Primary" width="100%" border="0" cellspacing="0" cellpadding="0">
	<TR><TD class="Border">&nbsp;</TD></TR>
	<TR> 
		<TD class="SubHeader">&nbsp;
			Resources
		</TD>
	</TR>
</TABLE>
</P>
<P>
<UL>
	<LI><a href="http://www.clarkware.com/articles/JUnitPrimer.html" 
		target="_blank">JUnit Primer</a>
		<BR>
		This article demonstrates how to write and run simple test cases 
		and test suites using the JUnit testing framework.
		<BR>
		<i>Mike Clark, Clarkware Consulting, Inc.</i>
		<P>
	</LI>
	<LI><A href="http://www.amazon.com/exec/obidos/ASIN/047120708X/qid=1010038430/sr=1-4/ref=
sr_1_15_4/107-9174541-2232556" target="_blank">Java Tools for Extreme Programming: Mastering Open Source Tools Including Ant, JUnit, and Cactus</A>,
		<BR>
		<I>Richard Hightower, Nicholas Lesiecki (John Wiley & Sons, 2001)</I>
		<BR>
		This book devotes a chapter to JUnitPerf and includes examples of integration
		with other JUnit extensions.
		<P>
	</LI>
	<LI><a href="http://www.junit.org/"target="_blank">http://www.junit.org</a>
		<BR>
		The JUnit Website
		<P>
	</LI>
</UL>
</P>
</TABLE>
</DIV>
</P>
<BR>


<!--

	Release History

-->
<P>
<A name="release"></A> 
<TABLE class="Primary" width="100%" border="0" cellspacing="0" cellpadding="0">
	<TR><TD class="Border">&nbsp;</TD></TR>
	<TR> 
		<TD class="SubHeader">&nbsp;
			Release History
		</TD>
	</TR>
</TABLE>
</P>
<P>
<TABLE width="75%"border="0" cellspacing="0" cellpadding="0">
<TR>
<TD class="Primary">
<TABLE class="Primary" width="100%" border="0" cellspacing="1" cellpadding="5">
	<TR>
		<TD class="SubHeader" width="25%" align="left">
			<B>Version</B>
		</TD>
		<TD align="left">
			1.8
		</TD>
	</TR>
	<TR>
		<TD class="SubHeader" width="25%" align="left">
			<B>Date</B>
		</TD>
		<TD align="left">
			09.03.2002
		</TD>
	</TR>
	<TR>
		<TD class="SubHeader" width="25%" align="left">
			<B>Description</B>
		</TD>
		<TD align="left">
			Usability upgrades
		</TD>
	</TR>
</TABLE>
</TD>
</TR>
<TR>
<TD class="Primary">
<TABLE class="Primary" width="100%" border="0" cellspacing="1" cellpadding="5">
	<TR>
		<TD class="SubHeader" align="left">
			<B>Changes</B>	  
		</TD>
	</TR>
	<TR>
		<TD align="left">
			<UL>
			<LI>Documentation edits and additions.
			</LI>
			<LI>Added the <code>ExampleThroughputUnderLoadTest</code> and
				<code>ExampleResponseTimeUnderLoadTest</code> to the samples.
			</LI>
			<LI>Added the <code>TimedTest.setQuiet()</code> method to
				optionally disable output of the test's elapsed time.
			</LI>
			</UL>
		</TD>
	</TR>
</TABLE>
</TD>
</TR>
</TABLE>
<BR>
</P>
<P>
<TABLE width="75%"border="0" cellspacing="0" cellpadding="0">
<TR>
<TD class="Primary">
<TABLE class="Primary" width="100%" border="0" cellspacing="1" cellpadding="5">
	<TR>
		<TD class="SubHeader" width="25%" align="left">
			<B>Version</B>
		</TD>
		<TD align="left">
			1.7
		</TD>
	</TR>
	<TR>
		<TD class="SubHeader" width="25%" align="left">
			<B>Date</B>
		</TD>
		<TD align="left">
			02.26.2002
		</TD>
	</TR>
	<TR>
		<TD class="SubHeader" width="25%" align="left">
			<B>Description</B>
		</TD>
		<TD align="left">
			Added TestMethodFactory  	
		</TD>
	</TR>
</TABLE>
</TD>
</TR>
<TR>
<TD class="Primary">
<TABLE class="Primary" width="100%" border="0" cellspacing="1" cellpadding="5">
	<TR>
		<TD class="SubHeader" align="left">
			<B>Changes</B>	  
		</TD>
	</TR>
	<TR>
		<TD align="left">
			<UL>
			<LI>The TestMethodFactory can be used to load test a single test method
				while ensuring that each concurrent user thread uses a thread-local 
				instance of the test.
			</LI>
			</UL>
		</TD>
	</TR>
</TABLE>
</TD>
</TR>
</TABLE>
<BR>
</P>
<P>
<TABLE width="75%"border="0" cellspacing="0" cellpadding="0">
<TR>
<TD class="Primary">
<TABLE class="Primary" width="100%" border="0" cellspacing="1" cellpadding="5">
	<TR>
		<TD class="SubHeader" width="25%" align="left">
			<B>Version</B>
		</TD>
		<TD align="left">
			1.6
		</TD>
	</TR>
	<TR>
		<TD class="SubHeader" width="25%" align="left">
			<B>Date</B>
		</TD>
		<TD align="left">
			11.23.2001
		</TD>
	</TR>
	<TR>
		<TD class="SubHeader" width="25%" align="left">
			<B>Description</B>
		</TD>
		<TD align="left">
			Maintenance upgrade. 	
		</TD>
	</TR>
</TABLE>
</TD>
</TR>
<TR>
<TD class="Primary">
<TABLE class="Primary" width="100%" border="0" cellspacing="1" cellpadding="5">
	<TR>
		<TD class="SubHeader" align="left">
			<B>Changes</B>	  
		</TD>
	</TR>
	<TR>
		<TD align="left">
			<UL>
			<LI>If a threaded test in a load test has been stopped, either by using the 
				"Stop" button of the Swing UI or using the <code>haltonfailure="yes"</code> 
				attribute of a JUnit Ant task, the stopped or failed test is 
				cancelled and the currently active threaded tests of the load test 
				are allowed to complete.  Prior to this upgrade, if a threaded test was 
				stopped, the load test would hang while waiting for the threaded
				test to report its completion.
			</LI>
			</UL>
		</TD>
	</TR>
</TABLE>
</TD>
</TR>
</TABLE>
<BR>
<P>
<TABLE width="75%"border="0" cellspacing="0" cellpadding="0">
<TR>
<TD class="Primary">
<TABLE class="Primary" width="100%" border="0" cellspacing="1" cellpadding="5">
	<TR>
		<TD class="SubHeader" width="25%" align="left">
			<B>Version</B>
		</TD>
		<TD align="left">
			1.5
		</TD>
	</TR>
	<TR>
		<TD class="SubHeader" width="25%" align="left">
			<B>Date</B>
		</TD>
		<TD align="left">
			09.08.2001
		</TD>
	</TR>
	<TR>
		<TD class="SubHeader" width="25%" align="left">
			<B>Description</B>
		</TD>
		<TD align="left">
			Usability upgrade. 	
		</TD>
	</TR>
</TABLE>
</TD>
</TR>
<TR>
<TD class="Primary">
<TABLE class="Primary" width="100%" border="0" cellspacing="1" cellpadding="5">
	<TR>
		<TD class="SubHeader" align="left">
			<B>Changes</B>
		</TD>
	</TR>
	<TR>
		<TD align="left">
			<UL>
			<LI>Added the <code>TestFactory</code> class to allow stateful tests 
				to be decorated as <code>LoadTest</code> instances.  Use of a 
				<code>TestFactory</code> ensures that each <code>LoadTest</code>
				thread uses its own decorated test instance.
			</LI>
			</UL>
		</TD>
	</TR>
</TABLE>
</TD>
</TR>
</TABLE>
<BR>
<P>
<TABLE width="75%"border="0" cellspacing="0" cellpadding="0">
<TR>
<TD class="Primary">
<TABLE class="Primary" width="100%" border="0" cellspacing="1" cellpadding="5">
	<TR>
		<TD class="SubHeader" width="25%" align="left">
			<B>Version</B>
		</TD>
		<TD align="left">
			1.4
		</TD>
	</TR>
	<TR>
		<TD class="SubHeader" width="25%" align="left">
			<B>Date</B>
		</TD>
		<TD align="left">
			06.12.2001
		</TD>
	</TR>
	<TR>
		<TD class="SubHeader" width="25%" align="left">
			<B>Description</B>
		</TD>
		<TD align="left">
			Design and usability upgrades. 	
		</TD>
	</TR>
</TABLE>
</TD>
</TR>
<TR>
<TD class="Primary">
<TABLE class="Primary" width="100%" border="0" cellspacing="1" cellpadding="5">
	<TR>
		<TD class="SubHeader" align="left">
			<B>Changes</B>
		</TD>
	</TR>
	<TR>
		<TD align="left">
			<UL>
			<LI>A <code>TimedTest</code> can now be constructed to fail 
			    immediately if the maximum elapsed time of the decorated test 
			    is exceeded.  In other words, the <code>TimedTest</code> will 
			    <b>not</b> wait for the decorated test to run to completion if 
			    the maximum elapsed time is exceeded.
			</LI>
            <LI>The <code>TimedTest.outOfTime()</code> method was added to 
                unambiguously determine whether the test failed due to the 
                maximum elapsed time being exceeded or the test itself failing.
            </LI>
			<LI>The <code>LoadTest</code> class now supports enforcing test
			    atomicity using the <code>setEnforceTestAtomicity()</code>
			    method.  By default, test atomicity is not enforced for 
                test cases that spawn threads, either directly or indirectly.
			</LI>
            <LI>The <code>TimedTest.toString()</code> method now includes an
                indication of whether the timed test will wait for test 
                completion (WAITING) or wait for the maximum elapsed time 
                to expire (NON-WAITING).
            </LI>
            <LI>The <code>LoadTest.toString()</code> method now includes an
                indication of whether the load test will enforce test
                atomicity by waiting for control to return (ATOMIC) or 
                waiting for all threaded tests to complete (NON-ATOMIC).
            </LI>
			</UL>
		</TD>
	</TR>
</TABLE>
</TD>
</TR>
</TABLE>
<BR>
<P>
<TABLE width="75%"border="0" cellspacing="0" cellpadding="0">
<TR>
<TD class="Primary">
<TABLE class="Primary" width="100%" border="0" cellspacing="1" cellpadding="5">
	<TR>
		<TD class="SubHeader" width="25%" align="left">
			<B>Version</B>
		</TD>
		<TD align="left">
			1.3
		</TD>
	</TR>
	<TR>
		<TD class="SubHeader" width="25%" align="left">
			<B>Date</B>
		</TD>
		<TD align="left">
			05.11.2001
		</TD>
	</TR>
	<TR>
		<TD class="SubHeader" width="25%" align="left">
			<B>Description</B>
		</TD>
		<TD align="left">
			Design and usability upgrades. 	
		</TD>
	</TR>
</TABLE>
</TD>
</TR>
<TR>
<TD class="Primary">
<TABLE class="Primary" width="100%" border="0" cellspacing="1" cellpadding="5">
	<TR>
		<TD class="SubHeader" align="left">
			<B>Changes</B>
		</TD>
	</TR>
	<TR>
		<TD align="left">
			<UL>
			<LI>The <code>LoadTest</code> class now employs a <code>ThreadBarrier</code> 
			    to allow threads spawned directly by a load test to properly signal 
			    their completion.  Threads spawned by decorated tests, either directly 
			    or indirectly, without a specified thread group are added to the 
			    <code>ThreadedTestGroup</code> by default.  This was causing the active 
			    count of the thread group to never fall to 0, thereby causing the load
				test to hang indefinitely.
			</LI>
			</UL>
		</TD>
	</TR>
</TABLE>
</TD>
</TR>
</TABLE>
<BR>
<TABLE width="75%"border="0" cellspacing="0" cellpadding="0">
<TR>
<TD class="Primary">
<TABLE class="Primary" width="100%" border="0" cellspacing="1" cellpadding="5">
	<TR>
		<TD class="SubHeader" width="25%" align="left">
			<B>Version</B>
		</TD>
		<TD align="left">
			1.2
		</TD>
	</TR>
	<TR>
		<TD class="SubHeader" width="25%" align="left">
			<B>Date</B>
		</TD>
		<TD align="left">
			04.23.2001
		</TD>
	</TR>
	<TR>
		<TD class="SubHeader" width="25%" align="left">
			<B>Description</B>
		</TD>
		<TD align="left">
			Design and usability upgrades.
		</TD>
	</TR>
</TABLE>
</TD>
</TR>
<TR>
<TD class="Primary">
<TABLE class="Primary" width="100%" border="0" cellspacing="1" cellpadding="5">
	<TR>
		<TD class="SubHeader" align="left">
			<B>Changes</B>
		</TD>
	</TR>
	<TR>
		<TD align="left">
			<UL>
			<LI>Replaced the <code>ThreadBarrier</code> with a 
				<code>ThreadedTestGroup</code> to catch and handle 
				uncaught exceptions thrown by threads spawned by 
				<code>ThreadedTest</code>.  This improves thread 
				safety and supports test atomicity (as defined by
				transaction processing) when enabled.
			</LI>
			<LI>Added several variants of <code>LoadTest</code> 
				constructors for convenience and extensibility.
			</LI>
			<LI>Updated <code>JUnitPerf.html</code> and 
				<code>ExampleLoadTest.java</code> to include more 
				examples for constructing <code>LoadTest</code>
				instances with various constructors.
			</LI>
			</UL>
		</TD>
	</TR>
</TABLE>
</TD>
</TR>
</TABLE>
<BR>
<TABLE width="75%"border="0" cellspacing="0" cellpadding="0">
<TR>
<TD class="Primary">
<TABLE class="Primary" width="100%" border="0" cellspacing="1" cellpadding="5">
	<TR>
		<TD class="SubHeader" width="25%" align="left">
			<B>Version</B>
		</TD>
		<TD align="left">
			1.1
		</TD>
	</TR>
	<TR>
		<TD class="SubHeader" width="25%" align="left">
			<B>Date</B>
		</TD>
		<TD align="left">
			03.03.2001
		</TD>
	</TR>
</TABLE>
</TD>
</TR>
<TR>
<TD class="Primary">
<TABLE class="Primary" width="100%" border="0" cellspacing="1" cellpadding="5">
	<TR>
		<TD class="SubHeader" width="25%" align="left">
			<B>Description</B>
		</TD>
		<TD align="left">
			Initial public release.
		</TD>
	</TR>
</TABLE>
</TD>
</TR>
</TABLE>
</P>

<BR>
<BR>
<HR>

<CENTER>
	<p class="Copyright">
	Copyright &copy; 1999-2003 
	<a href="http://www.clarkware.com" 
		target="_blank">Clarkware Consulting, Inc.</a>
	<BR>
	All Rights Reserved
</CENTER>

</BODY>
</HTML>