File: design.html

package info (click to toggle)
libtut 0.0.20070706-3
links: PTS
area: main
in suites: bookworm, bullseye
size: 1,144 kB
sloc: cpp: 3,588; xml: 137; makefile: 18; ansic: 9
file content (479 lines) | stat: -rw-r--r-- 17,004 bytes
parent folder | download | duplicates (3)
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">

<html>
<head>
<title>TUT Design - TUT: C++ Unit Test Framework</title>
<meta name="keywords" content="C++ test-driven unit tests TUT framework STL">
<meta name="description" content="TUT Design - what's hidden under the hood of tut.h, and why things are such as they
		are">
<style type="text/css">
<!--

body, p, td
{
	scrollbar-base-color: #eeeeee;
	scrollbar-track-color: #ffffff;
	scrollbar-arrow-color: #cccccc;
	scrollbar-shadow-color: #cccccc;
	scrollbar-highlight-color: #cccccc;
	scrollbar-darkshadow-color: #ffffff;
	scrollbar-3dlight-color: #ffffff;
	
	/*
	scrollbar-base-color: #cccccc;
	scrollbar-track-color: #ffffff;
	scrollbar-arrow-color: #eeeeee;
	*/
	
	font-family : Verdana, Arial, Helvetica, sans-serif;
	font-size : 14px;
	color: #000000;
}

pre
{
	font-family : Courier, sans-serif;
	font-size : 12px;
}

strong, b
{
	font-weight : bold;
}

ul
{
	list-style-type : square;
}

.header
{
	font-size : 24px;
	font-weight : bold;
}

.subheader
{
	font-size : 16px;
	font-weight : bold;
}

.logo
{
	font-size : 24px;
	font-weight : bold;
}

.question
{
	font-size : 16px;
	font-weight : bold;
	color : #000000;
}

a
{
	font-weight : bold;
	color : #ff9900;
	text-decoration : none;
}

a:hover
{
	font-weight : bold;
	color : #ff9900;
	text-decoration : underline;
}

a:visited
{
	font-weight : bold;
	color : #ff9900;
}

a.menu
{
	font-weight : bold;
	color : #ff9900;
	text-decoration : none;
}

a.menu:hover
{
	font-weight : bold;
	color : #ff9900;
	text-decoration : underline;
}

a.menu:visited
{
	font-weight : bold;
	color : #ff9900;
}

//-->
</style>
</head>

<body text="#000000" link="#ff9900" alink="#ffcc00" vlink="#ff9900" bgcolor="#ffffff" leftmargin=0 topmargin=0 marginheight=0 marginwidth=0>

<img src="/_img/pixel.gif" alt="" width=1 height=20><br>
<table width="100%" cellspacing=0 cellpadding=0>
<tr>
<td width=20 align=left valign=top>
<img src="/_img/pixel.gif" alt="" width=20 height=1><br>
</td>
<td width="100%" align=left valign=top>
<!-- home table >> -->
<table width="100%" cellspacing=0 cellpadding=0>
<tr>
<td width="100%" align=center valign=center bgcolor="#cccccc">
<!-- >> -->
<table width="100%" cellspacing=1 cellpadding=10>
<tr>
<td width="100%" align=left valign=top bgcolor="#eeeeee">
<p class="logo"><a href="/">TUT: C++ Unit Test Framework</a></p>
</td>
</tr>
</table>
<!-- << -->
</td>
</tr>
</table>
<!-- home table << -->
</td>
<td width=20 align=left valign=top>
<img src="/_img/pixel.gif" alt="" width=20 height=1><br>
</td>
</tr>
</table>

<table border=0 width="100%" cellspacing=20 cellpadding=0>
<tr>
<td width="30%" align=left valign=top>
<!-- menu table >> -->
<table width="100%" cellspacing=0 cellpadding=0>
<tr>
<td width="100%" align=center valign=center bgcolor="#cccccc">
<!-- >> -->
<table width="100%" cellspacing=1 cellpadding=10>
<tr>
<td width="100%" align=left valign=top bgcolor="#eeeeee">
<p class="subheader">Documentation</p>
<ul>
<li><p><a href="/howto/" class="menu">TUT How-To</a><br>minimum steps to make TUT work for you</p></li>
<li><p><strong>TUT Design</strong><br>what's hidden under the hood of tut.h, and why things are such as they
		are</p></li>
<li><p><a href="/example/" class="menu">TUT Usage Example</a><br>it's better to see once...</p></li>
<li><p><a href="/whole/" class="menu">TUT As Is</a><br>complete source of TUT</p></li>
</ul>
<p class="subheader">Distribution</p>
<ul>
<li><p><a href="/copyright/" class="menu">The TUT License</a><br>almost no restrictions to use</p></li>
<li><p><a href="/download/" class="menu">TUT Downloads</a><br>latest version of TUT as well as other related stuff</p></li>
</ul>
<p class="subheader">Support</p>
<ul>
<li><p><a href="/faq/" class="menu">TUT Frequently Asked Questions</a><br>often asked questions and answers for them</p></li>
<li><p><a href="/links/" class="menu">Links</a><br>related projects and concepts</p></li>
<li><p><a href="/author/" class="menu">TUT Author</a><br>who is the author</p></li>
</ul>
</td>
</tr>
</table>
<!-- << -->
</td>
</tr>
</table>
<!-- menu table << -->
</td>
<td width="70%" valign=top>
<!-- content table >> -->
<table width="100%" cellspacing=0 cellpadding=0>
<tr>
<td width="100%" align=center valign=center bgcolor="#cccccc">
<!-- >> -->
<table width="100%" cellspacing=1 cellpadding=10>
<tr>
<td width="100%" align=left valign=top bgcolor="#eeeeee">
<p class="header">TUT Design</p>
<p>In this document I attempt to explain the decisions made while developing
			TUT.</p>
		
		<p class="subheader">Requirements</p>
		
		<p> One day I ran into need of unit test framework for C++. So, I've made a small research
			and discovered C++Unit, boost::test and a bunch of similar libraries.. Though
			they were usable, I was not satisfied with the approach they offered; so I
			designed my own variant of unit test framework based on the following
			restrictions: </p>
		
		<ul>
			<li>No C-style macros</li>
			<li>No manual registration for test groups and methods</li>
			<li>No libraries of any kind</li>
			<li>Neutrality to user interface</li>
			<li>No Javisms</li>
		</ul>
		
		<p class="subheader">C-style macros and what's wrong with them</p>
		
		<p> Usually C++ Unit test frameworks define a lot of macroses to achieve the goals
			other languages have as built-in features: for example, Java is able to show you
			the whole exception stack; and C++ cannot. So, to achieve the same (or similar)
			results, C++ frameworks often declare a macro to catch any exception and trace
			__FILE__ and __LINE__ variables. </p>
		
		<p> The problem is that it turns the C++ code into something that is hard to read, where
			"hard to read" actually means "hard to maintain". </p>
		
		<p> Macros don't recognize namespace borders, so a simple macro can expand in the user
			code into something unexpected. To avoid this, we have to give macros unique
			prefixes, and this, in turn, reduces code readability even more. </p>
		
		<p> From bad to worse, C-style macros can't handle modern C++ templates, so comma
			separated template arguments will break the macro, since preprocessor will
			handle the template as two arguments (separated by the comma used in the
			template) to this macro. </p>
		
		<p> And the final contra for macros is that even if used they cannot achieve the same
			usability level as the native language tools; for example, macros cannot
			generate a full stack trace (at least, in a platform-independent manner). So it
			looks like we loose readability and get almost nothing for this. </p>
		
		<p> See also Bjarne Stroustrup notices about macros harmness:
			<a href="http://www.research.att.com/~bs/bs_faq2.html#macro"> So,
				what's wrong with using macros?</a> </p>
		
		<p class="subheader">Manual registration and why it annoys me</p>
		
		<p> In JUnit (Java-based Unit Tests framework) reflection is used to recognize
			user-written test methods. C++ has no reflection or similar mechanism, so user
			must somehow tell the framework that "this, this and that" methods should be
			considered as test methods, and others are just helpers for them. </p>
		
		<p> The same can be said about test groups, which have to be registered in test runner
			object. </p>
		
		<p> Again, most C++ frameworks introduce macros or at least methods to register a
			freestanding function or method as a test instance. I find writing redundant
			code rather annoying: I have to write test itself and then I have to write more code
			to mark that my test is a test. Even more, if I forget to register the method,
			nothing will warn me or somehow indicate that I have not done what I should. </p>
		
		<p class="subheader">Library and integration problems</p>
		
		<p> Most of C++ frameworks require building a library that user must link to test
			application to be able to run tests. The problem is that various platforms imply
			different ways for building libraries. One popular C++ framework has more than
			60% bugs in their bug database that sound like "cannot build library on platform
			XXX" or "built library doesn't work on platform YYY". </p>
		
		<p> Besides, some platforms has complexities in library memory management (e.g.
			Win32). </p>
		
		<p class="subheader">User interface</p>
		
		<p> Some frameworks provide predefined formatters for output results, for example
			CSV or XML. This restricts users in test results presentation options. What if a
			user wants some completely different format? Of course, he can implement his own
			formatter, but why frameworks provide useless formatters then? </p>
		
		<p> The ideal test framework must do only one thing: run tests. Anything beyond that is
			the responsibility of the user code. Framework provides the test results, and
			the user code then represents them in any desired form. </p>
		
		<p class="subheader">Javisms</p>
		
		<p>Most implementors of C++ test frameworks know about JUnit and inspired by
			this exciting tool. But, carelessly copying a Java implementation to C++, we can
			get strange and ugly design.</p>
		
		<p>Rather obvious example: JUnit has methods for setting up a test (setUp) and for
			cleaning after it (tearDown). I know at least two C++ frameworks that have these
			methods with the same semantics and names. But in C++ the job these methods do is
			the responsibility of constructor and destructor! In Java we don't have
			guaranteed destruction, so JUnit authors had to invent their own replacement
			for it - tearDown(); and it was natural then to introduce constructing
			counterpart - setUp(). Doing the same in C++ is absolutely redundant
		</p>
		
		<p>C++ has its own way of working, and whenever possible, I am going to stay at the C++
			roads, and will not repeat Java implementation just because it is really good for
			Java. </p>
		
		<p class="subheader">Decisions</p>
		
		<p class="subheader">No C-style macros</p>
		
		<p>The solution is that simple: just
			<b>do not</b> use any macros. I personally never needed a macro during
			development. </p>
		
		<p class="subheader">No manual registration</p>
		
		<p>Since C++ has no reflection, the only way to mark a method as a test is to give it a kind
			of predefined name.</p>
		
		<p>There would be a simple solution: create a set of virtual methods in test object base
			class, and allow user to overwrite them. The code might look like:</p>
		
		<pre> struct a_test_group : public test_group { virtual void test1() { ... } virtual
			void test2() { ... } }; </pre>
		
		<p>Unfortunately, this approach has major drawbacks:</p>
		
		<ul>
			<li>It scales badly. Consider, we have created 100 virtual test methods in a test
				group, but user needs 200. How can he achieve that? There is no proper way.
				Frankly speaking, such a demand will arise rarely (mostly in
				script-generated tests), but even the possibility of it makes this kind of
				design seemingly poor. </li>
			
			<li>There is no way to iterate virtual methods automatically. We
				would end up writing code that calls test1(), then test2(), and so on, each
				with its own exception handling and reporting.</li>
		</ul>
		
		<p>Another possible solution is to substitute reflection with a dynamic loading.
			User then would write static functions with predefined names, and TUT would use
			dlsym()/GetProcAddress() to find out the implemented tests. </p>
		
		<p>But I rejected the solution due to its platform and library operations
			dependencies. As I described above, the library operations are quite different
			on various platform. </p>
		
		<p>There was also an idea to have a small parser, that can scan the user code and
			generate registration procedure. This solution only looks simple; parsing
			free-standing user code can be a tricky procedure, and might easily overgrow twelve TUTs
			in complexity.</p>
		
		<p>Fortunately, modern C++ compilers already have a tool that can parse the user code
			and iterate methods. It is compiler template processing engine. To be more
			precise, it is template specialization technique.</p>
		
		<p>The following code iterates all methods named test&lt;N&gt; ranging from n to 0,
			and takes the address of each:</p>
		
		<pre> 
template &lt;class Test,class Group,int n&gt; 
struct tests_registerer 
{
	static void reg(Group&amp; group) 
	{ 
		group.reg(n,&amp;Test::template	test&lt;n&gt;); 
		tests_registerer&lt;Test,Group,n-1&gt;::reg(group); 
	}
}; 
			
template&lt;class Test,class Group&gt; 
struct tests_registerer&lt;Test,Group,0&gt; 
{ 
	static void reg(Group&amp;){}; 
};
... 
test_registerer&lt;test,group,100&gt;.reg(grp); 
		</pre>
		
		<p>This code generates recursive template instantiations until it reaches
			tests_registerer&lt;Test,Group,0&gt; which has empty specialization.
			There the recursion stops.</p>
		
		<p>The code is suitable for our needs because in the specialization preference is
			given to the user-written code, not to the generic one. Suppose we have a default
			method test&lt;N&gt; in our test group, and the user-written specialization
			for test&lt;1&gt;. So while iterating, compiler will get the address of the
			default method for all N, except 1, since user has supplied a special version of
			that method. </p>
		
		<pre> 
template&lt;int N&gt; 
void test() { }; 
... 
template&lt;&gt; 
void test_group::test&lt;1&gt;() { // user code here } 
		</pre>
		
		<p>This approach can be regarded as kind of compile-time virtual functions, since
			the user-written methods replace the default implementation. At the same time,
			it scales well - one just has to specify another test number upper bound at compile
			time. The method also allows iteration of methods, keeping code compact.</p>
		
		<p class="subheader">Library</p>
		
		<p>Since we dig into the template processing, it is natural to not build any
			libraries, therefor this problem mostly disappeares. Unfortunately, not
			completely: our code still needs some central point where it could register
			itself. But that point (singleton) is so small that it would be an overkill to
			create library just to put there one single object. Instead, we assume that the
			user code will contain our singleton somewhere in the main module of test
			application.</p>
		
		<p class="subheader">User interface. Callbacks.</p>
		
		<p>Our code will perform only minimamum set of tasks: TUT shall run tests. But we still
			need a way to adapt the end-user presentation requirements. For some of users it
			would be enough to see only failed tests in listing; others would like to see the
			complete plain-text report; some would prefer to get XML reports, and some would
			not want to get any reports at all since they draw their test execution log in GUI
			plugin for an IDE. </p>
		
		<p>"Many users" means "many demands", and satisfying all of them is quite a hard task.
			Attempt to use a kind of silver bullet (like XML) is not the right solution, since
			user would lack XML parser in his environment, or just would not want to put it into
			his project due to integration complexities.</p>
		
		<p>The decision was made to allow users to form their reports by themselfs. TUT will
			report an event, and the user code will form some kind of an artefact based on this
			event.</p>
		
		<p>The implementation of this decision is interface tut::callback. The user code
			creates a callback object, and passes it to the runner. When an appropriate event
			occures, the test runner invokes callback methods. User code can do anything,
			from dumping test results to std::cout to drawing 3D images, if desired. </p>
		
		<p class="subheader">STL</p>
		
		<p>Initially, there were plans to make TUT traits-based in order not to restrict
			it with STL only, but have a possibility to use other kinds of strings
			(TString, CString), containers and intercepted exceptions. </p>
		
		<p>In the current version, these plans are not implemented due to relative
			complexity of the task. For example, the actual set of operations can be quite
			different for various map implementations and this makes writing generic code
			much harder. </p>
		
		<p>Thus so far TUT is completely STL-based, since STL is the only library existing
			virtually on every platform. </p>
<img src="/_img/pixel.gif" alt="" width=1 height=300><br>

<center>
<!--RAX counter-->
<script language="JavaScript">
<!--
document.write('<a href="http://www.rax.ru/click" '+
'target=_blank><img src="http://counter.yadro.ru/hit?t13.4;r'+
escape(document.referrer)+((typeof(screen)=='undefined')?'':
';s'+screen.width+'*'+screen.height+'*'+(screen.colorDepth?
screen.colorDepth:screen.pixelDepth))+';'+Math.random()+
'" alt="rax.ru:     24 ,   24    \" '+
'border=0 width=88 height=31></a><br>')
//-->
</script>
<!--/RAX-->
</center>

</td>
</tr>
</table>
<!-- << -->
</td>
</tr>
</table>
<!-- content table << -->
</td>
</tr>
</table>

</body>
</html>