Dart Client Usage - Concepts

The Dart client operates under a number of Models:

• Nightly - Build/test sequences in lockstep with all other nightly builds. Nightly builds are the backbone of the Dart SQA process. Nightly builds provide the best opportunity to compare build/test sequences across multiple hardware or software configurations.
• Continuous - Build/test sequences started whenever something changes in the repository. Continuous builds allow the developers to monitor the impact of changes made to the software throughout the day.
• Experimental - Developer driven testing. This model is used by any developer to test their local copy of the software. Results from either form of experimental build/test sequences can be submitted to the nightly dashboard for all developers to see.

Under each model, the Dart client accepts a number of commands. The Commands for Nightly, Continuous and Experimental models are

• DashboardStart [date] - Starts a dashboard for the given date. Yesterday is default, only for nightly. Note that it triggers an Update automatically. DashboardStart only needs to be run on the Dart server.
• Start - Create a directory to store the build/test results.
• Update - For a Nightly build, update the code to the nightly timestamp. For a Continuous or Experimental build, update the code to reflect the current contents of the repository. For Experimental builds, Update reports on locally modified files. IMPORTANT: in the Nightly model the cvs update is done against the previous day's repository snapshot, not against the latest sources (Continuous or Experimental models). For Continuous builds, if no changes are detected in the repository, all remaining commands are skipped (no build is done and no tests are run) and no data is transmitted to the Dart server.
• Configure - Configure the build (could run CMake or a configure script).
• Build - Build the software and place reports in the proper directory (created by the Start command).
• Test - Test the software and place reports in the proper directory (created by the Start command).
• Coverage - Summarize the source code coverage of the test cases.
• Purify - Summarize dynamic memory usage of the test cases. Search for memory leaks, array bound reads, etc..
• Submit - Submit the last set of reports to the Dart server.
• DashboardEnd [date] -- Ends a dashboard for the given date. Yesterday is default, only for nightly. DashboardEnd only needs to be run on the Dart server.

Dart Client Usage - CMake

If Dart is configured to use CMake, then CMake will produce a number of make targets (Unix) or project files (Windows) to control Dart.

Unix Make Targets

• make Nightly - This make target performs a Start, Update, Build, Test, Submit for a nightly build.
• make NightlyStart - This make target performs a Start for a nightly build.
• make NightlyUpdate - This make target performs an Update for a nightly build.
• make NightlyBuild - This make target performs a Build for a nightly build.
• make NightlyTest - This make target Test's a nightly build.
• make NightlyCoverage - This make target computes Coverage for a nightly build.
• make NightlyPurify - This make target computes Purify (memory leaks, array bound reads, etc.) for a nightly build.
• make NightlySubmit - This make target Submit's nightly build/test results to the Dart server.

• make Continuous - This make target performs a Start, Update, Build, Test, Submit for a continuous build.
• make ContinuousStart - This make target performs a Start for a continuous build.
• make ContinuousUpdate - This make target performs an Update for a continuous build.
• make ContinuousBuild - This make target performs a Build for a continuous build.
• make ContinuousTest - This make target Test's a continuous build.
• make ContinuousCoverage - This make target computes Coverage for a continuous build.
• make ContinuousPurify - This make target computes Purify (memory leaks, array bound reads, etc.) for a continuous build.
• make ContinuousSubmit - This make target Submit's continuous build/test results to the Dart server.

• make Experimental - This make target performs a Start, Build, Test for an experimental build.
• make ExperimentalBuild - This make target performs a Start and Build for an experimental build.
• make ExperimentalTest - The make target only performs a Test of an experimental build.
• make ExperimentalCoverage - This make target computes Coverage for an experimental build.
• make ExperimentalSubmit - This make target Submit's an experimental build.

• make NightlyDashboardEnd - This make target rolls up a dashboard. Summarizing any information submitted since the last time the dashboard was generated.
• make NightlyDashboardStart - This make target opens a dashboard for submissions.

Windows MS Developer Projects

• Experimental.dsp - This project performs a Start, Build, Test for an experimental build.
• ExperimentalSubmit.dsp - This project Submit's an experimental build.

Dart Client Usage - Tcl

Dart can controlled directly using the Tcl scripts in Dart/Source/Client. These scripts are run from your project's build directory and have the form

tclsh Dart/Source/Client/DashboardManager.tcl DartConfiguration.tcl Model Command [Command ...]

The file DartConfiguration.tcl must exist in the project build directory. If you are using Dart with CMake, CMake will construct this file automatically, using Dart/Source/Client/Utility.conf.in as a template. If CMake is not being used, DartConfiguration.tcl must be constructed by hand.

Setting up a Nightly Client

Each Dart client should have a separate checkout of a project's source code and a separate build tree. Configure the build as you normally would. Define the SITE and BUILDNAME via CMake (or in DartConfiguration.tcl if not using CMake). Also make sure the CVSCOMMAND, MAKECOMMAND, TCL_TCLSH are defined properly. When using Microsoft Visual Studio as your compiler, the MAKECOMMAND will specify which build configuration (Release, Debug, etc.) to use. Set MAKECOMMAND to use the build configuration you want. Dart will search for test executables in a prescribed order (Release, Debug, MinSizeRel, RelWithDebInfo). Dart will run a test from the first configuration it finds. Dart clients, therefore, should only have a single configuration built. All other configurations should be empty. Future versions of Dart may allow the client to specify which configuration to run tests from. Finally, if your testing machine is behind a firewall, you may have set the environment variables HTTP_PROXY and HTTP_PROXY_PORT.

Once a build is configured, schedule a job to run your nightly build. The scheduled time can be anytime after the NIGHTLY_START_TIME for your project. Your scheduled job should set any necessary environment variables you need to build and run. Running a nightly build/test sequence can be accomplished with the following commands

cd ProjectNightlyBuildDirectory
tclsh Dart/Source/Client/DashboardManager.tcl DartConfiguration.tcl \
	Nightly Start Update Configure Build Test Submit

If you would like to submit results to the Dartboard as commands finish, you can break the command into several Dart commands. The following will submit the update and build results to the Dart server, then run the tests and submit those results.

cd ProjectNightlyBuildDirectory
tclsh Dart/Source/Client/DashboardManager.tcl DartConfiguration.tcl \
	Nightly Start Update Configure Build 
tclsh Dart/Source/Client/DashboardManager.tcl DartConfiguration.tcl \
	Nightly Test Submit

Setting up a Continuous Client

A Continuous client is setup similar to a Nightly client. It should have its own source checkout and its own build area. Whereas a Nighly client is driven by a scheduled job, a Continuous client is usually started once and runs forever. A script to run a client in Continuous mode would look like:

cd ProjectContinuousBuildDirectory
while (1)
   tclsh Dart/Source/Client/DashboardManager.tcl DartConfiguration.tcl \
	Continuous Start Update Configure Build Test Submit
   sleep 300
end

Here, the Dart commands are inside a "while" loop and the client is instructed to sleep for 5 minutes after completing a build/test sequence. Also note that we do not break the Dart commands into several command sequences like we can do for a Nightly. The reason is that an "Update" on a Continuous build will terminate the Dart client when nothing has changed in the repository. This avoids building and running tests when nothing has changed. This feature is only available when a Dart Continuous client runs all the Dart commands in one operation.

Hint: Turn off compiler optimisation with continuous testing. Unless your tests are very compute heavy, this continuous build cycle is dominated by the compiler and optimisation will just make it take longer. If compute resources are scarce, the same suggestion applies to nightly builds - however, it is useful to check for errors with the optimisation settings you will use to build your application for normal use.