File: buildbot.html

package info (click to toggle)
buildbot 0.7.4-3
links: PTS
area: main
in suites: etch, etch-m68k
size: 2,968 kB
ctags: 3,745
sloc: python: 20,806; sh: 194; makefile: 59
file content (6265 lines) | stat: -rw-r--r-- 306,755 bytes
<html lang="en">
<head>
<title>BuildBot Manual 0.7.4</title>
<meta http-equiv="Content-Type" content="text/html">
<meta name="description" content="BuildBot Manual 0.7.4">
<meta name="generator" content="makeinfo 4.8">
<link title="Top" rel="top" href="#Top">
<link href="http://www.gnu.org/software/texinfo/" rel="generator-home" title="Texinfo Homepage">
<!--
This is the BuildBot manual.

Copyright (C) 2005,2006 Brian Warner

Copying and distribution of this file, with or without
modification, are permitted in any medium without royalty
provided the copyright notice and this notice are preserved.-->
<meta http-equiv="Content-Style-Type" content="text/css">
<style type="text/css"><!--
  pre.display { font-family:inherit }
  pre.format  { font-family:inherit }
  pre.smalldisplay { font-family:inherit; font-size:smaller }
  pre.smallformat  { font-family:inherit; font-size:smaller }
  pre.smallexample { font-size:smaller }
  pre.smalllisp    { font-size:smaller }
  span.sc    { font-variant:small-caps }
  span.roman { font-family:serif; font-weight:normal; } 
  span.sansserif { font-family:sans-serif; font-weight:normal; } 
--></style>
</head>
<body>
<h1 class="settitle">BuildBot Manual 0.7.4</h1>
   <div class="contents">
<h2>Table of Contents</h2>
<ul>
<li><a name="toc_Top" href="#Top">BuildBot</a>
<li><a name="toc_Introduction" href="#Introduction">1 Introduction</a>
<ul>
<li><a href="#History-and-Philosophy">1.1 History and Philosophy</a>
<li><a href="#System-Architecture">1.2 System Architecture</a>
<ul>
<li><a href="#BuildSlave-Connections">1.2.1 BuildSlave Connections</a>
<li><a href="#Buildmaster-Architecture">1.2.2 Buildmaster Architecture</a>
<li><a href="#Status-Delivery-Architecture">1.2.3 Status Delivery Architecture</a>
</li></ul>
<li><a href="#Control-Flow">1.3 Control Flow</a>
</li></ul>
<li><a name="toc_Installation" href="#Installation">2 Installation</a>
<ul>
<li><a href="#Requirements">2.1 Requirements</a>
<li><a href="#Installing-the-code">2.2 Installing the code</a>
<li><a href="#Creating-a-buildmaster">2.3 Creating a buildmaster</a>
<li><a href="#Creating-a-buildslave">2.4 Creating a buildslave</a>
<ul>
<li><a href="#Buildslave-Options">2.4.1 Buildslave Options</a>
</li></ul>
<li><a href="#Launching-the-daemons">2.5 Launching the daemons</a>
<li><a href="#Logfiles">2.6 Logfiles</a>
<li><a href="#Shutdown">2.7 Shutdown</a>
<li><a href="#Maintenance">2.8 Maintenance</a>
<li><a href="#Troubleshooting">2.9 Troubleshooting</a>
<ul>
<li><a href="#Starting-the-buildslave">2.9.1 Starting the buildslave</a>
<li><a href="#Connecting-to-the-buildmaster">2.9.2 Connecting to the buildmaster</a>
<li><a href="#Forcing-Builds">2.9.3 Forcing Builds</a>
</li></ul>
</li></ul>
<li><a name="toc_Concepts" href="#Concepts">3 Concepts</a>
<ul>
<li><a href="#Version-Control-Systems">3.1 Version Control Systems</a>
<ul>
<li><a href="#Generalizing-VC-Systems">3.1.1 Generalizing VC Systems</a>
<li><a href="#Source-Tree-Specifications">3.1.2 Source Tree Specifications</a>
<li><a href="#How-Different-VC-Systems-Specify-Sources">3.1.3 How Different VC Systems Specify Sources</a>
<li><a href="#Attributes-of-Changes">3.1.4 Attributes of Changes</a>
</li></ul>
<li><a href="#Schedulers">3.2 Schedulers</a>
<li><a href="#BuildSet">3.3 BuildSet</a>
<li><a href="#BuildRequest">3.4 BuildRequest</a>
<li><a href="#Builder">3.5 Builder</a>
<li><a href="#Users">3.6 Users</a>
<ul>
<li><a href="#Doing-Things-With-Users">3.6.1 Doing Things With Users</a>
<li><a href="#Email-Addresses">3.6.2 Email Addresses</a>
<li><a href="#IRC-Nicknames">3.6.3 IRC Nicknames</a>
<li><a href="#Live-Status-Clients">3.6.4 Live Status Clients</a>
</li></ul>
</li></ul>
<li><a name="toc_Configuration" href="#Configuration">4 Configuration</a>
<ul>
<li><a href="#Config-File-Format">4.1 Config File Format</a>
<li><a href="#Loading-the-Config-File">4.2 Loading the Config File</a>
<li><a href="#Defining-the-Project">4.3 Defining the Project</a>
<li><a href="#Listing-Change-Sources-and-Schedulers">4.4 Listing Change Sources and Schedulers</a>
<ul>
<li><a href="#Scheduler-Types">4.4.1 Scheduler Types</a>
<li><a href="#Build-Dependencies">4.4.2 Build Dependencies</a>
</li></ul>
<li><a href="#Setting-the-slaveport">4.5 Setting the slaveport</a>
<li><a href="#Buildslave-Specifiers">4.6 Buildslave Specifiers</a>
<li><a href="#Defining-Builders">4.7 Defining Builders</a>
<li><a href="#Defining-Status-Targets">4.8 Defining Status Targets</a>
<li><a href="#Debug-options">4.9 Debug options</a>
</li></ul>
<li><a name="toc_Getting-Source-Code-Changes" href="#Getting-Source-Code-Changes">5 Getting Source Code Changes</a>
<ul>
<li><a href="#Change-Sources">5.1 Change Sources</a>
<ul>
<li><a href="#Choosing-ChangeSources">5.1.1 Choosing ChangeSources</a>
<li><a href="#CVSToys-_002d-PBService">5.1.2 CVSToys - PBService</a>
<li><a href="#CVSToys-_002d-mail-notification">5.1.3 CVSToys - mail notification</a>
<li><a href="#Other-mail-notification-ChangeSources">5.1.4 Other mail notification ChangeSources</a>
<li><a href="#PBChangeSource">5.1.5 PBChangeSource</a>
<li><a href="#P4Source">5.1.6 P4Source</a>
</li></ul>
</li></ul>
<li><a name="toc_Build-Process" href="#Build-Process">6 Build Process</a>
<ul>
<li><a href="#Build-Steps">6.1 Build Steps</a>
<ul>
<li><a href="#Common-Parameters">6.1.1 Common Parameters</a>
<li><a href="#Source-Checkout">6.1.2 Source Checkout</a>
<ul>
<li><a href="#CVS">6.1.2.1 CVS</a>
<li><a href="#SVN">6.1.2.2 SVN</a>
<li><a href="#Darcs">6.1.2.3 Darcs</a>
<li><a href="#Mercurial">6.1.2.4 Mercurial</a>
<li><a href="#Arch">6.1.2.5 Arch</a>
<li><a href="#Bazaar">6.1.2.6 Bazaar</a>
<li><a href="#P4">6.1.2.7 P4</a>
</li></ul>
<li><a href="#ShellCommand">6.1.3 ShellCommand</a>
<li><a href="#Simple-ShellCommand-Subclasses">6.1.4 Simple ShellCommand Subclasses</a>
<ul>
<li><a href="#Configure">6.1.4.1 Configure</a>
<li><a href="#Compile">6.1.4.2 Compile</a>
<li><a href="#Test">6.1.4.3 Test</a>
<li><a href="#Build-Properties">6.1.4.4 Build Properties</a>
</li></ul>
<li><a href="#Writing-New-BuildSteps">6.1.5 Writing New BuildSteps</a>
<ul>
<li><a href="#BuildStep-LogFiles">6.1.5.1 BuildStep LogFiles</a>
<li><a href="#Adding-LogObservers">6.1.5.2 Adding LogObservers</a>
<li><a href="#BuildStep-URLs">6.1.5.3 BuildStep URLs</a>
</li></ul>
</li></ul>
<li><a href="#Interlocks">6.2 Interlocks</a>
<li><a href="#Build-Factories">6.3 Build Factories</a>
<ul>
<li><a href="#BuildStep-Objects">6.3.1 BuildStep Objects</a>
<li><a href="#BuildFactory">6.3.2 BuildFactory</a>
<ul>
<li><a href="#BuildFactory-Attributes">6.3.2.1 BuildFactory Attributes</a>
<li><a href="#Quick-builds">6.3.2.2 Quick builds</a>
</li></ul>
<li><a href="#Process_002dSpecific-build-factories">6.3.3 Process-Specific build factories</a>
<ul>
<li><a href="#GNUAutoconf">6.3.3.1 GNUAutoconf</a>
<li><a href="#CPAN">6.3.3.2 CPAN</a>
<li><a href="#Python-distutils">6.3.3.3 Python distutils</a>
<li><a href="#Python_002fTwisted_002ftrial-projects">6.3.3.4 Python/Twisted/trial projects</a>
</li></ul>
</li></ul>
</li></ul>
<li><a name="toc_Status-Delivery" href="#Status-Delivery">7 Status Delivery</a>
<ul>
<li><a href="#HTML-Waterfall">7.1 HTML Waterfall</a>
<li><a href="#IRC-Bot">7.2 IRC Bot</a>
<li><a href="#PBListener">7.3 PBListener</a>
<li><a href="#Writing-New-Status-Plugins">7.4 Writing New Status Plugins</a>
</li></ul>
<li><a name="toc_Command_002dline-tool" href="#Command_002dline-tool">8 Command-line tool</a>
<ul>
<li><a href="#Administrator-Tools">8.1 Administrator Tools</a>
<li><a href="#Developer-Tools">8.2 Developer Tools</a>
<ul>
<li><a href="#statuslog">8.2.1 statuslog</a>
<li><a href="#statusgui">8.2.2 statusgui</a>
<li><a href="#try">8.2.3 try</a>
</li></ul>
<li><a href="#Other-Tools">8.3 Other Tools</a>
<ul>
<li><a href="#sendchange">8.3.1 sendchange</a>
<li><a href="#debugclient">8.3.2 debugclient</a>
</li></ul>
<li><a href="#_002ebuildbot-config-directory">8.4 .buildbot config directory</a>
</li></ul>
<li><a name="toc_Resources" href="#Resources">9 Resources</a>
<li><a name="toc_Developer_0027s-Appendix" href="#Developer_0027s-Appendix">Developer's Appendix</a>
<li><a name="toc_Index-of-Useful-Classes" href="#Index-of-Useful-Classes">Index of Useful Classes</a>
<li><a name="toc_Index-of-master_002ecfg-keys" href="#Index-of-master_002ecfg-keys">Index of master.cfg keys</a>
<li><a name="toc_Index" href="#Index">Index</a>
</li></ul>
</div>



<div class="node">
<p><hr>
<a name="Top"></a>
Next:&nbsp;<a rel="next" accesskey="n" href="#Introduction">Introduction</a>,
Previous:&nbsp;<a rel="previous" accesskey="p" href="#dir">(dir)</a>,
Up:&nbsp;<a rel="up" accesskey="u" href="#dir">(dir)</a>

</div>

<h2 class="unnumbered">BuildBot</h2>

<p>This is the BuildBot manual.

   <p>Copyright (C) 2005,2006 Brian Warner

   <p>Copying and distribution of this file, with or without
modification, are permitted in any medium without royalty
provided the copyright notice and this notice are preserved.

<ul class="menu">
<li><a accesskey="1" href="#Introduction">Introduction</a>:                 What the BuildBot does. 
<li><a accesskey="2" href="#Installation">Installation</a>:                 Creating a buildmaster and buildslaves,
                                running them. 
<li><a accesskey="3" href="#Concepts">Concepts</a>:                     What goes on in the buildbot's little mind. 
<li><a accesskey="4" href="#Configuration">Configuration</a>:                Controlling the buildbot. 
<li><a accesskey="5" href="#Getting-Source-Code-Changes">Getting Source Code Changes</a>:   Discovering when to run a build. 
<li><a accesskey="6" href="#Build-Process">Build Process</a>:                Controlling how each build is run. 
<li><a accesskey="7" href="#Status-Delivery">Status Delivery</a>:              Telling the world about the build's results. 
<li><a accesskey="8" href="#Command_002dline-tool">Command-line tool</a>
<li><a accesskey="9" href="#Resources">Resources</a>:                    Getting help. 
<li><a href="#Developer_0027s-Appendix">Developer's Appendix</a>
<li><a href="#Index-of-Useful-Classes">Index of Useful Classes</a>
<li><a href="#Index-of-master_002ecfg-keys">Index of master.cfg keys</a>
<li><a href="#Index">Index</a>:                        Complete index.

</li></ul>
<p>--- The Detailed Node Listing ---

<p>Introduction

</p>
<ul class="menu">
<li><a href="#History-and-Philosophy">History and Philosophy</a>
<li><a href="#System-Architecture">System Architecture</a>
<li><a href="#Control-Flow">Control Flow</a>

</li></ul>
<p>System Architecture

</p>
<ul class="menu">
<li><a href="#BuildSlave-Connections">BuildSlave Connections</a>
<li><a href="#Buildmaster-Architecture">Buildmaster Architecture</a>
<li><a href="#Status-Delivery-Architecture">Status Delivery Architecture</a>

</li></ul>
<p>Installation

</p>
<ul class="menu">
<li><a href="#Requirements">Requirements</a>
<li><a href="#Installing-the-code">Installing the code</a>
<li><a href="#Creating-a-buildmaster">Creating a buildmaster</a>
<li><a href="#Creating-a-buildslave">Creating a buildslave</a>
<li><a href="#Launching-the-daemons">Launching the daemons</a>
<li><a href="#Logfiles">Logfiles</a>
<li><a href="#Shutdown">Shutdown</a>
<li><a href="#Maintenance">Maintenance</a>
<li><a href="#Troubleshooting">Troubleshooting</a>

</li></ul>
<p>Creating a buildslave

</p>
<ul class="menu">
<li><a href="#Buildslave-Options">Buildslave Options</a>

</li></ul>
<p>Troubleshooting

</p>
<ul class="menu">
<li><a href="#Starting-the-buildslave">Starting the buildslave</a>
<li><a href="#Connecting-to-the-buildmaster">Connecting to the buildmaster</a>
<li><a href="#Forcing-Builds">Forcing Builds</a>

</li></ul>
<p>Concepts

</p>
<ul class="menu">
<li><a href="#Version-Control-Systems">Version Control Systems</a>
<li><a href="#Schedulers">Schedulers</a>
<li><a href="#BuildSet">BuildSet</a>
<li><a href="#BuildRequest">BuildRequest</a>
<li><a href="#Builder">Builder</a>
<li><a href="#Users">Users</a>

</li></ul>
<p>Version Control Systems

</p>
<ul class="menu">
<li><a href="#Generalizing-VC-Systems">Generalizing VC Systems</a>
<li><a href="#Source-Tree-Specifications">Source Tree Specifications</a>
<li><a href="#How-Different-VC-Systems-Specify-Sources">How Different VC Systems Specify Sources</a>
<li><a href="#Attributes-of-Changes">Attributes of Changes</a>

</li></ul>
<p>Users

</p>
<ul class="menu">
<li><a href="#Doing-Things-With-Users">Doing Things With Users</a>
<li><a href="#Email-Addresses">Email Addresses</a>
<li><a href="#IRC-Nicknames">IRC Nicknames</a>
<li><a href="#Live-Status-Clients">Live Status Clients</a>

</li></ul>
<p>Configuration

</p>
<ul class="menu">
<li><a href="#Config-File-Format">Config File Format</a>
<li><a href="#Loading-the-Config-File">Loading the Config File</a>
<li><a href="#Defining-the-Project">Defining the Project</a>
<li><a href="#Listing-Change-Sources-and-Schedulers">Listing Change Sources and Schedulers</a>
<li><a href="#Setting-the-slaveport">Setting the slaveport</a>
<li><a href="#Buildslave-Specifiers">Buildslave Specifiers</a>
<li><a href="#Defining-Builders">Defining Builders</a>
<li><a href="#Defining-Status-Targets">Defining Status Targets</a>
<li><a href="#Debug-options">Debug options</a>

</li></ul>
<p>Listing Change Sources and Schedulers

</p>
<ul class="menu">
<li><a href="#Scheduler-Types">Scheduler Types</a>
<li><a href="#Build-Dependencies">Build Dependencies</a>

</li></ul>
<p>Getting Source Code Changes

</p>
<ul class="menu">
<li><a href="#Change-Sources">Change Sources</a>

</li></ul>
<p>Change Sources

</p>
<ul class="menu">
<li><a href="#Choosing-ChangeSources">Choosing ChangeSources</a>
<li><a href="#CVSToys-_002d-PBService">CVSToys - PBService</a>
<li><a href="#CVSToys-_002d-mail-notification">CVSToys - mail notification</a>
<li><a href="#Other-mail-notification-ChangeSources">Other mail notification ChangeSources</a>
<li><a href="#PBChangeSource">PBChangeSource</a>
<li><a href="#P4Source">P4Source</a>

</li></ul>
<p>Build Process

</p>
<ul class="menu">
<li><a href="#Build-Steps">Build Steps</a>
<li><a href="#Interlocks">Interlocks</a>
<li><a href="#Build-Factories">Build Factories</a>

</li></ul>
<p>Build Steps

</p>
<ul class="menu">
<li><a href="#Common-Parameters">Common Parameters</a>
<li><a href="#Source-Checkout">Source Checkout</a>
<li><a href="#ShellCommand">ShellCommand</a>
<li><a href="#Simple-ShellCommand-Subclasses">Simple ShellCommand Subclasses</a>
<li><a href="#Writing-New-BuildSteps">Writing New BuildSteps</a>

</li></ul>
<p>Source Checkout

</p>
<ul class="menu">
<li><a href="#CVS">CVS</a>
<li><a href="#SVN">SVN</a>
<li><a href="#Darcs">Darcs</a>
<li><a href="#Mercurial">Mercurial</a>
<li><a href="#Arch">Arch</a>
<li><a href="#Bazaar">Bazaar</a>
<li><a href="#P4">P4</a>

</li></ul>
<p>Simple ShellCommand Subclasses

</p>
<ul class="menu">
<li><a href="#Configure">Configure</a>
<li><a href="#Compile">Compile</a>
<li><a href="#Test">Test</a>
<li><a href="#Build-Properties">Build Properties</a>

</li></ul>
<p>Writing New BuildSteps

</p>
<ul class="menu">
<li><a href="#BuildStep-LogFiles">BuildStep LogFiles</a>
<li><a href="#Adding-LogObservers">Adding LogObservers</a>
<li><a href="#BuildStep-URLs">BuildStep URLs</a>

</li></ul>
<p>Build Factories

</p>
<ul class="menu">
<li><a href="#BuildStep-Objects">BuildStep Objects</a>
<li><a href="#BuildFactory">BuildFactory</a>
<li><a href="#Process_002dSpecific-build-factories">Process-Specific build factories</a>

</li></ul>
<p>BuildStep Objects

</p>
<ul class="menu">
<li><a href="#BuildFactory-Attributes">BuildFactory Attributes</a>
<li><a href="#Quick-builds">Quick builds</a>

</li></ul>
<p>BuildFactory

</p>
<ul class="menu">
<li><a href="#BuildFactory-Attributes">BuildFactory Attributes</a>
<li><a href="#Quick-builds">Quick builds</a>

</li></ul>
<p>Process-Specific build factories

</p>
<ul class="menu">
<li><a href="#GNUAutoconf">GNUAutoconf</a>
<li><a href="#CPAN">CPAN</a>
<li><a href="#Python-distutils">Python distutils</a>
<li><a href="#Python_002fTwisted_002ftrial-projects">Python/Twisted/trial projects</a>

</li></ul>
<p>Status Delivery

</p>
<ul class="menu">
<li><a href="#HTML-Waterfall">HTML Waterfall</a>
<li><a href="#IRC-Bot">IRC Bot</a>
<li><a href="#PBListener">PBListener</a>
<li><a href="#Writing-New-Status-Plugins">Writing New Status Plugins</a>

</li></ul>
<p>Command-line tool

</p>
<ul class="menu">
<li><a href="#Administrator-Tools">Administrator Tools</a>
<li><a href="#Developer-Tools">Developer Tools</a>
<li><a href="#Other-Tools">Other Tools</a>
<li><a href="#g_t_002ebuildbot-config-directory">.buildbot config directory</a>

</li></ul>
<p>Developer Tools

</p>
<ul class="menu">
<li><a href="#statuslog">statuslog</a>
<li><a href="#statusgui">statusgui</a>
<li><a href="#try">try</a>

</li></ul>
<p>Other Tools

</p>
<ul class="menu">
<li><a href="#sendchange">sendchange</a>
<li><a href="#debugclient">debugclient</a>

   </ul>

<div class="node">
<p><hr>
<a name="Introduction"></a>
Next:&nbsp;<a rel="next" accesskey="n" href="#Installation">Installation</a>,
Previous:&nbsp;<a rel="previous" accesskey="p" href="#Top">Top</a>,
Up:&nbsp;<a rel="up" accesskey="u" href="#Top">Top</a>

</div>

<h2 class="chapter">1 Introduction</h2>

<p><a name="index-introduction-1"></a>
The BuildBot is a system to automate the compile/test cycle required by most
software projects to validate code changes. By automatically rebuilding and
testing the tree each time something has changed, build problems are
pinpointed quickly, before other developers are inconvenienced by the
failure. The guilty developer can be identified and harassed without human
intervention. By running the builds on a variety of platforms, developers
who do not have the facilities to test their changes everywhere before
checkin will at least know shortly afterwards whether they have broken the
build or not. Warning counts, lint checks, image size, compile time, and
other build parameters can be tracked over time, are more visible, and
are therefore easier to improve.

   <p>The overall goal is to reduce tree breakage and provide a platform to
run tests or code-quality checks that are too annoying or pedantic for
any human to waste their time with. Developers get immediate (and
potentially public) feedback about their changes, encouraging them to
be more careful about testing before checkin.

   <p>Features:

     <ul>
<li>run builds on a variety of slave platforms
<li>arbitrary build process: handles projects using C, Python, whatever
<li>minimal host requirements: python and Twisted
<li>slaves can be behind a firewall if they can still do checkout
<li>status delivery through web page, email, IRC, other protocols
<li>track builds in progress, provide estimated completion time
<li>flexible configuration by subclassing generic build process classes
<li>debug tools to force a new build, submit fake Changes, query slave status
<li>released under the GPL
</ul>

<ul class="menu">
<li><a accesskey="1" href="#History-and-Philosophy">History and Philosophy</a>
<li><a accesskey="2" href="#System-Architecture">System Architecture</a>
<li><a accesskey="3" href="#Control-Flow">Control Flow</a>
</ul>

<div class="node">
<p><hr>
<a name="History-and-Philosophy"></a>
Next:&nbsp;<a rel="next" accesskey="n" href="#System-Architecture">System Architecture</a>,
Previous:&nbsp;<a rel="previous" accesskey="p" href="#Introduction">Introduction</a>,
Up:&nbsp;<a rel="up" accesskey="u" href="#Introduction">Introduction</a>

</div>

<h3 class="section">1.1 History and Philosophy</h3>

<p><a name="index-Philosophy-of-operation-2"></a>
The Buildbot was inspired by a similar project built for a development
team writing a cross-platform embedded system. The various components
of the project were supposed to compile and run on several flavors of
unix (linux, solaris, BSD), but individual developers had their own
preferences and tended to stick to a single platform. From time to
time, incompatibilities would sneak in (some unix platforms want to
use <code>string.h</code>, some prefer <code>strings.h</code>), and then the tree
would compile for some developers but not others. The buildbot was
written to automate the human process of walking into the office,
updating a tree, compiling (and discovering the breakage), finding the
developer at fault, and complaining to them about the problem they had
introduced. With multiple platforms it was difficult for developers to
do the right thing (compile their potential change on all platforms);
the buildbot offered a way to help.

   <p>Another problem was when programmers would change the behavior of a
library without warning its users, or change internal aspects that
other code was (unfortunately) depending upon. Adding unit tests to
the codebase helps here: if an application's unit tests pass despite
changes in the libraries it uses, you can have more confidence that
the library changes haven't broken anything. Many developers
complained that the unit tests were inconvenient or took too long to
run: having the buildbot run them reduces the developer's workload to
a minimum.

   <p>In general, having more visibility into the project is always good,
and automation makes it easier for developers to do the right thing. 
When everyone can see the status of the project, developers are
encouraged to keep the tree in good working order. Unit tests that
aren't run on a regular basis tend to suffer from bitrot just like
code does: exercising them on a regular basis helps to keep them
functioning and useful.

   <p>The current version of the Buildbot is additionally targeted at
distributed free-software projects, where resources and platforms are
only available when provided by interested volunteers. The buildslaves
are designed to require an absolute minimum of configuration, reducing
the effort a potential volunteer needs to expend to be able to
contribute a new test environment to the project. The goal is for
anyone who wishes that a given project would run on their favorite
platform should be able to offer that project a buildslave, running on
that platform, where they can verify that their portability code
works, and keeps working.

<div class="node">
<p><hr>
<a name="System-Architecture"></a>
Next:&nbsp;<a rel="next" accesskey="n" href="#Control-Flow">Control Flow</a>,
Previous:&nbsp;<a rel="previous" accesskey="p" href="#History-and-Philosophy">History and Philosophy</a>,
Up:&nbsp;<a rel="up" accesskey="u" href="#Introduction">Introduction</a>

</div>

<!-- node-name,  next,  previous,  up -->
<h3 class="section">1.2 System Architecture</h3>

<p>The Buildbot consists of a single <code>buildmaster</code> and one or more
<code>buildslaves</code>, connected in a star topology. The buildmaster
makes all decisions about what, when, and how to build. It sends
commands to be run on the build slaves, which simply execute the
commands and return the results. (certain steps involve more local
decision making, where the overhead of sending a lot of commands back
and forth would be inappropriate, but in general the buildmaster is
responsible for everything).

   <p>The buildmaster is usually fed <code>Changes</code> by some sort of version
control system (see <a href="#Change-Sources">Change Sources</a>), which may cause builds to be
run. As the builds are performed, various status messages are
produced, which are then sent to any registered Status Targets
(see <a href="#Status-Delivery">Status Delivery</a>).

<!-- @image{FILENAME, WIDTH, HEIGHT, ALTTEXT, EXTENSION} -->
<div class="block-image"><img src="images/overview.png" alt="Overview Diagram"></div>

   <p>The buildmaster is configured and maintained by the &ldquo;buildmaster
admin&rdquo;, who is generally the project team member responsible for
build process issues. Each buildslave is maintained by a &ldquo;buildslave
admin&rdquo;, who do not need to be quite as involved. Generally slaves are
run by anyone who has an interest in seeing the project work well on
their favorite platform.

<ul class="menu">
<li><a accesskey="1" href="#BuildSlave-Connections">BuildSlave Connections</a>
<li><a accesskey="2" href="#Buildmaster-Architecture">Buildmaster Architecture</a>
<li><a accesskey="3" href="#Status-Delivery-Architecture">Status Delivery Architecture</a>
</ul>

<div class="node">
<p><hr>
<a name="BuildSlave-Connections"></a>
Next:&nbsp;<a rel="next" accesskey="n" href="#Buildmaster-Architecture">Buildmaster Architecture</a>,
Previous:&nbsp;<a rel="previous" accesskey="p" href="#System-Architecture">System Architecture</a>,
Up:&nbsp;<a rel="up" accesskey="u" href="#System-Architecture">System Architecture</a>

</div>

<h4 class="subsection">1.2.1 BuildSlave Connections</h4>

<p>The buildslaves are typically run on a variety of separate machines,
at least one per platform of interest. These machines connect to the
buildmaster over a TCP connection to a publically-visible port. As a
result, the buildslaves can live behind a NAT box or similar
firewalls, as long as they can get to buildmaster. The TCP connections
are initiated by the buildslave and accepted by the buildmaster, but
commands and results travel both ways within this connection. The
buildmaster is always in charge, so all commands travel exclusively
from the buildmaster to the buildslave.

   <p>To perform builds, the buildslaves must typically obtain source code
from a CVS/SVN/etc repository. Therefore they must also be able to
reach the repository. The buildmaster provides instructions for
performing builds, but does not provide the source code itself.

   <div class="block-image"><img src="images/slaves.png" alt="BuildSlave Connections"></div>

<div class="node">
<p><hr>
<a name="Buildmaster-Architecture"></a>
Next:&nbsp;<a rel="next" accesskey="n" href="#Status-Delivery-Architecture">Status Delivery Architecture</a>,
Previous:&nbsp;<a rel="previous" accesskey="p" href="#BuildSlave-Connections">BuildSlave Connections</a>,
Up:&nbsp;<a rel="up" accesskey="u" href="#System-Architecture">System Architecture</a>

</div>

<h4 class="subsection">1.2.2 Buildmaster Architecture</h4>

<p>The Buildmaster consists of several pieces:

   <div class="block-image"><img src="images/master.png" alt="BuildMaster Architecture"></div>

     <ul>
<li>Change Sources, which create a Change object each time something is
modified in the VC repository. Most ChangeSources listen for messages
from a hook script of some sort. Some sources actively poll the
repository on a regular basis. All Changes are fed to the Schedulers.

     <li>Schedulers, which decide when builds should be performed. They collect
Changes into BuildRequests, which are then queued for delivery to
Builders until a buildslave is available.

     <li>Builders, which control exactly <em>how</em> each build is performed
(with a series of BuildSteps, configured in a BuildFactory). Each
Build is run on a single buildslave.

     <li>Status plugins, which deliver information about the build results
through protocols like HTTP, mail, and IRC.

   </ul>

   <div class="block-image"><img src="images/slavebuilder.png" alt="SlaveBuilders"></div>

   <p>Each Builder is configured with a list of BuildSlaves that it will use
for its builds. These buildslaves are expected to behave identically:
the only reason to use multiple BuildSlaves for a single Builder is to
provide a measure of load-balancing.

   <p>Within a single BuildSlave, each Builder creates its own SlaveBuilder
instance. These SlaveBuilders operate independently from each other. 
Each gets its own base directory to work in. It is quite common to
have many Builders sharing the same buildslave. For example, there
might be two buildslaves: one for i386, and a second for PowerPC. 
There may then be a pair of Builders that do a full compile/test run,
one for each architecture, and a lone Builder that creates snapshot
source tarballs if the full builders complete successfully. The full
builders would each run on a single buildslave, whereas the tarball
creation step might run on either buildslave (since the platform
doesn't matter when creating source tarballs). In this case, the
mapping would look like:

<pre class="example">     Builder(full-i386)  -&gt;  BuildSlaves(slave-i386)
     Builder(full-ppc)   -&gt;  BuildSlaves(slave-ppc)
     Builder(source-tarball) -&gt; BuildSlaves(slave-i386, slave-ppc)
</pre>
   <p>and each BuildSlave would have two SlaveBuilders inside it, one for a
full builder, and a second for the source-tarball builder.

   <p>Once a SlaveBuilder is available, the Builder pulls one or more
BuildRequests off its incoming queue. (It may pull more than one if it
determines that it can merge the requests together; for example, there
may be multiple requests to build the current HEAD revision). These
requests are merged into a single Build instance, which includes the
SourceStamp that describes what exact version of the source code
should be used for the build. The Build is then assigned to a
SlaveBuilder and the build begins.

<div class="node">
<p><hr>
<a name="Status-Delivery-Architecture"></a>
Previous:&nbsp;<a rel="previous" accesskey="p" href="#Buildmaster-Architecture">Buildmaster Architecture</a>,
Up:&nbsp;<a rel="up" accesskey="u" href="#System-Architecture">System Architecture</a>

</div>

<h4 class="subsection">1.2.3 Status Delivery Architecture</h4>

<p>The buildmaster maintains a central Status object, to which various
status plugins are connected. Through this Status object, a full
hierarchy of build status objects can be obtained.

   <div class="block-image"><img src="images/status.png" alt="Status Delivery"></div>

   <p>The configuration file controls which status plugins are active. Each
status plugin gets a reference to the top-level Status object. From
there they can request information on each Builder, Build, Step, and
LogFile. This query-on-demand interface is used by the html.Waterfall
plugin to create the main status page each time a web browser hits the
main URL.

   <p>The status plugins can also subscribe to hear about new Builds as they
occur: this is used by the MailNotifier to create new email messages
for each recently-completed Build.

   <p>The Status object records the status of old builds on disk in the
buildmaster's base directory. This allows it to return information
about historical builds.

   <p>There are also status objects that correspond to Schedulers and
BuildSlaves. These allow status plugins to report information about
upcoming builds, and the online/offline status of each buildslave.

<div class="node">
<p><hr>
<a name="Control-Flow"></a>
Previous:&nbsp;<a rel="previous" accesskey="p" href="#System-Architecture">System Architecture</a>,
Up:&nbsp;<a rel="up" accesskey="u" href="#Introduction">Introduction</a>

</div>

<!-- node-name,  next,  previous,  up -->
<h3 class="section">1.3 Control Flow</h3>

<p>A day in the life of the buildbot:

     <ul>
<li>A developer commits some source code changes to the repository. A hook
script or commit trigger of some sort sends information about this
change to the buildmaster through one of its configured Change
Sources. This notification might arrive via email, or over a network
connection (either initiated by the buildmaster as it &ldquo;subscribes&rdquo;
to changes, or by the commit trigger as it pushes Changes towards the
buildmaster). The Change contains information about who made the
change, what files were modified, which revision contains the change,
and any checkin comments.

     <li>The buildmaster distributes this change to all of its configured
Schedulers. Any &ldquo;important&rdquo; changes cause the &ldquo;tree-stable-timer&rdquo;
to be started, and the Change is added to a list of those that will go
into a new Build. When the timer expires, a Build is started on each
of a set of configured Builders, all compiling/testing the same source
code. Unless configured otherwise, all Builds run in parallel on the
various buildslaves.

     <li>The Build consists of a series of Steps. Each Step causes some number
of commands to be invoked on the remote buildslave associated with
that Builder. The first step is almost always to perform a checkout of
the appropriate revision from the same VC system that produced the
Change. The rest generally perform a compile and run unit tests. As
each Step runs, the buildslave reports back command output and return
status to the buildmaster.

     <li>As the Build runs, status messages like &ldquo;Build Started&rdquo;, &ldquo;Step
Started&rdquo;, &ldquo;Build Finished&rdquo;, etc, are published to a collection of
Status Targets. One of these targets is usually the HTML &ldquo;Waterfall&rdquo;
display, which shows a chronological list of events, and summarizes
the results of the most recent build at the top of each column. 
Developers can periodically check this page to see how their changes
have fared. If they see red, they know that they've made a mistake and
need to fix it. If they see green, they know that they've done their
duty and don't need to worry about their change breaking anything.

     <li>If a MailNotifier status target is active, the completion of a build
will cause email to be sent to any developers whose Changes were
incorporated into this Build. The MailNotifier can be configured to
only send mail upon failing builds, or for builds which have just
transitioned from passing to failing. Other status targets can provide
similar real-time notification via different communication channels,
like IRC.

   </ul>

<div class="node">
<p><hr>
<a name="Installation"></a>
Next:&nbsp;<a rel="next" accesskey="n" href="#Concepts">Concepts</a>,
Previous:&nbsp;<a rel="previous" accesskey="p" href="#Introduction">Introduction</a>,
Up:&nbsp;<a rel="up" accesskey="u" href="#Top">Top</a>

</div>

<h2 class="chapter">2 Installation</h2>

<ul class="menu">
<li><a accesskey="1" href="#Requirements">Requirements</a>
<li><a accesskey="2" href="#Installing-the-code">Installing the code</a>
<li><a accesskey="3" href="#Creating-a-buildmaster">Creating a buildmaster</a>
<li><a accesskey="4" href="#Creating-a-buildslave">Creating a buildslave</a>
<li><a accesskey="5" href="#Launching-the-daemons">Launching the daemons</a>
<li><a accesskey="6" href="#Logfiles">Logfiles</a>
<li><a accesskey="7" href="#Shutdown">Shutdown</a>
<li><a accesskey="8" href="#Maintenance">Maintenance</a>
<li><a accesskey="9" href="#Troubleshooting">Troubleshooting</a>
</ul>

<div class="node">
<p><hr>
<a name="Requirements"></a>
Next:&nbsp;<a rel="next" accesskey="n" href="#Installing-the-code">Installing the code</a>,
Previous:&nbsp;<a rel="previous" accesskey="p" href="#Installation">Installation</a>,
Up:&nbsp;<a rel="up" accesskey="u" href="#Installation">Installation</a>

</div>

<h3 class="section">2.1 Requirements</h3>

<p>At a bare minimum, you'll need the following (for both the buildmaster
and a buildslave):

     <ul>
<li>Python: http://www.python.org

     <p>Buildbot requires python-2.2 or later, and is primarily developed
against python-2.3. The buildmaster uses generators, a feature which
is not available in python-2.1, and both master and slave require a
version of Twisted which only works with python-2.2 or later. Certain
features (like the inclusion of build logs in status emails) require
python-2.2.2 or later. The IRC &ldquo;force build&rdquo; command requires
python-2.3 (for the shlex.split function).

     <li>Twisted: http://twistedmatrix.com

     <p>Both the buildmaster and the buildslaves require Twisted-1.3.0 or
later. It has been mainly developed against Twisted-2.0.1, but has
been tested against Twisted-2.1.0 (the most recent as of this
writing), and might even work on versions as old as Twisted-1.1.0, but
as always the most recent version is recommended.

     <p>Twisted-1.3.0 and earlier were released as a single monolithic
package. When you run Buildbot against Twisted-2.0.0 or later (which
are split into a number of smaller subpackages), you'll need at least
"Twisted" (the core package), and you'll also want TwistedMail,
TwistedWeb, and TwistedWords (for sending email, serving a web status
page, and delivering build status via IRC, respectively). 
</ul>

   <p>Certain other packages may be useful on the system running the
buildmaster:

     <ul>
<li>CVSToys: http://purl.net/net/CVSToys

     <p>If your buildmaster uses FreshCVSSource to receive change notification
from a cvstoys daemon, it will require CVSToys be installed (tested
with CVSToys-1.0.10). If the it doesn't use that source (i.e. if you
only use a mail-parsing change source, or the SVN notification
script), you will not need CVSToys.

   </ul>

   <p>And of course, your project's build process will impose additional
requirements on the buildslaves. These hosts must have all the tools
necessary to compile and test your project's source code.

<div class="node">
<p><hr>
<a name="Installing-the-code"></a>
Next:&nbsp;<a rel="next" accesskey="n" href="#Creating-a-buildmaster">Creating a buildmaster</a>,
Previous:&nbsp;<a rel="previous" accesskey="p" href="#Requirements">Requirements</a>,
Up:&nbsp;<a rel="up" accesskey="u" href="#Installation">Installation</a>

</div>

<h3 class="section">2.2 Installing the code</h3>

<p><a name="index-installation-3"></a>
The Buildbot is installed using the standard python <code>distutils</code>
module. After unpacking the tarball, the process is:

<pre class="example">     python setup.py build
     python setup.py install
</pre>
   <p>where the install step may need to be done as root. This will put the
bulk of the code in somewhere like
/usr/lib/python2.3/site-packages/buildbot . It will also install the
<code>buildbot</code> command-line tool in /usr/bin/buildbot.

   <p>To test this, shift to a different directory (like /tmp), and run:

<pre class="example">     buildbot --version
</pre>
   <p>If it shows you the versions of Buildbot and Twisted, the install went
ok. If it says <code>no such command</code> or it gets an <code>ImportError</code>
when it tries to load the libaries, then something went wrong. 
<code>pydoc buildbot</code> is another useful diagnostic tool.

   <p>Windows users will find these files in other places. You will need to
make sure that python can find the libraries, and will probably find
it convenient to have <code>buildbot</code> on your PATH.

   <p>If you wish, you can run the buildbot unit test suite like this:

<pre class="example">     PYTHONPATH=. trial buildbot.test
</pre>
   <p>This should run up to 192 tests, depending upon what VC tools you have
installed. On my desktop machine it takes about five minutes to
complete. Nothing should fail, a few might be skipped. If any of the
tests fail, you should stop and investigate the cause before
continuing the installation process, as it will probably be easier to
track down the bug early.

   <p>If you cannot or do not wish to install the buildbot into a site-wide
location like <samp><span class="file">/usr</span></samp> or <samp><span class="file">/usr/local</span></samp>, you can also install
it into the account's home directory. Do the install command like
this:

<pre class="example">     python setup.py install --home=~
</pre>
   <p>That will populate <samp><span class="file">~/lib/python</span></samp> and create
<samp><span class="file">~/bin/buildbot</span></samp>. Make sure this lib directory is on your
<code>PYTHONPATH</code>.

<div class="node">
<p><hr>
<a name="Creating-a-buildmaster"></a>
Next:&nbsp;<a rel="next" accesskey="n" href="#Creating-a-buildslave">Creating a buildslave</a>,
Previous:&nbsp;<a rel="previous" accesskey="p" href="#Installing-the-code">Installing the code</a>,
Up:&nbsp;<a rel="up" accesskey="u" href="#Installation">Installation</a>

</div>

<h3 class="section">2.3 Creating a buildmaster</h3>

<p>As you learned earlier (see <a href="#System-Architecture">System Architecture</a>), the buildmaster
runs on a central host (usually one that is publically visible, so
everybody can check on the status of the project), and controls all
aspects of the buildbot system. Let us call this host
<code>buildbot.example.org</code>.

   <p>You may wish to create a separate user account for the buildmaster,
perhaps named <code>buildmaster</code>. This can help keep your personal
configuration distinct from that of the buildmaster and is useful if
you have to use a mail-based notification system (see <a href="#Change-Sources">Change Sources</a>). However, the Buildbot will work just fine with your regular
user account.

   <p>You need to choose a directory for the buildmaster, called the
<code>basedir</code>. This directory will be owned by the buildmaster, which
will use configuration files therein, and create status files as it
runs. <samp><span class="file">~/Buildbot</span></samp> is a likely value. If you run multiple
buildmasters in the same account, or if you run both masters and
slaves, you may want a more distinctive name like
<samp><span class="file">~/Buildbot/master/gnomovision</span></samp> or
<samp><span class="file">~/Buildmasters/fooproject</span></samp>. If you are using a separate user
account, this might just be <samp><span class="file">~buildmaster/masters/fooproject</span></samp>.

   <p>Once you've picked a directory, use the <samp><span class="command">buildbot
create-master</span></samp> command to create the directory and populate it with
startup files:

<pre class="example">     buildbot create-master <var>basedir</var>
</pre>
   <p>You will need to create a configuration file (see <a href="#Configuration">Configuration</a>)
before starting the buildmaster. Most of the rest of this manual is
dedicated to explaining how to do this. A sample configuration file is
placed in the working directory, named <samp><span class="file">master.cfg.sample</span></samp>, which
can be copied to <samp><span class="file">master.cfg</span></samp> and edited to suit your purposes.

   <p>(Internal details: This command creates a file named
<samp><span class="file">buildbot.tac</span></samp> that contains all the state necessary to create
the buildmaster. Twisted has a tool called <code>twistd</code> which can use
this .tac file to create and launch a buildmaster instance. twistd
takes care of logging and daemonization (running the program in the
background). <samp><span class="file">/usr/bin/buildbot</span></samp> is a front end which runs twistd
for you.)

   <p>In addition to <samp><span class="file">buildbot.tac</span></samp>, a small <samp><span class="file">Makefile.sample</span></samp> is
installed. This can be used as the basis for customized daemon startup,
See <a href="#Launching-the-daemons">Launching the daemons</a>.

<div class="node">
<p><hr>
<a name="Creating-a-buildslave"></a>
Next:&nbsp;<a rel="next" accesskey="n" href="#Launching-the-daemons">Launching the daemons</a>,
Previous:&nbsp;<a rel="previous" accesskey="p" href="#Creating-a-buildmaster">Creating a buildmaster</a>,
Up:&nbsp;<a rel="up" accesskey="u" href="#Installation">Installation</a>

</div>

<h3 class="section">2.4 Creating a buildslave</h3>

<p>Typically, you will be adding a buildslave to an existing buildmaster,
to provide additional architecture coverage. The buildbot
administrator will give you several pieces of information necessary to
connect to the buildmaster. You should also be somewhat familiar with
the project being tested, so you can troubleshoot build problems
locally.

   <p>The buildbot exists to make sure that the project's stated &ldquo;how to
build it&rdquo; process actually works. To this end, the buildslave should
run in an environment just like that of your regular developers. 
Typically the project build process is documented somewhere
(<samp><span class="file">README</span></samp>, <samp><span class="file">INSTALL</span></samp>, etc), in a document that should
mention all library dependencies and contain a basic set of build
instructions. This document will be useful as you configure the host
and account in which the buildslave runs.

   <p>Here's a good checklist for setting up a buildslave:

     <ol type=1 start=1>
<li>Set up the account

     <p>It is recommended (although not mandatory) to set up a separate user
account for the buildslave. This account is frequently named
<code>buildbot</code> or <code>buildslave</code>. This serves to isolate your
personal working environment from that of the slave's, and helps to
minimize the security threat posed by letting possibly-unknown
contributors run arbitrary code on your system. The account should
have a minimum of fancy init scripts.

     <li>Install the buildbot code

     <p>Follow the instructions given earlier (see <a href="#Installing-the-code">Installing the code</a>). 
If you use a separate buildslave account, and you didn't install the
buildbot code to a shared location, then you will need to install it
with <code>--home=~</code> for each account that needs it.

     <li>Set up the host

     <p>Make sure the host can actually reach the buildmaster. Usually the
buildmaster is running a status webserver on the same machine, so
simply point your web browser at it and see if you can get there. 
Install whatever additional packages or libraries the project's
INSTALL document advises. (or not: if your buildslave is supposed to
make sure that building without optional libraries still works, then
don't install those libraries).

     <p>Again, these libraries don't necessarily have to be installed to a
site-wide shared location, but they must be available to your build
process. Accomplishing this is usually very specific to the build
process, so installing them to <samp><span class="file">/usr</span></samp> or <samp><span class="file">/usr/local</span></samp> is
usually the best approach.

     <li>Test the build process

     <p>Follow the instructions in the INSTALL document, in the buildslave's
account. Perform a full CVS (or whatever) checkout, configure, make,
run tests, etc. Confirm that the build works without manual fussing. 
If it doesn't work when you do it by hand, it will be unlikely to work
when the buildbot attempts to do it in an automated fashion.

     <li>Choose a base directory

     <p>This should be somewhere in the buildslave's account, typically named
after the project which is being tested. The buildslave will not touch
any file outside of this directory. Something like <samp><span class="file">~/Buildbot</span></samp>
or <samp><span class="file">~/Buildslaves/fooproject</span></samp> is appropriate.

     <li>Get the buildmaster host/port, botname, and password

     <p>When the buildbot admin configures the buildmaster to accept and use
your buildslave, they will provide you with the following pieces of
information:

          <ul>
<li>your buildslave's name
<li>the password assigned to your buildslave
<li>the hostname and port number of the buildmaster, i.e. buildbot.example.org:8007
</ul>

     <li>Create the buildslave

     <p>Now run the 'buildbot' command as follows:

     <pre class="example">          buildbot create-slave <var>BASEDIR</var> <var>MASTERHOST</var>:<var>PORT</var> <var>SLAVENAME</var> <var>PASSWORD</var>
     </pre>
     <p>This will create the base directory and a collection of files inside,
including the <samp><span class="file">buildbot.tac</span></samp> file that contains all the
information you passed to the <code>buildbot</code> command.

     <li>Fill in the hostinfo files

     <p>When it first connects, the buildslave will send a few files up to the
buildmaster which describe the host that it is running on. These files
are presented on the web status display so that developers have more
information to reproduce any test failures that are witnessed by the
buildbot. There are sample files in the <samp><span class="file">info</span></samp> subdirectory of
the buildbot's base directory. You should edit these to correctly
describe you and your host.

     <p><samp><span class="file">BASEDIR/info/admin</span></samp> should contain your name and email address. 
This is the &ldquo;buildslave admin address&rdquo;, and will be visible from the
build status page (so you may wish to munge it a bit if
address-harvesting spambots are a concern).

     <p><samp><span class="file">BASEDIR/info/host</span></samp> should be filled with a brief description of
the host: OS, version, memory size, CPU speed, versions of relevant
libraries installed, and finally the version of the buildbot code
which is running the buildslave.

     <p>If you run many buildslaves, you may want to create a single
<samp><span class="file">~buildslave/info</span></samp> file and share it among all the buildslaves
with symlinks.

        </ol>

<ul class="menu">
<li><a accesskey="1" href="#Buildslave-Options">Buildslave Options</a>
</ul>

<div class="node">
<p><hr>
<a name="Buildslave-Options"></a>
Previous:&nbsp;<a rel="previous" accesskey="p" href="#Creating-a-buildslave">Creating a buildslave</a>,
Up:&nbsp;<a rel="up" accesskey="u" href="#Creating-a-buildslave">Creating a buildslave</a>

</div>

<h4 class="subsection">2.4.1 Buildslave Options</h4>

<p>There are a handful of options you might want to use when creating the
buildslave with the <samp><span class="command">buildbot create-slave &lt;options&gt; DIR &lt;params&gt;</span></samp>
command. You can type <samp><span class="command">buildbot create-slave --help</span></samp> for a summary. 
To use these, just include them on the <samp><span class="command">buildbot create-slave</span></samp>
command line, like this:

<pre class="example">     buildbot create-slave --umask=022 ~/buildslave buildmaster.example.org:42012 myslavename mypasswd
</pre>
     <dl>
<dt><code>--usepty</code><dd>This is a boolean flag that tells the buildslave whether to launch
child processes in a PTY (the default) or with regular pipes. The
advantage of using a PTY is that &ldquo;grandchild&rdquo; processes are more
likely to be cleaned up if the build is interrupted or times out
(since it enables the use of a &ldquo;process group&rdquo; in which all child
processes will be placed). The disadvantages: some forms of Unix have
problems with PTYs, some of your unit tests may behave differently
when run under a PTY (generally those which check to see if they are
being run interactively), and PTYs will merge the stdout and stderr
streams into a single output stream (which means the red-vs-black
coloring in the logfiles will be lost). If you encounter problems, you
can add <code>--usepty=0</code> to disable the use of PTYs. Note that
windows buildslaves never use PTYs.

     <br><dt><code>--umask</code><dd>This is a string (generally an octal representation of an integer)
which will cause the buildslave process' &ldquo;umask&rdquo; value to be set
shortly after initialization. The &ldquo;twistd&rdquo; daemonization utility
forces the umask to 077 at startup (which means that all files created
by the buildslave or its child processes will be unreadable by any
user other than the buildslave account). If you want build products to
be readable by other accounts, you can add <code>--umask=022</code> to tell
the buildslave to fix the umask after twistd clobbers it. If you want
build products to be <em>writable</em> by other accounts too, use
<code>--umask=000</code>, but this is likely to be a security problem.

     <br><dt><code>--keepalive</code><dd>This is a number that indicates how frequently &ldquo;keepalive&rdquo; messages
should be sent from the buildslave to the buildmaster, expressed in
seconds. The default (600) causes a message to be sent to the
buildmaster at least once every 10 minutes. To set this to a lower
value, use e.g. <code>--keepalive=120</code>.

     <p>If the buildslave is behind a NAT box or stateful firewall, these
messages may help to keep the connection alive: some NAT boxes tend to
forget about a connection if it has not been used in a while. When
this happens, the buildmaster will think that the buildslave has
disappeared, and builds will time out. Meanwhile the buildslave will
not realize than anything is wrong.

   </dl>

<div class="node">
<p><hr>
<a name="Launching-the-daemons"></a>
Next:&nbsp;<a rel="next" accesskey="n" href="#Logfiles">Logfiles</a>,
Previous:&nbsp;<a rel="previous" accesskey="p" href="#Creating-a-buildslave">Creating a buildslave</a>,
Up:&nbsp;<a rel="up" accesskey="u" href="#Installation">Installation</a>

</div>

<h3 class="section">2.5 Launching the daemons</h3>

<p>Both the buildmaster and the buildslave run as daemon programs. To
launch them, pass the working directory to the <code>buildbot</code>
command:

<pre class="example">     buildbot start <var>BASEDIR</var>
</pre>
   <p>This command will start the daemon and then return, so normally it
will not produce any output. To verify that the programs are indeed
running, look for a pair of files named <samp><span class="file">twistd.log</span></samp> and
<samp><span class="file">twistd.pid</span></samp> that should be created in the working directory. 
<samp><span class="file">twistd.pid</span></samp> contains the process ID of the newly-spawned daemon.

   <p>When the buildslave connects to the buildmaster, new directories will
start appearing in its base directory. The buildmaster tells the slave
to create a directory for each Builder which will be using that slave. 
All build operations are performed within these directories: CVS
checkouts, compiles, and tests.

   <p>Once you get everything running, you will want to arrange for the
buildbot daemons to be started at boot time. One way is to use
<code>cron</code>, by putting them in a @reboot crontab entry<a rel="footnote" href="#fn-1" name="fnd-1"><sup>1</sup></a>:

<pre class="example">     @reboot buildbot start <var>BASEDIR</var>
</pre>
   <p>When you run <samp><span class="command">crontab</span></samp> to set this up, remember to do it as
the buildmaster or buildslave account! If you add this to your crontab
when running as your regular account (or worse yet, root), then the
daemon will run as the wrong user, quite possibly as one with more
authority than you intended to provide.

   <p>It is important to remember that the environment provided to cron jobs
and init scripts can be quite different that your normal runtime. 
There may be fewer environment variables specified, and the PATH may
be shorter than usual. It is a good idea to test out this method of
launching the buildslave by using a cron job with a time in the near
future, with the same command, and then check <samp><span class="file">twistd.log</span></samp> to
make sure the slave actually started correctly. Common problems here
are for <samp><span class="file">/usr/local</span></samp> or <samp><span class="file">~/bin</span></samp> to not be on your
<code>PATH</code>, or for <code>PYTHONPATH</code> to not be set correctly. 
Sometimes <code>HOME</code> is messed up too.

   <p>To modify the way the daemons are started (perhaps you want to set
some environment variables first, or perform some cleanup each time),
you can create a file named <samp><span class="file">Makefile.buildbot</span></samp> in the base
directory. When the <samp><span class="file">buildbot</span></samp> front-end tool is told to
<samp><span class="command">start</span></samp> the daemon, and it sees this file (and
<samp><span class="file">/usr/bin/make</span></samp> exists), it will do <samp><span class="command">make -f
Makefile.buildbot start</span></samp> instead of its usual action (which involves
running <samp><span class="command">twistd</span></samp>). When the buildmaster or buildslave is
installed, a <samp><span class="file">Makefile.sample</span></samp> is created which implements the
same behavior as the the <samp><span class="file">buildbot</span></samp> tool uses, so if you want to
customize the process, just copy <samp><span class="file">Makefile.sample</span></samp> to
<samp><span class="file">Makefile.buildbot</span></samp> and edit it as necessary.

<div class="node">
<p><hr>
<a name="Logfiles"></a>
Next:&nbsp;<a rel="next" accesskey="n" href="#Shutdown">Shutdown</a>,
Previous:&nbsp;<a rel="previous" accesskey="p" href="#Launching-the-daemons">Launching the daemons</a>,
Up:&nbsp;<a rel="up" accesskey="u" href="#Installation">Installation</a>

</div>

<h3 class="section">2.6 Logfiles</h3>

<p><a name="index-logfiles-4"></a>
While a buildbot daemon runs, it emits text to a logfile, named
<samp><span class="file">twistd.log</span></samp>. A command like <code>tail -f twistd.log</code> is useful
to watch the command output as it runs.

   <p>The buildmaster will announce any errors with its configuration file
in the logfile, so it is a good idea to look at the log at startup
time to check for any problems. Most buildmaster activities will cause
lines to be added to the log.

<div class="node">
<p><hr>
<a name="Shutdown"></a>
Next:&nbsp;<a rel="next" accesskey="n" href="#Maintenance">Maintenance</a>,
Previous:&nbsp;<a rel="previous" accesskey="p" href="#Logfiles">Logfiles</a>,
Up:&nbsp;<a rel="up" accesskey="u" href="#Installation">Installation</a>

</div>

<h3 class="section">2.7 Shutdown</h3>

<p>To stop a buildmaster or buildslave manually, use:

<pre class="example">     buildbot stop <var>BASEDIR</var>
</pre>
   <p>This simply looks for the <samp><span class="file">twistd.pid</span></samp> file and kills whatever
process is identified within.

   <p>At system shutdown, all processes are sent a <code>SIGKILL</code>. The
buildmaster and buildslave will respond to this by shutting down
normally.

   <p>The buildmaster will respond to a <code>SIGHUP</code> by re-reading its
config file. The following shortcut is available:

<pre class="example">     buildbot sighup <var>BASEDIR</var>
</pre>
   <p>When you update the Buildbot code to a new release, you will need to
restart the buildmaster and/or buildslave before it can take advantage
of the new code. You can do a <code>buildbot stop </code><var>BASEDIR</var> and
<code>buildbot start </code><var>BASEDIR</var> in quick succession, or you can
use the <code>restart</code> shortcut, which does both steps for you:

<pre class="example">     buildbot restart <var>BASEDIR</var>
</pre>
   <div class="node">
<p><hr>
<a name="Maintenance"></a>
Next:&nbsp;<a rel="next" accesskey="n" href="#Troubleshooting">Troubleshooting</a>,
Previous:&nbsp;<a rel="previous" accesskey="p" href="#Shutdown">Shutdown</a>,
Up:&nbsp;<a rel="up" accesskey="u" href="#Installation">Installation</a>

</div>

<h3 class="section">2.8 Maintenance</h3>

<p>It is a good idea to check the buildmaster's status page every once in
a while, to see if your buildslave is still online. Eventually the
buildbot will probably be enhanced to send you email (via the
<samp><span class="file">info/admin</span></samp> email address) when the slave has been offline for
more than a few hours.

   <p>If you find you can no longer provide a buildslave to the project, please
let the project admins know, so they can put out a call for a
replacement.

   <p>The Buildbot records status and logs output continually, each time a
build is performed. The status tends to be small, but the build logs
can become quite large. Each build and log are recorded in a separate
file, arranged hierarchically under the buildmaster's base directory. 
To prevent these files from growing without bound, you should
periodically delete old build logs. A simple cron job to delete
anything older than, say, two weeks should do the job. The only trick
is to leave the <samp><span class="file">buildbot.tac</span></samp> and other support files alone, for
which find's <code>-mindepth</code> argument helps skip everything in the
top directory. You can use something like the following:

<pre class="example">     @weekly cd BASEDIR &amp;&amp; find . -mindepth 2 -type f -mtime +14 -exec rm {} \;
     @weekly cd BASEDIR &amp;&amp; find twistd.log* -mtime +14 -exec rm {} \;
</pre>
   <div class="node">
<p><hr>
<a name="Troubleshooting"></a>
Previous:&nbsp;<a rel="previous" accesskey="p" href="#Maintenance">Maintenance</a>,
Up:&nbsp;<a rel="up" accesskey="u" href="#Installation">Installation</a>

</div>

<h3 class="section">2.9 Troubleshooting</h3>

<p>Here are a few hints on diagnosing common problems.

<ul class="menu">
<li><a accesskey="1" href="#Starting-the-buildslave">Starting the buildslave</a>
<li><a accesskey="2" href="#Connecting-to-the-buildmaster">Connecting to the buildmaster</a>
<li><a accesskey="3" href="#Forcing-Builds">Forcing Builds</a>
</ul>

<div class="node">
<p><hr>
<a name="Starting-the-buildslave"></a>
Next:&nbsp;<a rel="next" accesskey="n" href="#Connecting-to-the-buildmaster">Connecting to the buildmaster</a>,
Previous:&nbsp;<a rel="previous" accesskey="p" href="#Troubleshooting">Troubleshooting</a>,
Up:&nbsp;<a rel="up" accesskey="u" href="#Troubleshooting">Troubleshooting</a>

</div>

<h4 class="subsection">2.9.1 Starting the buildslave</h4>

<p>Cron jobs are typically run with a minimal shell (<samp><span class="file">/bin/sh</span></samp>, not
<samp><span class="file">/bin/bash</span></samp>), and tilde expansion is not always performed in such
commands. You may want to use explicit paths, because the <code>PATH</code>
is usually quite short and doesn't include anything set by your
shell's startup scripts (<samp><span class="file">.profile</span></samp>, <samp><span class="file">.bashrc</span></samp>, etc). If
you've installed buildbot (or other python libraries) to an unusual
location, you may need to add a <code>PYTHONPATH</code> specification (note
that python will do tilde-expansion on <code>PYTHONPATH</code> elements by
itself). Sometimes it is safer to fully-specify everything:

<pre class="example">     @reboot PYTHONPATH=~/lib/python /usr/local/bin/buildbot start /usr/home/buildbot/basedir
</pre>
   <p>Take the time to get the @reboot job set up. Otherwise, things will work
fine for a while, but the first power outage or system reboot you have will
stop the buildslave with nothing but the cries of sorrowful developers to
remind you that it has gone away.

<div class="node">
<p><hr>
<a name="Connecting-to-the-buildmaster"></a>
Next:&nbsp;<a rel="next" accesskey="n" href="#Forcing-Builds">Forcing Builds</a>,
Previous:&nbsp;<a rel="previous" accesskey="p" href="#Starting-the-buildslave">Starting the buildslave</a>,
Up:&nbsp;<a rel="up" accesskey="u" href="#Troubleshooting">Troubleshooting</a>

</div>

<h4 class="subsection">2.9.2 Connecting to the buildmaster</h4>

<p>If the buildslave cannot connect to the buildmaster, the reason should
be described in the <samp><span class="file">twistd.log</span></samp> logfile. Some common problems
are an incorrect master hostname or port number, or a mistyped bot
name or password. If the buildslave loses the connection to the
master, it is supposed to attempt to reconnect with an
exponentially-increasing backoff. Each attempt (and the time of the
next attempt) will be logged. If you get impatient, just manually stop
and re-start the buildslave.

   <p>When the buildmaster is restarted, all slaves will be disconnected,
and will attempt to reconnect as usual. The reconnect time will depend
upon how long the buildmaster is offline (i.e. how far up the
exponential backoff curve the slaves have travelled). Again,
<code>buildbot stop </code><var>BASEDIR</var><code>; buildbot start </code><var>BASEDIR</var> will
speed up the process.

<div class="node">
<p><hr>
<a name="Forcing-Builds"></a>
Previous:&nbsp;<a rel="previous" accesskey="p" href="#Connecting-to-the-buildmaster">Connecting to the buildmaster</a>,
Up:&nbsp;<a rel="up" accesskey="u" href="#Troubleshooting">Troubleshooting</a>

</div>

<h4 class="subsection">2.9.3 Forcing Builds</h4>

<p>From the buildmaster's main status web page, you can force a build to
be run on your build slave. Figure out which column is for a builder
that runs on your slave, click on that builder's name, and the page
that comes up will have a &ldquo;Force Build&rdquo; button. Fill in the form,
hit the button, and a moment later you should see your slave's
<samp><span class="file">twistd.log</span></samp> filling with commands being run. Using <code>pstree</code>
or <code>top</code> should also reveal the cvs/make/gcc/etc processes being
run by the buildslave. Note that the same web page should also show
the <samp><span class="file">admin</span></samp> and <samp><span class="file">host</span></samp> information files that you configured
earlier.

<div class="node">
<p><hr>
<a name="Concepts"></a>
Next:&nbsp;<a rel="next" accesskey="n" href="#Configuration">Configuration</a>,
Previous:&nbsp;<a rel="previous" accesskey="p" href="#Installation">Installation</a>,
Up:&nbsp;<a rel="up" accesskey="u" href="#Top">Top</a>

</div>

<h2 class="chapter">3 Concepts</h2>

<p>This chapter defines some of the basic concepts that the Buildbot
uses. You'll need to understand how the Buildbot sees the world to
configure it properly.

<ul class="menu">
<li><a accesskey="1" href="#Version-Control-Systems">Version Control Systems</a>
<li><a accesskey="2" href="#Schedulers">Schedulers</a>
<li><a accesskey="3" href="#BuildSet">BuildSet</a>
<li><a accesskey="4" href="#BuildRequest">BuildRequest</a>
<li><a accesskey="5" href="#Builder">Builder</a>
<li><a accesskey="6" href="#Users">Users</a>
</ul>

<div class="node">
<p><hr>
<a name="Version-Control-Systems"></a>
Next:&nbsp;<a rel="next" accesskey="n" href="#Schedulers">Schedulers</a>,
Previous:&nbsp;<a rel="previous" accesskey="p" href="#Concepts">Concepts</a>,
Up:&nbsp;<a rel="up" accesskey="u" href="#Concepts">Concepts</a>

</div>

<h3 class="section">3.1 Version Control Systems</h3>

<p><a name="index-Version-Control-5"></a>
These source trees come from a Version Control System of some kind. 
CVS and Subversion are two popular ones, but the Buildbot supports
others. All VC systems have some notion of an upstream
<code>repository</code> which acts as a server<a rel="footnote" href="#fn-2" name="fnd-2"><sup>2</sup></a>, from which clients
can obtain source trees according to various parameters. The VC
repository provides source trees of various projects, for different
branches, and from various points in time. The first thing we have to
do is to specify which source tree we want to get.

<ul class="menu">
<li><a accesskey="1" href="#Generalizing-VC-Systems">Generalizing VC Systems</a>
<li><a accesskey="2" href="#Source-Tree-Specifications">Source Tree Specifications</a>
<li><a accesskey="3" href="#How-Different-VC-Systems-Specify-Sources">How Different VC Systems Specify Sources</a>
<li><a accesskey="4" href="#Attributes-of-Changes">Attributes of Changes</a>
</ul>

<div class="node">
<p><hr>
<a name="Generalizing-VC-Systems"></a>
Next:&nbsp;<a rel="next" accesskey="n" href="#Source-Tree-Specifications">Source Tree Specifications</a>,
Previous:&nbsp;<a rel="previous" accesskey="p" href="#Version-Control-Systems">Version Control Systems</a>,
Up:&nbsp;<a rel="up" accesskey="u" href="#Version-Control-Systems">Version Control Systems</a>

</div>

<h4 class="subsection">3.1.1 Generalizing VC Systems</h4>

<p>For the purposes of the Buildbot, we will try to generalize all VC
systems as having repositories that each provide sources for a variety
of projects. Each project is defined as a directory tree with source
files. The individual files may each have revisions, but we ignore
that and treat the project as a whole as having a set of revisions. 
Each time someone commits a change to the project, a new revision
becomes available. These revisions can be described by a tuple with
two items: the first is a branch tag, and the second is some kind of
timestamp or revision stamp. Complex projects may have multiple branch
tags, but there is always a default branch. The timestamp may be an
actual timestamp (such as the -D option to CVS), or it may be a
monotonically-increasing transaction number (such as the change number
used by SVN and P4, or the revision number used by Arch, or a labeled
tag used in CVS)<a rel="footnote" href="#fn-3" name="fnd-3"><sup>3</sup></a>. The SHA1 revision ID used by Monotone and
Mercurial is also a kind of revision stamp, in that it specifies a
unique copy of the source tree, as does a Darcs &ldquo;context&rdquo; file.

   <p>When we aren't intending to make any changes to the sources we check out
(at least not any that need to be committed back upstream), there are two
basic ways to use a VC system:

     <ul>
<li>Retrieve a specific set of source revisions: some tag or key is used
to index this set, which is fixed and cannot be changed by subsequent
developers committing new changes to the tree. Releases are built from
tagged revisions like this, so that they can be rebuilt again later
(probably with controlled modifications). 
<li>Retrieve the latest sources along a specific branch: some tag is used
to indicate which branch is to be used, but within that constraint we want
to get the latest revisions. 
</ul>

   <p>Build personnel or CM staff typically use the first approach: the
build that results is (ideally) completely specified by the two
parameters given to the VC system: repository and revision tag. This
gives QA and end-users something concrete to point at when reporting
bugs. Release engineers are also reportedly fond of shipping code that
can be traced back to a concise revision tag of some sort.

   <p>Developers are more likely to use the second approach: each morning
the developer does an update to pull in the changes committed by the
team over the last day. These builds are not easy to fully specify: it
depends upon exactly when you did a checkout, and upon what local
changes the developer has in their tree. Developers do not normally
tag each build they produce, because there is usually significant
overhead involved in creating these tags. Recreating the trees used by
one of these builds can be a challenge. Some VC systems may provide
implicit tags (like a revision number), while others may allow the use
of timestamps to mean &ldquo;the state of the tree at time X&rdquo; as opposed
to a tree-state that has been explicitly marked.

   <p>The Buildbot is designed to help developers, so it usually works in
terms of <em>the latest</em> sources as opposed to specific tagged
revisions. However, it would really prefer to build from reproducible
source trees, so implicit revisions are used whenever possible.

<div class="node">
<p><hr>
<a name="Source-Tree-Specifications"></a>
Next:&nbsp;<a rel="next" accesskey="n" href="#How-Different-VC-Systems-Specify-Sources">How Different VC Systems Specify Sources</a>,
Previous:&nbsp;<a rel="previous" accesskey="p" href="#Generalizing-VC-Systems">Generalizing VC Systems</a>,
Up:&nbsp;<a rel="up" accesskey="u" href="#Version-Control-Systems">Version Control Systems</a>

</div>

<h4 class="subsection">3.1.2 Source Tree Specifications</h4>

<p>So for the Buildbot's purposes we treat each VC system as a server
which can take a list of specifications as input and produce a source
tree as output. Some of these specifications are static: they are
attributes of the builder and do not change over time. Others are more
variable: each build will have a different value. The repository is
changed over time by a sequence of Changes, each of which represents a
single developer making changes to some set of files. These Changes
are cumulative<a rel="footnote" href="#fn-4" name="fnd-4"><sup>4</sup></a>.

   <p>For normal builds, the Buildbot wants to get well-defined source trees
that contain specific Changes, and exclude other Changes that may have
occurred after the desired ones. We assume that the Changes arrive at
the buildbot (through one of the mechanisms described in see <a href="#Change-Sources">Change Sources</a>) in the same order in which they are committed to the
repository. The Buildbot waits for the tree to become &ldquo;stable&rdquo;
before initiating a build, for two reasons. The first is that
developers frequently make multiple related commits in quick
succession, even when the VC system provides ways to make atomic
transactions involving multiple files at the same time. Running a
build in the middle of these sets of changes would use an inconsistent
set of source files, and is likely to fail (and is certain to be less
useful than a build which uses the full set of changes). The
tree-stable-timer is intended to avoid these useless builds that
include some of the developer's changes but not all. The second reason
is that some VC systems (i.e. CVS) do not provide repository-wide
transaction numbers, so that timestamps are the only way to refer to
a specific repository state. These timestamps may be somewhat
ambiguous, due to processing and notification delays. By waiting until
the tree has been stable for, say, 10 minutes, we can choose a
timestamp from the middle of that period to use for our source
checkout, and then be reasonably sure that any clock-skew errors will
not cause the build to be performed on an inconsistent set of source
files.

   <p>The Schedulers always use the tree-stable-timer, with a timeout that
is configured to reflect a reasonable tradeoff between build latency
and change frequency. When the VC system provides coherent
repository-wide revision markers (such as Subversion's revision
numbers, or in fact anything other than CVS's timestamps), the
resulting Build is simply performed against a source tree defined by
that revision marker. When the VC system does not provide this, a
timestamp from the middle of the tree-stable period is used to
generate the source tree<a rel="footnote" href="#fn-5" name="fnd-5"><sup>5</sup></a>.

<div class="node">
<p><hr>
<a name="How-Different-VC-Systems-Specify-Sources"></a>
Next:&nbsp;<a rel="next" accesskey="n" href="#Attributes-of-Changes">Attributes of Changes</a>,
Previous:&nbsp;<a rel="previous" accesskey="p" href="#Source-Tree-Specifications">Source Tree Specifications</a>,
Up:&nbsp;<a rel="up" accesskey="u" href="#Version-Control-Systems">Version Control Systems</a>

</div>

<h4 class="subsection">3.1.3 How Different VC Systems Specify Sources</h4>

<p>For CVS, the static specifications are <code>repository</code> and
<code>module</code>. In addition to those, each build uses a timestamp (or
omits the timestamp to mean <code>the latest</code>) and <code>branch tag</code>
(which defaults to HEAD). These parameters collectively specify a set
of sources from which a build may be performed.

   <p><a href="http://subversion.tigris.org">Subversion</a> combines the
repository, module, and branch into a single <code>Subversion URL</code>
parameter. Within that scope, source checkouts can be specified by a
numeric <code>revision number</code> (a repository-wide
monotonically-increasing marker, such that each transaction that
changes the repository is indexed by a different revision number), or
a revision timestamp. When branches are used, the repository and
module form a static <code>baseURL</code>, while each build has a
<code>revision number</code> and a <code>branch</code> (which defaults to a
statically-specified <code>defaultBranch</code>). The <code>baseURL</code> and
<code>branch</code> are simply concatenated together to derive the
<code>svnurl</code> to use for the checkout.

   <p><a href="http://www.perforce.com/">Perforce</a> is similar. The server
is specified through a <code>P4PORT</code> parameter. Module and branch
are specified in a single depot path, and revisions are
depot-wide. When branches are used, the <code>p4base</code> and
<code>defaultBranch</code> are concatenated together to produce the depot
path.

   <p><a href="http://wiki.gnuarch.org/">Arch</a> and
<a href="http://bazaar.canonical.com/">Bazaar</a> specify a repository by
URL, as well as a <code>version</code> which is kind of like a branch name. 
Arch uses the word <code>archive</code> to represent the repository. Arch
lets you push changes from one archive to another, removing the strict
centralization required by CVS and SVN. It retains the distinction
between repository and working directory that most other VC systems
use. For complex multi-module directory structures, Arch has a
built-in <code>build config</code> layer with which the checkout process has
two steps. First, an initial bootstrap checkout is performed to
retrieve a set of build-config files. Second, one of these files is
used to figure out which archives/modules should be used to populate
subdirectories of the initial checkout.

   <p>Builders which use Arch and Bazaar therefore have a static archive
<code>url</code>, and a default &ldquo;branch&rdquo; (which is a string that specifies
a complete category&ndash;branch&ndash;version triple). Each build can have its
own branch (the category&ndash;branch&ndash;version string) to override the
default, as well as a revision number (which is turned into a
&ndash;patch-NN suffix when performing the checkout).

   <p><a href="http://abridgegame.org/darcs/">Darcs</a> doesn't really have the
notion of a single master repository. Nor does it really have
branches. In Darcs, each working directory is also a repository, and
there are operations to push and pull patches from one of these
<code>repositories</code> to another. For the Buildbot's purposes, all you
need to do is specify the URL of a repository that you want to build
from. The build slave will then pull the latest patches from that
repository and build them. Multiple branches are implemented by using
multiple repositories (possibly living on the same server).

   <p>Builders which use Darcs therefore have a static <code>repourl</code> which
specifies the location of the repository. If branches are being used,
the source Step is instead configured with a <code>baseURL</code> and a
<code>defaultBranch</code>, and the two strings are simply concatenated
together to obtain the repository's URL. Each build then has a
specific branch which replaces <code>defaultBranch</code>, or just uses the
default one. Instead of a revision number, each build can have a
&ldquo;context&rdquo;, which is a string that records all the patches that are
present in a given tree (this is the output of <samp><span class="command">darcs changes
--context</span></samp>, and is considerably less concise than, e.g. Subversion's
revision number, but the patch-reordering flexibility of Darcs makes
it impossible to provide a shorter useful specification).

   <p><a href="http://selenic.com/mercurial">Mercurial</a> is like Darcs, in that
each branch is stored in a separate repository. The <code>repourl</code>,
<code>baseURL</code>, and <code>defaultBranch</code> arguments are all handled the
same way as with Darcs. The &ldquo;revision&rdquo;, however, is the hash
identifier returned by <samp><span class="command">hg identify</span></samp>.

<div class="node">
<p><hr>
<a name="Attributes-of-Changes"></a>
Previous:&nbsp;<a rel="previous" accesskey="p" href="#How-Different-VC-Systems-Specify-Sources">How Different VC Systems Specify Sources</a>,
Up:&nbsp;<a rel="up" accesskey="u" href="#Version-Control-Systems">Version Control Systems</a>

</div>

<h4 class="subsection">3.1.4 Attributes of Changes</h4>

<h3 class="heading">Who</h3>

<p>Each Change has a <code>who</code> attribute, which specifies which
developer is responsible for the change. This is a string which comes
from a namespace controlled by the VC repository. Frequently this
means it is a username on the host which runs the repository, but not
all VC systems require this (Arch, for example, uses a fully-qualified
<code>Arch ID</code>, which looks like an email address, as does Darcs). 
Each StatusNotifier will map the <code>who</code> attribute into something
appropriate for their particular means of communication: an email
address, an IRC handle, etc.

<h3 class="heading">Files</h3>

<p>It also has a list of <code>files</code>, which are just the tree-relative
filenames of any files that were added, deleted, or modified for this
Change. These filenames are used by the <code>isFileImportant</code>
function (in the Scheduler) to decide whether it is worth triggering a
new build or not, e.g. the function could use
<code>filename.endswith(".c")</code> to only run a build if a C file were
checked in. Certain BuildSteps can also use the list of changed files
to run a more targeted series of tests, e.g. the
<code>step_twisted.Trial</code> step can run just the unit tests that
provide coverage for the modified .py files instead of running the
full test suite.

<h3 class="heading">Comments</h3>

<p>The Change also has a <code>comments</code> attribute, which is a string
containing any checkin comments.

<h3 class="heading">Revision</h3>

<p>Each Change can have a <code>revision</code> attribute, which describes how
to get a tree with a specific state: a tree which includes this Change
(and all that came before it) but none that come after it. If this
information is unavailable, the <code>.revision</code> attribute will be
<code>None</code>. These revisions are provided by the ChangeSource, and
consumed by the <code>computeSourceRevision</code> method in the appropriate
<code>step.Source</code> class.

     <dl>
<dt>`<samp><span class="samp">CVS</span></samp>'<dd><code>revision</code> is an int, seconds since the epoch
<br><dt>`<samp><span class="samp">SVN</span></samp>'<dd><code>revision</code> is an int, a transation number (r%d)
<br><dt>`<samp><span class="samp">Darcs</span></samp>'<dd><code>revision</code> is a large string, the output of <code>darcs changes --context</code>
<br><dt>`<samp><span class="samp">Mercurial</span></samp>'<dd><code>revision</code> is a short string (a hash ID), the output of <code>hg identify</code>
<br><dt>`<samp><span class="samp">Arch/Bazaar</span></samp>'<dd><code>revision</code> is the full revision ID (ending in &ndash;patch-%d)
<br><dt>`<samp><span class="samp">P4</span></samp>'<dd><code>revision</code> is an int, the transaction number
</dl>

<h3 class="heading">Branches</h3>

<p>The Change might also have a <code>branch</code> attribute. This indicates
that all of the Change's files are in the same named branch. The
Schedulers get to decide whether the branch should be built or not.

   <p>For VC systems like CVS, Arch, and Monotone, the <code>branch</code> name is
unrelated to the filename. (that is, the branch name and the filename
inhabit unrelated namespaces). For SVN, branches are expressed as
subdirectories of the repository, so the file's &ldquo;svnurl&rdquo; is a
combination of some base URL, the branch name, and the filename within
the branch. (In a sense, the branch name and the filename inhabit the
same namespace). Darcs branches are subdirectories of a base URL just
like SVN. Mercurial branches are the same as Darcs.

     <dl>
<dt>`<samp><span class="samp">CVS</span></samp>'<dd>branch='warner-newfeature', files=['src/foo.c']
<br><dt>`<samp><span class="samp">SVN</span></samp>'<dd>branch='branches/warner-newfeature', files=['src/foo.c']
<br><dt>`<samp><span class="samp">Darcs</span></samp>'<dd>branch='warner-newfeature', files=['src/foo.c']
<br><dt>`<samp><span class="samp">Mercurial</span></samp>'<dd>branch='warner-newfeature', files=['src/foo.c']
<br><dt>`<samp><span class="samp">Arch/Bazaar</span></samp>'<dd>branch='buildbot&ndash;usebranches&ndash;0', files=['buildbot/master.py']
</dl>

<h3 class="heading">Links</h3>

<!-- TODO: who is using 'links'? how is it being used? -->
<p>Finally, the Change might have a <code>links</code> list, which is intended
to provide a list of URLs to a <em>viewcvs</em>-style web page that
provides more detail for this Change, perhaps including the full file
diffs.

<div class="node">
<p><hr>
<a name="Schedulers"></a>
Next:&nbsp;<a rel="next" accesskey="n" href="#BuildSet">BuildSet</a>,
Previous:&nbsp;<a rel="previous" accesskey="p" href="#Version-Control-Systems">Version Control Systems</a>,
Up:&nbsp;<a rel="up" accesskey="u" href="#Concepts">Concepts</a>

</div>

<h3 class="section">3.2 Schedulers</h3>

<p><a name="index-Scheduler-6"></a>
Each Buildmaster has a set of <code>Scheduler</code> objects, each of which
gets a copy of every incoming Change. The Schedulers are responsible
for deciding when Builds should be run. Some Buildbot installations
might have a single Scheduler, while others may have several, each for
a different purpose.

   <p>For example, a &ldquo;quick&rdquo; scheduler might exist to give immediate
feedback to developers, hoping to catch obvious problems in the code
that can be detected quickly. These typically do not run the full test
suite, nor do they run on a wide variety of platforms. They also
usually do a VC update rather than performing a brand-new checkout
each time. You could have a &ldquo;quick&rdquo; scheduler which used a 30 second
timeout, and feeds a single &ldquo;quick&rdquo; Builder that uses a VC
<code>mode='update'</code> setting.

   <p>A separate &ldquo;full&rdquo; scheduler would run more comprehensive tests a
little while later, to catch more subtle problems. This scheduler
would have a longer tree-stable-timer, maybe 30 minutes, and would
feed multiple Builders (with a <code>mode=</code> of <code>'copy'</code>,
<code>'clobber'</code>, or <code>'export'</code>).

   <p>The <code>tree-stable-timer</code> and <code>isFileImportant</code> decisions are
made by the Scheduler. Dependencies are also implemented here. 
Periodic builds (those which are run every N seconds rather than after
new Changes arrive) are triggered by a special <code>Periodic</code>
Scheduler subclass. The default Scheduler class can also be told to
watch for specific branches, ignoring Changes on other branches. This
may be useful if you have a trunk and a few release branches which
should be tracked, but when you don't want to have the Buildbot pay
attention to several dozen private user branches.

   <p>Some Schedulers may trigger builds for other reasons, other than
recent Changes. For example, a Scheduler subclass could connect to a
remote buildmaster and watch for builds of a library to succeed before
triggering a local build that uses that library.

   <p>Each Scheduler creates and submits <code>BuildSet</code> objects to the
<code>BuildMaster</code>, which is then responsible for making sure the
individual <code>BuildRequests</code> are delivered to the target
<code>Builders</code>.

   <p><code>Scheduler</code> instances are activated by placing them in the
<code>c['schedulers']</code> list in the buildmaster config file. Each
Scheduler has a unique name.

<div class="node">
<p><hr>
<a name="BuildSet"></a>
Next:&nbsp;<a rel="next" accesskey="n" href="#BuildRequest">BuildRequest</a>,
Previous:&nbsp;<a rel="previous" accesskey="p" href="#Schedulers">Schedulers</a>,
Up:&nbsp;<a rel="up" accesskey="u" href="#Concepts">Concepts</a>

</div>

<h3 class="section">3.3 BuildSet</h3>

<p><a name="index-BuildSet-7"></a>
A <code>BuildSet</code> is the name given to a set of Builds that all
compile/test the same version of the tree on multiple Builders. In
general, all these component Builds will perform the same sequence of
Steps, using the same source code, but on different platforms or
against a different set of libraries.

   <p>The <code>BuildSet</code> is tracked as a single unit, which fails if any of
the component Builds have failed, and therefore can succeed only if
<em>all</em> of the component Builds have succeeded. There are two kinds
of status notification messages that can be emitted for a BuildSet:
the <code>firstFailure</code> type (which fires as soon as we know the
BuildSet will fail), and the <code>Finished</code> type (which fires once
the BuildSet has completely finished, regardless of whether the
overall set passed or failed).

   <p>A <code>BuildSet</code> is created with a <em>source stamp</em> tuple of
(branch, revision, changes, patch), some of which may be None, and a
list of Builders on which it is to be run. They are then given to the
BuildMaster, which is responsible for creating a separate
<code>BuildRequest</code> for each Builder.

   <p>There are a couple of different likely values for the
<code>SourceStamp</code>:

     <dl>
<dt><code>(revision=None, changes=[CHANGES], patch=None)</code><dd>This is a <code>SourceStamp</code> used when a series of Changes have
triggered a build. The VC step will attempt to check out a tree that
contains CHANGES (and any changes that occurred before CHANGES, but
not any that occurred after them).

     <br><dt><code>(revision=None, changes=None, patch=None)</code><dd>This builds the most recent code on the default branch. This is the
sort of <code>SourceStamp</code> that would be used on a Build that was
triggered by a user request, or a Periodic scheduler. It is also
possible to configure the VC Source Step to always check out the
latest sources rather than paying attention to the Changes in the
SourceStamp, which will result in same behavior as this.

     <br><dt><code>(branch=BRANCH, revision=None, changes=None, patch=None)</code><dd>This builds the most recent code on the given BRANCH. Again, this is
generally triggered by a user request or Periodic build.

     <br><dt><code>(revision=REV, changes=None, patch=(LEVEL, DIFF))</code><dd>This checks out the tree at the given revision REV, then applies a
patch (using <code>diff -pLEVEL &lt;DIFF</code>). The <a href="#try">try</a> feature uses
this kind of <code>SourceStamp</code>. If <code>patch</code> is None, the patching
step is bypassed.

   </dl>

   <p>The buildmaster is responsible for turning the <code>BuildSet</code> into a
set of <code>BuildRequest</code> objects and queueing them on the
appropriate Builders.

<div class="node">
<p><hr>
<a name="BuildRequest"></a>
Next:&nbsp;<a rel="next" accesskey="n" href="#Builder">Builder</a>,
Previous:&nbsp;<a rel="previous" accesskey="p" href="#BuildSet">BuildSet</a>,
Up:&nbsp;<a rel="up" accesskey="u" href="#Concepts">Concepts</a>

</div>

<h3 class="section">3.4 BuildRequest</h3>

<p><a name="index-BuildRequest-8"></a>
A <code>BuildRequest</code> is a request to build a specific set of sources
on a single specific Builder. Each Builder runs the
<code>BuildRequest</code> as soon as it can (i.e. when an associated
buildslave becomes free).

   <p>The <code>BuildRequest</code> contains the <code>SourceStamp</code> specification. 
The actual process of running the build (the series of Steps that will
be executed) is implemented by the <code>Build</code> object. In this future
this might be changed, to have the <code>Build</code> define <em>what</em>
gets built, and a separate <code>BuildProcess</code> (provided by the
Builder) to define <em>how</em> it gets built.

   <p>The <code>BuildRequest</code> may be mergeable with other compatible
<code>BuildRequest</code>s. Builds that are triggered by incoming Changes
will generally be mergeable. Builds that are triggered by user
requests are generally not, unless they are multiple requests to build
the <em>latest sources</em> of the same branch.

<div class="node">
<p><hr>
<a name="Builder"></a>
Next:&nbsp;<a rel="next" accesskey="n" href="#Users">Users</a>,
Previous:&nbsp;<a rel="previous" accesskey="p" href="#BuildRequest">BuildRequest</a>,
Up:&nbsp;<a rel="up" accesskey="u" href="#Concepts">Concepts</a>

</div>

<h3 class="section">3.5 Builder</h3>

<p><a name="index-Builder-9"></a>
The <code>Builder</code> is a long-lived object which controls all Builds of
a given type. Each one is created when the config file is first
parsed, and lives forever (or rather until it is removed from the
config file). It mediates the connections to the buildslaves that do
all the work, and is responsible for creating the <code>Build</code> objects
that decide <em>how</em> a build is performed (i.e., which steps are
executed in what order).

   <p>Each <code>Builder</code> gets a unique name, and the path name of a
directory where it gets to do all its work (there is a
buildmaster-side directory for keeping status information, as well as
a buildslave-side directory where the actual checkout/compile/test
commands are executed). It also gets a <code>BuildFactory</code>, which is
responsible for creating new <code>Build</code> instances: because the
<code>Build</code> instance is what actually performs each build, choosing
the <code>BuildFactory</code> is the way to specify what happens each time a
build is done.

   <p>Each <code>Builder</code> is associated with one of more <code>BuildSlaves</code>. 
A <code>Builder</code> which is used to perform OS-X builds (as opposed to
Linux or Solaris builds) should naturally be associated with an
OS-X-based buildslave.

<div class="node">
<p><hr>
<a name="Users"></a>
Previous:&nbsp;<a rel="previous" accesskey="p" href="#Builder">Builder</a>,
Up:&nbsp;<a rel="up" accesskey="u" href="#Concepts">Concepts</a>

</div>

<h3 class="section">3.6 Users</h3>

<p><a name="index-Users-10"></a>
Buildbot has a somewhat limited awareness of <em>users</em>. It assumes
the world consists of a set of developers, each of whom can be
described by a couple of simple attributes. These developers make
changes to the source code, causing builds which may succeed or fail.

   <p>Each developer is primarily known through the source control system. Each
Change object that arrives is tagged with a <code>who</code> field that
typically gives the account name (on the repository machine) of the user
responsible for that change. This string is the primary key by which the
User is known, and is displayed on the HTML status pages and in each Build's
&ldquo;blamelist&rdquo;.

   <p>To do more with the User than just refer to them, this username needs to
be mapped into an address of some sort. The responsibility for this mapping
is left up to the status module which needs the address. The core code knows
nothing about email addresses or IRC nicknames, just user names.

<ul class="menu">
<li><a accesskey="1" href="#Doing-Things-With-Users">Doing Things With Users</a>
<li><a accesskey="2" href="#Email-Addresses">Email Addresses</a>
<li><a accesskey="3" href="#IRC-Nicknames">IRC Nicknames</a>
<li><a accesskey="4" href="#Live-Status-Clients">Live Status Clients</a>
</ul>

<div class="node">
<p><hr>
<a name="Doing-Things-With-Users"></a>
Next:&nbsp;<a rel="next" accesskey="n" href="#Email-Addresses">Email Addresses</a>,
Previous:&nbsp;<a rel="previous" accesskey="p" href="#Users">Users</a>,
Up:&nbsp;<a rel="up" accesskey="u" href="#Users">Users</a>

</div>

<h4 class="subsection">3.6.1 Doing Things With Users</h4>

<p>Each Change has a single User who is responsible for that Change. Most
Builds have a set of Changes: the Build represents the first time these
Changes have been built and tested by the Buildbot. The build has a
&ldquo;blamelist&rdquo; that consists of a simple union of the Users responsible
for all the Build's Changes.

   <p>The Build provides (through the IBuildStatus interface) a list of Users
who are &ldquo;involved&rdquo; in the build. For now this is equal to the
blamelist, but in the future it will be expanded to include a &ldquo;build
sheriff&rdquo; (a person who is &ldquo;on duty&rdquo; at that time and responsible for
watching over all builds that occur during their shift), as well as
per-module owners who simply want to keep watch over their domain (chosen by
subdirectory or a regexp matched against the filenames pulled out of the
Changes). The Involved Users are those who probably have an interest in the
results of any given build.

   <p>In the future, Buildbot will acquire the concept of &ldquo;Problems&rdquo;,
which last longer than builds and have beginnings and ends. For example, a
test case which passed in one build and then failed in the next is a
Problem. The Problem lasts until the test case starts passing again, at
which point the Problem is said to be &ldquo;resolved&rdquo;.

   <p>If there appears to be a code change that went into the tree at the
same time as the test started failing, that Change is marked as being
resposible for the Problem, and the user who made the change is added
to the Problem's &ldquo;Guilty&rdquo; list. In addition to this user, there may
be others who share responsibility for the Problem (module owners,
sponsoring developers). In addition to the Responsible Users, there
may be a set of Interested Users, who take an interest in the fate of
the Problem.

   <p>Problems therefore have sets of Users who may want to be kept aware of
the condition of the problem as it changes over time. If configured, the
Buildbot can pester everyone on the Responsible list with increasing
harshness until the problem is resolved, with the most harshness reserved
for the Guilty parties themselves. The Interested Users may merely be told
when the problem starts and stops, as they are not actually responsible for
fixing anything.

<div class="node">
<p><hr>
<a name="Email-Addresses"></a>
Next:&nbsp;<a rel="next" accesskey="n" href="#IRC-Nicknames">IRC Nicknames</a>,
Previous:&nbsp;<a rel="previous" accesskey="p" href="#Doing-Things-With-Users">Doing Things With Users</a>,
Up:&nbsp;<a rel="up" accesskey="u" href="#Users">Users</a>

</div>

<h4 class="subsection">3.6.2 Email Addresses</h4>

<p>The <code>buildbot.status.mail.MailNotifier</code> class provides a
status target which can send email about the results of each build. It
accepts a static list of email addresses to which each message should be
delivered, but it can also be configured to send mail to the Build's
Interested Users. To do this, it needs a way to convert User names into
email addresses.

   <p>For many VC systems, the User Name is actually an account name on the
system which hosts the repository. As such, turning the name into an
email address is a simple matter of appending
&ldquo;@repositoryhost.com&rdquo;. Some projects use other kinds of mappings
(for example the preferred email address may be at &ldquo;project.org&rdquo;
despite the repository host being named &ldquo;cvs.project.org&rdquo;), and some
VC systems have full separation between the concept of a user and that
of an account on the repository host (like Perforce). Some systems
(like Arch) put a full contact email address in every change.

   <p>To convert these names to addresses, the MailNotifier uses an EmailLookup
object. This provides a .getAddress method which accepts a name and
(eventually) returns an address. The default <code>MailNotifier</code>
module provides an EmailLookup which simply appends a static string,
configurable when the notifier is created. To create more complex behaviors
(perhaps using an LDAP lookup, or using &ldquo;finger&rdquo; on a central host to
determine a preferred address for the developer), provide a different object
as the <code>lookup</code> argument.

   <p>In the future, when the Problem mechanism has been set up, the Buildbot
will need to send mail to arbitrary Users. It will do this by locating a
MailNotifier-like object among all the buildmaster's status targets, and
asking it to send messages to various Users. This means the User-to-address
mapping only has to be set up once, in your MailNotifier, and every email
message the buildbot emits will take advantage of it.

<div class="node">
<p><hr>
<a name="IRC-Nicknames"></a>
Next:&nbsp;<a rel="next" accesskey="n" href="#Live-Status-Clients">Live Status Clients</a>,
Previous:&nbsp;<a rel="previous" accesskey="p" href="#Email-Addresses">Email Addresses</a>,
Up:&nbsp;<a rel="up" accesskey="u" href="#Users">Users</a>

</div>

<h4 class="subsection">3.6.3 IRC Nicknames</h4>

<p>Like MailNotifier, the <code>buildbot.status.words.IRC</code> class
provides a status target which can announce the results of each build. It
also provides an interactive interface by responding to online queries
posted in the channel or sent as private messages.

   <p>In the future, the buildbot can be configured map User names to IRC
nicknames, to watch for the recent presence of these nicknames, and to
deliver build status messages to the interested parties. Like
<code>MailNotifier</code> does for email addresses, the <code>IRC</code> object
will have an <code>IRCLookup</code> which is responsible for nicknames. The
mapping can be set up statically, or it can be updated by online users
themselves (by claiming a username with some kind of &ldquo;buildbot: i am
user warner&rdquo; commands).

   <p>Once the mapping is established, the rest of the buildbot can ask the
<code>IRC</code> object to send messages to various users. It can report on
the likelihood that the user saw the given message (based upon how long the
user has been inactive on the channel), which might prompt the Problem
Hassler logic to send them an email message instead.

<div class="node">
<p><hr>
<a name="Live-Status-Clients"></a>
Previous:&nbsp;<a rel="previous" accesskey="p" href="#IRC-Nicknames">IRC Nicknames</a>,
Up:&nbsp;<a rel="up" accesskey="u" href="#Users">Users</a>

</div>

<h4 class="subsection">3.6.4 Live Status Clients</h4>

<p>The Buildbot also offers a PB-based status client interface which can
display real-time build status in a GUI panel on the developer's desktop. 
This interface is normally anonymous, but it could be configured to let the
buildmaster know <em>which</em> developer is using the status client. The
status client could then be used as a message-delivery service, providing an
alternative way to deliver low-latency high-interruption messages to the
developer (like &ldquo;hey, you broke the build&rdquo;).

<div class="node">
<p><hr>
<a name="Configuration"></a>
Next:&nbsp;<a rel="next" accesskey="n" href="#Getting-Source-Code-Changes">Getting Source Code Changes</a>,
Previous:&nbsp;<a rel="previous" accesskey="p" href="#Concepts">Concepts</a>,
Up:&nbsp;<a rel="up" accesskey="u" href="#Top">Top</a>

</div>

<h2 class="chapter">4 Configuration</h2>

<p><a name="index-Configuration-11"></a>
The buildbot's behavior is defined by the &ldquo;config file&rdquo;, which
normally lives in the <samp><span class="file">master.cfg</span></samp> file in the buildmaster's base
directory (but this can be changed with an option to the
<code>buildbot create-master</code> command). This file completely specifies
which Builders are to be run, which slaves they should use, how
Changes should be tracked, and where the status information is to be
sent. The buildmaster's <samp><span class="file">buildbot.tac</span></samp> file names the base
directory; everything else comes from the config file.

   <p>A sample config file was installed for you when you created the
buildmaster, but you will need to edit it before your buildbot will do
anything useful.

   <p>This chapter gives an overview of the format of this file and the
various sections in it. You will need to read the later chapters to
understand how to fill in each section properly.

<ul class="menu">
<li><a accesskey="1" href="#Config-File-Format">Config File Format</a>
<li><a accesskey="2" href="#Loading-the-Config-File">Loading the Config File</a>
<li><a accesskey="3" href="#Defining-the-Project">Defining the Project</a>
<li><a accesskey="4" href="#Listing-Change-Sources-and-Schedulers">Listing Change Sources and Schedulers</a>
<li><a accesskey="5" href="#Setting-the-slaveport">Setting the slaveport</a>
<li><a accesskey="6" href="#Buildslave-Specifiers">Buildslave Specifiers</a>
<li><a accesskey="7" href="#Defining-Builders">Defining Builders</a>
<li><a accesskey="8" href="#Defining-Status-Targets">Defining Status Targets</a>
<li><a accesskey="9" href="#Debug-options">Debug options</a>
</ul>

<div class="node">
<p><hr>
<a name="Config-File-Format"></a>
Next:&nbsp;<a rel="next" accesskey="n" href="#Loading-the-Config-File">Loading the Config File</a>,
Previous:&nbsp;<a rel="previous" accesskey="p" href="#Configuration">Configuration</a>,
Up:&nbsp;<a rel="up" accesskey="u" href="#Configuration">Configuration</a>

</div>

<h3 class="section">4.1 Config File Format</h3>

<p>The config file is, fundamentally, just a piece of Python code which
defines a dictionary named <code>BuildmasterConfig</code>, with a number of
keys that are treated specially. You don't need to know Python to do
basic configuration, though, you can just copy the syntax of the
sample file. If you <em>are</em> comfortable writing Python code,
however, you can use all the power of a full programming language to
achieve more complicated configurations.

   <p>The <code>BuildmasterConfig</code> name is the only one which matters: all
other names defined during the execution of the file are discarded. 
When parsing the config file, the Buildmaster generally compares the
old configuration with the new one and performs the minimum set of
actions necessary to bring the buildbot up to date: Builders which are
not changed are left untouched, and Builders which are modified get to
keep their old event history.

   <p>Basic Python syntax: comments start with a hash character (&ldquo;#&rdquo;),
tuples are defined with <code>(parenthesis, pairs)</code>, arrays are
defined with <code>[square, brackets]</code>, tuples and arrays are mostly
interchangeable. Dictionaries (data structures which map &ldquo;keys&rdquo; to
&ldquo;values&rdquo;) are defined with curly braces: <code>{'key1': 'value1',
'key2': 'value2'} </code>. Function calls (and object instantiation) can use
named parameters, like <code>w = html.Waterfall(http_port=8010)</code>.

   <p>The config file starts with a series of <code>import</code> statements,
which make various kinds of Steps and Status targets available for
later use. The main <code>BuildmasterConfig</code> dictionary is created,
then it is populated with a variety of keys. These keys are broken
roughly into the following sections, each of which is documented in
the rest of this chapter:

     <ul>
<li>Project Definitions
<li>Change Sources / Schedulers
<li>Slaveport
<li>Buildslave Configuration
<li>Builders / Interlocks
<li>Status Targets
<li>Debug options
</ul>

   <p>The config file can use a few names which are placed into its namespace:

     <dl>
<dt><code>basedir</code><dd>the base directory for the buildmaster. This string has not been
expanded, so it may start with a tilde. It needs to be expanded before
use. The config file is located in
<code>os.path.expanduser(os.path.join(basedir, 'master.cfg'))</code>

   </dl>

<div class="node">
<p><hr>
<a name="Loading-the-Config-File"></a>
Next:&nbsp;<a rel="next" accesskey="n" href="#Defining-the-Project">Defining the Project</a>,
Previous:&nbsp;<a rel="previous" accesskey="p" href="#Config-File-Format">Config File Format</a>,
Up:&nbsp;<a rel="up" accesskey="u" href="#Configuration">Configuration</a>

</div>

<h3 class="section">4.2 Loading the Config File</h3>

<p>The config file is only read at specific points in time. It is first
read when the buildmaster is launched. Once it is running, there are
various ways to ask it to reload the config file. If you are on the
system hosting the buildmaster, you can send a <code>SIGHUP</code> signal to
it: the <samp><span class="command">buildbot</span></samp> tool has a shortcut for this:

<pre class="example">     buildbot sighup <var>BASEDIR</var>
</pre>
   <p>The debug tool (<code>buildbot debugclient --master HOST:PORT</code>) has a
&ldquo;Reload .cfg&rdquo; button which will also trigger a reload. In the
future, there will be other ways to accomplish this step (probably a
password-protected button on the web page, as well as a privileged IRC
command).

<div class="node">
<p><hr>
<a name="Defining-the-Project"></a>
Next:&nbsp;<a rel="next" accesskey="n" href="#Listing-Change-Sources-and-Schedulers">Listing Change Sources and Schedulers</a>,
Previous:&nbsp;<a rel="previous" accesskey="p" href="#Loading-the-Config-File">Loading the Config File</a>,
Up:&nbsp;<a rel="up" accesskey="u" href="#Configuration">Configuration</a>

</div>

<h3 class="section">4.3 Defining the Project</h3>

<p>There are a couple of basic settings that you use to tell the buildbot
what project it is working on. This information is used by status
reporters to let users find out more about the codebase being
exercised by this particular Buildbot installation.

<pre class="example">     c['projectName'] = "Buildbot"
     c['projectURL'] = "http://buildbot.sourceforge.net/"
     c['buildbotURL'] = "http://localhost:8010/"
</pre>
   <p><a name="index-c_005b_0027projectName_0027_005d-12"></a><code>projectName</code> is a short string will be used to describe the
project that this buildbot is working on. For example, it is used as
the title of the waterfall HTML page.

   <p><a name="index-c_005b_0027projectURL_0027_005d-13"></a><code>projectURL</code> is a string that gives a URL for the project as a
whole. HTML status displays will show <code>projectName</code> as a link to
<code>projectURL</code>, to provide a link from buildbot HTML pages to your
project's home page.

   <p><a name="index-c_005b_0027buildbotURL_0027_005d-14"></a>The <code>buildbotURL</code> string should point to the location where the
buildbot's internal web server (usually the <code>html.Waterfall</code>
page) is visible. This typically uses the port number set when you
create the <code>Waterfall</code> object: the buildbot needs your help to
figure out a suitable externally-visible host name.

   <p>When status notices are sent to users (either by email or over IRC),
<code>buildbotURL</code> will be used to create a URL to the specific build
or problem that they are being notified about. It will also be made
available to queriers (over IRC) who want to find out where to get
more information about this buildbot.

<div class="node">
<p><hr>
<a name="Listing-Change-Sources-and-Schedulers"></a>
Next:&nbsp;<a rel="next" accesskey="n" href="#Setting-the-slaveport">Setting the slaveport</a>,
Previous:&nbsp;<a rel="previous" accesskey="p" href="#Defining-the-Project">Defining the Project</a>,
Up:&nbsp;<a rel="up" accesskey="u" href="#Configuration">Configuration</a>

</div>

<h3 class="section">4.4 Listing Change Sources and Schedulers</h3>

<p><a name="index-c_005b_0027sources_0027_005d-15"></a>The <code>c['sources']</code> key is a list of ChangeSource
instances<a rel="footnote" href="#fn-6" name="fnd-6"><sup>6</sup></a>. 
This defines how the buildmaster learns about source code changes. 
More information about what goes here is available in See <a href="#Getting-Source-Code-Changes">Getting Source Code Changes</a>.

<pre class="example">     import buildbot.changes.pb
     c['sources'] = [buildbot.changes.pb.PBChangeSource()]
</pre>
   <p><a name="index-c_005b_0027schedulers_0027_005d-16"></a><code>c['schedulers']</code> is a list of Scheduler instances, each of which
causes builds to be started on a particular set of Builders. The two
basic Scheduler classes you are likely to start with are
<code>Scheduler</code> and <code>Periodic</code>, but you can write a customized
subclass to implement more complicated build scheduling.

   <p>The docstring for <code>buildbot.scheduler.Scheduler</code> is the best
place to see all the options that can be used. Type <code>pydoc
buildbot.scheduler.Scheduler</code> to see it, or look in
<samp><span class="file">buildbot/scheduler.py</span></samp> directly.

   <p>The basic Scheduler takes four arguments:

     <dl>
<dt><code>name</code><dd>Each Scheduler must have a unique name. This is only used in status
displays.

     <br><dt><code>branch</code><dd>This Scheduler will pay attention to a single branch, ignoring Changes
that occur on other branches. Setting <code>branch</code> equal to the
special value of <code>None</code> means it should only pay attention to the
default branch. Note that <code>None</code> is a keyword, not a string, so
you want to use <code>None</code> and not <code>"None"</code>.

     <br><dt><code>treeStableTimer</code><dd>The Scheduler will wait for this many seconds before starting the
build. If new changes are made during this interval, the timer will be
restarted, so really the build will be started after a change and then
after this many seconds of inactivity.

     <br><dt><code>builderNames</code><dd>When the tree-stable-timer finally expires, builds will be started on
these Builders. Each Builder gets a unique name: these strings must
match.

   </dl>

<pre class="example">     from buildbot import scheduler
     quick = scheduler.Scheduler("quick", None, 60,
                                 ["quick-linux", "quick-netbsd"])
     full = scheduler.Scheduler("full", None, 5*60,
                                ["full-linux", "full-netbsd", "full-OSX"])
     nightly = scheduler.Periodic("nightly", ["full-solaris"], 24*60*60)
     c['schedulers'] = [quick, full, nightly]
</pre>
   <p>In this example, the two &ldquo;quick&rdquo; builds are triggered 60 seconds
after the tree has been changed. The &ldquo;full&rdquo; builds do not run quite
so quickly (they wait 5 minutes), so hopefully if the quick builds
fail due to a missing file or really simple typo, the developer can
discover and fix the problem before the full builds are started. Both
Schedulers only pay attention to the default branch: any changes on
other branches are ignored by these Schedulers. Each Scheduler
triggers a different set of Builders, referenced by name.

   <p>The third Scheduler in this example just runs the full solaris build
once per day. (note that this Scheduler only lets you control the time
between builds, not the absolute time-of-day of each Build, so this
could easily wind up a &ldquo;daily&rdquo; or &ldquo;every afternoon&rdquo; scheduler
depending upon when it was first activated).

<ul class="menu">
<li><a accesskey="1" href="#Scheduler-Types">Scheduler Types</a>
<li><a accesskey="2" href="#Build-Dependencies">Build Dependencies</a>
</ul>

<div class="node">
<p><hr>
<a name="Scheduler-Types"></a>
Next:&nbsp;<a rel="next" accesskey="n" href="#Build-Dependencies">Build Dependencies</a>,
Previous:&nbsp;<a rel="previous" accesskey="p" href="#Listing-Change-Sources-and-Schedulers">Listing Change Sources and Schedulers</a>,
Up:&nbsp;<a rel="up" accesskey="u" href="#Listing-Change-Sources-and-Schedulers">Listing Change Sources and Schedulers</a>

</div>

<h4 class="subsection">4.4.1 Scheduler Types</h4>

<p><a name="index-buildbot_002escheduler_002eScheduler-17"></a><a name="index-buildbot_002escheduler_002eAnyBranchScheduler-18"></a><a name="index-buildbot_002escheduler_002ePeriodic-19"></a><a name="index-buildbot_002escheduler_002eNightly-20"></a>
Here is a brief catalog of the available Scheduler types. All these
Schedulers are classes in <code>buildbot.scheduler</code>, and the
docstrings there are the best source of documentation on the arguments
taken by each one.

     <dl>
<dt><code>Scheduler</code><dd>This is the default Scheduler class. It follows exactly one branch,
and starts a configurable tree-stable-timer after each change on that
branch. When the timer expires, it starts a build on some set of
Builders. The Scheduler accepts a <code>fileIsImportant</code> function
which can be used to ignore some Changes if they do not affect any
&ldquo;important&rdquo; files.

     <br><dt><code>AnyBranchScheduler</code><dd>This scheduler uses a tree-stable-timer like the default one, but
follows multiple branches at once. Each branch gets a separate timer.

     <br><dt><code>Dependent</code><dd>This scheduler watches an &ldquo;upstream&rdquo; Builder. When that Builder
successfully builds a particular set of Changes, it triggers builds of
the same code on a configured set of &ldquo;downstream&rdquo; builders. The next
section (see <a href="#Build-Dependencies">Build Dependencies</a>) describes this scheduler in more
detail.

     <br><dt><code>Periodic</code><dd>This simple scheduler just triggers a build every N seconds.

     <br><dt><code>Nightly</code><dd>This is highly configurable periodic build scheduler, which triggers a
build at particular times of day, week, month, or year. The
configuration syntax is very similar to the well-known <code>crontab</code>
format, in which you provide values for minute, hour, day, and month
(some of which can be wildcards), and a build is triggered whenever
the current time matches the given constraints. This can run a build
every night, every morning, every weekend, alternate Thursdays, on
your boss's birthday, etc.

     <br><dt><code>Try_Jobdir / Try_Userpass</code><dd>This scheduler allows developers to use the <code>buildbot try</code>
command to trigger builds of code they have not yet committed. See
<a href="#try">try</a> for complete details.

   </dl>

<div class="node">
<p><hr>
<a name="Build-Dependencies"></a>
Previous:&nbsp;<a rel="previous" accesskey="p" href="#Scheduler-Types">Scheduler Types</a>,
Up:&nbsp;<a rel="up" accesskey="u" href="#Listing-Change-Sources-and-Schedulers">Listing Change Sources and Schedulers</a>

</div>

<h4 class="subsection">4.4.2 Build Dependencies</h4>

<p><a name="index-Dependent-21"></a><a name="index-Dependencies-22"></a><a name="index-buildbot_002escheduler_002eDependent-23"></a>
It is common to wind up with one kind of build which should only be
performed if the same source code was successfully handled by some
other kind of build first. An example might be a packaging step: you
might only want to produce .deb or RPM packages from a tree that was
known to compile successfully and pass all unit tests. You could put
the packaging step in the same Build as the compile and testing steps,
but there might be other reasons to not do this (in particular you
might have several Builders worth of compiles/tests, but only wish to
do the packaging once). Another example is if you want to skip the
&ldquo;full&rdquo; builds after a failing &ldquo;quick&rdquo; build of the same source
code. Or, if one Build creates a product (like a compiled library)
that is used by some other Builder, you'd want to make sure the
consuming Build is run <em>after</em> the producing one.

   <p>You can use <code>Dependencies</code> to express this relationship to the
Buildbot. There is a special kind of Scheduler named
<code>scheduler.Dependent</code> that will watch an &ldquo;upstream&rdquo; Scheduler
for builds to complete successfully (on all of its Builders). Each
time that happens, the same source code (i.e. the same
<code>SourceStamp</code>) will be used to start a new set of builds, on a
different set of Builders. This &ldquo;downstream&rdquo; scheduler doesn't pay
attention to Changes at all, it only pays attention to the upstream
scheduler.

   <p>If the SourceStamp fails on any of the Builders in the upstream set,
the downstream builds will not fire.

<pre class="example">     from buildbot import scheduler
     tests = scheduler.Scheduler("tests", None, 5*60,
                                 ["full-linux", "full-netbsd", "full-OSX"])
     package = scheduler.Dependent("package",
                                   tests, # upstream scheduler
                                   ["make-tarball", "make-deb", "make-rpm"])
     c['schedulers'] = [tests, package]
</pre>
   <p>Note that <code>Dependent</code>'s upstream scheduler argument is given as a
<code>Scheduler</code> <em>instance</em>, not a name. This makes it impossible
to create circular dependencies in the config file.

<div class="node">
<p><hr>
<a name="Setting-the-slaveport"></a>
Next:&nbsp;<a rel="next" accesskey="n" href="#Buildslave-Specifiers">Buildslave Specifiers</a>,
Previous:&nbsp;<a rel="previous" accesskey="p" href="#Listing-Change-Sources-and-Schedulers">Listing Change Sources and Schedulers</a>,
Up:&nbsp;<a rel="up" accesskey="u" href="#Configuration">Configuration</a>

</div>

<h3 class="section">4.5 Setting the slaveport</h3>

<p><a name="index-c_005b_0027slavePortnum_0027_005d-24"></a>
The buildmaster will listen on a TCP port of your choosing for
connections from buildslaves. It can also use this port for
connections from remote Change Sources, status clients, and debug
tools. This port should be visible to the outside world, and you'll
need to tell your buildslave admins about your choice.

   <p>It does not matter which port you pick, as long it is externally
visible, however you should probably use something larger than 1024,
since most operating systems don't allow non-root processes to bind to
low-numbered ports. If your buildmaster is behind a firewall or a NAT
box of some sort, you may have to configure your firewall to permit
inbound connections to this port.

<pre class="example">     c['slavePortnum'] = 10000
</pre>
   <p><code>c['slavePortnum']</code> is a <em>strports</em> specification string,
defined in the <code>twisted.application.strports</code> module (try
<samp><span class="command">pydoc twisted.application.strports</span></samp> to get documentation on
the format). This means that you can have the buildmaster listen on a
localhost-only port by doing:

<pre class="example">     c['slavePortnum'] = "tcp:10000:interface=127.0.0.1"
</pre>
   <p>This might be useful if you only run buildslaves on the same machine,
and they are all configured to contact the buildmaster at
<code>localhost:10000</code>.

<div class="node">
<p><hr>
<a name="Buildslave-Specifiers"></a>
Next:&nbsp;<a rel="next" accesskey="n" href="#Defining-Builders">Defining Builders</a>,
Previous:&nbsp;<a rel="previous" accesskey="p" href="#Setting-the-slaveport">Setting the slaveport</a>,
Up:&nbsp;<a rel="up" accesskey="u" href="#Configuration">Configuration</a>

</div>

<h3 class="section">4.6 Buildslave Specifiers</h3>

<p><a name="index-c_005b_0027bots_0027_005d-25"></a>
The <code>c['bots']</code> key is a list of known buildslaves. Each
buildslave is defined by a tuple of (slavename, slavepassword). These
are the same two values that need to be provided to the buildslave
administrator when they create the buildslave.

<pre class="example">     c['bots'] = [('bot-solaris', 'solarispasswd'),
                  ('bot-bsd', 'bsdpasswd'),
                 ]
</pre>
   <p>The slavenames must be unique, of course. The password exists to
prevent evildoers from interfering with the buildbot by inserting
their own (broken) buildslaves into the system and thus displacing the
real ones.

   <p>Buildslaves with an unrecognized slavename or a non-matching password
will be rejected when they attempt to connect, and a message
describing the problem will be put in the log file (see <a href="#Logfiles">Logfiles</a>).

<div class="node">
<p><hr>
<a name="Defining-Builders"></a>
Next:&nbsp;<a rel="next" accesskey="n" href="#Defining-Status-Targets">Defining Status Targets</a>,
Previous:&nbsp;<a rel="previous" accesskey="p" href="#Buildslave-Specifiers">Buildslave Specifiers</a>,
Up:&nbsp;<a rel="up" accesskey="u" href="#Configuration">Configuration</a>

</div>

<h3 class="section">4.7 Defining Builders</h3>

<p><a name="index-c_005b_0027builders_0027_005d-26"></a>
The <code>c['builders']</code> key is a list of dictionaries which specify
the Builders. The Buildmaster runs a collection of Builders, each of
which handles a single type of build (e.g. full versus quick), on a
single build slave. A Buildbot which makes sure that the latest code
(&ldquo;HEAD&rdquo;) compiles correctly across four separate architecture will
have four Builders, each performing the same build but on different
slaves (one per platform).

   <p>Each Builder gets a separate column in the waterfall display. In
general, each Builder runs independently (although various kinds of
interlocks can cause one Builder to have an effect on another).

   <p>Each Builder specification dictionary has several required keys:

     <dl>
<dt><code>name</code><dd>This specifies the Builder's name, which is used in status
reports.

     <br><dt><code>slavename</code><dd>This specifies which buildslave will be used by this Builder. 
<code>slavename</code> must appear in the <code>c['bots']</code> list. Each
buildslave can accomodate multiple Builders.

     <br><dt><code>slavenames</code><dd>If you provide <code>slavenames</code> instead of <code>slavename</code>, you can
give a list of buildslaves which are capable of running this Builder. 
If multiple buildslaves are available for any given Builder, you will
have some measure of redundancy: in case one slave goes offline, the
others can still keep the Builder working. In addition, multiple
buildslaves will allow multiple simultaneous builds for the same
Builder, which might be useful if you have a lot of forced or &ldquo;try&rdquo;
builds taking place.

     <p>If you use this feature, it is important to make sure that the
buildslaves are all, in fact, capable of running the given build. The
slave hosts should be configured similarly, otherwise you will spend a
lot of time trying (unsuccessfully) to reproduce a failure that only
occurs on some of the buildslaves and not the others. Different
platforms, operating systems, versions of major programs or libraries,
all these things mean you should use separate Builders.

     <br><dt><code>builddir</code><dd>This specifies the name of a subdirectory (under the base directory)
in which everything related to this builder will be placed. On the
buildmaster, this holds build status information. On the buildslave,
this is where checkouts, compiles, and tests are run.

     <br><dt><code>factory</code><dd>This is a <code>buildbot.process.factory.BuildFactory</code> instance which
controls how the build is performed. Full details appear in their own
chapter, See <a href="#Build-Process">Build Process</a>. Parameters like the location of the CVS
repository and the compile-time options used for the build are
generally provided as arguments to the factory's constructor.

   </dl>

   <p>Other optional keys may be set on each Builder:

     <dl>
<dt><code>category</code><dd>If provided, this is a string that identifies a category for the
builder to be a part of. Status clients can limit themselves to a
subset of the available categories. A common use for this is to add
new builders to your setup (for a new module, or for a new buildslave)
that do not work correctly yet and allow you to integrate them with
the active builders. You can put these new builders in a test
category, make your main status clients ignore them, and have only
private status clients pick them up. As soon as they work, you can
move them over to the active category.

   </dl>

<div class="node">
<p><hr>
<a name="Defining-Status-Targets"></a>
Next:&nbsp;<a rel="next" accesskey="n" href="#Debug-options">Debug options</a>,
Previous:&nbsp;<a rel="previous" accesskey="p" href="#Defining-Builders">Defining Builders</a>,
Up:&nbsp;<a rel="up" accesskey="u" href="#Configuration">Configuration</a>

</div>

<h3 class="section">4.8 Defining Status Targets</h3>

<p>The Buildmaster has a variety of ways to present build status to
various users. Each such delivery method is a &ldquo;Status Target&rdquo; object
in the configuration's <code>status</code> list. To add status targets, you
just append more objects to this list:

   <p><a name="index-c_005b_0027status_0027_005d-27"></a>
<pre class="example">     c['status'] = []
     
     from buildbot.status import html
     c['status'].append(html.Waterfall(http_port=8010))
     
     from buildbot.status import mail
     m = mail.MailNotifier(fromaddr="buildbot@localhost",
                           extraRecipients=["builds@lists.example.com"],
                           sendToInterestedUsers=False)
     c['status'].append(m)
     
     from buildbot.status import words
     c['status'].append(words.IRC(host="irc.example.com", nick="bb",
                                  channels=["#example"]))
</pre>
   <p>Status delivery has its own chapter, See <a href="#Status-Delivery">Status Delivery</a>, in which
all the built-in status targets are documented.

<div class="node">
<p><hr>
<a name="Debug-options"></a>
Previous:&nbsp;<a rel="previous" accesskey="p" href="#Defining-Status-Targets">Defining Status Targets</a>,
Up:&nbsp;<a rel="up" accesskey="u" href="#Configuration">Configuration</a>

</div>

<h3 class="section">4.9 Debug options</h3>

<p><a name="index-c_005b_0027debugPassword_0027_005d-28"></a>If you set <code>c['debugPassword']</code>, then you can connect to the
buildmaster with the diagnostic tool launched by <code>buildbot
debugclient MASTER:PORT</code>. From this tool, you can reload the config
file, manually force builds, and inject changes, which may be useful
for testing your buildmaster without actually commiting changes to
your repository (or before you have the Change Sources set up). The
debug tool uses the same port number as the slaves do:
<code>c['slavePortnum']</code>, and is authenticated with this password.

<pre class="example">     c['debugPassword'] = "debugpassword"
</pre>
   <p><a name="index-c_005b_0027manhole_0027_005d-29"></a>If you set <code>c['manhole']</code> to an instance of one of the classes in
<code>buildbot.manhole</code>, you can telnet or ssh into the buildmaster
and get an interactive Python shell, which may be useful for debugging
buildbot internals. It is probably only useful for buildbot
developers. It exposes full access to the buildmaster's account
(including the ability to modify and delete files), so it should not
be enabled with a weak or easily guessable password.

   <p>There are three separate <code>Manhole</code> classes. Two of them use SSH,
one uses unencrypted telnet. Two of them use a username+password
combination to grant access, one of them uses an SSH-style
<samp><span class="file">authorized_keys</span></samp> file which contains a list of ssh public keys.

     <dl>
<dt><code>manhole.AuthorizedKeysManhole</code><dd>You construct this with the name of a file that contains one SSH
public key per line, just like <samp><span class="file">~/.ssh/authorized_keys</span></samp>. If you
provide a non-absolute filename, it will be interpreted relative to
the buildmaster's base directory.

     <br><dt><code>manhole.PasswordManhole</code><dd>This one accepts SSH connections but asks for a username and password
when authenticating. It accepts only one such pair.

     <br><dt><code>manhole.TelnetManhole</code><dd>This accepts regular unencrypted telnet connections, and asks for a
username/password pair before providing access. Because this
username/password is transmitted in the clear, and because Manhole
access to the buildmaster is equivalent to granting full shell
privileges to both the buildmaster and all the buildslaves (and to all
accounts which then run code produced by the buildslaves), it is
highly recommended that you use one of the SSH manholes instead.

   </dl>

<pre class="example">     # some examples:
     from buildbot import manhole
     c['manhole'] = manhole.AuthorizedKeysManhole(1234, "authorized_keys")
     c['manhole'] = manhole.PasswordManhole(1234, "alice", "mysecretpassword")
     c['manhole'] = manhole.TelnetManhole(1234, "bob", "snoop_my_password_please")
</pre>
   <p>The <code>Manhole</code> instance can be configured to listen on a specific
port. You may wish to have this listening port bind to the loopback
interface (sometimes known as &ldquo;lo0&rdquo;, &ldquo;localhost&rdquo;, or 127.0.0.1) to
restrict access to clients which are running on the same host.

<pre class="example">     from buildbot.manhole import PasswordManhole
     c['manhole'] = PasswordManhole("tcp:9999:interface=127.0.0.1","admin","passwd")
</pre>
   <p>To have the <code>Manhole</code> listen on all interfaces, use
<code>"tcp:9999"</code> or simply 9999. This port specification uses
<code>twisted.application.strports</code>, so you can make it listen on SSL
or even UNIX-domain sockets if you want.

   <p>Note that using any Manhole requires that the TwistedConch package be
installed, and that you be using Twisted version 2.0 or later.

   <p>The buildmaster's SSH server will use a different host key than the
normal sshd running on a typical unix host. This will cause the ssh
client to complain about a &ldquo;host key mismatch&rdquo;, because it does not
realize there are two separate servers running on the same host. To
avoid this, use a clause like the following in your <samp><span class="file">.ssh/config</span></samp>
file:

<pre class="example">     Host remotehost-buildbot
      HostName remotehost
      HostKeyAlias remotehost-buildbot
      Port 9999
      # use 'user' if you use PasswordManhole and your name is not 'admin'.
      # if you use AuthorizedKeysManhole, this probably doesn't matter.
      User admin
</pre>
   <div class="node">
<p><hr>
<a name="Getting-Source-Code-Changes"></a>
Next:&nbsp;<a rel="next" accesskey="n" href="#Build-Process">Build Process</a>,
Previous:&nbsp;<a rel="previous" accesskey="p" href="#Configuration">Configuration</a>,
Up:&nbsp;<a rel="up" accesskey="u" href="#Top">Top</a>

</div>

<h2 class="chapter">5 Getting Source Code Changes</h2>

<p>The most common way to use the Buildbot is centered around the idea of
<code>Source Trees</code>: a directory tree filled with source code of some form
which can be compiled and/or tested. Some projects use languages that don't
involve any compilation step: nevertheless there may be a <code>build</code> phase
where files are copied or rearranged into a form that is suitable for
installation. Some projects do not have unit tests, and the Buildbot is
merely helping to make sure that the sources can compile correctly. But in
all of these cases, the thing-being-tested is a single source tree.

   <p>A Version Control System mantains a source tree, and tells the
buildmaster when it changes. The first step of each Build is typically
to acquire a copy of some version of this tree.

   <p>This chapter describes how the Buildbot learns about what Changes have
occurred. For more information on VC systems and Changes, see
<a href="#Version-Control-Systems">Version Control Systems</a>.

<ul class="menu">
<li><a accesskey="1" href="#Change-Sources">Change Sources</a>
</ul>

<div class="node">
<p><hr>
<a name="Change-Sources"></a>
Previous:&nbsp;<a rel="previous" accesskey="p" href="#Getting-Source-Code-Changes">Getting Source Code Changes</a>,
Up:&nbsp;<a rel="up" accesskey="u" href="#Getting-Source-Code-Changes">Getting Source Code Changes</a>

</div>

<h3 class="section">5.1 Change Sources</h3>

<!-- TODO: rework this, the one-buildmaster-one-tree thing isn't quite -->
<!-- so narrow-minded anymore -->
<p>Each Buildmaster watches a single source tree. Changes can be provided
by a variety of ChangeSource types, however any given project will
typically have only a single ChangeSource active. This section
provides a description of all available ChangeSource types and
explains how to set up each of them.

   <p>There are a variety of ChangeSources available, some of which are
meant to be used in conjunction with other tools to deliver Change
events from the VC repository to the buildmaster.

     <ul>
<li>CVSToys
This ChangeSource opens a TCP connection from the buildmaster to a
waiting FreshCVS daemon that lives on the repository machine, and
subscribes to hear about Changes.

     <li>MaildirSource
This one watches a local maildir-format inbox for email sent out by
the repository when a change is made. When a message arrives, it is
parsed to create the Change object. A variety of parsing functions are
available to accomodate different email-sending tools.

     <li>PBChangeSource
This ChangeSource listens on a local TCP socket for inbound
connections from a separate tool. Usually, this tool would be run on
the VC repository machine in a commit hook. It is expected to connect
to the TCP socket and send a Change message over the network
connection. The <samp><span class="command">buildbot sendchange</span></samp> command is one example
of a tool that knows how to send these messages, so you can write a
commit script for your VC system that calls it to deliver the Change. 
There are other tools in the contrib/ directory that use the same
protocol.

   </ul>

   <p>As a quick guide, here is a list of VC systems and the ChangeSources
that might be useful with them. All of these ChangeSources are in the
<code>buildbot.changes</code> module.

     <dl>
<dt><code>CVS</code><dd>
          <ul>
<li>freshcvs.FreshCVSSource (connected via TCP to the freshcvs daemon)
<li>mail.FCMaildirSource (watching for email sent by a freshcvs daemon)
<li>mail.BonsaiMaildirSource (watching for email sent by Bonsai)
<li>mail.SyncmailMaildirSource (watching for email sent by syncmail)
<li>pb.PBChangeSource (listening for connections from <code>buildbot
sendchange</code> run in a loginfo script)
<li>pb.PBChangeSource (listening for connections from a long-running
<code>contrib/viewcvspoll.py</code> polling process which examines the ViewCVS
database directly
</ul>

     <br><dt><code>SVN</code><dd>
          <ul>
<li>pb.PBChangeSource (listening for connections from
<code>contrib/svn_buildbot.py</code> run in a postcommit script)
<li>pb.PBChangeSource (listening for connections from a long-running
<code>contrib/svn_watcher.py</code> or <code>contrib/svnpoller.py</code> polling
process
</ul>

     <br><dt><code>Darcs</code><dd>
          <ul>
<li>pb.PBChangeSource (listening for connections from
<code>contrib/darcs_buildbot.py</code> in a commit script
</ul>

     <br><dt><code>Mercurial</code><dd>
          <ul>
<li>pb.PBChangeSource (listening for connections from
<code>contrib/hg_buildbot.py</code> run in an 'incoming' hook)
</ul>

     <br><dt><code>Arch/Bazaar</code><dd>
          <ul>
<li>pb.PBChangeSource (listening for connections from
<code>contrib/arch_buildbot.py</code> run in a commit hook)
</ul>

   </dl>

   <p>All VC systems can be driven by a PBChangeSource and the
<code>buildbot sendchange</code> tool run from some form of commit script. 
If you write an email parsing function, they can also all be driven by
a suitable <code>MaildirSource</code>.

<ul class="menu">
<li><a accesskey="1" href="#Choosing-ChangeSources">Choosing ChangeSources</a>
<li><a accesskey="2" href="#CVSToys-_002d-PBService">CVSToys - PBService</a>
<li><a accesskey="3" href="#CVSToys-_002d-mail-notification">CVSToys - mail notification</a>
<li><a accesskey="4" href="#Other-mail-notification-ChangeSources">Other mail notification ChangeSources</a>
<li><a accesskey="5" href="#PBChangeSource">PBChangeSource</a>
<li><a accesskey="6" href="#P4Source">P4Source</a>
</ul>

<div class="node">
<p><hr>
<a name="Choosing-ChangeSources"></a>
Next:&nbsp;<a rel="next" accesskey="n" href="#CVSToys-_002d-PBService">CVSToys - PBService</a>,
Previous:&nbsp;<a rel="previous" accesskey="p" href="#Change-Sources">Change Sources</a>,
Up:&nbsp;<a rel="up" accesskey="u" href="#Change-Sources">Change Sources</a>

</div>

<h4 class="subsection">5.1.1 Choosing ChangeSources</h4>

<p>The <code>master.cfg</code> configuration file has a dictionary key named
<code>BuildmasterConfig['sources']</code>, which holds a list of
<code>IChangeSource</code> objects. The config file will typically create an
object from one of the classes described below and stuff it into the
list.

<pre class="example">     s = FreshCVSSourceNewcred(host="host", port=4519,
                               user="alice", passwd="secret",
                               prefix="Twisted")
     BuildmasterConfig['sources'] = [s]
</pre>
   <p>Each source tree has a nominal <code>top</code>. Each Change has a list of
filenames, which are all relative to this top location. The
ChangeSource is responsible for doing whatever is necessary to
accomplish this. Most sources have a <code>prefix</code> argument: a partial
pathname which is stripped from the front of all filenames provided to
that <code>ChangeSource</code>. Files which are outside this sub-tree are
ignored by the changesource: it does not generate Changes for those
files.

<div class="node">
<p><hr>
<a name="CVSToys---PBService"></a>
<a name="CVSToys-_002d-PBService"></a>
Next:&nbsp;<a rel="next" accesskey="n" href="#CVSToys-_002d-mail-notification">CVSToys - mail notification</a>,
Previous:&nbsp;<a rel="previous" accesskey="p" href="#Choosing-ChangeSources">Choosing ChangeSources</a>,
Up:&nbsp;<a rel="up" accesskey="u" href="#Change-Sources">Change Sources</a>

</div>

<h4 class="subsection">5.1.2 CVSToys - PBService</h4>

<p><a name="index-buildbot_002echanges_002efreshcvs_002eFreshCVSSource-30"></a>
The <a href="http://purl.net/net/CVSToys">CVSToys</a> package provides a
server which runs on the machine that hosts the CVS repository it
watches. It has a variety of ways to distribute commit notifications,
and offers a flexible regexp-based way to filter out uninteresting
changes. One of the notification options is named <code>PBService</code> and
works by listening on a TCP port for clients. These clients subscribe
to hear about commit notifications.

   <p>The buildmaster has a CVSToys-compatible <code>PBService</code> client built
in. There are two versions of it, one for old versions of CVSToys
(1.0.9 and earlier) which used the <code>oldcred</code> authentication
framework, and one for newer versions (1.0.10 and later) which use
<code>newcred</code>. Both are classes in the
<code>buildbot.changes.freshcvs</code> package.

   <p><code>FreshCVSSourceNewcred</code> objects are created with the following
parameters:

     <dl>
<dt>`<samp><code>host</code><span class="samp"> and </span><code>port</code></samp>'<dd>these specify where the CVSToys server can be reached

     <br><dt>`<samp><code>user</code><span class="samp"> and </span><code>passwd</code></samp>'<dd>these specify the login information for the CVSToys server
(<code>freshcvs</code>). These must match the server's values, which are
defined in the <code>freshCfg</code> configuration file (which lives in the
CVSROOT directory of the repository).

     <br><dt>`<samp><code>prefix</code></samp>'<dd>this is the prefix to be found and stripped from filenames delivered
by the CVSToys server. Most projects live in sub-directories of the
main repository, as siblings of the CVSROOT sub-directory, so
typically this prefix is set to that top sub-directory name.

   </dl>

<h3 class="heading">Example</h3>

<p>To set up the freshCVS server, add a statement like the following to
your <samp><span class="file">freshCfg</span></samp> file:

<pre class="example">     pb = ConfigurationSet([
         (None, None, None, PBService(userpass=('foo', 'bar'), port=4519)),
         ])
</pre>
   <p>This will announce all changes to a client which connects to port 4519
using a username of 'foo' and a password of 'bar'.

   <p>Then add a clause like this to your buildmaster's <samp><span class="file">master.cfg</span></samp>:

<pre class="example">     BuildmasterConfig['sources'] = [FreshCVSSource("cvs.example.com", 4519,
                                     "foo", "bar",
                                     prefix="glib/")]
</pre>
   <p>where "cvs.example.com" is the host that is running the FreshCVS daemon, and
"glib" is the top-level directory (relative to the repository's root) where
all your source code lives. Most projects keep one or more projects in the
same repository (along with CVSROOT/ to hold admin files like loginfo and
freshCfg); the prefix= argument tells the buildmaster to ignore everything
outside that directory, and to strip that common prefix from all pathnames
it handles.

<div class="node">
<p><hr>
<a name="CVSToys---mail-notification"></a>
<a name="CVSToys-_002d-mail-notification"></a>
Next:&nbsp;<a rel="next" accesskey="n" href="#Other-mail-notification-ChangeSources">Other mail notification ChangeSources</a>,
Previous:&nbsp;<a rel="previous" accesskey="p" href="#CVSToys-_002d-PBService">CVSToys - PBService</a>,
Up:&nbsp;<a rel="up" accesskey="u" href="#Change-Sources">Change Sources</a>

</div>

<h4 class="subsection">5.1.3 CVSToys - mail notification</h4>

<p><a name="index-buildbot_002echanges_002email_002eFCMaildirSource-31"></a>
CVSToys also provides a <code>MailNotification</code> action which will send
email to a list of recipients for each commit. This tends to work
better than using <code>/bin/mail</code> from within the CVSROOT/loginfo
file directly, as CVSToys will batch together all files changed during
the same CVS invocation, and can provide more information (like
creating a ViewCVS URL for each file changed).

   <p>The Buildbot's <code>FCMaildirSource</code> is a ChangeSource which knows
how to parse these CVSToys messages and turn them into Change objects. 
It watches a Maildir for new messages. The usually installation
process looks like:

     <ol type=1 start=1>
<li>Create a mailing list, <code>projectname-commits</code>. 
<li>In CVSToys' freshCfg file, use a <code>MailNotification</code> action to
send commit mail to this mailing list. 
<li>Subscribe the buildbot user to the mailing list. 
<li>Configure your .qmail or .forward file to deliver these messages into
a maildir. 
<li>In the Buildbot's master.cfg file, use a <code>FCMaildirSource</code> to
watch the maildir for commit messages.
        </ol>

   <p>The <code>FCMaildirSource</code> is created with two parameters: the
directory name of the maildir root, and the prefix to strip.

<div class="node">
<p><hr>
<a name="Other-mail-notification-ChangeSources"></a>
Next:&nbsp;<a rel="next" accesskey="n" href="#PBChangeSource">PBChangeSource</a>,
Previous:&nbsp;<a rel="previous" accesskey="p" href="#CVSToys-_002d-mail-notification">CVSToys - mail notification</a>,
Up:&nbsp;<a rel="up" accesskey="u" href="#Change-Sources">Change Sources</a>

</div>

<h4 class="subsection">5.1.4 Other mail notification ChangeSources</h4>

<p><a name="index-buildbot_002echanges_002email_002eSyncmailMaildirSource-32"></a><a name="index-buildbot_002echanges_002email_002eBonsaiMaildirSource-33"></a>
There are other types of maildir-watching ChangeSources, which only
differ in the function used to parse the message body.

   <p><code>SyncmailMaildirSource</code> knows how to parse the message format
used in mail sent by Syncmail.

   <p><code>BonsaiMaildirSource</code> parses messages sent out by Bonsai.

<div class="node">
<p><hr>
<a name="PBChangeSource"></a>
Next:&nbsp;<a rel="next" accesskey="n" href="#P4Source">P4Source</a>,
Previous:&nbsp;<a rel="previous" accesskey="p" href="#Other-mail-notification-ChangeSources">Other mail notification ChangeSources</a>,
Up:&nbsp;<a rel="up" accesskey="u" href="#Change-Sources">Change Sources</a>

</div>

<h4 class="subsection">5.1.5 PBChangeSource</h4>

<p><a name="index-buildbot_002echanges_002epb_002ePBChangeSource-34"></a>
The last kind of ChangeSource actually listens on a TCP port for
clients to connect and push change notices <em>into</em> the
Buildmaster. This is used by the built-in <code>buildbot sendchange</code>
notification tool, as well as the VC-specific
<samp><span class="file">contrib/svn_buildbot.py</span></samp> and <samp><span class="file">contrib/arch_buildbot.py</span></samp>
tools. These tools are run by the repository (in a commit hook
script), and connect to the buildmaster directly each time a file is
comitted. This is also useful for creating new kinds of change sources
that work on a <code>push</code> model instead of some kind of subscription
scheme, for example a script which is run out of an email .forward
file.

   <p>This ChangeSource can be configured to listen on its own TCP port, or
it can share the port that the buildmaster is already using for the
buildslaves to connect. (This is possible because the
<code>PBChangeSource</code> uses the same protocol as the buildslaves, and
they can be distinguished by the <code>username</code> attribute used when
the initial connection is established). It might be useful to have it
listen on a different port if, for example, you wanted to establish
different firewall rules for that port. You could allow only the SVN
repository machine access to the <code>PBChangeSource</code> port, while
allowing only the buildslave machines access to the slave port. Or you
could just expose one port and run everything over it. <em>Note:
this feature is not yet implemented, the PBChangeSource will always
share the slave port and will always have a </em><code>user</code><em> name of
</em><code>change</code><em>, and a passwd of </em><code>changepw</code><em>. These limitations will
be removed in the future.</em>.

   <p>The <code>PBChangeSource</code> is created with the following arguments. All
are optional.

     <dl>
<dt>`<samp><code>port</code></samp>'<dd>which port to listen on. If <code>None</code> (which is the default), it
shares the port used for buildslave connections. <em>Not
Implemented, always set to </em><code>None</code>.

     <br><dt>`<samp><code>user</code><span class="samp"> and </span><code>passwd</code></samp>'<dd>The user/passwd account information that the client program must use
to connect. Defaults to <code>change</code> and <code>changepw</code>. <em>Not
Implemented, </em><code>user</code><em> is currently always set to </em><code>change</code><em>,
</em><code>passwd</code><em> is always set to </em><code>changepw</code>.

     <br><dt>`<samp><code>prefix</code></samp>'<dd>The prefix to be found and stripped from filenames delivered over the
connection. Any filenames which do not start with this prefix will be
removed. If all the filenames in a given Change are removed, the that
whole Change will be dropped. This string should probably end with a
directory separator.

     <p>This is useful for changes coming from version control systems that
represent branches as parent directories within the repository (like
SVN and Perforce). Use a prefix of 'trunk/' or
'project/branches/foobranch/' to only follow one branch and to get
correct tree-relative filenames. Without a prefix, the PBChangeSource
will probably deliver Changes with filenames like <samp><span class="file">trunk/foo.c</span></samp>
instead of just <samp><span class="file">foo.c</span></samp>. Of course this also depends upon the
tool sending the Changes in (like <samp><span class="command">buildbot sendchange</span></samp>) and
what filenames it is delivering: that tool may be filtering and
stripping prefixes at the sending end.

   </dl>

<div class="node">
<p><hr>
<a name="P4Source"></a>
Previous:&nbsp;<a rel="previous" accesskey="p" href="#PBChangeSource">PBChangeSource</a>,
Up:&nbsp;<a rel="up" accesskey="u" href="#Change-Sources">Change Sources</a>

</div>

<h4 class="subsection">5.1.6 P4Source</h4>

<p><a name="index-buildbot_002echanges_002ep4poller_002eP4Source-35"></a>
The <code>P4Source</code> periodically polls a <a href="http://www.perforce.com/">Perforce</a> depot for changes. It accepts the following arguments:

     <dl>
<dt>`<samp><code>p4base</code></samp>'<dd>The base depot path to watch, without the trailing '/...'.

     <br><dt>`<samp><code>p4port</code></samp>'<dd>The Perforce server to connect to (as host:port).

     <br><dt>`<samp><code>p4user</code></samp>'<dd>The Perforce user.

     <br><dt>`<samp><code>p4passwd</code></samp>'<dd>The Perforce password.

     <br><dt>`<samp><code>split_file</code></samp>'<dd>A function that maps a pathname, without the leading <code>p4base</code>, to a
(branch, filename) tuple. The default just returns (None, branchfile),
which effectively disables branch support. You should supply a function
which understands your repository structure.

     <br><dt>`<samp><code>pollinterval</code></samp>'<dd>How often to poll, in seconds. Defaults to 600 (10 minutes).

     <br><dt>`<samp><code>histmax</code></samp>'<dd>The maximum number of changes to inspect at a time. If more than this
number occur since the last poll, older changes will be silently
ignored. 
</dl>

<h3 class="heading">Example</h3>

<p>This configuration uses the <code>P4PORT</code>, <code>P4USER</code>, and <code>P4PASSWD</code>
specified in the buildmaster's environment. It watches a project in which the
branch name is simply the next path component, and the file is all path
components after.

<pre class="example">     import buildbot.changes.p4poller
     c['sources'].append(p4poller.P4Source(
             p4base='//depot/project/',
             split_file=lambda branchfile: branchfile.split('/',1)
     ))
</pre>
   <div class="node">
<p><hr>
<a name="Build-Process"></a>
Next:&nbsp;<a rel="next" accesskey="n" href="#Status-Delivery">Status Delivery</a>,
Previous:&nbsp;<a rel="previous" accesskey="p" href="#Getting-Source-Code-Changes">Getting Source Code Changes</a>,
Up:&nbsp;<a rel="up" accesskey="u" href="#Top">Top</a>

</div>

<h2 class="chapter">6 Build Process</h2>

<p>A <code>Build</code> object is responsible for actually performing a build. 
It gets access to a remote <code>SlaveBuilder</code> where it may run
commands, and a <code>BuildStatus</code> object where it must emit status
events. The <code>Build</code> is created by the Builder's
<code>BuildFactory</code>.

   <p>The default <code>Build</code> class is made up of a fixed sequence of
<code>BuildSteps</code>, executed one after another until all are complete
(or one of them indicates that the build should be halted early). The
default <code>BuildFactory</code> creates instances of this <code>Build</code>
class with a list of <code>BuildSteps</code>, so the basic way to configure
the build is to provide a list of <code>BuildSteps</code> to your
<code>BuildFactory</code>.

   <p>More complicated <code>Build</code> subclasses can make other decisions:
execute some steps only if certain files were changed, or if certain
previous steps passed or failed. The base class has been written to
allow users to express basic control flow without writing code, but
you can always subclass and customize to achieve more specialized
behavior.

<ul class="menu">
<li><a accesskey="1" href="#Build-Steps">Build Steps</a>
<li><a accesskey="2" href="#Interlocks">Interlocks</a>
<li><a accesskey="3" href="#Build-Factories">Build Factories</a>
</ul>

<div class="node">
<p><hr>
<a name="Build-Steps"></a>
Next:&nbsp;<a rel="next" accesskey="n" href="#Interlocks">Interlocks</a>,
Previous:&nbsp;<a rel="previous" accesskey="p" href="#Build-Process">Build Process</a>,
Up:&nbsp;<a rel="up" accesskey="u" href="#Build-Process">Build Process</a>

</div>

<h3 class="section">6.1 Build Steps</h3>

<p><code>BuildStep</code>s are usually specified in the buildmaster's
configuration file, in a list of &ldquo;step specifications&rdquo; that is used
to create the <code>BuildFactory</code>. These &ldquo;step specifications&rdquo; are
not actual steps, but rather a tuple of the <code>BuildStep</code> subclass
to be created and a dictionary of arguments. (the actual
<code>BuildStep</code> instances are not created until the Build is started,
so that each Build gets an independent copy of each BuildStep). The
preferred way to create these step specifications is with the
<code>BuildFactory</code>'s <code>addStep</code> method:

<pre class="example">     from buildbot.process import step, factory
     
     f = factory.BuildFactory()
     f.addStep(step.SVN, svnurl="http://svn.example.org/Trunk/")
     f.addStep(step.ShellCommand, command=["make", "all"])
     f.addStep(step.ShellCommand, command=["make", "test"])
</pre>
   <p>The rest of this section lists all the standard BuildStep objects
available for use in a Build, and the parameters which can be used to
control each.

<ul class="menu">
<li><a accesskey="1" href="#Common-Parameters">Common Parameters</a>
<li><a accesskey="2" href="#Source-Checkout">Source Checkout</a>
<li><a accesskey="3" href="#ShellCommand">ShellCommand</a>
<li><a accesskey="4" href="#Simple-ShellCommand-Subclasses">Simple ShellCommand Subclasses</a>
<li><a accesskey="5" href="#Writing-New-BuildSteps">Writing New BuildSteps</a>
</ul>

<div class="node">
<p><hr>
<a name="Common-Parameters"></a>
Next:&nbsp;<a rel="next" accesskey="n" href="#Source-Checkout">Source Checkout</a>,
Previous:&nbsp;<a rel="previous" accesskey="p" href="#Build-Steps">Build Steps</a>,
Up:&nbsp;<a rel="up" accesskey="u" href="#Build-Steps">Build Steps</a>

</div>

<h4 class="subsection">6.1.1 Common Parameters</h4>

<p>The standard <code>Build</code> runs a series of <code>BuildStep</code>s in order,
only stopping when it runs out of steps or if one of them requests
that the build be halted. It collects status information from each one
to create an overall build status (of SUCCESS, WARNINGS, or FAILURE).

   <p>All BuildSteps accept some common parameters. Some of these control
how their individual status affects the overall build. Others are used
to specify which <code>Locks</code> (see see <a href="#Interlocks">Interlocks</a>) should be
acquired before allowing the step to run.

   <p>Arguments common to all <code>BuildStep</code> subclasses:

     <dl>
<dt><code>name</code><dd>the name used to describe the step on the status display. It is also
used to give a name to any LogFiles created by this step.

     <br><dt><code>haltOnFailure</code><dd>if True, a FAILURE of this build step will cause the build to halt
immediately with an overall result of FAILURE.

     <br><dt><code>flunkOnWarnings</code><dd>when True, a WARNINGS or FAILURE of this build step will mark the
overall build as FAILURE. The remaining steps will still be executed.

     <br><dt><code>flunkOnFailure</code><dd>when True, a FAILURE of this build step will mark the overall build as
a FAILURE. The remaining steps will still be executed.

     <br><dt><code>warnOnWarnings</code><dd>when True, a WARNINGS or FAILURE of this build step will mark the
overall build as having WARNINGS. The remaining steps will still be
executed.

     <br><dt><code>warnOnFailure</code><dd>when True, a FAILURE of this build step will mark the overall build as
having WARNINGS. The remaining steps will still be executed.

     <br><dt><code>locks</code><dd>a list of Locks (instances of <code>buildbot.locks.SlaveLock</code> or
<code>buildbot.locks.MasterLock</code>) that should be acquired before
starting this Step. The Locks will be released when the step is
complete. Note that this is a list of actual Lock instances, not
names. Also note that all Locks must have unique names.

   </dl>

<div class="node">
<p><hr>
<a name="Source-Checkout"></a>
Next:&nbsp;<a rel="next" accesskey="n" href="#ShellCommand">ShellCommand</a>,
Previous:&nbsp;<a rel="previous" accesskey="p" href="#Common-Parameters">Common Parameters</a>,
Up:&nbsp;<a rel="up" accesskey="u" href="#Build-Steps">Build Steps</a>

</div>

<h4 class="subsection">6.1.2 Source Checkout</h4>

<p>The first step of any build is typically to acquire the source code
from which the build will be performed. There are several classes to
handle this, one for each of the different source control system that
Buildbot knows about. For a description of how Buildbot treats source
control in general, see <a href="#Version-Control-Systems">Version Control Systems</a>.

   <p>All source checkout steps accept some common parameters to control how
they get the sources and where they should be placed. The remaining
per-VC-system parameters are mostly to specify where exactly the
sources are coming from.

     <dl>
<dt><code>mode</code><dd>
a string describing the kind of VC operation that is desired. Defaults
to <code>update</code>.

          <dl>
<dt><code>update</code><dd>specifies that the CVS checkout/update should be performed directly
into the workdir. Each build is performed in the same directory,
allowing for incremental builds. This minimizes disk space, bandwidth,
and CPU time. However, it may encounter problems if the build process
does not handle dependencies properly (sometimes you must do a &ldquo;clean
build&rdquo; to make sure everything gets compiled), or if source files are
deleted but generated files can influence test behavior (e.g. python's
.pyc files), or when source directories are deleted but generated
files prevent CVS from removing them. Builds ought to be correct
regardless of whether they are done &ldquo;from scratch&rdquo; or incrementally,
but it is useful to test both kinds: this mode exercises the
incremental-build style.

          <br><dt><code>copy</code><dd>specifies that the CVS workspace should be maintained in a separate
directory (called the 'copydir'), using checkout or update as
necessary. For each build, a new workdir is created with a copy of the
source tree (rm -rf workdir; cp -r copydir workdir). This doubles the
disk space required, but keeps the bandwidth low (update instead of a
full checkout). A full 'clean' build is performed each time. This
avoids any generated-file build problems, but is still occasionally
vulnerable to CVS problems such as a repository being manually
rearranged, causing CVS errors on update which are not an issue with a
full checkout.

          <!-- TODO: something is screwy about this, revisit. Is it the source -->
<!-- directory or the working directory that is deleted each time? -->
<br><dt><code>clobber</code><dd>specifes that the working directory should be deleted each time,
necessitating a full checkout for each build. This insures a clean
build off a complete checkout, avoiding any of the problems described
above. This mode exercises the &ldquo;from-scratch&rdquo; build style.

          <br><dt><code>export</code><dd>this is like <code>clobber</code>, except that the 'cvs export' command is
used to create the working directory. This command removes all CVS
metadata files (the CVS/ directories) from the tree, which is
sometimes useful for creating source tarballs (to avoid including the
metadata in the tar file). 
</dl>

     <br><dt><code>workdir</code><dd>like all Steps, this indicates the directory where the build will take
place. Source Steps are special in that they perform some operations
outside of the workdir (like creating the workdir itself).

     <br><dt><code>alwaysUseLatest</code><dd>if True, bypass the usual &ldquo;update to the last Change&rdquo; behavior, and
always update to the latest changes instead.

     <br><dt><code>retry</code><dd>If set, this specifies a tuple of <code>(delay, repeats)</code> which means
that when a full VC checkout fails, it should be retried up to
<var>repeats</var> times, waiting <var>delay</var> seconds between attempts. If
you don't provide this, it defaults to <code>None</code>, which means VC
operations should not be retried. This is provided to make life easier
for buildslaves which are stuck behind poor network connections.

   </dl>

   <p>My habit as a developer is to do a <code>cvs update</code> and <code>make</code> each
morning. Problems can occur, either because of bad code being checked in, or
by incomplete dependencies causing a partial rebuild to fail where a
complete from-scratch build might succeed. A quick Builder which emulates
this incremental-build behavior would use the <code>mode='update'</code>
setting.

   <p>On the other hand, other kinds of dependency problems can cause a clean
build to fail where a partial build might succeed. This frequently results
from a link step that depends upon an object file that was removed from a
later version of the tree: in the partial tree, the object file is still
around (even though the Makefiles no longer know how to create it).

   <p>&ldquo;official&rdquo; builds (traceable builds performed from a known set of
source revisions) are always done as clean builds, to make sure it is
not influenced by any uncontrolled factors (like leftover files from a
previous build). A &ldquo;full&rdquo; Builder which behaves this way would want
to use the <code>mode='clobber'</code> setting.

   <p>Each VC system has a corresponding source checkout class: their
arguments are described on the following pages.

<ul class="menu">
<li><a accesskey="1" href="#CVS">CVS</a>
<li><a accesskey="2" href="#SVN">SVN</a>
<li><a accesskey="3" href="#Darcs">Darcs</a>
<li><a accesskey="4" href="#Mercurial">Mercurial</a>
<li><a accesskey="5" href="#Arch">Arch</a>
<li><a accesskey="6" href="#Bazaar">Bazaar</a>
<li><a accesskey="7" href="#P4">P4</a>
</ul>

<div class="node">
<p><hr>
<a name="CVS"></a>
Next:&nbsp;<a rel="next" accesskey="n" href="#SVN">SVN</a>,
Previous:&nbsp;<a rel="previous" accesskey="p" href="#Source-Checkout">Source Checkout</a>,
Up:&nbsp;<a rel="up" accesskey="u" href="#Source-Checkout">Source Checkout</a>

</div>

<h5 class="subsubsection">6.1.2.1 CVS</h5>

<p><a name="index-CVS-Checkout-36"></a><a name="index-buildbot_002eprocess_002estep_002eCVS-37"></a>

   <p>The <code>CVS</code> build step performs a <a href="http://www.nongnu.org/cvs/">CVS</a> checkout or update. It takes the following arguments:

     <dl>
<dt><code>cvsroot</code><dd>(required): specify the CVSROOT value, which points to a CVS
repository, probably on a remote machine. For example, the cvsroot
value you would use to get a copy of the Buildbot source code is
<code>:pserver:anonymous@cvs.sourceforge.net:/cvsroot/buildbot</code>

     <br><dt><code>cvsmodule</code><dd>(required): specify the cvs <code>module</code>, which is generally a
subdirectory of the CVSROOT. The cvsmodule for the Buildbot source
code is <code>buildbot</code>.

     <br><dt><code>branch</code><dd>a string which will be used in a <code>-r</code> argument. This is most
useful for specifying a branch to work on. Defaults to <code>HEAD</code>.

     <br><dt><code>global_options</code><dd>a list of flags to be put before the verb in the CVS command.

     <br><dt><code>checkoutDelay</code><dd>if set, the number of seconds to put between the timestamp of the last
known Change and the value used for the <code>-D</code> option. Defaults to
half of the parent Build's treeStableTimer.

   </dl>

<div class="node">
<p><hr>
<a name="SVN"></a>
Next:&nbsp;<a rel="next" accesskey="n" href="#Darcs">Darcs</a>,
Previous:&nbsp;<a rel="previous" accesskey="p" href="#CVS">CVS</a>,
Up:&nbsp;<a rel="up" accesskey="u" href="#Source-Checkout">Source Checkout</a>

</div>

<h5 class="subsubsection">6.1.2.2 SVN</h5>

<p><a name="index-SVN-Checkout-38"></a><a name="index-buildbot_002eprocess_002estep_002eSVN-39"></a>

   <p>The <code>SVN</code> build step performs a
<a href="http://subversion.tigris.org">Subversion</a> checkout or update. 
There are two basic ways of setting up the checkout step, depending
upon whether you are using multiple branches or not.

   <p>If all of your builds use the same branch, then you should create the
<code>SVN</code> step with the <code>svnurl</code> argument:

     <dl>
<dt><code>svnurl</code><dd>(required): this specifies the <code>URL</code> argument that will be given
to the <code>svn checkout</code> command. It dictates both where the
repository is located and which sub-tree should be extracted. In this
respect, it is like a combination of the CVS <code>cvsroot</code> and
<code>cvsmodule</code> arguments. For example, if you are using a remote
Subversion repository which is accessible through HTTP at a URL of
<code>http://svn.example.com/repos</code>, and you wanted to check out the
<code>trunk/calc</code> sub-tree, you would use
<code>svnurl="http://svn.example.com/repos/trunk/calc"</code> as an argument
to your <code>SVN</code> step. 
</dl>

   <p>If, on the other hand, you are building from multiple branches, then
you should create the <code>SVN</code> step with the <code>baseURL</code> and
<code>defaultBranch</code> arguments instead:

     <dl>
<dt><code>baseURL</code><dd>(required): this specifies the base repository URL, to which a branch
name will be appended. It should probably end in a slash.

     <br><dt><code>defaultBranch</code><dd>this specifies the name of the branch to use when a Build does not
provide one of its own. This will be appended to <code>baseURL</code> to
create the string that will be passed to the <code>svn checkout</code>
command. 
</dl>

   <p>If you are using branches, you must also make sure your
<code>ChangeSource</code> will report the correct branch names.

<h3 class="heading">branch example</h3>

<p>Let's suppose that the &ldquo;MyProject&rdquo; repository uses branches for the
trunk, for various users' individual development efforts, and for
several new features that will require some amount of work (involving
multiple developers) before they are ready to merge onto the trunk. 
Such a repository might be organized as follows:

<pre class="example">     svn://svn.example.org/MyProject/trunk
     svn://svn.example.org/MyProject/branches/User1/foo
     svn://svn.example.org/MyProject/branches/User1/bar
     svn://svn.example.org/MyProject/branches/User2/baz
     svn://svn.example.org/MyProject/features/newthing
     svn://svn.example.org/MyProject/features/otherthing
</pre>
   <p>Further assume that we want the Buildbot to run tests against the
trunk and against all the feature branches (i.e., do a
checkout/compile/build of branch X when a file has been changed on
branch X, when X is in the set [trunk, features/newthing,
features/otherthing]). We do not want the Buildbot to automatically
build any of the user branches, but it should be willing to build a
user branch when explicitly requested (most likely by the user who
owns that branch).

   <p>There are three things that need to be set up to accomodate this
system. The first is a ChangeSource that is capable of identifying the
branch which owns any given file. This depends upon a user-supplied
function, in an external program that runs in the SVN commit hook and
connects to the buildmaster's <code>PBChangeSource</code> over a TCP
connection. (you can use the &ldquo;<code>buildbot sendchange</code>&rdquo; utility
for this purpose, but you will still need an external program to
decide what value should be passed to the <code>--branch=</code> argument). 
For example, a change to a file with the SVN url of
&ldquo;svn://svn.example.org/MyProject/features/newthing/src/foo.c&rdquo; should
be broken down into a Change instance with
<code>branch='features/newthing'</code> and <code>file='src/foo.c'</code>.

   <p>The second piece is an <code>AnyBranchScheduler</code> which will pay
attention to the desired branches. It will not pay attention to the
user branches, so it will not automatically start builds in response
to changes there. The AnyBranchScheduler class requires you to
explicitly list all the branches you want it to use, but it would not
be difficult to write a subclass which used
<code>branch.startswith('features/'</code> to remove the need for this
explicit list. Or, if you want to build user branches too, you can use
AnyBranchScheduler with <code>branches=None</code> to indicate that you want
it to pay attention to all branches.

   <p>The third piece is an <code>SVN</code> checkout step that is configured to
handle the branches correctly, with a <code>baseURL</code> value that
matches the way the ChangeSource splits each file's URL into base,
branch, and file.

<pre class="example">     from buildbot.changes.pb import PBChangeSource
     from buildbot.scheduler import AnyBranchScheduler
     from buildbot.process import step, factory
     
     c['sources'] = [PBChangeSource()]
     s1 = AnyBranchScheduler('main',
                             ['trunk', 'features/newthing', 'features/otherthing'],
                             10*60, ['test-i386', 'test-ppc'])
     c['schedulers'] = [s1]
     
     f = factory.BuildFactory()
     f.addStep(step.SVN, mode='update',
               baseURL='svn://svn.example.org/MyProject/',
               defaultBranch='trunk')
     f.addStep(step.Compile, command="make all")
     f.addStep(step.Test, command="make test")
     
     c['builders'] = [
       {'name':'test-i386', 'slavename':'bot-i386', 'builddir':'test-i386',
                            'factory':f },
       {'name':'test-ppc', 'slavename':'bot-ppc', 'builddir':'test-ppc',
                           'factory':f },
      ]
</pre>
   <p>In this example, when a change arrives with a <code>branch</code> attribute
of &ldquo;trunk&rdquo;, the resulting build will have an SVN step that
concatenates &ldquo;svn://svn.example.org/MyProject/&rdquo; (the baseURL) with
&ldquo;trunk&rdquo; (the branch name) to get the correct svn command. If the
&ldquo;newthing&rdquo; branch has a change to &ldquo;src/foo.c&rdquo;, then the SVN step
will concatenate &ldquo;svn://svn.example.org/MyProject/&rdquo; with
&ldquo;features/newthing&rdquo; to get the svnurl for checkout.

<div class="node">
<p><hr>
<a name="Darcs"></a>
Next:&nbsp;<a rel="next" accesskey="n" href="#Mercurial">Mercurial</a>,
Previous:&nbsp;<a rel="previous" accesskey="p" href="#SVN">SVN</a>,
Up:&nbsp;<a rel="up" accesskey="u" href="#Source-Checkout">Source Checkout</a>

</div>

<h5 class="subsubsection">6.1.2.3 Darcs</h5>

<p><a name="index-Darcs-Checkout-40"></a><a name="index-buildbot_002eprocess_002estep_002eDarcs-41"></a>

   <p>The <code>Darcs</code> build step performs a
<a href="http://abridgegame.org/darcs/">Darcs</a> checkout or update.

   <p>Like See <a href="#SVN">SVN</a>, this step can either be configured to always check
out a specific tree, or set up to pull from a particular branch that
gets specified separately for each build. Also like SVN, the
repository URL given to Darcs is created by concatenating a
<code>baseURL</code> with the branch name, and if no particular branch is
requested, it uses a <code>defaultBranch</code>. The only difference in
usage is that each potential Darcs repository URL must point to a
fully-fledged repository, whereas SVN URLs usually point to sub-trees
of the main Subversion repository. In other words, doing an SVN
checkout of <code>baseURL</code> is legal, but silly, since you'd probably
wind up with a copy of every single branch in the whole repository. 
Doing a Darcs checkout of <code>baseURL</code> is just plain wrong, since
the parent directory of a collection of Darcs repositories is not
itself a valid repository.

   <p>The Darcs step takes the following arguments:

     <dl>
<dt><code>repourl</code><dd>(required unless <code>baseURL</code> is provided): the URL at which the
Darcs source repository is available.

     <br><dt><code>baseURL</code><dd>(required unless <code>repourl</code> is provided): the base repository URL,
to which a branch name will be appended. It should probably end in a
slash.

     <br><dt><code>defaultBranch</code><dd>(allowed if and only if <code>baseURL</code> is provided): this specifies
the name of the branch to use when a Build does not provide one of its
own. This will be appended to <code>baseURL</code> to create the string that
will be passed to the <code>darcs get</code> command. 
</dl>

<div class="node">
<p><hr>
<a name="Mercurial"></a>
Next:&nbsp;<a rel="next" accesskey="n" href="#Arch">Arch</a>,
Previous:&nbsp;<a rel="previous" accesskey="p" href="#Darcs">Darcs</a>,
Up:&nbsp;<a rel="up" accesskey="u" href="#Source-Checkout">Source Checkout</a>

</div>

<h5 class="subsubsection">6.1.2.4 Mercurial</h5>

<p><a name="index-Mercurial-Checkout-42"></a><a name="index-buildbot_002eprocess_002estep_002eMercurial-43"></a>

   <p>The <code>Mercurial</code> build step performs a
<a href="http://selenic.com/mercurial">Mercurial</a> (aka &ldquo;hg&rdquo;) checkout
or update.

   <p>Branches are handled just like See <a href="#Darcs">Darcs</a>.

   <p>The Mercurial step takes the following arguments:

     <dl>
<dt><code>repourl</code><dd>(required unless <code>baseURL</code> is provided): the URL at which the
Mercurial source repository is available.

     <br><dt><code>baseURL</code><dd>(required unless <code>repourl</code> is provided): the base repository URL,
to which a branch name will be appended. It should probably end in a
slash.

     <br><dt><code>defaultBranch</code><dd>(allowed if and only if <code>baseURL</code> is provided): this specifies
the name of the branch to use when a Build does not provide one of its
own. This will be appended to <code>baseURL</code> to create the string that
will be passed to the <code>hg clone</code> command. 
</dl>

<div class="node">
<p><hr>
<a name="Arch"></a>
Next:&nbsp;<a rel="next" accesskey="n" href="#Bazaar">Bazaar</a>,
Previous:&nbsp;<a rel="previous" accesskey="p" href="#Mercurial">Mercurial</a>,
Up:&nbsp;<a rel="up" accesskey="u" href="#Source-Checkout">Source Checkout</a>

</div>

<h5 class="subsubsection">6.1.2.5 Arch</h5>

<p><a name="index-Arch-Checkout-44"></a><a name="index-buildbot_002eprocess_002estep_002eArch-45"></a>

   <p>The <code>Arch</code> build step performs an <a href="http://gnuarch.org/">Arch</a> checkout or update using the <code>tla</code> client. It takes the
following arguments:

     <dl>
<dt><code>url</code><dd>(required): this specifies the URL at which the Arch source archive is
available.

     <br><dt><code>version</code><dd>(required): this specifies which &ldquo;development line&rdquo; (like a branch)
should be used. This provides the default branch name, but individual
builds may specify a different one.

     <br><dt><code>archive</code><dd>(optional): Each repository knows its own archive name. If this
parameter is provided, it must match the repository's archive name. 
The parameter is accepted for compatibility with the <code>Bazaar</code>
step, below.

   </dl>

<div class="node">
<p><hr>
<a name="Bazaar"></a>
Next:&nbsp;<a rel="next" accesskey="n" href="#P4">P4</a>,
Previous:&nbsp;<a rel="previous" accesskey="p" href="#Arch">Arch</a>,
Up:&nbsp;<a rel="up" accesskey="u" href="#Source-Checkout">Source Checkout</a>

</div>

<h5 class="subsubsection">6.1.2.6 Bazaar</h5>

<p><a name="index-Bazaar-Checkout-46"></a><a name="index-buildbot_002eprocess_002estep_002eBazaar-47"></a>

   <p><code>Bazaar</code> is an alternate implementation of the Arch VC system,
which uses a client named <code>baz</code>. The checkout semantics are just
different enough from <code>tla</code> that there is a separate BuildStep for
it.

   <p>It takes exactly the same arguments as <code>Arch</code>, except that the
<code>archive=</code> parameter is required. (baz does not emit the archive
name when you do <code>baz register-archive</code>, so we must provide it
ourselves).

<div class="node">
<p><hr>
<a name="P4"></a>
Previous:&nbsp;<a rel="previous" accesskey="p" href="#Bazaar">Bazaar</a>,
Up:&nbsp;<a rel="up" accesskey="u" href="#Source-Checkout">Source Checkout</a>

</div>

<h5 class="subsubsection">6.1.2.7 P4</h5>

<p><a name="index-Perforce-Update-48"></a><a name="index-buildbot_002eprocess_002estep_002eP4-49"></a><!-- TODO @bsindex buildbot.process.step.P4Sync -->

   <p>The <code>P4</code> build step creates a <a href="http://www.perforce.com/">Perforce</a> client specification and performs an update.

     <dl>
<dt><code>p4base</code><dd>A view into the Perforce depot without branch name or trailing "...". 
Typically "//depot/proj/". 
<br><dt><code>defaultBranch</code><dd>A branch name to append on build requests if none is specified. 
Typically "trunk". 
<br><dt><code>p4port</code><dd>(optional): the host:port string describing how to get to the P4 Depot
(repository), used as the -p argument for all p4 commands. 
<br><dt><code>p4user</code><dd>(optional): the Perforce user, used as the -u argument to all p4
commands. 
<br><dt><code>p4passwd</code><dd>(optional): the Perforce password, used as the -p argument to all p4
commands. 
<br><dt><code>p4extra_views</code><dd>(optional): a list of (depotpath, clientpath) tuples containing extra
views to be mapped into the client specification. Both will have
"/..." appended automatically. The client name and source directory
will be prepended to the client path. 
<br><dt><code>p4client</code><dd>(optional): The name of the client to use. In mode='copy' and
mode='update', it's particularly important that a unique name is used
for each checkout directory to avoid incorrect synchronization. For
this reason, Python percent substitution will be performed on this value
to replace %(slave)s with the slave name and %(builder)s with the
builder name. The default is "buildbot_%(slave)s_%(build)s". 
</dl>

<div class="node">
<p><hr>
<a name="ShellCommand"></a>
Next:&nbsp;<a rel="next" accesskey="n" href="#Simple-ShellCommand-Subclasses">Simple ShellCommand Subclasses</a>,
Previous:&nbsp;<a rel="previous" accesskey="p" href="#Source-Checkout">Source Checkout</a>,
Up:&nbsp;<a rel="up" accesskey="u" href="#Build-Steps">Build Steps</a>

</div>

<h4 class="subsection">6.1.3 ShellCommand</h4>

<p><a name="index-buildbot_002eprocess_002estep_002eShellCommand-50"></a><!-- TODO @bsindex buildbot.process.step.TreeSize -->

   <p>This is a useful base class for just about everything you might want
to do during a build (except for the initial source checkout). It runs
a single command in a child shell on the buildslave. All stdout/stderr
is recorded into a LogFile. The step finishes with a status of FAILURE
if the command's exit code is non-zero, otherwise it has a status of
SUCCESS.

   <p>The preferred way to specify the command is with a list of argv strings,
since this allows for spaces in filenames and avoids doing any fragile
shell-escaping. You can also specify the command with a single string, in
which case the string is given to '/bin/sh -c COMMAND' for parsing.

   <p>All ShellCommands are run by default in the &ldquo;workdir&rdquo;, which
defaults to the &ldquo;<samp><span class="file">build</span></samp>&rdquo; subdirectory of the slave builder's
base directory. The absolute path of the workdir will thus be the
slave's basedir (set as an option to <code>buildbot create-slave</code>,
see <a href="#Creating-a-buildslave">Creating a buildslave</a>) plus the builder's basedir (set in the
builder's <code>c['builddir']</code> key in master.cfg) plus the workdir
itself (a class-level attribute of the BuildFactory, defaults to
&ldquo;<samp><span class="file">build</span></samp>&rdquo;).

   <p><code>ShellCommand</code> arguments:

     <dl>
<dt><code>command</code><dd>a list of strings (preferred) or single string (discouraged) which
specifies the command to be run. A list of strings is preferred
because it can be used directly as an argv array. Using a single
string (with embedded spaces) requires the buildslave to pass the
string to /bin/sh for interpretation, which raises all sorts of
difficult questions about how to escape or interpret shell
metacharacters.

     <br><dt><code>env</code><dd>a dictionary of environment strings which will be added to the child
command's environment. For example, to run tests with a different i18n
language setting, you might use

     <pre class="example">          f.addStep(ShellCommand, command=["make", "test"],
                    env={'LANG': 'fr_FR'})
     </pre>
     <p>These variable settings will override any existing ones in the
buildslave's environment. The exception is PYTHONPATH, which is merged
with (actually prepended to) any existing $PYTHONPATH setting. The
value is treated as a list of directories to prepend, and a single
string is treated like a one-item list. For example, to prepend both
<samp><span class="file">/usr/local/lib/python2.3</span></samp> and <samp><span class="file">/home/buildbot/lib/python</span></samp>
to any existing $PYTHONPATH setting, you would do something like the
following:

     <pre class="example">          f.addStep(ShellCommand, command=["make", "test"],
                    env={'PYTHONPATH': ["/usr/local/lib/python2.3",
                                        "/home/buildbot/lib/python"] })
     </pre>
     <br><dt><code>want_stdout</code><dd>if False, stdout from the child process is discarded rather than being
sent to the buildmaster for inclusion in the step's LogFile.

     <br><dt><code>want_stderr</code><dd>like <code>want_stdout</code> but for stderr. Note that commands run through
a PTY do not have separate stdout/stderr streams: both are merged into
stdout.

     <br><dt><code>logfiles</code><dd>Sometimes commands will log interesting data to a local file, rather
than emitting everything to stdout or stderr. For example, Twisted's
&ldquo;trial&rdquo; command (which runs unit tests) only presents summary
information to stdout, and puts the rest into a file named
<samp><span class="file">_trial_temp/test.log</span></samp>. It is often useful to watch these files
as the command runs, rather than using <samp><span class="command">/bin/cat</span></samp> to dump
their contents afterwards.

     <p>The <code>logfiles=</code> argument allows you to collect data from these
secondary logfiles in near-real-time, as the step is running. It
accepts a dictionary which maps from a local Log name (which is how
the log data is presented in the build results) to a remote filename
(interpreted relative to the build's working directory). Each named
file will be polled on a regular basis (every couple of seconds) as
the build runs, and any new text will be sent over to the buildmaster.

     <pre class="example">          f.addStep(ShellCommand, command=["make", "test"],
                    logfiles={"triallog": "_trial_temp/test.log"})
     </pre>
     <br><dt><code>timeout</code><dd>if the command fails to produce any output for this many seconds, it
is assumed to be locked up and will be killed.

     <br><dt><code>description</code><dd>This will be used to describe the command (on the Waterfall display)
while the command is still running. It should be a single
imperfect-tense verb, like &ldquo;compiling&rdquo; or &ldquo;testing&rdquo;. The preferred
form is a list of short strings, which allows the HTML Waterfall
display to create narrower columns by emitting a &lt;br&gt; tag between each
word. You may also provide a single string.

     <br><dt><code>descriptionDone</code><dd>This will be used to describe the command once it has finished. A
simple noun like &ldquo;compile&rdquo; or &ldquo;tests&rdquo; should be used. Like
<code>description</code>, this may either be a list of short strings or a
single string.

     <p>If neither <code>description</code> nor <code>descriptionDone</code> are set, the
actual command arguments will be used to construct the description. 
This may be a bit too wide to fit comfortably on the Waterfall
display.

     <pre class="example">          f.addStep(ShellCommand, command=["make", "test"],
                    description=["testing"],
                    descriptionDone=["tests"])
     </pre>
     </dl>

<div class="node">
<p><hr>
<a name="Simple-ShellCommand-Subclasses"></a>
Next:&nbsp;<a rel="next" accesskey="n" href="#Writing-New-BuildSteps">Writing New BuildSteps</a>,
Previous:&nbsp;<a rel="previous" accesskey="p" href="#ShellCommand">ShellCommand</a>,
Up:&nbsp;<a rel="up" accesskey="u" href="#Build-Steps">Build Steps</a>

</div>

<h4 class="subsection">6.1.4 Simple ShellCommand Subclasses</h4>

<p>Several subclasses of ShellCommand are provided as starting points for
common build steps. These are all very simple: they just override a few
parameters so you don't have to specify them yourself, making the master.cfg
file less verbose.

<ul class="menu">
<li><a accesskey="1" href="#Configure">Configure</a>
<li><a accesskey="2" href="#Compile">Compile</a>
<li><a accesskey="3" href="#Test">Test</a>
<li><a accesskey="4" href="#Build-Properties">Build Properties</a>
</ul>

<div class="node">
<p><hr>
<a name="Configure"></a>
Next:&nbsp;<a rel="next" accesskey="n" href="#Compile">Compile</a>,
Previous:&nbsp;<a rel="previous" accesskey="p" href="#Simple-ShellCommand-Subclasses">Simple ShellCommand Subclasses</a>,
Up:&nbsp;<a rel="up" accesskey="u" href="#Simple-ShellCommand-Subclasses">Simple ShellCommand Subclasses</a>

</div>

<h5 class="subsubsection">6.1.4.1 Configure</h5>

<p><a name="index-buildbot_002eprocess_002estep_002eConfigure-51"></a>
This is intended to handle the <code>./configure</code> step from
autoconf-style projects, or the <code>perl Makefile.PL</code> step from perl
MakeMaker.pm-style modules. The default command is <code>./configure</code>
but you can change this by providing a <code>command=</code> parameter.

<div class="node">
<p><hr>
<a name="Compile"></a>
Next:&nbsp;<a rel="next" accesskey="n" href="#Test">Test</a>,
Previous:&nbsp;<a rel="previous" accesskey="p" href="#Configure">Configure</a>,
Up:&nbsp;<a rel="up" accesskey="u" href="#Simple-ShellCommand-Subclasses">Simple ShellCommand Subclasses</a>

</div>

<h5 class="subsubsection">6.1.4.2 Compile</h5>

<p><a name="index-buildbot_002eprocess_002estep_002eCompile-52"></a>
This is meant to handle compiling or building a project written in C. The
default command is <code>make all</code>. When the compile is finished, the
log file is scanned for GCC error/warning messages and a summary log is
created with any problems that were seen (TODO: the summary is not yet
created).

<div class="node">
<p><hr>
<a name="Test"></a>
Next:&nbsp;<a rel="next" accesskey="n" href="#Build-Properties">Build Properties</a>,
Previous:&nbsp;<a rel="previous" accesskey="p" href="#Compile">Compile</a>,
Up:&nbsp;<a rel="up" accesskey="u" href="#Simple-ShellCommand-Subclasses">Simple ShellCommand Subclasses</a>

</div>

<h5 class="subsubsection">6.1.4.3 Test</h5>

<p><a name="index-buildbot_002eprocess_002estep_002eTest-53"></a>
This is meant to handle unit tests. The default command is <code>make
test</code>, and the <code>warnOnFailure</code> flag is set.

<div class="node">
<p><hr>
<a name="Build-Properties"></a>
Previous:&nbsp;<a rel="previous" accesskey="p" href="#Test">Test</a>,
Up:&nbsp;<a rel="up" accesskey="u" href="#Simple-ShellCommand-Subclasses">Simple ShellCommand Subclasses</a>

</div>

<h5 class="subsubsection">6.1.4.4 Build Properties</h5>

<p><a name="index-build-properties-54"></a>
Each build has a set of &ldquo;Build Properties&rdquo;, which can be used by its
BuildStep to modify their actions. For example, the SVN revision
number of the source code being built is available as a build
property, and a ShellCommand step could incorporate this number into a
command which create a numbered release tarball.

   <p>Some build properties are set when the build starts, such as the
SourceStamp information. Other properties can be set by BuildSteps as
they run, for example the various Source steps will set the
<code>got_revision</code> property to the source revision that was actually
checked out (which can be useful when the SourceStamp in use merely
requested the &ldquo;latest revision&rdquo;: <code>got_revision</code> will tell you
what was actually built).

   <p>In custom BuildSteps, you can get and set the build properties with
the <code>getProperty</code>/<code>setProperty</code> methods. Each takes a string
for the name of the property, and returns or accepts an
arbitrary<a rel="footnote" href="#fn-7" name="fnd-7"><sup>7</sup></a> object. For example:

<pre class="example">     class MakeTarball(step.ShellCommand):
         def start(self):
             self.setCommand(["tar", "czf",
                              "build-%s.tar.gz" % self.getProperty("revision"),
                              "source"])
             step.ShellCommand.start(self)
</pre>
   <p><a name="index-WithProperties-55"></a>
You can use build properties in ShellCommands by using the
<code>WithProperties</code> wrapper when setting the arguments of the
ShellCommand. This interpolates the named build properties into the
generated shell command.

<pre class="example">     from buildbot.process.step import ShellCommand, WithProperties
     
     f.addStep(ShellCommand,
               command=["tar", "czf",
                        WithProperties("build-%s.tar.gz", "revision"),
                        "source"])
</pre>
   <p>If this BuildStep were used in a tree obtained from Subversion, it
would create a tarball with a name like <samp><span class="file">build-1234.tar.gz</span></samp>.

   <p>The <code>WithProperties</code> function does <code>printf</code>-style string
interpolation, using strings obtained by calling
<code>build.getProperty(propname)</code>. Note that for every <code>%s</code> (or
<code>%d</code>, etc), you must have exactly one additional argument to
indicate which build property you want to insert.

   <p>You can also use python dictionary-style string interpolation by using
the <code>%(propname)s</code> syntax. In this form, the property name goes
in the parentheses, and WithProperties takes <em>no</em> additional
arguments:

<pre class="example">     f.addStep(ShellCommand,
               command=["tar", "czf",
                        WithProperties("build-%(revision)s.tar.gz"),
                        "source"])
</pre>
   <p>Don't forget the extra &ldquo;s&rdquo; after the closing parenthesis! This is
the cause of many confusing errors.

   <p>Note that, like python, you can either do positional-argument
interpolation <em>or</em> keyword-argument interpolation, not both. Thus
you cannot use a string like
<code>WithProperties("foo-%(revision)s-%s", "branch")</code>.

   <p>At the moment, the only way to set build properties is by writing a
custom BuildStep.

<h3 class="heading">Common Build Properties</h3>

<p>The following build properties are set when the build is started, and
are available to all steps.

     <dl>
<dt><code>branch</code><dd>
This comes from the build's SourceStamp, and describes which branch is
being checked out. This will be <code>None</code> (which interpolates into
<code>WithProperties</code> as an empty string) if the build is on the
default branch, which is generally the trunk. Otherwise it will be a
string like &ldquo;branches/beta1.4&rdquo;. The exact syntax depends upon the VC
system being used.

     <br><dt><code>revision</code><dd>
This also comes from the SourceStamp, and is the revision of the
source code tree that was requested from the VC system. When a build
is requested of a specific revision (as is generally the case when the
build is triggered by Changes), this will contain the revision
specification. The syntax depends upon the VC system in use: for SVN
it is an integer, for Mercurial it is a short string, for Darcs it is
a rather large string, etc.

     <p>If the &ldquo;force build&rdquo; button was pressed, the revision will be
<code>None</code>, which means to use the most recent revision available. 
This is a &ldquo;trunk build&rdquo;. This will be interpolated as an empty
string.

     <br><dt><code>got_revision</code><dd>
This is set when a Source step checks out the source tree, and
provides the revision that was actually obtained from the VC system. 
In general this should be the same as <code>revision</code>, except for
trunk builds, where <code>got_revision</code> indicates what revision was
current when the checkout was performed. This can be used to rebuild
the same source code later.

     <p>Note that for some VC systems (Darcs in particular), the revision is a
large string containing newlines, and is not suitable for
interpolation into a filename.

     <br><dt><code>buildername</code><dd>
This is a string that indicates which Builder the build was a part of. 
The combination of buildername and buildnumber uniquely identify a
build.

     <br><dt><code>buildnumber</code><dd>
Each build gets a number, scoped to the Builder (so the first build
performed on any given Builder will have a build number of 0). This
integer property contains the build's number.

     <br><dt><code>slavename</code><dd>
This is a string which identifies which buildslave the build is
running on.

   </dl>

<div class="node">
<p><hr>
<a name="Writing-New-BuildSteps"></a>
Previous:&nbsp;<a rel="previous" accesskey="p" href="#Simple-ShellCommand-Subclasses">Simple ShellCommand Subclasses</a>,
Up:&nbsp;<a rel="up" accesskey="u" href="#Build-Steps">Build Steps</a>

</div>

<h4 class="subsection">6.1.5 Writing New BuildSteps</h4>

<p>While it is a good idea to keep your build process self-contained in
the source code tree, sometimes it is convenient to put more
intelligence into your Buildbot configuration. One was to do this is
to write a custom BuildStep. Once written, this Step can be used in
the <samp><span class="file">master.cfg</span></samp> file.

   <p>The best reason for writing a custom BuildStep is to better parse the
results of the command being run. For example, a BuildStep that knows
about JUnit could look at the logfiles to determine which tests had
been run, how many passed and how many failed, and then report more
detailed information than a simple <code>rc==0</code> -based &ldquo;good/bad&rdquo;
decision.

   <p>TODO: add more description of BuildSteps.

<ul class="menu">
<li><a accesskey="1" href="#BuildStep-LogFiles">BuildStep LogFiles</a>
<li><a accesskey="2" href="#Adding-LogObservers">Adding LogObservers</a>
<li><a accesskey="3" href="#BuildStep-URLs">BuildStep URLs</a>
</ul>

<div class="node">
<p><hr>
<a name="BuildStep-LogFiles"></a>
Next:&nbsp;<a rel="next" accesskey="n" href="#Adding-LogObservers">Adding LogObservers</a>,
Previous:&nbsp;<a rel="previous" accesskey="p" href="#Writing-New-BuildSteps">Writing New BuildSteps</a>,
Up:&nbsp;<a rel="up" accesskey="u" href="#Writing-New-BuildSteps">Writing New BuildSteps</a>

</div>

<h5 class="subsubsection">6.1.5.1 BuildStep LogFiles</h5>

<p>Each BuildStep has a collection of &ldquo;logfiles&rdquo;. Each one has a short
name, like &ldquo;stdio&rdquo; or &ldquo;warnings&rdquo;. Each LogFile contains an
arbitrary amount of text, usually the contents of some output file
generated during a build or test step, or a record of everything that
was printed to stdout/stderr during the execution of some command.

   <p>These LogFiles are stored to disk, so they can be retrieved later.

   <p>Each can contain multiple &ldquo;channels&rdquo;, generally limited to three
basic ones: stdout, stderr, and &ldquo;headers&rdquo;. For example, when a
ShellCommand runs, it writes a few lines to the &ldquo;headers&rdquo; channel to
indicate the exact argv strings being run, which directory the command
is being executed in, and the contents of the current environment
variables. Then, as the command runs, it adds a lot of &ldquo;stdout&rdquo; and
&ldquo;stderr&rdquo; messages. When the command finishes, a final &ldquo;header&rdquo;
line is added with the exit code of the process.

   <p>Status display plugins can format these different channels in
different ways. For example, the web page shows LogFiles as text/html,
with header lines in blue text, stdout in black, and stderr in red. A
different URL is available which provides a text/plain format, in
which stdout and stderr are collapsed together, and header lines are
stripped completely. This latter option makes it easy to save the
results to a file and run <samp><span class="command">grep</span></samp> or whatever against the
output.

   <p>Each BuildStep contains a mapping (implemented in a python dictionary)
from LogFile name to the actual LogFile objects. Status plugins can
get a list of LogFiles to display, for example, a list of HREF links
that, when clicked, provide the full contents of the LogFile.

<h3 class="heading">Using LogFiles in custom BuildSteps</h3>

<p>The most common way for a custom BuildStep to use a LogFile is to
summarize the results of a ShellCommand (after the command has
finished running). For example, a compile step with thousands of lines
of output might want to create a summary of just the warning messages. 
If you were doing this from a shell, you would use something like:

<pre class="example">     grep "warning:" output.log &gt;warnings.log
</pre>
   <p>In a custom BuildStep, you could instead create a &ldquo;warnings&rdquo; LogFile
that contained the same text. To do this, you would add code to your
<code>createSummary</code> method that pulls lines from the main output log
and creates a new LogFile with the results:

<pre class="example">         def createSummary(self, log):
             warnings = []
             for line in log.readlines():
                 if "warning:" in line:
                     warnings.append()
             self.addCompleteLog('warnings', "".join(warnings))
</pre>
   <p>This example uses the <code>addCompleteLog</code> method, which creates a
new LogFile, puts some text in it, and then &ldquo;closes&rdquo; it, meaning
that no further contents will be added. This LogFile will appear in
the HTML display under an HREF with the name &ldquo;warnings&rdquo;, since that
is the name of the LogFile.

   <p>You can also use <code>addHTMLLog</code> to create a complete (closed)
LogFile that contains HTML instead of plain text. The normal LogFile
will be HTML-escaped if presented through a web page, but the HTML
LogFile will not. At the moment this is only used to present a pretty
HTML representation of an otherwise ugly exception traceback when
something goes badly wrong during the BuildStep.

   <p>In contrast, you might want to create a new LogFile at the beginning
of the step, and add text to it as the command runs. You can create
the LogFile and attach it to the build by calling <code>addLog</code>, which
returns the LogFile object. You then add text to this LogFile by
calling methods like <code>addStdout</code> and <code>addHeader</code>. When you
are done, you must call the <code>finish</code> method so the LogFile can be
closed. It may be useful to create and populate a LogFile like this
from a LogObserver method See <a href="#Adding-LogObservers">Adding LogObservers</a>.

   <p>The <code>logfiles=</code> argument to <code>ShellCommand</code> (see
see <a href="#ShellCommand">ShellCommand</a>) creates new LogFiles and fills them in realtime
by asking the buildslave to watch a actual file on disk. The
buildslave will look for additions in the target file and report them
back to the BuildStep. These additions will be added to the LogFile by
calling <code>addStdout</code>. These secondary LogFiles can be used as the
source of a LogObserver just like the normal &ldquo;stdio&rdquo; LogFile.

<div class="node">
<p><hr>
<a name="Adding-LogObservers"></a>
Next:&nbsp;<a rel="next" accesskey="n" href="#BuildStep-URLs">BuildStep URLs</a>,
Previous:&nbsp;<a rel="previous" accesskey="p" href="#BuildStep-LogFiles">BuildStep LogFiles</a>,
Up:&nbsp;<a rel="up" accesskey="u" href="#Writing-New-BuildSteps">Writing New BuildSteps</a>

</div>

<h5 class="subsubsection">6.1.5.2 Adding LogObservers</h5>

<p><a name="index-LogObserver-56"></a><a name="index-LogLineObserver-57"></a>
Most shell commands emit messages to stdout or stderr as they operate,
especially if you ask them nicely with a <code>--verbose</code> flag of some
sort. They may also write text to a log file while they run. Your
BuildStep can watch this output as it arrives, to keep track of how
much progress the command has made. You can get a better measure of
progress by counting the number of source files compiled or test cases
run than by merely tracking the number of bytes that have been written
to stdout. This improves the accuracy and the smoothness of the ETA
display.

   <p>To accomplish this, you will need to attach a <code>LogObserver</code> to
one of the log channels, most commonly to the &ldquo;stdio&rdquo; channel but
perhaps to another one which tracks a log file. This observer is given
all text as it is emitted from the command, and has the opportunity to
parse that output incrementally. Once the observer has decided that
some event has occurred (like a source file being compiled), it can
use the <code>setProgress</code> method to tell the BuildStep about the
progress that this event represents.

   <p>There are a number of pre-built <code>LogObserver</code> classes that you
can choose from, and of course you can subclass them to add further
customization. The <code>LogLineObserver</code> class handles the grunt work
of buffering and scanning for end-of-line delimiters, allowing your
parser to operate on complete stdout/stderr lines.

   <p>For example, let's take a look at the <code>TrialTestCaseCounter</code>,
which is used by the Trial step to count test cases as they are run. 
As Trial executes, it emits lines like the following:

<pre class="example">     buildbot.test.test_config.ConfigTest.testDebugPassword ... [OK]
     buildbot.test.test_config.ConfigTest.testEmpty ... [OK]
     buildbot.test.test_config.ConfigTest.testIRC ... [FAIL]
     buildbot.test.test_config.ConfigTest.testLocks ... [OK]
</pre>
   <p>When the tests are finished, trial emits a long line of &ldquo;======&rdquo; and
then some lines which summarize the tests that failed. We want to
avoid parsing these trailing lines, because their format is less
well-defined than the &ldquo;[OK]&rdquo; lines.

   <p>The parser class looks like this:

<pre class="example">     class TrialTestCaseCounter(step.LogLineObserver):
         _line_re = re.compile(r'^([\w\.]+) \.\.\. \[([^\]]+)\]$')
         numTests = 0
         finished = False
     
         def outLineReceived(self, line):
             if self.finished:
                 return
             if line.startswith("=" * 40):
                 self.finished = True
                 return
     
             m = self._line_re.search(line.strip())
             if m:
                 testname, result = m.groups()
                 self.numTests += 1
                 self.step.setProgress('tests', self.numTests)
</pre>
   <p>This parser only pays attention to stdout, since that's where trial
writes the progress lines. It has a mode flag named <code>finished</code> to
ignore everything after the &ldquo;====&rdquo; marker, and a scary-looking
regular expression to match each line while hopefully ignoring other
messages that might get displayed as the test runs.

   <p>Each time it identifies a test has been completed, it increments its
counter and delivers the new progress value to the step with
<code>self.step.setProgress</code>. This class is specifically measuring
progress along the &ldquo;tests&rdquo; metric, in units of test cases (as
opposed to other kinds of progress like the &ldquo;output&rdquo; metric, which
measures in units of bytes). The Progress-tracking code uses each
progress metric separately to come up with an overall completion
percentage and an ETA value.

   <p>To connect this parser into the <code>Trial</code> BuildStep,
<code>Trial.__init__</code> ends with the following clause:

<pre class="example">             # this counter will feed Progress along the 'test cases' metric
             counter = TrialTestCaseCounter()
             self.addLogObserver('stdio', counter)
</pre>
   <p>This creates a TrialTestCaseCounter and tells the step that the
counter wants to watch the &ldquo;stdio&rdquo; log. The observer is
automatically given a reference to the step in its <code>.step</code>
attribute.

<div class="node">
<p><hr>
<a name="BuildStep-URLs"></a>
Previous:&nbsp;<a rel="previous" accesskey="p" href="#Adding-LogObservers">Adding LogObservers</a>,
Up:&nbsp;<a rel="up" accesskey="u" href="#Writing-New-BuildSteps">Writing New BuildSteps</a>

</div>

<h5 class="subsubsection">6.1.5.3 BuildStep URLs</h5>

<p><a name="index-links-58"></a><a name="index-BuildStep-URLs-59"></a><a name="index-addURL-60"></a>
Each BuildStep has a collection of &ldquo;links&rdquo;. Like its collection of
LogFiles, each link has a name and a target URL. The web status page
creates HREFs for each link in the same box as it does for LogFiles,
except that the target of the link is the external URL instead of an
internal link to a page that shows the contents of the LogFile.

   <p>These external links can be used to point at build information hosted
on other servers. For example, the test process might produce an
intricate description of which tests passed and failed, or some sort
of code coverage data in HTML form, or a PNG or GIF image with a graph
of memory usage over time. The external link can provide an easy way
for users to navigate from the buildbot's status page to these
external web sites or file servers. Note that the step itself is
responsible for insuring that there will be a document available at
the given URL (perhaps by using <samp><span class="command">scp</span></samp> to copy the HTML output
to a <samp><span class="file">~/public_html/</span></samp> directory on a remote web server). Calling
<code>addURL</code> does not magically populate a web server.

   <p>To set one of these links, the BuildStep should call the <code>addURL</code>
method with the name of the link and the target URL. Multiple URLs can
be set.

   <p>In this example, we assume that the <samp><span class="command">make test</span></samp> command causes
a collection of HTML files to be created and put somewhere on the
coverage.example.org web server, in a filename that incorporates the
build number.

<pre class="example">     class TestWithCodeCoverage(BuildStep):
         command = ["make", "test",
                    WithProperties("buildnum=%s" % "buildnumber")]
     
         def createSummary(self, log):
             buildnumber = self.getProperty("buildnumber")
             url = "http://coverage.example.org/builds/%s.html" % buildnumber
             self.addURL("coverage", url)
</pre>
   <p>You might also want to extract the URL from some special message
output by the build process itself:

<pre class="example">     class TestWithCodeCoverage(BuildStep):
         command = ["make", "test",
                    WithProperties("buildnum=%s" % "buildnumber")]
     
         def createSummary(self, log):
             output = StringIO(log.getText())
             for line in output.readlines():
                 if line.startswith("coverage-url:"):
                     url = line[len("coverage-url:"):].strip()
                     self.addURL("coverage", url)
                     return
</pre>
   <p>Note that a build process which emits both stdout and stderr might
cause this line to be split or interleaved between other lines. It
might be necessary to restrict the getText() call to only stdout with
something like this:

<pre class="example">             output = StringIO("".join([c[1]
                                        for c in log.getChunks()
                                        if c[0] == LOG_CHANNEL_STDOUT]))
</pre>
   <p>Of course if the build is run under a PTY, then stdout and stderr will
be merged before the buildbot ever sees them, so such interleaving
will be unavoidable.

<div class="node">
<p><hr>
<a name="Interlocks"></a>
Next:&nbsp;<a rel="next" accesskey="n" href="#Build-Factories">Build Factories</a>,
Previous:&nbsp;<a rel="previous" accesskey="p" href="#Build-Steps">Build Steps</a>,
Up:&nbsp;<a rel="up" accesskey="u" href="#Build-Process">Build Process</a>

</div>

<h3 class="section">6.2 Interlocks</h3>

<p><a name="index-locks-61"></a><a name="index-buildbot_002elocks_002eMasterLock-62"></a><a name="index-buildbot_002elocks_002eSlaveLock-63"></a>
For various reasons, you may want to prevent certain Steps (or perhaps
entire Builds) from running simultaneously. Limited CPU speed or
network bandwidth to the VC server, problems with simultaneous access
to a database server used by unit tests, or multiple Builds which
access shared state may all require some kind of interlock to prevent
corruption, confusion, or resource overload.

   <p><code>Locks</code> are the mechanism used to express these kinds of
constraints on when Builds or Steps can be run. There are two kinds of
<code>Locks</code>, each with their own scope: <code>SlaveLock</code>s are scoped
to a single buildslave, while <code>MasterLock</code> instances are scoped
to the buildbot as a whole. Each <code>Lock</code> is created with a unique
name.

   <p>To use a lock, simply include it in the <code>locks=</code> argument of the
<code>BuildStep</code> object that should obtain the lock before it runs. 
This argument accepts a list of <code>Lock</code> objects: the Step will
acquire all of them before it runs.

   <p>To claim a lock for the whole Build, add a <code>'locks'</code> key to the
builder specification dictionary with the same list of <code>Lock</code>
objects. (This is the dictionary that has the <code>'name'</code>,
<code>'slavename'</code>, <code>'builddir'</code>, and <code>'factory'</code> keys). The
<code>Build</code> object also accepts a <code>locks=</code> argument, but unless
you are writing your own <code>BuildFactory</code> subclass then it will be
easier to set the locks in the builder dictionary.

   <p>Note that there are no partial-acquire or partial-release semantics:
this prevents deadlocks caused by two Steps each waiting for a lock
held by the other<a rel="footnote" href="#fn-8" name="fnd-8"><sup>8</sup></a>. This also means
that waiting to acquire a <code>Lock</code> can take an arbitrarily long
time: if the buildmaster is very busy, a Step or Build which requires
only one <code>Lock</code> may starve another that is waiting for that
<code>Lock</code> plus some others.

   <p>In the following example, we run the same build on three different
platforms. The unit-test steps of these builds all use a common
database server, and would interfere with each other if allowed to run
simultaneously. The <code>Lock</code> prevents more than one of these builds
from happening at the same time.

<pre class="example">     from buildbot import locks
     from buildbot.process import step, factory
     
     db_lock = locks.MasterLock("database")
     f = factory.BuildFactory()
     f.addStep(step.SVN, svnurl="http://example.org/svn/Trunk")
     f.addStep(step.ShellCommand, command="make all")
     f.addStep(step.ShellCommand, command="make test", locks=[db_lock])
     b1 = {'name': 'full1', 'slavename': 'bot-1', builddir='f1', 'factory': f}
     b2 = {'name': 'full2', 'slavename': 'bot-2', builddir='f2', 'factory': f}
     b3 = {'name': 'full3', 'slavename': 'bot-3', builddir='f3', 'factory': f}
     c['builders'] = [b1, b2, b3]
</pre>
   <p>In the next example, we have one buildslave hosting three separate
Builders (each running tests against a different version of Python). 
The machine which hosts this buildslave is not particularly fast, so
we want to prevent the builds from all happening at the same time. We
use a <code>SlaveLock</code> because the builds happening on the slow slave
do not affect builds running on other slaves, and we use the lock on
the build as a whole because the slave is so slow that even multiple
SVN checkouts would be taxing.

<pre class="example">     from buildbot import locks
     from buildbot.process import s, step, factory
     
     slow_lock = locks.SlaveLock("cpu")
     source = s(step.SVN, svnurl="http://example.org/svn/Trunk")
     f22 = factory.Trial(source, trialpython=["python2.2"])
     f23 = factory.Trial(source, trialpython=["python2.3"])
     f24 = factory.Trial(source, trialpython=["python2.4"])
     b1 = {'name': 'p22', 'slavename': 'bot-1', builddir='p22', 'factory': f22,
                          'locks': [slow_lock] }
     b2 = {'name': 'p23', 'slavename': 'bot-1', builddir='p23', 'factory': f23,
                          'locks': [slow_lock] }
     b3 = {'name': 'p24', 'slavename': 'bot-1', builddir='p24', 'factory': f24,
                          'locks': [slow_lock] }
     c['builders'] = [b1, b2, b3]
</pre>
   <p>In the last example, we use two Locks at the same time. In this case,
we're concerned about both of the previous constraints, but we'll say
that only the tests are computationally intensive, and that they have
been split into those which use the database and those which do not. 
In addition, two of the Builds run on a fast machine which does not
need to worry about the cpu lock, but which still must be prevented
from simultaneous database access.

<pre class="example">     from buildbot import locks
     from buildbot.process import step, factory
     
     db_lock = locks.MasterLock("database")
     cpu_lock = locks.SlaveLock("cpu")
     slow_f = factory.BuildFactory()
     f.addStep(step.SVN, svnurl="http://example.org/svn/Trunk")
     f.addStep(step.ShellCommand, command="make all", locks=[cpu_lock])
     f.addStep(step.ShellCommand, command="make test", locks=[cpu_lock])
     f.addStep(step.ShellCommand, command="make db-test",
                                  locks=[db_lock, cpu_lock])
     
     fast_factory = factory.BuildFactory()
     f.addStep(step.SVN, svnurl="http://example.org/svn/Trunk")
     f.addStep(step.ShellCommand, command="make all", locks=[])
     f.addStep(step.ShellCommand, command="make test", locks=[])
     f.addStep(step.ShellCommand, command="make db-test", locks=[db_lock])
     
     b1 = {'name': 'full1', 'slavename': 'bot-slow', builddir='full1',
                            'factory': slow_factory}
     b2 = {'name': 'full2', 'slavename': 'bot-slow', builddir='full2',
                            'factory': slow_factory}
     b3 = {'name': 'full3', 'slavename': 'bot-fast', builddir='full3',
                            'factory': fast_factory}
     b4 = {'name': 'full4', 'slavename': 'bot-fast', builddir='full4',
                            'factory': fast_factory}
     c['builders'] = [b1, b2, b3, b4]
</pre>
   <p>As a final note, remember that a unit test system which breaks when
multiple people run it at the same time is fragile and should be
fixed. Asking your human developers to serialize themselves when
running unit tests will just discourage them from running the unit
tests at all. Find a way to fix this: change the database tests to
create a new (uniquely-named) user or table for each test run, don't
use fixed listening TCP ports for network tests (instead listen on
port 0 to let the kernel choose a port for you and then query the
socket to find out what port was allocated). <code>MasterLock</code>s can be
used to accomodate broken test systems like this, but are really
intended for other purposes: build processes that store or retrieve
products in shared directories, or which do things that human
developers would not (or which might slow down or break in ways that
require human attention to deal with).

   <p><code>SlaveLocks</code>s can be used to keep automated performance tests
from interfering with each other, when there are multiple Builders all
using the same buildslave. But they can't prevent other users from
running CPU-intensive jobs on that host while the tests are running.

<div class="node">
<p><hr>
<a name="Build-Factories"></a>
Previous:&nbsp;<a rel="previous" accesskey="p" href="#Interlocks">Interlocks</a>,
Up:&nbsp;<a rel="up" accesskey="u" href="#Build-Process">Build Process</a>

</div>

<h3 class="section">6.3 Build Factories</h3>

<p>Each Builder is equipped with a &ldquo;build factory&rdquo;, which is
responsible for producing the actual <code>Build</code> objects that perform
each build. This factory is created in the configuration file, and
attached to a Builder through the <code>factory</code> element of its
dictionary.

   <p>The standard <code>BuildFactory</code> object creates <code>Build</code> objects
by default. These Builds will each execute a collection of BuildSteps
in a fixed sequence. Each step can affect the results of the build,
but in general there is little intelligence to tie the different steps
together. You can create subclasses of <code>Build</code> to implement more
sophisticated build processes, and then use a subclass of
<code>BuildFactory</code> (or simply set the <code>buildClass</code> attribute) to
create instances of your new Build subclass.

<ul class="menu">
<li><a accesskey="1" href="#BuildStep-Objects">BuildStep Objects</a>
<li><a accesskey="2" href="#BuildFactory">BuildFactory</a>
<li><a accesskey="3" href="#Process_002dSpecific-build-factories">Process-Specific build factories</a>
</ul>

<div class="node">
<p><hr>
<a name="BuildStep-Objects"></a>
Next:&nbsp;<a rel="next" accesskey="n" href="#BuildFactory">BuildFactory</a>,
Previous:&nbsp;<a rel="previous" accesskey="p" href="#Build-Factories">Build Factories</a>,
Up:&nbsp;<a rel="up" accesskey="u" href="#Build-Factories">Build Factories</a>

</div>

<h4 class="subsection">6.3.1 BuildStep Objects</h4>

<p>The steps used by these builds are all subclasses of <code>BuildStep</code>. 
The standard ones provided with Buildbot are documented later,
See <a href="#Build-Steps">Build Steps</a>. You can also write your own subclasses to use in
builds.

   <p>The basic behavior for a <code>BuildStep</code> is to:

     <ul>
<li>run for a while, then stop
<li>possibly invoke some RemoteCommands on the attached build slave
<li>possibly produce a set of log files
<li>finish with a status described by one of four values defined in
buildbot.status.builder: SUCCESS, WARNINGS, FAILURE, SKIPPED
<li>provide a list of short strings to describe the step
<li>define a color (generally green, orange, or red) with which the
step should be displayed
</ul>

   <p>More sophisticated steps may produce additional information and
provide it to later build steps, or store it in the factory to provide
to later builds.

<ul class="menu">
<li><a accesskey="1" href="#BuildFactory-Attributes">BuildFactory Attributes</a>
<li><a accesskey="2" href="#Quick-builds">Quick builds</a>
</ul>

<div class="node">
<p><hr>
<a name="BuildFactory"></a>
Next:&nbsp;<a rel="next" accesskey="n" href="#Process_002dSpecific-build-factories">Process-Specific build factories</a>,
Previous:&nbsp;<a rel="previous" accesskey="p" href="#BuildStep-Objects">BuildStep Objects</a>,
Up:&nbsp;<a rel="up" accesskey="u" href="#Build-Factories">Build Factories</a>

</div>

<h4 class="subsection">6.3.2 BuildFactory</h4>

<p><a name="index-buildbot_002eprocess_002efactory_002eBuildFactory-64"></a><a name="index-buildbot_002eprocess_002efactory_002eBasicBuildFactory-65"></a><!-- TODO: what is BasicSVN anyway? -->
<a name="index-buildbot_002eprocess_002efactory_002eBasicSVN-66"></a>
The default <code>BuildFactory</code>, provided in the
<code>buildbot.process.factory</code> module, contains a list of &ldquo;BuildStep
specifications&rdquo;: a list of <code>(step_class, kwargs)</code> tuples for
each. When asked to create a Build, it loads the list of steps into
the new Build object. When the Build is actually started, these step
specifications are used to create the actual set of BuildSteps, which
are then executed one at a time. For example, a build which consists
of a CVS checkout followed by a <code>make build</code> would be constructed
as follows:

<pre class="example">     from buildbot.process import step, factory
     
     f = factory.BuildFactory()
     f.addStep(step.CVS, cvsroot=CVSROOT, cvsmodule="project", mode="update")
     f.addStep(step.Compile, command=["make", "build"])
</pre>
   <p>It is also possible to pass a list of step specifications into the
<code>BuildFactory</code> when it is created. Using <code>addStep</code> is
usually simpler, but there are cases where is is more convenient to
create the list of steps ahead of time. To make this approach easier,
a convenience function named <code>s</code> is available:

<pre class="example">     from buildbot.process import step, factory
     from buildbot.factory import s
     # s is a convenience function, defined with:
     # def s(steptype, **kwargs): return (steptype, kwargs)
     
     all_steps = [s(step.CVS, cvsroot=CVSROOT, cvsmodule="project", mode="update"),
                  s(step.Compile, command=["make", "build"]),
                 ]
     f = factory.BuildFactory(all_steps)
</pre>
   <p>Each step can affect the build process in the following ways:

     <ul>
<li>If the step's <code>haltOnFailure</code> attribute is True, then a failure
in the step (i.e. if it completes with a result of FAILURE) will cause
the whole build to be terminated immediately: no further steps will be
executed. This is useful for setup steps upon which the rest of the
build depends: if the CVS checkout or <code>./configure</code> process
fails, there is no point in trying to compile or test the resulting
tree.

     <li>If the <code>flunkOnFailure</code> or <code>flunkOnWarnings</code> flag is set,
then a result of FAILURE or WARNINGS will mark the build as a whole as
FAILED. However, the remaining steps will still be executed. This is
appropriate for things like multiple testing steps: a failure in any
one of them will indicate that the build has failed, however it is
still useful to run them all to completion.

     <li>Similarly, if the <code>warnOnFailure</code> or <code>warnOnWarnings</code> flag
is set, then a result of FAILURE or WARNINGS will mark the build as
having WARNINGS, and the remaining steps will still be executed. This
may be appropriate for certain kinds of optional build or test steps. 
For example, a failure experienced while building documentation files
should be made visible with a WARNINGS result but not be serious
enough to warrant marking the whole build with a FAILURE.

   </ul>

   <p>In addition, each Step produces its own results, may create logfiles,
etc. However only the flags described above have any effect on the
build as a whole.

   <p>The pre-defined BuildSteps like <code>CVS</code> and <code>Compile</code> have
reasonably appropriate flags set on them already. For example, without
a source tree there is no point in continuing the build, so the
<code>CVS</code> class has the <code>haltOnFailure</code> flag set to True. Look
in <samp><span class="file">buildbot/process/step.py</span></samp> to see how the other Steps are
marked.

   <p>Each Step is created with an additional <code>workdir</code> argument that
indicates where its actions should take place. This is specified as a
subdirectory of the slave builder's base directory, with a default
value of <code>build</code>. This is only implemented as a step argument (as
opposed to simply being a part of the base directory) because the
CVS/SVN steps need to perform their checkouts from the parent
directory.

<ul class="menu">
<li><a accesskey="1" href="#BuildFactory-Attributes">BuildFactory Attributes</a>
<li><a accesskey="2" href="#Quick-builds">Quick builds</a>
</ul>

<div class="node">
<p><hr>
<a name="BuildFactory-Attributes"></a>
Next:&nbsp;<a rel="next" accesskey="n" href="#Quick-builds">Quick builds</a>,
Previous:&nbsp;<a rel="previous" accesskey="p" href="#BuildFactory">BuildFactory</a>,
Up:&nbsp;<a rel="up" accesskey="u" href="#BuildFactory">BuildFactory</a>

</div>

<h5 class="subsubsection">6.3.2.1 BuildFactory Attributes</h5>

<p>Some attributes from the BuildFactory are copied into each Build.

   <p><a name="index-treeStableTimer-67"></a>
     <dl>
<dt><code>useProgress</code><dd>(defaults to True): if True, the buildmaster keeps track of how long
each step takes, so it can provide estimates of how long future builds
will take. If builds are not expected to take a consistent amount of
time (such as incremental builds in which a random set of files are
recompiled or tested each time), this should be set to False to
inhibit progress-tracking.

   </dl>

<div class="node">
<p><hr>
<a name="Quick-builds"></a>
Previous:&nbsp;<a rel="previous" accesskey="p" href="#BuildFactory-Attributes">BuildFactory Attributes</a>,
Up:&nbsp;<a rel="up" accesskey="u" href="#BuildFactory">BuildFactory</a>

</div>

<h5 class="subsubsection">6.3.2.2 Quick builds</h5>

<p><a name="index-buildbot_002eprocess_002efactory_002eQuickBuildFactory-68"></a>
The difference between a &ldquo;full build&rdquo; and a &ldquo;quick build&rdquo; is that
quick builds are generally done incrementally, starting with the tree
where the previous build was performed. That simply means that the
source-checkout step should be given a <code>mode='update'</code> flag, to
do the source update in-place.

   <p>In addition to that, the <code>useProgress</code> flag should be set to
False. Incremental builds will (or at least the ought to) compile as
few files as necessary, so they will take an unpredictable amount of
time to run. Therefore it would be misleading to claim to predict how
long the build will take.

<div class="node">
<p><hr>
<a name="Process-Specific-build-factories"></a>
<a name="Process_002dSpecific-build-factories"></a>
Previous:&nbsp;<a rel="previous" accesskey="p" href="#BuildFactory">BuildFactory</a>,
Up:&nbsp;<a rel="up" accesskey="u" href="#Build-Factories">Build Factories</a>

</div>

<h4 class="subsection">6.3.3 Process-Specific build factories</h4>

<p>Many projects use one of a few popular build frameworks to simplify
the creation and maintenance of Makefiles or other compilation
structures. Buildbot provides several pre-configured BuildFactory
subclasses which let you build these projects with a minimum of fuss.

<ul class="menu">
<li><a accesskey="1" href="#GNUAutoconf">GNUAutoconf</a>
<li><a accesskey="2" href="#CPAN">CPAN</a>
<li><a accesskey="3" href="#Python-distutils">Python distutils</a>
<li><a accesskey="4" href="#Python_002fTwisted_002ftrial-projects">Python/Twisted/trial projects</a>
</ul>

<div class="node">
<p><hr>
<a name="GNUAutoconf"></a>
Next:&nbsp;<a rel="next" accesskey="n" href="#CPAN">CPAN</a>,
Previous:&nbsp;<a rel="previous" accesskey="p" href="#Process_002dSpecific-build-factories">Process-Specific build factories</a>,
Up:&nbsp;<a rel="up" accesskey="u" href="#Process_002dSpecific-build-factories">Process-Specific build factories</a>

</div>

<h5 class="subsubsection">6.3.3.1 GNUAutoconf</h5>

<p><a name="index-buildbot_002eprocess_002efactory_002eGNUAutoconf-69"></a>
<a href="http://www.gnu.org/software/autoconf/">GNU Autoconf</a> is a
software portability tool, intended to make it possible to write
programs in C (and other languages) which will run on a variety of
UNIX-like systems. Most GNU software is built using autoconf. It is
frequently used in combination with GNU automake. These tools both
encourage a build process which usually looks like this:

<pre class="example">     % CONFIG_ENV=foo ./configure --with-flags
     % make all
     % make check
     # make install
</pre>
   <p>(except of course the Buildbot always skips the <code>make install</code>
part).

   <p>The Buildbot's <code>buildbot.process.factory.GNUAutoconf</code> factory is
designed to build projects which use GNU autoconf and/or automake. The
configuration environment variables, the configure flags, and command
lines used for the compile and test are all configurable, in general
the default values will be suitable.

   <p>Example:

<pre class="example">     # use the s() convenience function defined earlier
     f = factory.GNUAutoconf(source=s(step.SVN, svnurl=URL, mode="copy"),
                             flags=["--disable-nls"])
</pre>
   <p>Required Arguments:

     <dl>
<dt><code>source</code><dd>This argument must be a step specification tuple that provides a
BuildStep to generate the source tree. 
</dl>

   <p>Optional Arguments:

     <dl>
<dt><code>configure</code><dd>The command used to configure the tree. Defaults to
<code>./configure</code>. Accepts either a string or a list of shell argv
elements.

     <br><dt><code>configureEnv</code><dd>The environment used for the initial configuration step. This accepts
a dictionary which will be merged into the buildslave's normal
environment. This is commonly used to provide things like
<code>CFLAGS="-O2 -g"</code> (to turn off debug symbols during the compile). 
Defaults to an empty dictionary.

     <br><dt><code>configureFlags</code><dd>A list of flags to be appended to the argument list of the configure
command. This is commonly used to enable or disable specific features
of the autoconf-controlled package, like <code>["--without-x"]</code> to
disable windowing support. Defaults to an empty list.

     <br><dt><code>compile</code><dd>this is a shell command or list of argv values which is used to
actually compile the tree. It defaults to <code>make all</code>. If set to
None, the compile step is skipped.

     <br><dt><code>test</code><dd>this is a shell command or list of argv values which is used to run
the tree's self-tests. It defaults to <code>make check</code>. If set to
None, the test step is skipped.

   </dl>

<div class="node">
<p><hr>
<a name="CPAN"></a>
Next:&nbsp;<a rel="next" accesskey="n" href="#Python-distutils">Python distutils</a>,
Previous:&nbsp;<a rel="previous" accesskey="p" href="#GNUAutoconf">GNUAutoconf</a>,
Up:&nbsp;<a rel="up" accesskey="u" href="#Process_002dSpecific-build-factories">Process-Specific build factories</a>

</div>

<h5 class="subsubsection">6.3.3.2 CPAN</h5>

<p><a name="index-buildbot_002eprocess_002efactory_002eCPAN-70"></a>
Most Perl modules available from the <a href="http://www.cpan.org/">CPAN</a>
archive use the <code>MakeMaker</code> module to provide configuration,
build, and test services. The standard build routine for these modules
looks like:

<pre class="example">     % perl Makefile.PL
     % make
     % make test
     # make install
</pre>
   <p>(except again Buildbot skips the install step)

   <p>Buildbot provides a <code>CPAN</code> factory to compile and test these
projects.

   <p>Arguments:
     <dl>
<dt><code>source</code><dd>(required): A step specification tuple, that that used by GNUAutoconf.

     <br><dt><code>perl</code><dd>A string which specifies the <code>perl</code> executable to use. Defaults
to just <code>perl</code>.

   </dl>

<div class="node">
<p><hr>
<a name="Python-distutils"></a>
Next:&nbsp;<a rel="next" accesskey="n" href="#Python_002fTwisted_002ftrial-projects">Python/Twisted/trial projects</a>,
Previous:&nbsp;<a rel="previous" accesskey="p" href="#CPAN">CPAN</a>,
Up:&nbsp;<a rel="up" accesskey="u" href="#Process_002dSpecific-build-factories">Process-Specific build factories</a>

</div>

<h5 class="subsubsection">6.3.3.3 Python distutils</h5>

<p><a name="index-buildbot_002eprocess_002efactory_002eDistutils-71"></a>
Most Python modules use the <code>distutils</code> package to provide
configuration and build services. The standard build process looks
like:

<pre class="example">     % python ./setup.py build
     % python ./setup.py install
</pre>
   <p>Unfortunately, although Python provides a standard unit-test framework
named <code>unittest</code>, to the best of my knowledge <code>distutils</code>
does not provide a standardized target to run such unit tests. (please
let me know if I'm wrong, and I will update this factory).

   <p>The <code>Distutils</code> factory provides support for running the build
part of this process. It accepts the same <code>source=</code> parameter as
the other build factories.

   <p>Arguments:
     <dl>
<dt><code>source</code><dd>(required): A step specification tuple, that that used by GNUAutoconf.

     <br><dt><code>python</code><dd>A string which specifies the <code>python</code> executable to use. Defaults
to just <code>python</code>.

     <br><dt><code>test</code><dd>Provides a shell command which runs unit tests. This accepts either a
string or a list. The default value is None, which disables the test
step (since there is no common default command to run unit tests in
distutils modules).

   </dl>

<div class="node">
<p><hr>
<a name="Python%2fTwisted%2ftrial-projects"></a>
<a name="Python_002fTwisted_002ftrial-projects"></a>
Previous:&nbsp;<a rel="previous" accesskey="p" href="#Python-distutils">Python distutils</a>,
Up:&nbsp;<a rel="up" accesskey="u" href="#Process_002dSpecific-build-factories">Process-Specific build factories</a>

</div>

<h5 class="subsubsection">6.3.3.4 Python/Twisted/trial projects</h5>

<p><a name="index-buildbot_002eprocess_002efactory_002eTrial-72"></a><!-- TODO: document these steps better -->
<a name="index-buildbot_002eprocess_002estep_005ftwisted_002eHLint-73"></a><a name="index-buildbot_002eprocess_002estep_005ftwisted_002eTrial-74"></a><a name="index-buildbot_002eprocess_002estep_005ftwisted_002eProcessDocs-75"></a><a name="index-buildbot_002eprocess_002estep_005ftwisted_002eBuildDebs-76"></a><a name="index-buildbot_002eprocess_002estep_005ftwisted_002eRemovePYCs-77"></a>
Twisted provides a unit test tool named <code>trial</code> which provides a
few improvements over Python's built-in <code>unittest</code> module. Many
python projects which use Twisted for their networking or application
services also use trial for their unit tests. These modules are
usually built and tested with something like the following:

<pre class="example">     % python ./setup.py build
     % PYTHONPATH=build/lib.linux-i686-2.3 trial -v PROJECTNAME.test
     % python ./setup.py install
</pre>
   <p>Unfortunately, the <samp><span class="file">build/lib</span></samp> directory into which the
built/copied .py files are placed is actually architecture-dependent,
and I do not yet know of a simple way to calculate its value. For many
projects it is sufficient to import their libraries &ldquo;in place&rdquo; from
the tree's base directory (<code>PYTHONPATH=.</code>).

   <p>In addition, the <var>PROJECTNAME</var> value where the test files are
located is project-dependent: it is usually just the project's
top-level library directory, as common practice suggests the unit test
files are put in the <code>test</code> sub-module. This value cannot be
guessed, the <code>Trial</code> class must be told where to find the test
files.

   <p>The <code>Trial</code> class provides support for building and testing
projects which use distutils and trial. If the test module name is
specified, trial will be invoked. The library path used for testing
can also be set.

   <p>One advantage of trial is that the Buildbot happens to know how to
parse trial output, letting it identify which tests passed and which
ones failed. The Buildbot can then provide fine-grained reports about
how many tests have failed, when individual tests fail when they had
been passing previously, etc.

   <p>Another feature of trial is that you can give it a series of source
.py files, and it will search them for special <code>test-case-name</code>
tags that indicate which test cases provide coverage for that file. 
Trial can then run just the appropriate tests. This is useful for
quick builds, where you want to only run the test cases that cover the
changed functionality.

   <p>Arguments:
     <dl>
<dt><code>source</code><dd>(required): A step specification tuple, like that used by GNUAutoconf.

     <br><dt><code>buildpython</code><dd>A list (argv array) of strings which specifies the <code>python</code>
executable to use when building the package. Defaults to just
<code>['python']</code>. It may be useful to add flags here, to supress
warnings during compilation of extension modules. This list is
extended with <code>['./setup.py', 'build']</code> and then executed in a
ShellCommand.

     <br><dt><code>testpath</code><dd>Provides a directory to add to <code>PYTHONPATH</code> when running the unit
tests, if tests are being run. Defaults to <code>.</code> to include the
project files in-place. The generated build library is frequently
architecture-dependent, but may simply be <samp><span class="file">build/lib</span></samp> for
pure-python modules.

     <br><dt><code>trialpython</code><dd>Another list of strings used to build the command that actually runs
trial. This is prepended to the contents of the <code>trial</code> argument
below. It may be useful to add <code>-W</code> flags here to supress
warnings that occur while tests are being run. Defaults to an empty
list, meaning <code>trial</code> will be run without an explicit
interpreter, which is generally what you want if you're using
<samp><span class="file">/usr/bin/trial</span></samp> instead of, say, the <samp><span class="file">./bin/trial</span></samp> that
lives in the Twisted source tree.

     <br><dt><code>trial</code><dd>provides the name of the <code>trial</code> command. It is occasionally
useful to use an alternate executable, such as <code>trial2.2</code> which
might run the tests under an older version of Python. Defaults to
<code>trial</code>.

     <br><dt><code>tests</code><dd>Provides a module name or names which contain the unit tests for this
project. Accepts a string, typically <code>PROJECTNAME.test</code>, or a
list of strings. Defaults to None, indicating that no tests should be
run. You must either set this or <code>useTestCaseNames</code> to do anyting
useful with the Trial factory.

     <br><dt><code>useTestCaseNames</code><dd>Tells the Step to provide the names of all changed .py files to trial,
so it can look for test-case-name tags and run just the matching test
cases. Suitable for use in quick builds. Defaults to False.

     <br><dt><code>randomly</code><dd>If <code>True</code>, tells Trial (with the <code>--random=0</code> argument) to
run the test cases in random order, which sometimes catches subtle
inter-test dependency bugs. Defaults to <code>False</code>.

     <br><dt><code>recurse</code><dd>If <code>True</code>, tells Trial (with the <code>--recurse</code> argument) to
look in all subdirectories for additional test cases. It isn't clear
to me how this works, but it may be useful to deal with the
unknown-PROJECTNAME problem described above, and is currently used in
the Twisted buildbot to accomodate the fact that test cases are now
distributed through multiple twisted.SUBPROJECT.test directories.

   </dl>

   <p>Unless one of <code>trialModule</code> or <code>useTestCaseNames</code>
are set, no tests will be run.

   <p>Some quick examples follow. Most of these examples assume that the
target python code (the &ldquo;code under test&rdquo;) can be reached directly
from the root of the target tree, rather than being in a <samp><span class="file">lib/</span></samp>
subdirectory.

<pre class="example">     #  Trial(source, tests="toplevel.test") does:
     #   python ./setup.py build
     #   PYTHONPATH=. trial -to toplevel.test
     
     #  Trial(source, tests=["toplevel.test", "other.test"]) does:
     #   python ./setup.py build
     #   PYTHONPATH=. trial -to toplevel.test other.test
     
     #  Trial(source, useTestCaseNames=True) does:
     #   python ./setup.py build
     #   PYTHONPATH=. trial -to --testmodule=foo/bar.py..  (from Changes)
     
     #  Trial(source, buildpython=["python2.3", "-Wall"], tests="foo.tests"):
     #   python2.3 -Wall ./setup.py build
     #   PYTHONPATH=. trial -to foo.tests
     
     #  Trial(source, trialpython="python2.3", trial="/usr/bin/trial",
     #        tests="foo.tests") does:
     #   python2.3 -Wall ./setup.py build
     #   PYTHONPATH=. python2.3 /usr/bin/trial -to foo.tests
     
     # For running trial out of the tree being tested (only useful when the
     # tree being built is Twisted itself):
     #  Trial(source, trialpython=["python2.3", "-Wall"], trial="./bin/trial",
     #        tests="foo.tests") does:
     #   python2.3 -Wall ./setup.py build
     #   PYTHONPATH=. python2.3 -Wall ./bin/trial -to foo.tests
</pre>
   <p>If the output directory of <code>./setup.py build</code> is known, you can
pull the python code from the built location instead of the source
directories. This should be able to handle variations in where the
source comes from, as well as accomodating binary extension modules:

<pre class="example">     # Trial(source,tests="toplevel.test",testpath='build/lib.linux-i686-2.3')
     # does:
     #  python ./setup.py build
     #  PYTHONPATH=build/lib.linux-i686-2.3 trial -to toplevel.test
</pre>
   <div class="node">
<p><hr>
<a name="Status-Delivery"></a>
Next:&nbsp;<a rel="next" accesskey="n" href="#Command_002dline-tool">Command-line tool</a>,
Previous:&nbsp;<a rel="previous" accesskey="p" href="#Build-Process">Build Process</a>,
Up:&nbsp;<a rel="up" accesskey="u" href="#Top">Top</a>

</div>

<h2 class="chapter">7 Status Delivery</h2>

<p>More details are available in the docstrings for each class, use
<code>pydoc buildbot.status.html.Waterfall</code> to see them. Most status
delivery objects take a <code>categories=</code> argument, which can contain
a list of &ldquo;category&rdquo; names: in this case, it will only show status
for Builders that are in one of the named categories.

   <p>(implementor's note: each of these objects should be a
service.MultiService which will be attached to the BuildMaster object
when the configuration is processed. They should use
<code>self.parent.getStatus()</code> to get access to the top-level IStatus
object, either inside <code>startService</code> or later. They may call
<code>status.subscribe()</code> in <code>startService</code> to receive
notifications of builder events, in which case they must define
<code>builderAdded</code> and related methods. See the docstrings in
<samp><span class="file">buildbot/interfaces.py</span></samp> for full details.)

<ul class="menu">
<li><a accesskey="1" href="#HTML-Waterfall">HTML Waterfall</a>
<li><a accesskey="2" href="#IRC-Bot">IRC Bot</a>
<li><a accesskey="3" href="#PBListener">PBListener</a>
<li><a accesskey="4" href="#Writing-New-Status-Plugins">Writing New Status Plugins</a>
</ul>

<!-- @node Email Delivery,  , Status Delivery, Status Delivery -->
<!-- @subsection Email Delivery -->
<!-- DOCUMENT THIS -->
<div class="node">
<p><hr>
<a name="HTML-Waterfall"></a>
Next:&nbsp;<a rel="next" accesskey="n" href="#IRC-Bot">IRC Bot</a>,
Previous:&nbsp;<a rel="previous" accesskey="p" href="#Status-Delivery">Status Delivery</a>,
Up:&nbsp;<a rel="up" accesskey="u" href="#Status-Delivery">Status Delivery</a>

</div>

<h3 class="section">7.1 HTML Waterfall</h3>

<p><a name="index-Waterfall-78"></a><a name="index-buildbot_002estatus_002ehtml_002eWaterfall-79"></a>

<pre class="example">     from buildbot.status import html
     w = html.Waterfall(http_port=8080)
     c['status'].append(w)
</pre>
   <p>The <code>buildbot.status.html.Waterfall</code> status target creates an
HTML &ldquo;waterfall display&rdquo;, which shows a time-based chart of events. 
This display provides detailed information about all steps of all
recent builds, and provides hyperlinks to look at individual build
logs and source changes. If the <code>http_port</code> argument is provided,
it provides a strports specification for the port that the web server
should listen on. This can be a simple port number, or a string like
<code>tcp:8080:interface=127.0.0.1</code> (to limit connections to the
loopback interface, and therefore to clients running on the same
host)<a rel="footnote" href="#fn-9" name="fnd-9"><sup>9</sup></a>.

   <p>If instead (or in addition) you provide the <code>distrib_port</code>
argument, a twisted.web distributed server will be started either on a
TCP port (if <code>distrib_port</code> is like <code>"tcp:12345"</code>) or more
likely on a UNIX socket (if <code>distrib_port</code> is like
<code>"unix:/path/to/socket"</code>).

   <p>The <code>distrib_port</code> option means that, on a host with a
suitably-configured twisted-web server, you do not need to consume a
separate TCP port for the buildmaster's status web page. When the web
server is constructed with <code>mktap web --user</code>, URLs that point to
<code>http://host/~username/</code> are dispatched to a sub-server that is
listening on a UNIX socket at <code>~username/.twisted-web-pb</code>. On
such a system, it is convenient to create a dedicated <code>buildbot</code>
user, then set <code>distrib_port</code> to
<code>"unix:"+os.path.expanduser("~/.twistd-web-pb")</code>. This
configuration will make the HTML status page available at
<code>http://host/~buildbot/</code> . Suitable URL remapping can make it
appear at <code>http://host/buildbot/</code>, and the right virtual host
setup can even place it at <code>http://buildbot.host/</code> .

   <p>Other arguments:

     <dl>
<dt><code>allowForce</code><dd>If set to True (the default), then the web page will provide a &ldquo;Force
Build&rdquo; button that allows visitors to manually trigger builds. This
is useful for developers to re-run builds that have failed because of
intermittent problems in the test suite, or because of libraries that
were not installed at the time of the previous build. You may not wish
to allow strangers to cause a build to run: in that case, set this to
False to remove these buttons.

     <br><dt><code>favicon</code><dd>If set to a string, this will be interpreted as a filename containing
a &ldquo;favicon&rdquo;: a small image that contains an icon for the web site. 
This is returned to browsers that request the <code>favicon.ico</code> file,
and should point to a .png or .ico image file. The default value uses
the buildbot/buildbot.png image (a small hex nut) contained in the
buildbot distribution. You can set this to None to avoid using a
favicon at all.

     <br><dt><code>robots_txt</code><dd>If set to a string, this will be interpreted as a filename containing
the contents of &ldquo;robots.txt&rdquo;. Many search engine spiders request
this file before indexing the site. Setting it to a file which
contains:
     <pre class="example">          User-agent: *
          Disallow: /
     </pre>
     <p>will prevent most search engines from trawling the (voluminous)
generated status pages.

   </dl>

<div class="node">
<p><hr>
<a name="IRC-Bot"></a>
Next:&nbsp;<a rel="next" accesskey="n" href="#PBListener">PBListener</a>,
Previous:&nbsp;<a rel="previous" accesskey="p" href="#HTML-Waterfall">HTML Waterfall</a>,
Up:&nbsp;<a rel="up" accesskey="u" href="#Status-Delivery">Status Delivery</a>

</div>

<h3 class="section">7.2 IRC Bot</h3>

<p><a name="index-IRC-80"></a><a name="index-buildbot_002estatus_002ewords_002eIRC-81"></a>

   <p>The <code>buildbot.status.words.IRC</code> status target creates an IRC bot
which will attach to certain channels and be available for status
queries. It can also be asked to announce builds as they occur, or be
told to shut up.

<pre class="example">     from twisted.status import words
     irc = words.IRC("irc.example.org", "botnickname",
                     channels=["channel1", "channel2"],
                     password="mysecretpassword")
     c['status'].append(irc)
</pre>
   <p>Take a look at the docstring for <code>words.IRC</code> for more details on
configuring this service. The <code>password</code> argument, if provided,
will be sent to Nickserv to claim the nickname: some IRC servers will
not allow clients to send private messages until they have logged in
with a password.

   <p>To use the service, you address messages at the buildbot, either
normally (<code>botnickname: status</code>) or with private messages
(<code>/msg botnickname status</code>). The buildbot will respond in kind.

   <p>Some of the commands currently available:

     <dl>
<dt><code>list builders</code><dd>Emit a list of all configured builders
<br><dt><code>status BUILDER</code><dd>Announce the status of a specific Builder: what it is doing right now. 
<br><dt><code>status all</code><dd>Announce the status of all Builders
<br><dt><code>watch BUILDER</code><dd>If the given Builder is currently running, wait until the Build is
finished and then announce the results. 
<br><dt><code>last BUILDER</code><dd>Return the results of the last build to run on the given Builder.

     <br><dt><code>help COMMAND</code><dd>Describe a command. Use <code>help commands</code> to get a list of known
commands. 
<br><dt><code>source</code><dd>Announce the URL of the Buildbot's home page. 
<br><dt><code>version</code><dd>Announce the version of this Buildbot. 
</dl>

   <p>If the <code>allowForce=True</code> option was used, some addtional commands
will be available:

     <dl>
<dt><code>force build BUILDER REASON</code><dd>Tell the given Builder to start a build of the latest code. The user
requesting the build and REASON are recorded in the Build status. The
buildbot will announce the build's status when it finishes.

     <br><dt><code>stop build BUILDER REASON</code><dd>Terminate any running build in the given Builder. REASON will be added
to the build status to explain why it was stopped. You might use this
if you committed a bug, corrected it right away, and don't want to
wait for the first build (which is destined to fail) to complete
before starting the second (hopefully fixed) build. 
</dl>

<div class="node">
<p><hr>
<a name="PBListener"></a>
Next:&nbsp;<a rel="next" accesskey="n" href="#Writing-New-Status-Plugins">Writing New Status Plugins</a>,
Previous:&nbsp;<a rel="previous" accesskey="p" href="#IRC-Bot">IRC Bot</a>,
Up:&nbsp;<a rel="up" accesskey="u" href="#Status-Delivery">Status Delivery</a>

</div>

<h3 class="section">7.3 PBListener</h3>

<p><a name="index-PBListener-82"></a><a name="index-buildbot_002estatus_002eclient_002ePBListener-83"></a>

<pre class="example">     import buildbot.status.client
     pbl = buildbot.status.client.PBListener(port=int, user=str,
                                             passwd=str)
     c['status'].append(pbl)
</pre>
   <p>This sets up a PB listener on the given TCP port, to which a PB-based
status client can connect and retrieve status information. 
<code>buildbot statusgui</code> (see <a href="#statusgui">statusgui</a>) is an example of such a
status client. The <code>port</code> argument can also be a strports
specification string.

<div class="node">
<p><hr>
<a name="Writing-New-Status-Plugins"></a>
Previous:&nbsp;<a rel="previous" accesskey="p" href="#PBListener">PBListener</a>,
Up:&nbsp;<a rel="up" accesskey="u" href="#Status-Delivery">Status Delivery</a>

</div>

<h3 class="section">7.4 Writing New Status Plugins</h3>

<p>TODO: this needs a lot more examples

   <p>Each status plugin is an object which provides the
<code>twisted.application.service.IService</code> interface, which creates a
tree of Services with the buildmaster at the top [not strictly true]. 
The status plugins are all children of an object which implements
<code>buildbot.interfaces.IStatus</code>, the main status object. From this
object, the plugin can retrieve anything it wants about current and
past builds. It can also subscribe to hear about new and upcoming
builds.

   <p>Status plugins which only react to human queries (like the Waterfall
display) never need to subscribe to anything: they are idle until
someone asks a question, then wake up and extract the information they
need to answer it, then they go back to sleep. Plugins which need to
act spontaneously when builds complete (like the Mail plugin) need to
subscribe to hear about new builds.

   <p>If the status plugin needs to run network services (like the HTTP
server used by the Waterfall plugin), they can be attached as Service
children of the plugin itself, using the <code>IServiceCollection</code>
interface.

<div class="node">
<p><hr>
<a name="Command-line-tool"></a>
<a name="Command_002dline-tool"></a>
Next:&nbsp;<a rel="next" accesskey="n" href="#Resources">Resources</a>,
Previous:&nbsp;<a rel="previous" accesskey="p" href="#Status-Delivery">Status Delivery</a>,
Up:&nbsp;<a rel="up" accesskey="u" href="#Top">Top</a>

</div>

<h2 class="chapter">8 Command-line tool</h2>

<p>The <samp><span class="command">buildbot</span></samp> command-line tool can be used to start or stop a
buildmaster or buildbot, and to interact with a running buildmaster. 
Some of its subcommands are intended for buildmaster admins, while
some are for developers who are editing the code that the buildbot is
monitoring.

<ul class="menu">
<li><a accesskey="1" href="#Administrator-Tools">Administrator Tools</a>
<li><a accesskey="2" href="#Developer-Tools">Developer Tools</a>
<li><a accesskey="3" href="#Other-Tools">Other Tools</a>
<li><a accesskey="4" href="#g_t_002ebuildbot-config-directory">.buildbot config directory</a>
</ul>

<div class="node">
<p><hr>
<a name="Administrator-Tools"></a>
Next:&nbsp;<a rel="next" accesskey="n" href="#Developer-Tools">Developer Tools</a>,
Previous:&nbsp;<a rel="previous" accesskey="p" href="#Command_002dline-tool">Command-line tool</a>,
Up:&nbsp;<a rel="up" accesskey="u" href="#Command_002dline-tool">Command-line tool</a>

</div>

<h3 class="section">8.1 Administrator Tools</h3>

<p>The following <samp><span class="command">buildbot</span></samp> sub-commands are intended for
buildmaster administrators:

<h3 class="heading">create-master</h3>

<p>This creates a new directory and populates it with files that allow it
to be used as a buildmaster's base directory.

<pre class="example">     buildbot create-master BASEDIR
</pre>
   <h3 class="heading">create-slave</h3>

<p>This creates a new directory and populates it with files that let it
be used as a buildslave's base directory. You must provide several
arguments, which are used to create the initial <samp><span class="file">buildbot.tac</span></samp>
file.

<pre class="example">     buildbot create-slave <var>BASEDIR</var> <var>MASTERHOST</var>:<var>PORT</var> <var>SLAVENAME</var> <var>PASSWORD</var>
</pre>
   <h3 class="heading">start</h3>

<p>This starts a buildmaster or buildslave which was already created in
the given base directory. The daemon is launched in the background,
with events logged to a file named <samp><span class="file">twistd.log</span></samp>.

<pre class="example">     buildbot start BASEDIR
</pre>
   <h3 class="heading">stop</h3>

<p>This terminates the daemon (either buildmaster or buildslave) running
in the given directory.

<pre class="example">     buildbot stop BASEDIR
</pre>
   <h3 class="heading">sighup</h3>

<p>This sends a SIGHUP to the buildmaster running in the given directory,
which causes it to re-read its <samp><span class="file">master.cfg</span></samp> file.

<pre class="example">     buildbot sighup BASEDIR
</pre>
   <div class="node">
<p><hr>
<a name="Developer-Tools"></a>
Next:&nbsp;<a rel="next" accesskey="n" href="#Other-Tools">Other Tools</a>,
Previous:&nbsp;<a rel="previous" accesskey="p" href="#Administrator-Tools">Administrator Tools</a>,
Up:&nbsp;<a rel="up" accesskey="u" href="#Command_002dline-tool">Command-line tool</a>

</div>

<h3 class="section">8.2 Developer Tools</h3>

<p>These tools are provided for use by the developers who are working on
the code that the buildbot is monitoring.

<ul class="menu">
<li><a accesskey="1" href="#statuslog">statuslog</a>
<li><a accesskey="2" href="#statusgui">statusgui</a>
<li><a accesskey="3" href="#try">try</a>
</ul>

<div class="node">
<p><hr>
<a name="statuslog"></a>
Next:&nbsp;<a rel="next" accesskey="n" href="#statusgui">statusgui</a>,
Previous:&nbsp;<a rel="previous" accesskey="p" href="#Developer-Tools">Developer Tools</a>,
Up:&nbsp;<a rel="up" accesskey="u" href="#Developer-Tools">Developer Tools</a>

</div>

<h4 class="subsection">8.2.1 statuslog</h4>

<pre class="example">     buildbot statuslog --master <var>MASTERHOST</var>:<var>PORT</var>
</pre>
   <p>This command starts a simple text-based status client, one which just
prints out a new line each time an event occurs on the buildmaster.

   <p>The <samp><span class="option">--master</span></samp> option provides the location of the
<code>client.PBListener</code> status port, used to deliver build
information to realtime status clients. The option is always in the
form of a string, with hostname and port number separated by a colon
(<code>HOSTNAME:PORTNUM</code>). Note that this port is <em>not</em> the same
as the slaveport (although a future version may allow the same port
number to be used for both purposes).

   <p>The <samp><span class="option">--master</span></samp> option can also be provided by the
<code>masterstatus</code> name in <samp><span class="file">.buildbot/options</span></samp> (see <a href="#g_t_002ebuildbot-config-directory">.buildbot config directory</a>).

<div class="node">
<p><hr>
<a name="statusgui"></a>
Next:&nbsp;<a rel="next" accesskey="n" href="#try">try</a>,
Previous:&nbsp;<a rel="previous" accesskey="p" href="#statuslog">statuslog</a>,
Up:&nbsp;<a rel="up" accesskey="u" href="#Developer-Tools">Developer Tools</a>

</div>

<h4 class="subsection">8.2.2 statusgui</h4>

<p><a name="index-statusgui-84"></a>
If you have set up a PBListener (see <a href="#PBListener">PBListener</a>), you will be able
to monitor your Buildbot using a simple Gtk+ application invoked with
the <code>buildbot statusgui</code> command:

<pre class="example">     buildbot statusgui --master <var>MASTERHOST</var>:<var>PORT</var>
</pre>
   <p>This command starts a simple Gtk+-based status client, which contains
a few boxes for each Builder that change color as events occur. It
uses the same <samp><span class="option">--master</span></samp> argument as the <samp><span class="command">buildbot
statuslog</span></samp> command (see <a href="#statuslog">statuslog</a>).

<div class="node">
<p><hr>
<a name="try"></a>
Previous:&nbsp;<a rel="previous" accesskey="p" href="#statusgui">statusgui</a>,
Up:&nbsp;<a rel="up" accesskey="u" href="#Developer-Tools">Developer Tools</a>

</div>

<h4 class="subsection">8.2.3 try</h4>

<p>This lets a developer to ask the question &ldquo;What would happen if I
committed this patch right now?&rdquo;. It runs the unit test suite (across
multiple build platforms) on the developer's current code, allowing
them to make sure they will not break the tree when they finally
commit their changes.

   <p>The <samp><span class="command">buildbot try</span></samp> command is meant to be run from within a
developer's local tree, and starts by figuring out the base revision
of that tree (what revision was current the last time the tree was
updated), and a patch that can be applied to that revision of the tree
to make it match the developer's copy. This (revision, patch) pair is
then sent to the buildmaster, which runs a build with that
SourceStamp. If you want, the tool will emit status messages as the
builds run, and will not terminate until the first failure has been
detected (or the last success).

   <p>For this command to work, several pieces must be in place:

<h3 class="heading">TryScheduler</h3>

<p><a name="index-buildbot_002escheduler_002eTry_005fJobdir-85"></a><a name="index-buildbot_002escheduler_002eTry_005fUserpass-86"></a>
The buildmaster must have a <code>scheduler.Try</code> instance in
the config file's <code>c['schedulers']</code> list. This lets the
administrator control who may initiate these &ldquo;trial&rdquo; builds, which
branches are eligible for trial builds, and which Builders should be
used for them.

   <p>The <code>TryScheduler</code> has various means to accept build requests:
all of them enforce more security than the usual buildmaster ports do. 
Any source code being built can be used to compromise the buildslave
accounts, but in general that code must be checked out from the VC
repository first, so only people with commit privileges can get
control of the buildslaves. The usual force-build control channels can
waste buildslave time but do not allow arbitrary commands to be
executed by people who don't have those commit privileges. However,
the source code patch that is provided with the trial build does not
have to go through the VC system first, so it is important to make
sure these builds cannot be abused by a non-committer to acquire as
much control over the buildslaves as a committer has. Ideally, only
developers who have commit access to the VC repository would be able
to start trial builds, but unfortunately the buildmaster does not, in
general, have access to VC system's user list.

   <p>As a result, the <code>TryScheduler</code> requires a bit more
configuration. There are currently two ways to set this up:

     <dl>
<dt><strong>jobdir (ssh)</strong><dd>
This approach creates a command queue directory, called the
&ldquo;jobdir&rdquo;, in the buildmaster's working directory. The buildmaster
admin sets the ownership and permissions of this directory to only
grant write access to the desired set of developers, all of whom must
have accounts on the machine. The <code>buildbot try</code> command creates
a special file containing the source stamp information and drops it in
the jobdir, just like a standard maildir. When the buildmaster notices
the new file, it unpacks the information inside and starts the builds.

     <p>The config file entries used by 'buildbot try' either specify a local
queuedir (for which write and mv are used) or a remote one (using scp
and ssh).

     <p>The advantage of this scheme is that it is quite secure, the
disadvantage is that it requires fiddling outside the buildmaster
config (to set the permissions on the jobdir correctly). If the
buildmaster machine happens to also house the VC repository, then it
can be fairly easy to keep the VC userlist in sync with the
trial-build userlist. If they are on different machines, this will be
much more of a hassle. It may also involve granting developer accounts
on a machine that would not otherwise require them.

     <p>To implement this, the buildslave invokes 'ssh -l username host
buildbot tryserver ARGS', passing the patch contents over stdin. The
arguments must include the inlet directory and the revision
information.

     <br><dt><strong>user+password (PB)</strong><dd>
In this approach, each developer gets a username/password pair, which
are all listed in the buildmaster's configuration file. When the
developer runs <code>buildbot try</code>, their machine connects to the
buildmaster via PB and authenticates themselves using that username
and password, then sends a PB command to start the trial build.

     <p>The advantage of this scheme is that the entire configuration is
performed inside the buildmaster's config file. The disadvantages are
that it is less secure (while the &ldquo;cred&rdquo; authentication system does
not expose the password in plaintext over the wire, it does not offer
most of the other security properties that SSH does). In addition, the
buildmaster admin is responsible for maintaining the username/password
list, adding and deleting entries as developers come and go.

   </dl>

   <p>For example, to set up the &ldquo;jobdir&rdquo; style of trial build, using a
command queue directory of <samp><span class="file">MASTERDIR/jobdir</span></samp> (and assuming that
all your project developers were members of the <code>developers</code> unix
group), you would first create that directory (with <samp><span class="command">mkdir
MASTERDIR/jobdir MASTERDIR/jobdir/new MASTERDIR/jobdir/cur
MASTERDIR/jobdir/tmp; chgrp developers MASTERDIR/jobdir
MASTERDIR/jobdir/*; chmod g+rwx,o-rwx MASTERDIR/jobdir
MASTERDIR/jobdir/*</span></samp>), and then use the following scheduler in the
buildmaster's config file:

<pre class="example">     from buildbot.scheduler import Try_Jobdir
     s = Try_Jobdir("try1", ["full-linux", "full-netbsd", "full-OSX"],
                    jobdir="jobdir")
     c['schedulers'] = [s]
</pre>
   <p>Note that you must create the jobdir before telling the buildmaster to
use this configuration, otherwise you will get an error. Also remember
that the buildmaster must be able to read and write to the jobdir as
well. Be sure to watch the <samp><span class="file">twistd.log</span></samp> file (see <a href="#Logfiles">Logfiles</a>)
as you start using the jobdir, to make sure the buildmaster is happy
with it.

   <p>To use the username/password form of authentication, create a
<code>Try_Userpass</code> instance instead. It takes the same
<code>builderNames</code> argument as the <code>Try_Jobdir</code> form, but
accepts an addtional <code>port</code> argument (to specify the TCP port to
listen on) and a <code>userpass</code> list of username/password pairs to
accept. Remember to use good passwords for this: the security of the
buildslave accounts depends upon it:

<pre class="example">     from buildbot.scheduler import Try_Userpass
     s = Try_Userpass("try2", ["full-linux", "full-netbsd", "full-OSX"],
                      port=8031, userpass=[("alice","pw1"), ("bob", "pw2")] )
     c['schedulers'] = [s]
</pre>
   <p>Like most places in the buildbot, the <code>port</code> argument takes a
strports specification. See <code>twisted.application.strports</code> for
details.

<h3 class="heading">locating the master</h3>

<p>The <samp><span class="command">try</span></samp> command needs to be told how to connect to the
<code>TryScheduler</code>, and must know which of the authentication
approaches described above is in use by the buildmaster. You specify
the approach by using <samp><span class="option">--connect=ssh</span></samp> or <samp><span class="option">--connect=pb</span></samp>
(or <code>try_connect = 'ssh'</code> or <code>try_connect = 'pb'</code> in
<samp><span class="file">.buildbot/options</span></samp>).

   <p>For the PB approach, the command must be given a <samp><span class="option">--master</span></samp>
argument (in the form HOST:PORT) that points to TCP port that you
picked in the <code>Try_Userpass</code> scheduler. It also takes a
<samp><span class="option">--username</span></samp> and <samp><span class="option">--passwd</span></samp> pair of arguments that match
one of the entries in the buildmaster's <code>userpass</code> list. These
arguments can also be provided as <code>try_master</code>,
<code>try_username</code>, and <code>try_password</code> entries in the
<samp><span class="file">.buildbot/options</span></samp> file.

   <p>For the SSH approach, the command must be given <samp><span class="option">--tryhost</span></samp>,
<samp><span class="option">--username</span></samp>, and optionally <samp><span class="option">--password</span></samp> (TODO:
really?) to get to the buildmaster host. It must also be given
<samp><span class="option">--trydir</span></samp>, which points to the inlet directory configured
above. The trydir can be relative to the user's home directory, but
most of the time you will use an explicit path like
<samp><span class="file">~buildbot/project/trydir</span></samp>. These arguments can be provided in
<samp><span class="file">.buildbot/options</span></samp> as <code>try_host</code>, <code>try_username</code>,
<code>try_password</code>, and <code>try_dir</code>.

   <p>In addition, the SSH approach needs to connect to a PBListener status
port, so it can retrieve and report the results of the build (the PB
approach uses the existing connection to retrieve status information,
so this step is not necessary). This requires a <samp><span class="option">--master</span></samp>
argument, or a <code>masterstatus</code> entry in <samp><span class="file">.buildbot/options</span></samp>,
in the form of a HOSTNAME:PORT string.

<h3 class="heading">choosing the Builders</h3>

<p>A trial build is performed on multiple Builders at the same time, and
the developer gets to choose which Builders are used (limited to a set
selected by the buildmaster admin with the TryScheduler's
<code>builderNames=</code> argument). The set you choose will depend upon
what your goals are: if you are concerned about cross-platform
compatibility, you should use multiple Builders, one from each
platform of interest. You might use just one builder if that platform
has libraries or other facilities that allow better test coverage than
what you can accomplish on your own machine, or faster test runs.

   <p>The set of Builders to use can be specified with multiple
<samp><span class="option">--builder</span></samp> arguments on the command line. It can also be
specified with a single <code>try_builders</code> option in
<samp><span class="file">.buildbot/options</span></samp> that uses a list of strings to specify all
the Builder names:

<pre class="example">     try_builders = ["full-OSX", "full-win32", "full-linux"]
</pre>
   <h3 class="heading">specifying the VC system</h3>

<p>The <samp><span class="command">try</span></samp> command also needs to know how to take the
developer's current tree and extract the (revision, patch)
source-stamp pair. Each VC system uses a different process, so you
start by telling the <samp><span class="command">try</span></samp> command which VC system you are
using, with an argument like <samp><span class="option">--vc=cvs</span></samp> or <samp><span class="option">--vc=tla</span></samp>. 
This can also be provided as <code>try_vc</code> in
<samp><span class="file">.buildbot/options</span></samp>.

   <p>The following names are recognized: <code>cvs</code> <code>svn</code> <code>baz</code>
<code>tla</code> <code>hg</code> <code>darcs</code>

<h3 class="heading">finding the top of the tree</h3>

<p>Some VC systems (notably CVS and SVN) track each directory
more-or-less independently, which means the <samp><span class="command">try</span></samp> command
needs to move up to the top of the project tree before it will be able
to construct a proper full-tree patch. To accomplish this, the
<samp><span class="command">try</span></samp> command will crawl up through the parent directories
until it finds a marker file. The default name for this marker file is
<samp><span class="file">.buildbot-top</span></samp>, so when you are using CVS or SVN you should
<code>touch .buildbot-top</code> from the top of your tree before running
<samp><span class="command">buildbot try</span></samp>. Alternatively, you can use a filename like
<samp><span class="file">ChangeLog</span></samp> or <samp><span class="file">README</span></samp>, since many projects put one of
these files in their top-most directory (and nowhere else). To set
this filename, use <samp><span class="option">--try-topfile=ChangeLog</span></samp>, or set it in the
options file with <code>try_topfile = 'ChangeLog'</code>.

   <p>You can also manually set the top of the tree with
<samp><span class="option">--try-topdir=~/trees/mytree</span></samp>, or <code>try_topdir =
'~/trees/mytree'</code>. If you use <code>try_topdir</code>, in a
<samp><span class="file">.buildbot/options</span></samp> file, you will need a separate options file
for each tree you use, so it may be more convenient to use the
<code>try_topfile</code> approach instead.

   <p>Other VC systems which work on full projects instead of individual
directories (tla, baz, darcs, monotone, mercurial) do not require
<samp><span class="command">try</span></samp> to know the top directory, so the <samp><span class="option">--try-topfile</span></samp>
and <samp><span class="option">--try-topdir</span></samp> arguments will be ignored. 
<!-- is this true? I think I currently require topdirs all the time. -->

   <p>If the <samp><span class="command">try</span></samp> command cannot find the top directory, it will
abort with an error message.

<h3 class="heading">determining the branch name</h3>

<p>Some VC systems record the branch information in a way that &ldquo;try&rdquo;
can locate it, in particular Arch (both <samp><span class="command">tla</span></samp> and
<samp><span class="command">baz</span></samp>). For the others, if you are using something other than
the default branch, you will have to tell the buildbot which branch
your tree is using. You can do this with either the <samp><span class="option">--branch</span></samp>
argument, or a <samp><span class="option">try_branch</span></samp> entry in the
<samp><span class="file">.buildbot/options</span></samp> file.

<h3 class="heading">determining the revision and patch</h3>

<p>Each VC system has a separate approach for determining the tree's base
revision and computing a patch.

     <dl>
<dt><code>CVS</code><dd>
<samp><span class="command">try</span></samp> pretends that the tree is up to date. It converts the
current time into a <code>-D</code> time specification, uses it as the base
revision, and computes the diff between the upstream tree as of that
point in time versus the current contents. This works, more or less,
but requires that the local clock be in reasonably good sync with the
repository.

     <br><dt><code>SVN</code><dd><samp><span class="command">try</span></samp> does a <code>svn status -u</code> to find the latest
repository revision number (emitted on the last line in the &ldquo;Status
against revision: NN&rdquo; message). It then performs an <code>svn diff
-rNN</code> to find out how your tree differs from the repository version,
and sends the resulting patch to the buildmaster. If your tree is not
up to date, this will result in the &ldquo;try&rdquo; tree being created with
the latest revision, then <em>backwards</em> patches applied to bring it
&ldquo;back&rdquo; to the version you actually checked out (plus your actual
code changes), but this will still result in the correct tree being
used for the build.

     <br><dt><code>baz</code><dd><samp><span class="command">try</span></samp> does a <code>baz tree-id</code> to determine the
fully-qualified version and patch identifier for the tree
(ARCHIVE/VERSION&ndash;patch-NN), and uses the VERSION&ndash;patch-NN component
as the base revision. It then does a <code>baz diff</code> to obtain the
patch.

     <br><dt><code>tla</code><dd><samp><span class="command">try</span></samp> does a <code>tla tree-version</code> to get the
fully-qualified version identifier (ARCHIVE/VERSION), then takes the
first line of <code>tla logs --reverse</code> to figure out the base
revision. Then it does <code>tla changes --diffs</code> to obtain the patch.

     <br><dt><code>Darcs</code><dd><code>darcs changes --context</code> emits a text file that contains a list
of all patches back to and including the last tag was made. This text
file (plus the location of a repository that contains all these
patches) is sufficient to re-create the tree. Therefore the contents
of this &ldquo;context&rdquo; file <em>are</em> the revision stamp for a
Darcs-controlled source tree.

     <p>So <samp><span class="command">try</span></samp> does a <code>darcs changes --context</code> to determine
what your tree's base revision is, and then does a <code>darcs diff
-u</code> to compute the patch relative to that revision.

     <br><dt><code>Mercurial</code><dd><code>hg identify</code> emits a short revision ID (basically a truncated
SHA1 hash of the current revision's contents), which is used as the
base revision. <code>hg diff</code> then provides the patch relative to that
revision. For <samp><span class="command">try</span></samp> to work, your working directory must only
have patches that are available from the same remotely-available
repository that the build process' <code>step.Mercurial</code> will use.

     <!-- TODO: monotone, git -->
</dl>

<h3 class="heading">waiting for results</h3>

<p>If you provide the <samp><span class="option">--wait</span></samp> option (or <code>try_wait = True</code>
in <samp><span class="file">.buildbot/options</span></samp>), the <samp><span class="command">buildbot try</span></samp> command will
wait until your changes have either been proven good or bad before
exiting. Unless you use the <samp><span class="option">--quiet</span></samp> option (or
<code>try_quiet=True</code>), it will emit a progress message every 60
seconds until the builds have completed.

<div class="node">
<p><hr>
<a name="Other-Tools"></a>
Next:&nbsp;<a rel="next" accesskey="n" href="#g_t_002ebuildbot-config-directory">.buildbot config directory</a>,
Previous:&nbsp;<a rel="previous" accesskey="p" href="#Developer-Tools">Developer Tools</a>,
Up:&nbsp;<a rel="up" accesskey="u" href="#Command_002dline-tool">Command-line tool</a>

</div>

<h3 class="section">8.3 Other Tools</h3>

<p>These tools are generally used by buildmaster administrators.

<ul class="menu">
<li><a accesskey="1" href="#sendchange">sendchange</a>
<li><a accesskey="2" href="#debugclient">debugclient</a>
</ul>

<div class="node">
<p><hr>
<a name="sendchange"></a>
Next:&nbsp;<a rel="next" accesskey="n" href="#debugclient">debugclient</a>,
Previous:&nbsp;<a rel="previous" accesskey="p" href="#Other-Tools">Other Tools</a>,
Up:&nbsp;<a rel="up" accesskey="u" href="#Other-Tools">Other Tools</a>

</div>

<h4 class="subsection">8.3.1 sendchange</h4>

<p>This command is used to tell the buildmaster about source changes. It
is intended to be used from within a commit script, installed on the
VC server.

<pre class="example">     buildbot sendchange --master <var>MASTERHOST</var>:<var>PORT</var> --username <var>USER</var> <var>FILENAMES..</var>
</pre>
   <p>There are other (optional) arguments which can influence the
<code>Change</code> that gets submitted:

     <dl>
<dt><code>--branch</code><dd>This provides the (string) branch specifier. If omitted, it defaults
to None, indicating the &ldquo;default branch&rdquo;. All files included in this
Change must be on the same branch.

     <br><dt><code>--revision_number</code><dd>This provides a (numeric) revision number for the change, used for VC systems
that use numeric transaction numbers (like Subversion).

     <br><dt><code>--revision</code><dd>This provides a (string) revision specifier, for VC systems that use
strings (Arch would use something like patch-42 etc).

     <br><dt><code>--revision_file</code><dd>This provides a filename which will be opened and the contents used as
the revision specifier. This is specifically for Darcs, which uses the
output of <samp><span class="command">darcs changes --context</span></samp> as a revision specifier. 
This context file can be a couple of kilobytes long, spanning a couple
lines per patch, and would be a hassle to pass as a command-line
argument.

     <br><dt><code>--comments</code><dd>This provides the change comments as a single argument. You may want
to use <samp><span class="option">--logfile</span></samp> instead.

     <br><dt><code>--logfile</code><dd>This instructs the tool to read the change comments from the given
file. If you use <code>-</code> as the filename, the tool will read the
change comments from stdin. 
</dl>

<div class="node">
<p><hr>
<a name="debugclient"></a>
Previous:&nbsp;<a rel="previous" accesskey="p" href="#sendchange">sendchange</a>,
Up:&nbsp;<a rel="up" accesskey="u" href="#Other-Tools">Other Tools</a>

</div>

<h4 class="subsection">8.3.2 debugclient</h4>

<pre class="example">     buildbot debugclient --master <var>MASTERHOST</var>:<var>PORT</var> --passwd <var>DEBUGPW</var>
</pre>
   <p>This launches a small Gtk+/Glade-based debug tool, connecting to the
buildmaster's &ldquo;debug port&rdquo;. This debug port shares the same port
number as the slaveport (see <a href="#Setting-the-slaveport">Setting the slaveport</a>), but the
<code>debugPort</code> is only enabled if you set a debug password in the
buildmaster's config file (see <a href="#Debug-options">Debug options</a>). The
<samp><span class="option">--passwd</span></samp> option must match the <code>c['debugPassword']</code>
value.

   <p><samp><span class="option">--master</span></samp> can also be provided in <samp><span class="file">.debug/options</span></samp> by the
<code>master</code> key. <samp><span class="option">--passwd</span></samp> can be provided by the
<code>debugPassword</code> key.

   <p>The <code>Connect</code> button must be pressed before any of the other
buttons will be active. This establishes the connection to the
buildmaster. The other sections of the tool are as follows:

     <dl>
<dt><code>Reload .cfg</code><dd>Forces the buildmaster to reload its <samp><span class="file">master.cfg</span></samp> file. This is
equivalent to sending a SIGHUP to the buildmaster, but can be done
remotely through the debug port. Note that it is a good idea to be
watching the buildmaster's <samp><span class="file">twistd.log</span></samp> as you reload the config
file, as any errors which are detected in the config file will be
announced there.

     <br><dt><code>Rebuild .py</code><dd>(not yet implemented). The idea here is to use Twisted's &ldquo;rebuild&rdquo;
facilities to replace the buildmaster's running code with a new
version. Even if this worked, it would only be used by buildbot
developers.

     <br><dt><code>poke IRC</code><dd>This locates a <code>words.IRC</code> status target and causes it to emit a
message on all the channels to which it is currently connected. This
was used to debug a problem in which the buildmaster lost the
connection to the IRC server and did not attempt to reconnect.

     <br><dt><code>Commit</code><dd>This allows you to inject a Change, just as if a real one had been
delivered by whatever VC hook you are using. You can set the name of
the committed file and the name of the user who is doing the commit. 
Optionally, you can also set a revision for the change. If the
revision you provide looks like a number, it will be sent as an
integer, otherwise it will be sent as a string.

     <br><dt><code>Force Build</code><dd>This lets you force a Builder (selected by name) to start a build of
the current source tree.

     <br><dt><code>Currently</code><dd>(obsolete). This was used to manually set the status of the given
Builder, but the status-assignment code was changed in an incompatible
way and these buttons are no longer meaningful.

   </dl>

<div class="node">
<p><hr>
<a name=".buildbot-config-directory"></a>
<a name="g_t_002ebuildbot-config-directory"></a>
Previous:&nbsp;<a rel="previous" accesskey="p" href="#Other-Tools">Other Tools</a>,
Up:&nbsp;<a rel="up" accesskey="u" href="#Command_002dline-tool">Command-line tool</a>

</div>

<h3 class="section">8.4 .buildbot config directory</h3>

<p>Many of the <samp><span class="command">buildbot</span></samp> tools must be told how to contact the
buildmaster that they interact with. This specification can be
provided as a command-line argument, but most of the time it will be
easier to set them in an &ldquo;options&rdquo; file. The <samp><span class="command">buildbot</span></samp>
command will look for a special directory named <samp><span class="file">.buildbot</span></samp>,
starting from the current directory (where the command was run) and
crawling upwards, eventually looking in the user's home directory. It
will look for a file named <samp><span class="file">options</span></samp> in this directory, and will
evaluate it as a python script, looking for certain names to be set. 
You can just put simple <code>name = 'value'</code> pairs in this file to
set the options.

   <p>For a description of the names used in this file, please see the
documentation for the individual <samp><span class="command">buildbot</span></samp> sub-commands. The
following is a brief sample of what this file's contents could be.

<pre class="example">     # for status-reading tools
     masterstatus = 'buildbot.example.org:12345'
     # for 'sendchange' or the debug port
     master = 'buildbot.example.org:18990'
     debugPassword = 'eiv7Po'
</pre>
     <dl>
<dt><code>masterstatus</code><dd>Location of the <code>client.PBListener</code> status port, used by
<samp><span class="command">statuslog</span></samp> and <samp><span class="command">statusgui</span></samp>.

     <br><dt><code>master</code><dd>Location of the <code>debugPort</code> (for <samp><span class="command">debugclient</span></samp>). Also the
location of the <code>pb.PBChangeSource</code> (for <samp><span class="command">sendchange</span></samp>). 
Usually shares the slaveport, but a future version may make it
possible to have these listen on a separate port number.

     <br><dt><code>debugPassword</code><dd>Must match the value of <code>c['debugPassword']</code>, used to protect the
debug port, for the <samp><span class="command">debugclient</span></samp> command.

     <br><dt><code>username</code><dd>Provides a default username for the <samp><span class="command">sendchange</span></samp> command.

   </dl>

<div class="node">
<p><hr>
<a name="Resources"></a>
Next:&nbsp;<a rel="next" accesskey="n" href="#Developer_0027s-Appendix">Developer's Appendix</a>,
Previous:&nbsp;<a rel="previous" accesskey="p" href="#Command_002dline-tool">Command-line tool</a>,
Up:&nbsp;<a rel="up" accesskey="u" href="#Top">Top</a>

</div>

<h2 class="chapter">9 Resources</h2>

<p>The Buildbot's home page is at <a href="http://buildbot.sourceforge.net/">http://buildbot.sourceforge.net/</a>

   <p>For configuration questions and general discussion, please use the
<code>buildbot-devel</code> mailing list. The subscription instructions and
archives are available at
<a href="http://lists.sourceforge.net/lists/listinfo/buildbot-devel">http://lists.sourceforge.net/lists/listinfo/buildbot-devel</a>

<div class="node">
<p><hr>
<a name="Developer's-Appendix"></a>
<a name="Developer_0027s-Appendix"></a>
Next:&nbsp;<a rel="next" accesskey="n" href="#Index-of-Useful-Classes">Index of Useful Classes</a>,
Previous:&nbsp;<a rel="previous" accesskey="p" href="#Resources">Resources</a>,
Up:&nbsp;<a rel="up" accesskey="u" href="#Top">Top</a>

</div>

<h2 class="unnumbered">Developer's Appendix</h2>

<p>This appendix contains random notes about the implementation of the
Buildbot, and is likely to only be of use to people intending to
extend the Buildbot's internals.

   <p>The buildmaster consists of a tree of Service objects, which is shaped
as follows:

<pre class="example">     BuildMaster
      ChangeMaster  (in .change_svc)
       [IChangeSource instances]
      [IScheduler instances]  (in .schedulers)
      BotMaster  (in .botmaster)
      [IStatusTarget instances]  (in .statusTargets)
</pre>
   <p>The BotMaster has a collection of Builder objects as values of its
<code>.builders</code> dictionary.

<div class="node">
<p><hr>
<a name="Index-of-Useful-Classes"></a>
Next:&nbsp;<a rel="next" accesskey="n" href="#Index-of-master_002ecfg-keys">Index of master.cfg keys</a>,
Previous:&nbsp;<a rel="previous" accesskey="p" href="#Developer_0027s-Appendix">Developer's Appendix</a>,
Up:&nbsp;<a rel="up" accesskey="u" href="#Top">Top</a>

</div>

<h2 class="unnumbered">Index of Useful Classes</h2>

<p>This is a list of all user-visible classes. There are the ones that
are useful in <samp><span class="file">master.cfg</span></samp>, the buildmaster's configuration file. 
Classes that are not listed here are generally internal things that
admins are unlikely to have much use for.

<h3 class="heading">Change Sources</h3>

<ul class="index-cs" compact>
<li><a href="#index-buildbot_002echanges_002efreshcvs_002eFreshCVSSource-30"><code>buildbot.changes.freshcvs.FreshCVSSource</code></a>: <a href="#CVSToys-_002d-PBService">CVSToys - PBService</a></li>
<li><a href="#index-buildbot_002echanges_002email_002eBonsaiMaildirSource-33"><code>buildbot.changes.mail.BonsaiMaildirSource</code></a>: <a href="#Other-mail-notification-ChangeSources">Other mail notification ChangeSources</a></li>
<li><a href="#index-buildbot_002echanges_002email_002eFCMaildirSource-31"><code>buildbot.changes.mail.FCMaildirSource</code></a>: <a href="#CVSToys-_002d-mail-notification">CVSToys - mail notification</a></li>
<li><a href="#index-buildbot_002echanges_002email_002eSyncmailMaildirSource-32"><code>buildbot.changes.mail.SyncmailMaildirSource</code></a>: <a href="#Other-mail-notification-ChangeSources">Other mail notification ChangeSources</a></li>
<li><a href="#index-buildbot_002echanges_002ep4poller_002eP4Source-35"><code>buildbot.changes.p4poller.P4Source</code></a>: <a href="#P4Source">P4Source</a></li>
<li><a href="#index-buildbot_002echanges_002epb_002ePBChangeSource-34"><code>buildbot.changes.pb.PBChangeSource</code></a>: <a href="#PBChangeSource">PBChangeSource</a></li>
   </ul><h3 class="heading">Schedulers and Locks</h3>



<ul class="index-sl" compact>
<li><a href="#index-buildbot_002elocks_002eMasterLock-62"><code>buildbot.locks.MasterLock</code></a>: <a href="#Interlocks">Interlocks</a></li>
<li><a href="#index-buildbot_002elocks_002eSlaveLock-63"><code>buildbot.locks.SlaveLock</code></a>: <a href="#Interlocks">Interlocks</a></li>
<li><a href="#index-buildbot_002escheduler_002eAnyBranchScheduler-18"><code>buildbot.scheduler.AnyBranchScheduler</code></a>: <a href="#Scheduler-Types">Scheduler Types</a></li>
<li><a href="#index-buildbot_002escheduler_002eDependent-23"><code>buildbot.scheduler.Dependent</code></a>: <a href="#Build-Dependencies">Build Dependencies</a></li>
<li><a href="#index-buildbot_002escheduler_002eNightly-20"><code>buildbot.scheduler.Nightly</code></a>: <a href="#Scheduler-Types">Scheduler Types</a></li>
<li><a href="#index-buildbot_002escheduler_002ePeriodic-19"><code>buildbot.scheduler.Periodic</code></a>: <a href="#Scheduler-Types">Scheduler Types</a></li>
<li><a href="#index-buildbot_002escheduler_002eScheduler-17"><code>buildbot.scheduler.Scheduler</code></a>: <a href="#Scheduler-Types">Scheduler Types</a></li>
<li><a href="#index-buildbot_002escheduler_002eTry_005fJobdir-85"><code>buildbot.scheduler.Try_Jobdir</code></a>: <a href="#try">try</a></li>
<li><a href="#index-buildbot_002escheduler_002eTry_005fUserpass-86"><code>buildbot.scheduler.Try_Userpass</code></a>: <a href="#try">try</a></li>
   </ul><h3 class="heading">Build Factories</h3>



<ul class="index-bf" compact>
<li><a href="#index-buildbot_002eprocess_002efactory_002eBasicBuildFactory-65"><code>buildbot.process.factory.BasicBuildFactory</code></a>: <a href="#BuildFactory">BuildFactory</a></li>
<li><a href="#index-buildbot_002eprocess_002efactory_002eBasicSVN-66"><code>buildbot.process.factory.BasicSVN</code></a>: <a href="#BuildFactory">BuildFactory</a></li>
<li><a href="#index-buildbot_002eprocess_002efactory_002eBuildFactory-64"><code>buildbot.process.factory.BuildFactory</code></a>: <a href="#BuildFactory">BuildFactory</a></li>
<li><a href="#index-buildbot_002eprocess_002efactory_002eCPAN-70"><code>buildbot.process.factory.CPAN</code></a>: <a href="#CPAN">CPAN</a></li>
<li><a href="#index-buildbot_002eprocess_002efactory_002eDistutils-71"><code>buildbot.process.factory.Distutils</code></a>: <a href="#Python-distutils">Python distutils</a></li>
<li><a href="#index-buildbot_002eprocess_002efactory_002eGNUAutoconf-69"><code>buildbot.process.factory.GNUAutoconf</code></a>: <a href="#GNUAutoconf">GNUAutoconf</a></li>
<li><a href="#index-buildbot_002eprocess_002efactory_002eQuickBuildFactory-68"><code>buildbot.process.factory.QuickBuildFactory</code></a>: <a href="#Quick-builds">Quick builds</a></li>
<li><a href="#index-buildbot_002eprocess_002efactory_002eTrial-72"><code>buildbot.process.factory.Trial</code></a>: <a href="#Python_002fTwisted_002ftrial-projects">Python/Twisted/trial projects</a></li>
   </ul><h3 class="heading">Build Steps</h3>



<ul class="index-bs" compact>
<li><a href="#index-buildbot_002eprocess_002emaxq_002eMaxQ-88"><code>buildbot.process.maxq.MaxQ</code></a>: <a href="#Index-of-Useful-Classes">Index of Useful Classes</a></li>
<li><a href="#index-buildbot_002eprocess_002estep_002eArch-45"><code>buildbot.process.step.Arch</code></a>: <a href="#Arch">Arch</a></li>
<li><a href="#index-buildbot_002eprocess_002estep_002eBazaar-47"><code>buildbot.process.step.Bazaar</code></a>: <a href="#Bazaar">Bazaar</a></li>
<li><a href="#index-buildbot_002eprocess_002estep_002eCompile-52"><code>buildbot.process.step.Compile</code></a>: <a href="#Compile">Compile</a></li>
<li><a href="#index-buildbot_002eprocess_002estep_002eConfigure-51"><code>buildbot.process.step.Configure</code></a>: <a href="#Configure">Configure</a></li>
<li><a href="#index-buildbot_002eprocess_002estep_002eCVS-37"><code>buildbot.process.step.CVS</code></a>: <a href="#CVS">CVS</a></li>
<li><a href="#index-buildbot_002eprocess_002estep_002eDarcs-41"><code>buildbot.process.step.Darcs</code></a>: <a href="#Darcs">Darcs</a></li>
<li><a href="#index-buildbot_002eprocess_002estep_002eGit-87"><code>buildbot.process.step.Git</code></a>: <a href="#Index-of-Useful-Classes">Index of Useful Classes</a></li>
<li><a href="#index-buildbot_002eprocess_002estep_002eMercurial-43"><code>buildbot.process.step.Mercurial</code></a>: <a href="#Mercurial">Mercurial</a></li>
<li><a href="#index-buildbot_002eprocess_002estep_002eP4-49"><code>buildbot.process.step.P4</code></a>: <a href="#P4">P4</a></li>
<li><a href="#index-buildbot_002eprocess_002estep_002eShellCommand-50"><code>buildbot.process.step.ShellCommand</code></a>: <a href="#ShellCommand">ShellCommand</a></li>
<li><a href="#index-buildbot_002eprocess_002estep_002eSVN-39"><code>buildbot.process.step.SVN</code></a>: <a href="#SVN">SVN</a></li>
<li><a href="#index-buildbot_002eprocess_002estep_002eTest-53"><code>buildbot.process.step.Test</code></a>: <a href="#Test">Test</a></li>
<li><a href="#index-buildbot_002eprocess_002estep_005ftwisted_002eBuildDebs-76"><code>buildbot.process.step_twisted.BuildDebs</code></a>: <a href="#Python_002fTwisted_002ftrial-projects">Python/Twisted/trial projects</a></li>
<li><a href="#index-buildbot_002eprocess_002estep_005ftwisted_002eHLint-73"><code>buildbot.process.step_twisted.HLint</code></a>: <a href="#Python_002fTwisted_002ftrial-projects">Python/Twisted/trial projects</a></li>
<li><a href="#index-buildbot_002eprocess_002estep_005ftwisted_002eProcessDocs-75"><code>buildbot.process.step_twisted.ProcessDocs</code></a>: <a href="#Python_002fTwisted_002ftrial-projects">Python/Twisted/trial projects</a></li>
<li><a href="#index-buildbot_002eprocess_002estep_005ftwisted_002eRemovePYCs-77"><code>buildbot.process.step_twisted.RemovePYCs</code></a>: <a href="#Python_002fTwisted_002ftrial-projects">Python/Twisted/trial projects</a></li>
<li><a href="#index-buildbot_002eprocess_002estep_005ftwisted_002eTrial-74"><code>buildbot.process.step_twisted.Trial</code></a>: <a href="#Python_002fTwisted_002ftrial-projects">Python/Twisted/trial projects</a></li>
   </ul><!-- undocumented steps -->
<p><a name="index-buildbot_002eprocess_002estep_002eGit-87"></a><a name="index-buildbot_002eprocess_002emaxq_002eMaxQ-88"></a>

<h3 class="heading">Status Targets</h3>



<ul class="index-st" compact>
<li><a href="#index-buildbot_002estatus_002eclient_002ePBListener-83"><code>buildbot.status.client.PBListener</code></a>: <a href="#PBListener">PBListener</a></li>
<li><a href="#index-buildbot_002estatus_002ehtml_002eWaterfall-79"><code>buildbot.status.html.Waterfall</code></a>: <a href="#HTML-Waterfall">HTML Waterfall</a></li>
<li><a href="#index-buildbot_002estatus_002email_002eMailNotifier-89"><code>buildbot.status.mail.MailNotifier</code></a>: <a href="#Index-of-Useful-Classes">Index of Useful Classes</a></li>
<li><a href="#index-buildbot_002estatus_002ewords_002eIRC-81"><code>buildbot.status.words.IRC</code></a>: <a href="#IRC-Bot">IRC Bot</a></li>
   </ul><!-- TODO: undocumented targets -->
<p><a name="index-buildbot_002estatus_002email_002eMailNotifier-89"></a>
<div class="node">
<p><hr>
<a name="Index-of-master.cfg-keys"></a>
<a name="Index-of-master_002ecfg-keys"></a>
Next:&nbsp;<a rel="next" accesskey="n" href="#Index">Index</a>,
Previous:&nbsp;<a rel="previous" accesskey="p" href="#Index-of-Useful-Classes">Index of Useful Classes</a>,
Up:&nbsp;<a rel="up" accesskey="u" href="#Top">Top</a>

</div>

<h2 class="unnumbered">Index of master.cfg keys</h2>

<p>This is a list of all of the significant keys in master.cfg . Recall
that master.cfg is effectively a small python program one
responsibility: create a dictionary named <code>BuildmasterConfig</code>. 
The keys of this dictionary are listed here. The beginning of the
master.cfg file typically starts with something like:

<pre class="example">     BuildmasterConfig = c = {}
</pre>
   <p>Therefore a config key of <code>sources</code> will usually appear in
master.cfg as <code>c['sources']</code>.



<ul class="index-bc" compact>
<li><a href="#index-c_005b_0027bots_0027_005d-25"><code>c['bots']</code></a>: <a href="#Buildslave-Specifiers">Buildslave Specifiers</a></li>
<li><a href="#index-c_005b_0027buildbotURL_0027_005d-14"><code>c['buildbotURL']</code></a>: <a href="#Defining-the-Project">Defining the Project</a></li>
<li><a href="#index-c_005b_0027builders_0027_005d-26"><code>c['builders']</code></a>: <a href="#Defining-Builders">Defining Builders</a></li>
<li><a href="#index-c_005b_0027debugPassword_0027_005d-28"><code>c['debugPassword']</code></a>: <a href="#Debug-options">Debug options</a></li>
<li><a href="#index-c_005b_0027manhole_0027_005d-29"><code>c['manhole']</code></a>: <a href="#Debug-options">Debug options</a></li>
<li><a href="#index-c_005b_0027projectName_0027_005d-12"><code>c['projectName']</code></a>: <a href="#Defining-the-Project">Defining the Project</a></li>
<li><a href="#index-c_005b_0027projectURL_0027_005d-13"><code>c['projectURL']</code></a>: <a href="#Defining-the-Project">Defining the Project</a></li>
<li><a href="#index-c_005b_0027schedulers_0027_005d-16"><code>c['schedulers']</code></a>: <a href="#Listing-Change-Sources-and-Schedulers">Listing Change Sources and Schedulers</a></li>
<li><a href="#index-c_005b_0027slavePortnum_0027_005d-24"><code>c['slavePortnum']</code></a>: <a href="#Setting-the-slaveport">Setting the slaveport</a></li>
<li><a href="#index-c_005b_0027sources_0027_005d-15"><code>c['sources']</code></a>: <a href="#Listing-Change-Sources-and-Schedulers">Listing Change Sources and Schedulers</a></li>
<li><a href="#index-c_005b_0027status_0027_005d-27"><code>c['status']</code></a>: <a href="#Defining-Status-Targets">Defining Status Targets</a></li>
   </ul><div class="node">
<p><hr>
<a name="Index"></a>
Previous:&nbsp;<a rel="previous" accesskey="p" href="#Index-of-master_002ecfg-keys">Index of master.cfg keys</a>,
Up:&nbsp;<a rel="up" accesskey="u" href="#Top">Top</a>

</div>

<h2 class="unnumbered">Index</h2>



<ul class="index-cp" compact>
<li><a href="#index-addURL-60">addURL</a>: <a href="#BuildStep-URLs">BuildStep URLs</a></li>
<li><a href="#index-Arch-Checkout-44">Arch Checkout</a>: <a href="#Arch">Arch</a></li>
<li><a href="#index-Bazaar-Checkout-46">Bazaar Checkout</a>: <a href="#Bazaar">Bazaar</a></li>
<li><a href="#index-build-properties-54">build properties</a>: <a href="#Build-Properties">Build Properties</a></li>
<li><a href="#index-Builder-9">Builder</a>: <a href="#Builder">Builder</a></li>
<li><a href="#index-BuildRequest-8">BuildRequest</a>: <a href="#BuildRequest">BuildRequest</a></li>
<li><a href="#index-BuildSet-7">BuildSet</a>: <a href="#BuildSet">BuildSet</a></li>
<li><a href="#index-BuildStep-URLs-59">BuildStep URLs</a>: <a href="#BuildStep-URLs">BuildStep URLs</a></li>
<li><a href="#index-Configuration-11">Configuration</a>: <a href="#Configuration">Configuration</a></li>
<li><a href="#index-CVS-Checkout-36">CVS Checkout</a>: <a href="#CVS">CVS</a></li>
<li><a href="#index-Darcs-Checkout-40">Darcs Checkout</a>: <a href="#Darcs">Darcs</a></li>
<li><a href="#index-Dependencies-22">Dependencies</a>: <a href="#Build-Dependencies">Build Dependencies</a></li>
<li><a href="#index-Dependent-21">Dependent</a>: <a href="#Build-Dependencies">Build Dependencies</a></li>
<li><a href="#index-installation-3">installation</a>: <a href="#Installing-the-code">Installing the code</a></li>
<li><a href="#index-introduction-1">introduction</a>: <a href="#Introduction">Introduction</a></li>
<li><a href="#index-IRC-80">IRC</a>: <a href="#IRC-Bot">IRC Bot</a></li>
<li><a href="#index-links-58">links</a>: <a href="#BuildStep-URLs">BuildStep URLs</a></li>
<li><a href="#index-locks-61">locks</a>: <a href="#Interlocks">Interlocks</a></li>
<li><a href="#index-logfiles-4">logfiles</a>: <a href="#Logfiles">Logfiles</a></li>
<li><a href="#index-LogLineObserver-57">LogLineObserver</a>: <a href="#Adding-LogObservers">Adding LogObservers</a></li>
<li><a href="#index-LogObserver-56">LogObserver</a>: <a href="#Adding-LogObservers">Adding LogObservers</a></li>
<li><a href="#index-Mercurial-Checkout-42">Mercurial Checkout</a>: <a href="#Mercurial">Mercurial</a></li>
<li><a href="#index-PBListener-82">PBListener</a>: <a href="#PBListener">PBListener</a></li>
<li><a href="#index-Perforce-Update-48">Perforce Update</a>: <a href="#P4">P4</a></li>
<li><a href="#index-Philosophy-of-operation-2">Philosophy of operation</a>: <a href="#History-and-Philosophy">History and Philosophy</a></li>
<li><a href="#index-Scheduler-6">Scheduler</a>: <a href="#Schedulers">Schedulers</a></li>
<li><a href="#index-statusgui-84">statusgui</a>: <a href="#statusgui">statusgui</a></li>
<li><a href="#index-SVN-Checkout-38">SVN Checkout</a>: <a href="#SVN">SVN</a></li>
<li><a href="#index-treeStableTimer-67">treeStableTimer</a>: <a href="#BuildFactory-Attributes">BuildFactory Attributes</a></li>
<li><a href="#index-Users-10">Users</a>: <a href="#Users">Users</a></li>
<li><a href="#index-Version-Control-5">Version Control</a>: <a href="#Version-Control-Systems">Version Control Systems</a></li>
<li><a href="#index-Waterfall-78">Waterfall</a>: <a href="#HTML-Waterfall">HTML Waterfall</a></li>
<li><a href="#index-WithProperties-55">WithProperties</a>: <a href="#Build-Properties">Build Properties</a></li>
   </ul><div class="footnote">
<hr>
<a name="texinfo-footnotes-in-document"></a><h4>Footnotes</h4><p class="footnote"><small>[<a name="fn-1" href="#fnd-1">1</a>]</small> this
@reboot syntax is understood by Vixie cron, which is the flavor
usually provided with linux systems. Other unices may have a cron that
doesn't understand @reboot</p>

   <p class="footnote"><small>[<a name="fn-2" href="#fnd-2">2</a>]</small> except Darcs, but
since the Buildbot never modifies its local source tree we can ignore
the fact that Darcs uses a less centralized model</p>

   <p class="footnote"><small>[<a name="fn-3" href="#fnd-3">3</a>]</small> many VC systems provide more complexity than
this: in particular the local views that P4 and ClearCase can assemble
out of various source directories are more complex than we're prepared
to take advantage of here</p>

   <p class="footnote"><small>[<a name="fn-4" href="#fnd-4">4</a>]</small> Monotone's <em>multiple heads</em> feature
violates this assumption of cumulative Changes, but in most situations
the changes don't occur frequently enough for this to be a significant
problem</p>

   <p class="footnote"><small>[<a name="fn-5" href="#fnd-5">5</a>]</small> this <code>checkoutDelay</code> defaults
to half the tree-stable timer, but it can be overridden with an
argument to the Source Step</p>

   <p class="footnote"><small>[<a name="fn-6" href="#fnd-6">6</a>]</small> To be precise, it is a list of objects which all
implement the <code>buildbot.interfaces.IChangeSource</code> Interface</p>

   <p class="footnote"><small>[<a name="fn-7" href="#fnd-7">7</a>]</small> Build properties are serialized along with the
build results, so they must be serializable. For this reason, the
value of any build property should be simple inert data: strings,
numbers, lists, tuples, and dictionaries. They should not contain
class instances.</p>

   <p class="footnote"><small>[<a name="fn-8" href="#fnd-8">8</a>]</small> Also note that a clever buildmaster admin
could still create the opportunity for deadlock: Build A obtains Lock
1, inside which Step A.two tries to acquire Lock 2 at the Step level. 
Meanwhile Build B obtains Lock 2, and has a Step B.two which wants to
acquire Lock 1 at the Step level. Don't Do That.</p>

   <p class="footnote"><small>[<a name="fn-9" href="#fnd-9">9</a>]</small> It may even be possible to provide SSL access by using
a specification like
<code>"ssl:12345:privateKey=mykey.pen:certKey=cert.pem"</code>, but this is
completely untested</p>

   <p><hr></div>

</body></html>