Buildbot Part IV: Customize the scheduler

My previous post was about customizing buildbot to display more information on the waterfall page. Now let’s customize the scheduler to trigger the knightly build only if the continuous one hasn’t failed.

One way to accomplish this is by adding a new scheduler called DependentNightly, this scheduler receives a new parameter called relatedBuilderNames containing a set of builders that the scheduler should watch. The startbuild method will now check, for every related builder, the last finished build status and start the knightly build if no failures are found. Bellow master.cfg changes.

from buildbot.schedulers.basic import SingleBranchScheduler
from buildbot.schedulers.timed import Nightly
from buildbot.changes import filter
from buildbot.status.builder import SUCCESS, WARNINGS, FAILURE
from twisted.python import log
from twisted.internet import defer

class DependentNightly(Nightly):

    relatedBuilderNames = []
    
    def __init__(self, relatedBuilderNames = [], **kwargs):
        Nightly.__init__(self, **kwargs)
        self.relatedBuilderNames = relatedBuilderNames

    def startBuild(self):
        startBuild = True
        for builderName in  self.relatedBuilderNames:
            builder_status = self.master.status.getBuilder(builderName)
            lastBuild = builder_status.getLastFinishedBuild()
            if lastBuild != None:
                startBuild = startBuild and (lastBuild.getResults() != FAILURE)

        if (startBuild):
            return Nightly.startBuild(self)
        else:
            log.msg(("Nightly Scheduler <%s>: skipping build. " +
                         "- Related builder <%s> failed last build")
                        % (self.name, ", ".join(self.relatedBuilderNames)))
            return defer.succeed(None)   

c['schedulers'] = []
c['schedulers'].append(SingleBranchScheduler(
                            name="Continuous",
                            change_filter=filter.ChangeFilter(branch='master'),
                            treeStableTimer=3*60,
                            builderNames=["Continuous"]))
c['schedulers'].append(DependentNightly(
                            name="Ninghtly",
                            branch='master',
                            hour=13,
                            minute=24,
                            onlyIfChanged=False,
                            builderNames=["Ninghtly"],
                            relatedBuilderNames=["Continuous"] ))

When the scheduler finds out that the build won’t run, a message will be log to twistd.log file as shown bellow.



I hope this series of blog posts were helpful to give you an overall idea about using buildbot for CI.

Buildbot Part I - Setting Up

Buildbot is a Python-based continuous integration system that consists of a buildmaster and one or more buildslaves, you can check it out on github.

The buildmaster decides what, when and how the system is build, it has a configuration file called master.cfg where the build process is defined. On the other hand, the buildslaves handle the execution of the commands and return results to the master.

The following is a graph (from buildbot’s wiki) shows buildbot architecture supported repositories and notifiers.

Setting up

To setup on windows, the master and slaves require:

• Python 2.7 http://www.python.org/download/ and setuptools http://pypi.python.org/pypi/setuptools/0.6c11.
• Twisted http://twistedmatrix.com (I’m using 12.1.0 for Python 2.7 64 bits).
• Twisted requires PyWin32 in order to spawn processes on Windows http://sourceforge.net/projects/pywin32/ (downloaded pywin32-217.win-amd64-py2.7).

Additionally, master requires:

• sqlite3: http://www.sqlite.org (sqlite-dll-win32-x64-3071300).
• Jinja2: http://jinja.pocoo.org/ (http://pypi.python.org/pypi/Jinja2/2.6).
• SQLAlchemy http://www.sqlalchemy.org/ (SQLAlchemy-0.7.8).
• SQLAlchemy-Migrate http://code.google.com/p/sqlalchemy-migrate/ (sqlalchemy-migrate-0.7.1.tar)

Install buildbotmaster (http://buildbot.googlecode.com/files/buildbot-0.8.6p1.tar.gz) and buildbotslave (http://buildbot.googlecode.com/files/buildbot-slave-0.8.6p1.tar.gz)

For this example, I installed both on same machine, if the installation went ok you should be able to check their versions:



The following steps creates and starts the master and slave:

Create BuildbotMaster

C:\Python27\Scripts>buildbot create-master -r C:\BuildMaster

Create BuildbotSlave

C:\Python27\Scripts>buildslave create-slave C:\BuildSlave localhost:9989 BuildSlave01 mysecretpwd

Start BuildbotMaster

C:\Python27\Scripts>buildbot start c:\BuildMaster

Start BuildbotSlave

C:\Python27\Scripts>buildslave start c:\BuildSlave

if everything went fine you should see a page like the following:



You can check for errors on C:\BuildMaster\twistd.log and C:\BuildSlave\twistd.log.

Who is using buildbot:  Python, Mozilla, Google Chromium and others. The following shows Python 2.7 waterfall page:



On my next post I’ll show you how to run automated builds and tests for a .net application hosted on git/codeplex using buildbot.