Continuous Integration Server Review: CruiseControl, Hudson, TeamCity and Cruise

By: Benjamin Booth

To come up with four or five good options for this review, I did a little digging.  Most of the prior reviews I found only covered open source alternatives - usually CC, Hudson, and sometimes Continuum.  But, unless you're down to your last dime, why be a an open-source only fool (Oof)?  Sometimes a little investment will way more than pay for itself.

Aside; if you're wondering what the heck continuous integration (CI) is at this point, read Martin Fowler's definitive treatise on the topic, here.

To narrow down paid alternatives for this review; I had used TeamCity on a previous project and loved it.  Mingle is our agile project management tool of choice.  Mingle's nice integration with Cruise, pricing discounts when bundling Mingle with Cruise, and Cruises' CC pedigree made Cruise also worth considering.

For open source options, I picked the two most popular: CruiseControl and Hudson.  'Popularity' was determined by more of my admittedly unscientific intuition, derived from countless Google searches and developer-on-the-street word of mouth.

Disclaimer: I have no hands-on experience with Cruise or Hudson.  Feel free to stop reading now.  My familiarity with CC and TeamCity, I felt, were sufficient to make objective inferences during the comparison.  If your actual experience contradicts this review, let me know!

For us, the following criteria were paramount:

A short note on the review format... 

For each alternative, the pros, cons and, finally, a score are given.  Each product was scored based on the scoring system in this spreadsheet.  Features were weighted according to project needs.  Feel free to steal the spreadsheet for your own customized analysis.

CruiseControl

Pros

CC is the father of all CI servers.  Like "Kleenex", "Coke", and "Google", "CruiseControl" and CI are practically synonymous.  To date, it has more extensions and configuration options than any alternative.  It's well documented and practically everyone knows about it or has used it some point.  Now, .Net and .rb versions of it have even been released.  For being free, it's still a compelling option.

Cons

At one point the web-based UI was clunky but no longer.  The current UI is clean and easy to navigate.

Even so, CC configuration suffers somewhat.  Most of the configuration is done in an XML file that is notoriously tedious to tinker with.  The cycle between making a configuration change and waiting to see if your change worked can be aggravating. 

Although CC does support distributed builds, getting this to work is a bit tricky.

For us, a fairly significant CC limitation was its lack of a strong security model.  Each of the alternatives to CC offers some kind of role-based access.  Many integrate nicely with LDAP.

Despite Cruise's great history and central role in the CI movement, the trend seems to be that most of the Oofs are going for Hudson these days.

Final Score: 74/151 (49%)

Hudson

Pros

Hudson, like Cruise and TeamCity, is about as full featured for a CI server as you can get.  Compared to CC, it's a major improvement in CI server usability.  All configuration is done with a GUI.

Read the interweb comment threads and you find that both Oofs and non-oofs seem to be flocking to Hudson.  My cursory review shows me why.  Other than being easy to start, feature-rich, and stable, the UI layout is simple.  There are lots of large icons placed in an intuitive UI layout.

Hudson allows you setup dependencies among build artifacts based in 'file fingerprints' (MD5 checksums).  This enables precise and reusable build artifact dependencies.

Hudson, like CC has a vibrant user community.  Lots of plug-ins have written for it, extending it's capability in diverse ways.  The vibrancy of the user community is a benefit when trying to work with your CI server.  More documentation is available and Google searches return better results.  Licensed alternatives such as Cruise and TeamCity don't even compare.

Cons

For us, no direct integration with Mingle was a drawback compared to Cruise.  I realize for most people, this wouldn't be a deciding factor.

Final Score: 115/151 (76%)

Cruise

Pros

Cruise is an interesting, recent addition to the CI server family.  Thoughtworks' Cruise has the benefit of retaining lessons learned from previous CC development.  Cruise development benefits by the application of Thoughtworks' own strong agile/Scrum methodology.  In nice but subtle ways, this shows up in the way that new Cruise features are being rolled out.

An interesting innovation with Cruise is the ability to manage release pipelines.  Pipelines are release workflows that give everyone involved visibility and control over the entire release process.  Separate releases can be created for the same source code 'baseline' (in the repository), greatly simplifying configuration management by avoiding direct manipulation of the version control system (VCS).

Cruise also integrates nicely with Mingle.  Mingle, an up and coming agile management tool, can display Cruise build history and status.  First-hand word from Thoughtworks is that there are plans to provide even tighter integration between the two products, making build and release management just another Mingle workflow 'wall' of sorts.  Makes sense to me.

Cons

Surprisingly, a good amount of Cruise's configuration UI is with XML.  I'm guessing this was a carry over from CC.  I was a bit surprised.  Given the amount of hype Hudson received for putting all configuration in the GUI, I would've expected Thoughtworks to do the same.

To give Thougthworks the benefit of the doubt, I can only guess that their argument may have been, "You configure your build project, roughly, once.  Before giving you a fancy GUI configuration, we wanted to make the other 95% of your CI server user interactions great."

At least the XML config is done in a web page, avoiding the need to log onto the server directly.  Either way, it's not a big knock against it.

Like most licensed software, the amount of freely available information about how to use and extend Cruise is limited.  On the flip-side the documentation that is available is usually very good.

Final Score: 104/151 (69%)

TeamCity

Pros

TeamCity comes to us from JetBrains, the same ones that revolutionized the Integrated Development Environment (IDE) market with the introduction of IntelliJ sometime ago.  Times move fast.  By the time I'd heard of TeamCity, to my surprise, it had been out for quite awhile.  Somehow, I missed the marketing splash.  Now, in case you missed it as well, TeamCity has become a mature, highly tuned CI veteran.

TeamCity supports distributed builds, security, and all manner of CI goodness.  Remote build agents can be configured centrally, minimizing the distributed build configuration hassle.  The UI is clean and very easy to navigate.  Build updates are displayed live, a great thing for large build processes.  No one likes to wait for the entire 7.4 minute build duration in order to find out that what they just did to fix the build didn't work.  With TeamCity you can debug your builds incrementally.

Overall, TeamCity is a joy to work with, is stable, and does practically everything you'd need it to.  Compared to CC or Hudson, it may not have as many available extensions but this should not be a big deterrent - it has more features than most will ever use.  Going into this review, it was probably my number one candidate for our next CI server. 

Cons

For some, TeamCity's licensing may be a deterrent.  Full perpetual licenses are $2,000/yr with $2,000/yr in maintenance costs.  However smaller teams can still use TeamCity for free.  For free, they can even take advantage of its distributed build capability, though there is a limit to how many distributed build agents you can employ.

Like Cruise, the amount of information freely available for TeamCity compared to CC or Hudson is worth considering.  Google searches are generally more productive with widely adopted open source software.

Final Score: 106/151 (70%)

Conclusion

CruiseControl and Hudson both gain the benefits of any widely adopted open source software project: expansive set of freely available extensions, lots of documentation (for better or worse) and good internet search results.

CruiseControl, the father of all CI servers, seems to have peaked in popularity and is being replaced by Hudson.  CruiseControl's lack of a security model and configuration difficulty compare to its alternatives knock it out of contention for our selection.

The remaining three options, Hudson, TeamCity, and Cruise all scored pretty closely in our psuedo-scientific weighted analysis, although Hudson emerged as the clear leader.  Each is a fully featured CI server that would serve anyone well.  TeamCity, my old favorite, is a great option - practically equivalent to Cruise when you look at their spec sheets.  It is backed by the renowned JetBrains developers.  Compared to Cruise and Hudson, TeamCity also has the benefit of maturity.  This counts for a lot.  After all, your CI server is supposed to aid in a reduction of bug count.

Cruise, the newcomer, has the innovative 'pipeline' way of plugging into your agile release process.  For this, it stands apart.  Cruise's interface is clean and attractive, the only negative being use of XML in the web UI to configure it.

Given our previous adoption of Mingle and a desire to have paid support for build server issues, we nudged Cruise from a near tie into a slight advantage over TeamCity and Hudson.  To be honest, I'm a bit concerned about the maturity of Cruise but feel it's worth the risk in our case.

If money is a concern for your project (or even if it's not) or if you want complete control over your CI server, Hudson seems to be smart way to go.  We don't expect problems with Cruise but if it proves it's not quite ready, we'd be totally comfortable standing up TeamCity or Hudson.