Implementing a real build pipeline with Jenkins

Jenkins is a popular, open source, continuous integration (CI) server. I quite often find it being used in the wild. Equally often I find builds that are too complex, time consuming. Most teams try to break it down into a pipeline, either under the spirit of quick-feedback or as an attempt to kickstart some sort of continuous delivery process. The knee reflex reaction is to look for a plugin that implements a pipeline and I too ended up installing the BuildPipeline only to realise that it doesn't really do what I'd like it to. Hence I've decided to invest a bit of time and see whether it was possible to implement a real pipeline using the bits and bobs that are already available and avoiding to cut code.

Good news is that yes it's possible and it isn't too hard. The Jenkins community has made tremendous progress and we can leverage their efforts. The bad news is that there's a myriad of plugins out there and trawling through them all can be a laborious task, hopefully this blog entry will save you some time.

A pipeline to be a real one needs:

• to ensure that the change-set version that started the pipeline is the same used in all steps, this should be preferably done with only one check-out from source control
• steps that can be triggered independently and without the need to re-run previous steps (this is a controversial item but reality is that few builds are always stable)
• to trigger steps automatically or manually according to need (e.g. you may not want to trigger a deploy to a TEST environment an indadvertedly stuff up manual tests)
• only one number that represents a given build across all pipeline steps
• artefacts to be produced only once and reused in later steps that require them
• to allow easily identifying what code is in the build. I reckon that using the change-set number is good, conversely git SHA's aren't that readable

Here's the list of plug-ins I used to create a real pipeline in Jenkins:

• Token Macro Plugin
• Build Name Setter Plugin
• Clone Workspace SCM Plugin
• Parameterized Trigger Plugin
• Conditional BuildStep Plugin
• CopyArchiver Plugin
• Your favourite source control plugin to get the version number from

The Token Macro Plugin let's you provide an expression that evaluates to text. This is a dependency for Build Name Setter and it's installed by default in Jenkins. The gotcha here was that prior to version 1.432 it doesn't have the the key tag - ENV - that allows using environment variables. It also doesn't show it as a plugin on the Plugins page, when it fails it doesn't tell you why :( Solution here is to install a version higher than 1.432 which has the proper ENV implementation.

With Build Name Setter Plugin I chose the following number format: PL#123-456. PL# is for PipeLine number. 123 is the change-set number that triggered it. This way I know exactly which code is in the build just by looking at this number. The 456 part is the regular Jenkins BUILD_NUMBER and it gives the run of a given step. So imagine a pipe line with 3 steps that got triggered by commit number 4922. Individual step numbers look like this: PL#4922-7654, PL#4922-324 and PL#4922-8. Step 1 could be a fast unit-test step, hence builds a lot, step 2 could be functional tests, so it builds less often and step 3 deploys to staging, and rarely happens. To configure Build Name Setter find the 'Build Environment' section of the build configuration, check 'Set Build Name' and enter an expression that will be evaluated to text:

In 'PL#${ENV,var="CHANGESET_NUMBER"}-${BUILD_NUMBER}' CHANGESET_NUMBER is an environment variable created by the source control plugin and set at check out time. You may have to figure out the details of the specific source control plugin you're using. BUILD_NUMBER is given by Jenkins.

Clone Workspace SCM Plugin allows you to reuse the workspace of a job, in the pipeline, a step, in another one. This avoids going to SCM again and re-checking out the code. This is key to a real pipeline. Remember? One check-out only.

After you installed Clone Workspace a Post-build Action should be available. i.e.:

You have to think about the workspace as it progresses in the pipeline. If the pipeline makes modifications to the workspace and there modifications are needed in future steps, then you need to re-clone again - very pleonastic :) If the workspace never changes then one clone at the end of step 1 should do the trick.

The subsequent steps will need to refer to the cloned workspace. For instance in Step 2 you should set the Source Code Management section to use the cloned version instead of pulling it from source control, i.e.:

I used Parameterized Trigger Plugin to invoke downstream steps and to pass the needed information so that the step can run. In this example all we need to pass is the change-set number. To configure it look for the 'Trigger parameterized build on other projects' on the Post-build Actions:

The called step should also state that it needs a parameter to work. This parameter will be met from the invoking step when triggered automatically and if the step is triggered manually you'll need to provide it. To set it up look for the 'This build is parameterized' in the main section of the job configuration screen:

Finally to ensure that the version building is the expected one I used a conditional check. First in step one I have a simple script that writes a file with the version number in the workspace. The file name is the change-set number itself. Note that because we're cloning the workspace this file will always be copied across but won't exist in source control per se. Second I have Conditional BuildStep Plugin check that the change-set number file exists and that its name matches the passed change-set parameter. This way the step won't accidentally run the wrong thing, this is particularly relevant when manually triggering steps. The Conditional BuildStep can be found under the build section.

Other tasks I found important to do were:

• aggregation of test results. To do this you can configure the self explanatory 'Aggregate downstream test results' Post-build Action

• archive artefacts in a central location/share that can be accessed from Jenkins slaves. To do this I used CopyArchiver Plugin

There are downsides of using Jenkins to implement a pipeline. One of them is that there are heaps of configuration to do and lots of repetition. Another problem is that the configuration itself isn't maintained in source control and figuring out who changed what can be tricky. I recommend making sure that you at least have Jenkins back-ups configured.

Oz Citizenship iPhone released

I had a lot of fun developing my first iPhone app. I was a bit disappointed with the Objective-C/XCode community though, it's so different than the Ruby and Java ones. Anyways, kudos to Brian Knorr from UISPec, you rock dude!

Check the Oz Citizenship app in iTunes

I found these tools very useful while developing my app:

• Pivotal Tracker (agile project management)
• UISpec (functional tests - hope I find the time to blog about it, it's really cool!)
• GHUnit (unit tests)
• EGODatabase (sqlite3 ORMish)
• OCMock (mocking framework)

One piece of advice for devs: don't use IB at all and write the UI by hand. Over and over code generation proves to be a bad choice.

Thanks to Lawrence Song for the help with development, Kevin O'neill for tips and tricks and Christof Wildermuth for the german translations. Curious that it was indexed in german before any other language.

Yet Another Validator

Sick of writing validators? Sick of XML? Sick of framework specific validation? Sick of complex formats and extra stuff you never use? Well, I was too and now I've outsourced something that might help you too reducing these pains.
Check out YAV.
YAV is a simple, framework independent, lightweight Java validation tool. It applies validations directly to domain models through annotations. It checks for mandatoriness, email, range and regex. If you need, create your own methods to perform more complex validations. It is JSR-303 non-compliant - thank goodness :)
Currently integrates nicely with Spring2.5, Struts2 is on the way.
It also has ValidationTestCases that minimize annoyingly repetitive testing tasks.
As always, there are a few edges to trim, but it works.

Momentary Autism and Business Blindness on Software Development

I finished reading Blink a couple of weeks ago and the anecdotes keep on coming back to me all the time. The book has several support stories for the author’s arguments and in general they are very well constructed. There is one in especial that is closely related to software development. It is the one that talks about how police officers get into a state of momentary autism on situations of extreme pressure. Far from me saying that software developers are under the gun in the same fashion as police people, but there are similarities.

In the book Gladwell discusses why highly trained professionals can bluntly fail to recognize true danger. The most probable reason is that once inside of complex situations the human mind tends to fail intercepting external stimuli. He suggests that a way to avoid getting caught on such mind blanking traps is to repeat safety steps. Most people can handle complex situations but few know how to avoid creating them in the first place. In software production, frequent codebase commits and updates, rigorous test writing (preferably in TDD way) and the use a continuous fast build are as important as keeping a gun away from suspects or holding the torch with the weak hand. If the process provides safety use it always.

In his research Gladwell found that police officers working in pairs do things differently than those working alone. Actually partners tend to speed up when working together. For the police speed can be bad because failing to correctly interpret signals in life threatening situations may be disastrous. Software, on the other hand, needs to be produced quickly, so let’s pair. Developers working together are more confident and like police officers tend to be bolder and charge more. It's a pity that police doesn’t have test frameworks to find out if the suspect was holding a gun or a wallet by the time they decided to open fire.

The autistic condition reported by officers has striking resemblance with the business blindness that plagues developers. Officers say that the fear of gunfire is such that they react without considering external factors. They are trained gunmen but the last thing they want is to fire a gun. Don’t go on writing code unless you know it has business value. It is every developer’s job to challenge requirements for purpose and need. Don’t allow yourself into a condition where everything is self-justified. Don’t use your experience to simply write more code instead make sure that every line delivers value.

This is a list of things to note when developing Rails/JRuby applications that will be deployed to java web containers such as Tomcat, WebSphere et alli. It's obviously incomplete but it is much longer than I thought it would be when I first started.

• RAILS_ROOT becomes the context therefore your traditional Rails folders aren't where you'd expect them to be. In a java container the public area is usually context's root. Warbler helps be copying the files from the application's Rails public folder and you can also modify how it packs the app by changing the entries in warbler.rb.
• Rails caching usually depends on where public files are and things tend to go bezerk unless you set where the public folder lives. So, place this in your environment.rb: Rails.public_path="#{RAILS_ROOT}/.."
  
• Try to stick to one interpreter only. It's quite confusing managing dependencies and versions when having too many rubys installed, and it's very easy to end up with:
  

1. the IDE's JRuby
2. the JRuby you installed
3. the JRuby that comes with warbler
4. the C-ruby install

• Assuming the app will run in JRuby prefer NOT to use a C-ruby interpreter at all, not even in development. This doesn't necessarily means giving up Mongrel/Webrick. Java containers won't refresh changes to view files so it is very handy to use Mongrel/Webrick for the minor css/html changes, otherwise you'll loose rails quick feedback cycle.
• Make sure that the version you're developing in is the same the app will be running when in production.
  
• Don't use warbler's (v. 0.9.9) JRuby version (1.1.1) it will most certainly cause you grief. Delete it from the installed GEM and get the latest jruby-complete.jar, then place it in the lib directory so warbler picks it up when packing the war.
• Check config.webxml.rails.env in warble.rb to make sure the warbled app is configured with the environment you want it to be. This variable creates an entry in the generated web.xml that defines which RAILS_ENV will be executed.
• JDBC adapters seem to be a problem still. When running RSpec the test environment crashes because recreate-database method isn't implemented yet. It seems that Derby is the only one fully implemented.
• Gems dependencies (Rails 2.1) aren't working properly. It seems that it has issues with java gems. I failed to install/unpack hpricot and activerecord-jdbc. So use other ways to freeze gems into the project. Don't forget to use config.load_paths to tell where the gems are in relation to the app in the java container.
  
• Deploy as frequently as possible to the target java container to avoid surprises. I can assure you that there will bad surprises if you don't do it.
• Autotest (ZenTest) doesn't seem to like the script/spec command. A cheap way to overcome this is to create a softlink called script in JRuby's bin directory that points to the bin directory itself.
• I'm using H2 (file based) in development and it is a cool database but there is one annoyance in Windows. Windows locks H2 log file even if the connection is explicitly saying that it shouldn't - url: jdbc:h2:file:db/development;FILE_LOCK=NO. In Linux/MacOS it behaves properly.
• JNDI resources are available for databases only, if you need to grab a hold of other JNDI resources some code has to be cooked to do the lookup. I wrote this very simple JNDI adapter.
• Let database adapters do their jobs, it's a little more code but it guarantees you won't run into type conversion problems. For instance:

1. GOOD -> Person.find_by_name name, :conditions => ["alive = :breathing", {:breathing => true}]
2. BAD -> Person.find_by_name name, :conditions => ["alive = true"]

• Use Exception Notification. It provides a very nice snapshot of the state of affairs and you'll need it.
• Use HAML, it really makes things simpler. This not a JRuby thing but a still a good recommendation.
• Pay attention to links. Strive to use rails resolvers to create links and urls, as traditionally, in a java container you'll usually have a context. In regular rails it looks like this: http://localhost:3000/persons and in java-land most likely this: http://localhost:8080/appName/persons
• Netbeans 6.1 made me smile after frustrated weeks using Aptana and E. I know IDE's are religious war starters but this is my take on it. Netbeans works the others don't.
  
• If possible prefer NOT to develop in Windows, especially because it is definitely much slower. If you must use Cygwin. BTW there is this cool tool called Terminator, it keeps all terminals together in tabs.
• I had problems making controllers see some modules. More specifically cache sweeping. In my controller I had to add these 2 lines, or it doesn't work:
  

1. require File.expand_path("#{RAILS_ROOT}/vendor/rails/actionpack/lib/action_controller/caching/sweeping") unless defined? cache_sweeper
2. include ActionController::Caching unless defined? cache_sweeper

• Protect from forgery has problems with WebSphere. It always complains about the generated tokens not been compatible. Is any one using it?
  

Slight difference on defined? results

Some piece of ruby logic was depending on a defined? check to identify local-variables. It was working fine in C-Ruby but it failed when ported to JRuby.
So what we found out was that the defined? keywork/method/check/thingy in JRuby gives a slightly different response.
Try it out. In your terminal bring up irb and type:


Then try it with Jruby:


JRuby spits out a local-variable(in-block) whilst C-ruby gives back local-variable only. I suppose this is intentional, although I didn't go down the rabbit hole to confirm it. If you know the rationale I'd like to know it too. Anyways, just a heads up that they are different.

Reading JNDI resources in JRuby

The most common use of a JNDI resource in JRuby-land is for configuring datasources. But now I wanted to have ActionMailer send emails using a JNDI config. So how to do it?
Do not fear! JRuby is your friend and every old java trick can be used.

So let's create the JNDI resource which is an javax.mail.Session pointing to localhost. This is what one would normally find in Tomcat's context.xml file (pastie):


Then let's create an adapter to do look ups for us (pastie):


And finally a simple test to see if it works (pastie):


Pretty easy huh?