Notes on Selenium-grid

The fact that web tests, let’s call them here Selenium tests, are time consuming seems to hurt a lot of agile projects. Long build times are hard to live with.

In the early stages of the project there are a few Selenium tests, usually not very well structured and testing on a per story basis. As more stories are played more Selenium tests are created, usually with a certain degree of repetition, causing the build time to increase. That’s when it starts to sting.

In my last client the build time started off in the thirty minutes range and after four iterations it was costing the team a horrendous seventy minutes. Selenium tests alone were responsible for almost fifty minutes.

There were a few ways to reduce time. The first obvious one is ditching some tests. If you are playing by the book and doing TDD, really caring about your unit and integration tests, and have a decent test coverage, giving up a couple of redundant tests might not be so bad. Another option is to rewrite tests to eliminate redundancy and test using a scenario approach instead of the story one. Scenario based tests are a Good Idea™ but time consuming and hard to guarantee that everything that was tested before is still been tested using the conceived scenarios. A third option was to parallelize the tests and Selenium-grid allows doing just that.

Lawrence Song and I had Selenium-grid up and running with no traumas. It’s quite straight forward to install it but not all is roses. Web tests that run on a farm need to be more carefully constructed that those running on a single threaded environment.

Tests must run atomically and the rule of one behaviour per test should be strictly observed. For instance, if tests perform database operations on the same data they can have problems when running concurrently.

Don’t leave Selenium objects hanging in there. Selenium objects that aren’t closed properly causes the hub to stop sending RC commands until you have only one Selenium server available. So the Selenium lifecycle needs to be managed more tightly.

You need to think about the test list. In order to use the grid’s capabilities tests need to be run in a multi-thread fashion. TestNG does a better job that Parallel JUnit. Some Gargoyle like facility would help make it more manageable. As a last resource, which seems to be the sensible one especially when needing to avoid two or more tests running concurrently, is to use parallel ant task and pick your tests using a pattern.

Web tests are generally CPU intense and it’s a Good Idea™ to think about how many Selenium servers should be running on each box, especially if they are also running other apps like your container for example. We saw flaky results when boxes where running close to 100% CPU all the time.

Bottom line is that Selenium-grid is a great help to solve a common problem, in our client the build time dropped significantly and now takes around twenty five minutes with a few hours of effort. Yay! And we can always add more boxes. Selenium-grid has a very positive side effect which is to reinforce better test design.

Nevertheless I’d really enjoy if Selenium-grid could provide the community with the ability to utilise idle computers to run tests and a friendlier interface to manage the hub.