To create good web applications, you need to test them. To test effectively, you need a good Web UI testing framework (which I will abbreviate as WUITF for the rest of this post. Catchy, no?).
So, What Makes a Good WUITF?
At its most basic, a WUITF should allow you to write tests using some kind of API (or DSL) and run those tests in the browser. Tests you write should be able to properly exercise the UI (click things that can be clicked, locate HTML elements on a page, etc).
A good WUITF should also allow you to do things like this:
- record tests by interacting with the browser
- support testing on multiple browsers
- be pluggable into your CI (so that your CI knows whether the tests passed or failed)
Finally, it would be nice if the tests you write were compatible with your existing eco-system. For example, if you’re in a .NET shop, it would be nice to be able to write these tests in C# and run them from Visual Studio.
We evaluated WaitN, Sahi, and Selenium using the criteria I described above. These are by no means the only three WUITFs available, but they do represent a good sampling. Somewhat surprisingly, we ended up choosing Sahi. I say surprisingly because Sahi is not nearly as widespread as Selenium nor does it play well with .NET like WaitN.
Sahi’s triumph had a lot to do with the unique way it’s implemented. Most WUITFs rely on browser APIs to execute tests against the browser. For example, Selenium’s WebDriver API uses specific implementations for IE (using IE Automation API), FireFox (using XPCOM), and Chrome. This approach can be limiting for non-modern browsers (i.e. IE6) because the API they expose may not give you access to everything.
Tell Me More About This Sahi
Sahi also has a very cool recorder which you can use to automatically generate tests by simply interacting with the browser. This feature is pretty nice because you can create a lot of tests quickly. That said, you should plan on refactoring these tests in order to make them less brittle and more maintainable.
Finally, Sahi is open source, although it has a Pro version as well. So far, the free version is meeting our needs.
What About Microsoft?
A special dishonorable mention goes to Microsoft’s answer to web tests, Coded UI Tests in Visual Studio 2010. On paper they sound great: write your tests in C#, run them from VS or TFS natively, integrate with Test Manager, etc. I mean, just imagine: your QA person defines a test case in TFS (with all the steps and such), runs it once to generate C# code, passes that to devs for optimization and BAM, you’ve got an automated test case! Fantastic, except for one problem.
Coded UI Tests only work on IE 7+ and maybe Firefox. No Chrome, or Safari, or Opera, or Mobile Safari, or even IE6. Even worse, Microsoft has no plans to add support. The official party line is “We have API’s which can be developed against by 3rd parties“. That’s wonderful news, thanks Microsoft.
You may also like:
Did you love / hate / were unmoved by this post?
Then show your support / disgust / indifference by following me on Twitter!