Automated UI Testing Framework… A Real Example
Michael Feathers recently blogged a very interesting topic he titled “UI Test Automation Tools are Snake Oil”. This seems to have stirred up again the very controversial topic of automated UI testing. I had previously posted about the pro's and con's of using a recording tool versus creating a framework. Now I wanted to expand a little more on using a framework by giving a concrete example of a framework I have written and am using to actually write good automated UI tests that are not fragile.
The project I am working on is mostly written in Java. It is a case management system which workers use to enter applicants' information and then run their eligibility to qualify them for benefits. The basic use of the system is to create a new case, create people on the case with their information: income, relationships etc, and then run the eligibility and save the results. One other important piece here I want to mention before I get flamed about technology choices is that the product was a legacy product and only supported on IE.
Based on most of the developers being versed in Java and the system requiring IE, I chose to use Watij for the framework. Watij is basically a framework built on top of IE that directly interacts with the browser through the DOM through the COM interface. When you are testing in Watij you are actually interacting with the browser, but since you are going through the com layer instead of clicking directly through the screens you are able to hide the browser window and run tests with the machine locked. I also used JUnit as the test driver to execute the tests since it is easy to run from eclipse and can easily be run from an ant build to report the results and run on the build server.
The basic idea behind this framework is that I wanted to make it so a non programmer could write the tests and maintain the tests without having to know much about Java or Watij. Even though Watij happens to be being used behind the scenes, I wanted the person writing the test scripts to only need to have domain knowledge of the application. With that in mind, here is a picture of the stack.
You can see in the diagram that I show a technology and how you interact with it. IE uses HTML and CSS, Watij is built on top of IE and lets you work against the DOM. My application framework is build on top of Watij and lets you work with domain specific concepts such as a specific screen, work flows and navigation. The test scripts are written against the domain specific application framework.
Using this type of a framework, most tests can be written in as few a 5 lines of code. Here is an example of a test:
Now this isn't going to make much sense to most people who read it, and I actually consider that a good thing. If you looked at the screens in the application it would make much more sense. Someone writing these tests only needs to know about the screens and they can figure out how to do something. There is actually a large amount of things going on in the background when you run this test. It will create a complete case, add the person to the case, then click through a task workflow to add a specific task, then it will go through another workflow to search by the task to see that it was added. There are probably about 20-25 different screen that this test ends up going through with these very few and simple lines of code.
The secret behind these kinds of a simple UI automation tests is abstraction. The framework is designed to essentially be a domain specific language. By abstracting away anything that has to do with the browser, html and even the dom, I am able to present a simple and stable interface for the tester to do exactly what they want and not know about all the other technology going on behind the wheel. Think about how complex a motor vehicle is, yet how simple the interface to the user is. I see many automation strategies that require the person writing the test scripts to know about all nuances of browsers and html.
What about change?
Most automation strategies break down when encountering UI changes. I welcome UI changes. Go ahead and change the UI, it won't break all my tests, it will break one specific place in the framework that knows about what kind of things are on a particular screen. With this framework it is very easy to fix a broken UI in one place and all the tests will work again. Contrast this to many of the automation scenarios you have seen where a single renaming of a button requires 50 tests to be reworked. All of the Watij interaction is in one namespace called “UI”. All of the screens use UI to interact with Watij, so that I could even swap out Watij for another browser automation framework.
Back to the snake oil
I do have to say I agree with many of the points Michael made in his post, but I have seen them overcome. A tool is never going to be a magic bullet. Yet at the same time we should not run away from UI automation. We just need to correctly apply the same principles we apply to create good abstractions in our source code to our automation code.