Category Archives: UI

Getting Up to BAT: Designing an Automation Framework

Now that you’ve gotten an automation lead and decided on the browser automation tool you are going to use, the next step is to design an actual automation framework.

This is one of the most critical components of the overall success of your automation strategy, so you will want to make sure you invest properly in this area.

I have seen a large number of automation projects go forth, but each time the critical component determining their success or failure was having a good automation framework.

Solid design and excellent technical resources are absolutely critical to success in this area!

What is an automation framework?

An automation framework is essentially an API that all of your BATs (Blackbox Automated Tests) are written against.

This API can be as simple as the API that is exposed by your browser automation tool (WatiN, Selenium, etc.), but I would highly recommend building your own layer on top of the browser automation tool’s API that will act as a DSL (Domain Specific Language) for automating your application.

Let me break this down a bit further by using an example.

Think about the task of making coffee.

There are several “API” levels we can interact with.

  1. We can go with a very low level API where we take whole coffee beans and grind them down.  Then we take some water, get it hot.  We take our filter put the ground beans in it, put it over a cup and pour the water into the filter.
  2. We can go with a higher level API where we use a traditional coffee maker.  In this case we load the coffee maker with a filter, ground coffee beans and water and push a “brew” button.  We could also set it to start at a certain time.
  3. We can go with a very high level API where we use a Keurig machine or similar device.  In this case we only make sure the machine has water in it, and we just insert a little pod and press brew.  We can make different kinds of coffee, cider or hot cocoa just by changing what pod we use.

unnamed thumb Getting Up to BAT: Designing an Automation Framework

Using the API provided by the browser driver is like making the coffee by hand.  It’s going to take a large amount of effort each time you do it.

We want our automation framework to be more like the Keurig machine.  We want to be able to compartmentalize our tests in little pods that are small and don’t require many hooks into the automation framework.

To rehash this one more time, basically our automation framework will be a framework we build on top of the browser driver framework, which is designed to make it easy to write tests which automate our application.

What makes a good automation framework design?

The true measure of an automation framework is the size of the tests that are written against them. The less lines of code in each test, the better the automation framework captures the domain of the application being automated.

In my earlier post about an example of an automation framework, I talked a bit about the strategy I used in a real implementation of an automation framework.

Here is a diagram showing the different layers.  You can see the framework layer is in green here.

automated testing framework 2 thumb Getting Up to BAT: Designing an Automation Framework

You can also see in this diagram that on the right hand side, I have screens, workflows and navigation.

This is one of the common design patterns you can use to build your automation framework, since it closely models that of most web applications.

A good way to design your framework is to use a pattern like this and create classes for each page in your application, create classes which might represent workflows in your application, and create some classes for navigating around the application.

The basic idea you are shooting for with your automation framework is to make it so the tests do not have to know anything about the browser.  Only your automation framework itself should be dealing with the DOM and HTML and CSS.  Your tests should be dealing with concepts which any user would be familiar with.  That is why I commonly use pages and workflows.

Let me give you an example.

Let’s say we ignored the advice of creating an automation framework and decided to program our tests directly against the browser driver layer.  In this case, let’s say that we want to automate a login page.  Our test might look something like this pseudo-code.

Element userName = browser.getElementById("login_userName");

Element userPassword = browser.getElementById("login_userPassword");

Element loginButton = browser.


This is not very readable.  It is not something an average tester will be able to pick up and start writing, and it is extremely fragile.  If you have 500 tests like this and the id changes for one of the elements on the page, many tests will break.

Now, contrast it to this pseudo-code:



Where did all the code go?  You don’t need it!  Actually it went into the framework, where it can be reused.  Now your test code is simple, can be understood by just about anyone, and won’t break with changes to the UI.  (You will just have to change the framework method instead of 500 tests.)

If you want some more detailed examples take a look at my Boise Code Camp slides on the subject here.

Let me offer up some general guidelines for creating a good design for an automation framework.

  1. NEVER require the tests to declare variables.
  2. NEVER require the tests to use the new keyword or create new objects.
  3. NEVER require the tests to manage state on their own.
  4. ALWAYS reduce the number of parameters for API calls when possible.
  5. ALWAYS use default values instead of requiring a parameter when possible.
  6. PREFER to make the API easier to use over making the internals of the API less complex.
  7. PREFER using enumerations and constants to requiring the test to pass in primitive types.  (Make the input bounded when possible.)
  8. NEVER expose the browser or DOM to the tests or let them manipulate it directly.

You can see that these guidelines will almost force your API to be entirely static methods!  Don’t freak out.  It is perfectly fine.  There is nothing wrong with using static methods in this case.

Stop… take a breath, think about why you think static methods are bad, because I probably agree with you.

But, in this case we are HIGHLY valuing making the tests simple and easy to write over anything else, and because of this emphasis, static methods are going to be the way to go.

Some advice on getting started

Start with the tests in plain human English.

Take those tests and make them into code preserving as much English as possible.  Then work down from there implementing the methods needed to make those tests work.  Only add things as you need them.  Do not overdesign!

In my example above, we might have started with something like this:

1. Goto the login page.

2. Login with the default user.

The actual code looks just like that.  It should always be a 1 to 1 mapping.

This is where having someone that has some experience doing this is going to come in handy.

Just keep in mind the whole time your 2 big goals:

  1. Easy to write tests
  2. Tests are concise
As always, you can subscribe to this RSS feed to follow my posts on Making the Complex Simple.  Feel free to check out where I post about the topic of writing elegant code about once a week.  Also, you can follow me on twitter here.

Back to Basics: Becoming BAT Man

In my Back to Basics post on my conclusions about blackbox automated tests (BATs) and unit testing, I said that we should:

Spend a majority of your effort, all the time you would have spent writing unit tests, instead writing what I will call blackbox automated tests or BATs.

In this post, I am going to outline what I think it takes for a team to go from BAT-less to BAT-man (or BAT-woman) without going batty.

I want to take a practical approach to getting from 0 blackbox automated tests to building a sustainable process of developing BATs as a integral component of the acceptance criteria for calling a backlog item “done.”

Let me paint a picture for you

human paint brush thumb Back to Basics: Becoming BAT Man

I want to paint the picture for you of the end goal that we are seeking to achieve, from start to finish, of a single backlog.

  1. A backlog is selected for work.
  2. Developers, QA and product owner work together to clearly define the acceptance criteria which will enable that backlog to be signed off or called “done.”
  3. QA and developers work together to transform the acceptance criteria into a high level description of BATs which will be created to verify each aspect of the acceptance criteria.
  4. Developer begins coding with this high level set of BATs in mind (BATs drive the development), QA begins developing BATs.
  5. As developer completes enough functionality to pass a BAT and not break existing BATs that functionality is checked into trunk.
  6. Each check-in causes the entire battery of BATs to be kicked off split across multiple VMs which run 100s of hours worth of automated tests in minutes.
  7. Once all BATs for a backlog are passing, that backlog is done and those BATs become part of the battery of tests which are run with each continuous integration build.

We should keep this simple process in mind and strive to reach it, knowing that when we are able to reach this process we will be able to have a HUGE amount of confidence in the functionality and non-regression of our system.

Put aside for a second your notion that unit testing is the correct thing to do.  Put aside the idea that blackbox automated testing is too hard and too fragile, and imagine the world of software development that flows like the picture I painted above.

Now listen up, because this part is important…

Before you put forth the effort to write one more unit test, before you dip your test double pen in the ink well of “mock,” make sure you are first taking efforts to develop a process to create BATs.

The value proposition

We really need to take the effort to understand the value proposition being presented here.

I’m going to use some real fake numbers to try and really convey this important point that I think is likely to be overlooked.

Consider first the amount of time that you spend doing two things:

  1. Creating infrastructure to allow your code to be unit tested in isolation.  (Dependency injection, interfaces, etc.)
  2. Time spent writing unit tests.

Now, before we go any further, I just want to reiterate.  I am not advocating completely abolishing the writing of unit tests.  See my original conclusions post, or my post on unit testing without mocks for a better understanding of what I am advocating in regards to unit testing.

So back to my point.  Think about all the time it takes to do this.  To properly isolate and unit test code most developers would probably estimate that for every hour worth of work writing the code there is about another hour to two hours worth of unit testing and unit test prep time.

There is a reason why unit tests take longer to write than the code they test; it takes MORE lines of code to test a line of code in isolation.

BATs developed using a good automation framework are just the opposite.  While unit test code might have a ratio of 4 lines of unit test code to 1 line of production code or a 4:1 ratio, BAT code has a completely opposite ratio.  It is very likely that 1 line of BAT code can test 20 lines or more of production code, a 1:20 ratio.  (Where do I get these numbers?  Looking at some of my real production code being tested with unit tests and BATs.)

Even if unit tests and BATs were equally effective in preventing the regression of your software system, and equally effective at providing an absolute assurance of meeting the acceptance criteria (which I would argue that BATs are much more effective at both), you can see easily that you are going to get a much higher return on your investment by investing your time in BATs vs. investing your time in unit tests.

I don’t make these statements lightly or in a theoretical tone.  I have real world experience with successfully implementing automation frameworks for writing BATs that reinforce these conclusions.

How to get there

If I have done my job, you are now at least convinced that getting to the point of having BATs for all backlogs is a good goal, but if you are like most people I talk to, you are skeptical of the costs and feasibility of doing it.

What you need is a good guide.  A step by step guide on how to do it.

I am going to conclude the Back to Basics posts and segue into a new series with the goal of providing a step by step guide to getting to fully automated blackbox testing.

Let’s take a look at the steps that I will cover in the next series of posts.

  1. Hiring an automation lead – it’s important to either find someone who’s done this before, or a developer resource to this role.
  2. Deciding on the underlying browser driver or browser automation tool – assuming web apps here, there are several to choose from, WatiX, Selenium and more.
  3. Designing an automation framework – building a framework tailored specifically to your application under test will create an effective domain specific language for testing your application.
  4. Creating your first smoke tests – building some smoke tests will help you prove and build out your framework and provide you the highest value tests first.
  5. Adding smoke tests to your build – with smoke tests being run as part of the build, you can immediately start seeing value.
  6. Adding BATs to your acceptance criteria – we need to start out slowly here, but eventually make all backlogs require BATs for each acceptance criterion.
  7. Scaling out – as you start to get a large amount of BATs you’ll need to figure out a way to virtualize more servers and parallelize the test runs to be able to run all the BATs in a short amount of time.
  8. Building a true DSL – once you get this far, it may be time to start thinking about creating a domain specific language that even business analysts can write tests with.
As always, you can subscribe to this RSS feed to follow my posts on Making the Complex Simple.  Feel free to check out where I post about the topic of writing elegant code about once a week.  Also, you can follow me on twitter here.

Back to Basics: Unit Testing, Automated Blackbox Testing, and Conclusions!

If you’ve been following me from the beginning of the Back to Basics series, you’ll know that I set out to reevaluate some of the commonly held truths of what best practices are, especially in regards to unit testing, dependency injection and inversion of control containers.

We’ve talked about what an interface is, cohesion and coupling, and even went a little bit off track to talk about sorting for a bit.

One of the reoccurring themes that kept showing up in most of the posts was unit testing.  I talked about why unit testing is hard, and I defined three levels of unit testing.

  • Level 1 – we have a single class with no external dependencies and no state.  We are just testing an algorithm.
  • Level 2 – we have a single class with no external dependencies but it does have state.  We are setting up an object and testing it as a whole.
  • Level 3 – we have a single class with at least one external dependency, but it does not depend on its own internal state.
  • Level 4 – we have a single class with at least one external dependency and depends on its own internal state.

Throughout this series I ended up tearing down using interfaces with only single non-unit test implementation.  I criticized the overuse of dependency injection for the sole purpose of unit testing.  I attacked a large portion of best practices that I felt were only really being used in order to be able to unit test classes in isolation.

But, I never offered a solution.  I told you what was bad, but I never told you what was good.

I said don’t create all these extra interfaces, use IoC containers all over your app, and mocks everywhere just for the purpose of being able to isolate a class you want to unit test, but when you asked me what to do instead, I said “I don’t know, I just know what we are doing is wrong and we need to stop.”

Well, that is no answer, but I intend to give one now.  I’ve been thinking about this for months, researching the topic and experimenting on my own.

cool experiment thumb Back to Basics: Unit Testing, Automated Blackbox Testing, and Conclusions!

I finally have an answer

But, before I give you it, I want to give you a little background on my position on the subject matter.

I come from a pretty solid background of unit testing and test driven development.  I have been preaching both for at least the last 7 years.

I was on board from the beginning with dependency injection and IoC containers.  I had even rolled my own as a way to facilitate isolating dependencies for true unit tests.

I think unit testing and TDD are very good skills to have.  I think everyone should learn them.  TDD truly helps you write object oriented code with small concentrated areas of responsibility.

But, after all this time I have finally concluded, for the most part, that unit tests and practicing TDD in general do more good for the coder than the software.

What?  How can I speak such blasphemy?

The truth of the matter is that I have personally grown as a developer by learning and practicing TDD, which has lead me to build better software, but not because the unit tests themselves did much. 

What happened is that while I was feeling all that pain of creating mocks for dependencies and trying to unit test code after I had written it, I was learning to reduce dependencies and how to create proper abstractions. 

I feel like I learned the most when the IoC frameworks were the weakest, because I was forced to minimize dependencies for the pain of trying to create so many mocks or not being able to unit test a class in isolation at all.

I’ve gotten to the point now where two things have happened:

  1. I don’t need the TDD training wheels anymore.  I don’t pretend to be a coding god or demi-god of some sort, but in general the code I write that is done in a TDD or BDD style is almost exactly the same as the code I write without it.
  2. The IoC containers have made it so easy to pass 50 dependencies into my constructor that I am no longer feeling the pain that caused my unit tests to cause me to write better code.

What I find myself ending up with now when I write unit tests is 70% mocking code that verifies that my code calls certain methods in a certain order.

Many times I can’t even be sure if my unit test is actually testing what I think it is, because it is so complex.

Umm, did you say you had an answer, dude?

Yes, I do have an answer.  I just wanted to make sure you understand where I am coming from before I throw out all these years of practical knowledge and good practices.

I am not the enemy.

My answer to the problem of what to do if you shouldn’t be using IoC containers and interfaces all over your code base just for the purpose of unit testing, is to take a two pronged approach.

2prong thumb Back to Basics: Unit Testing, Automated Blackbox Testing, and Conclusions!

  1. Mostly only write level 1 or level 2 unit tests.  Occasionally write level 3 unit tests if you only have 1 or possibly 2 dependencies.  (I’ll talk about more how to do this in my next post)
  2. Spend a majority of your effort, all the time you would have spent writing unit tests, instead writing what I will call blackbox automated tests or BATs.  (I used to call this automated functional tests, but I think that name is too ambiguous.)

I intend to drill really deep into these approaches in some upcoming posts, but I want to briefly talk about why I am suggesting these two things in place of traditional BDD or TDD approaches.

What are the benefits?

The first obvious benefit is that you won’t be complicating your production code with complex frameworks for injecting dependencies and other clever things that really amount to making unit testing easier.

Again, I am not saying you shouldn’t ever use dependency injection, interfaces or IoC containers.  I am just saying you should use them when they provide a real tangible value (which most of the time is going to require alternate non-unit test implementations of an interface.)

Think about how much simpler your code would be if you just went ahead and new’d up a concrete class when you needed it.  If you didn’t create an extra interface for it, and then pass it in the constructor.  You just used it where you needed it and that was that.

The second benefit is that you won’t spend so much time writing hard unit tests.  I know that when I am writing code for a feature I usually spend at least half the amount of time writing unit tests.  This is mostly because I am writing level 3 and level 4 unit tests, which require a large number of mocks.

Mocks kill us.  Mocking has a negative ROI.  Not only is creating them expensive in terms of time, but it also strongly couples our test classes to the system and makes them very fragile.  Plus, mocking adds huge amounts of complexity to unit tests.  Mocking usually ends up causing our unit test code to become unreadable, which makes it almost worthless.

I’ve been writing mocks for years.  I know just about every trick in the book.  I can show you how to do it in Java, in C#, even in C++.  It is always painful, even with auto-mocking libraries.

By skipping the hard unit tests and finding smart ways to make more classes only require level 1 and level 2 unit tests, you are making your job a whole lot easier and maximizing on the activities that give you a high ROI.  Level 1 and level 2 unit tests, in my estimation, give very high ROIs.

The thirds benefit is that blackbox automated tests are the most valuable tests in your entire system and now you’ll be writing more of them.  There are many names for these tests, I am calling them BATs now, but basically this is what most companies call automation.  Unfortunately, most companies leave this job to a QA automation engineer instead of the development teams.  Don’t get me wrong, QA automation engineers are great, but there aren’t many of them, good ones are very expensive, and the responsibility shouldn’t lie squarely on their shoulders.

BATs test the whole system working together.  BATs are your automated regression tests for the entire system.  BATs are automated customer acceptance tests and the ROI for each line for code in a BAT can be much higher than the ROI of each line of production code.

Why?  How is this even possible?  It’s all about leverage baby.  Each line of code in a BAT may be exercising anywhere from 5 to 500 lines of production code, which is quite the opposite case of a unit test where each line of unit test code might only be testing a 1/8th or 1/16th a line of production code on average (depending on code coverage numbers being reached.)

I’ll save the detail for later posts, but it is my strong opinion that a majority of a development teams effort should be put in BATs, because BATs

  • Have high value to the customer
  • Regression test the entire system
  • Have a huge ROI per line of code (if you create a proper BAT framework)

Imagine how much higher quality your software would be if you had a BAT for each backlog item in your system which you could run every single iteration of your development process.  Imagine how confident you would be in making changes to the system, knowing that you have an automated set of tests that will catch almost any break in functionality.

Don’t you think that is worth giving up writing level 3 and level 4 unit tests, which are already painful and not very fun to begin with to achieve?

In my future posts on the Back to Basics series, I will cover in-depth how to push more of your code into level 1 and level 2 unit tests by extracting logic out to separate classes that have no dependencies, and I will talk more about BATs, and how to get started and be successful using them.  (Hint: you need a good BAT framework.)

As always, you can subscribe to this RSS feed to follow my posts on Making the Complex Simple.  Feel free to check out where I post about the topic of writing elegant code about once a week.  Also, you can follow me on twitter here.

Boise Code Camp 2010 Presentations

Went to Boise Code Camp this weekend and it was great!

There were some really good sessions and good discussions on Agile, Scrum and domain specific languages.

I think events like these are important because they help us to zoom out a little and see what kinds of things other people are doing in our field and what companies around us are interested in high quality software.

Code camps are also a great place to get some experience doing presentations and share what you are working on.  Networking opportunities abound.

I presented two talks this weekend.  I am posting the links to those here.

The first presentation I did was on Pair Programming.

You can find the presentation on Prezi here.

We also did a live pair programming session, which I thought went pretty well.  We ran into some of the issues that often come up with pair programming, like being afraid to tell someone you don’t like their method name.

The second presentation I did was on Creating an Internal DSL for Automated Functional Testing.

You can find the presentation on Slideshare here.

We went over how to go about creating a good automation framework which fills in the gaps between the business language and technical language.

Thanks to everyone who attended.

Automated UI Testing Framework… A Real Example

Michael Feathers recently blogged a very interesting topic he titled “UI Test Automation Tools are Snake Oil”.  This seems to have stirred up again the very controversial topic of automated UI testing.  I had previously posted about the pro’s and con’s of using a recording tool versus creating a framework.  Now I wanted to expand a little more on using a framework by giving a concrete example of a framework I have written and am using to actually write good automated UI tests that are not fragile.

The project

The project I am working on is mostly written in Java.  It is a case management system which workers use to enter applicants’ information and then run their eligibility to qualify them for benefits.  The basic use of the system is to create a new case, create people on the case with their information: income, relationships etc, and then run the eligibility and save the results.  One other important piece here I want to mention before I get flamed about technology choices is that the product was a legacy product and only supported on IE.

The tools

Based on most of the developers being versed in Java and the system requiring IE, I chose to use Watij for the framework.  Watij is basically a framework built on top of IE that directly interacts with the browser through the DOM through the COM interface.  When you are testing in Watij you are actually interacting with the browser, but since you are going through the com layer instead of clicking directly through the screens you are able to hide the browser window and run tests with the machine locked.  I also used JUnit as the test driver to execute the tests since it is easy to run from eclipse and can easily be run from an ant build to report the results and run on the build server.

The framework

The basic idea behind this framework is that I wanted to make it so a non programmer could write the tests and maintain the tests without having to know much about Java or Watij.  Even though Watij happens to be being used behind the scenes, I wanted the person writing the test scripts to only need to have domain knowledge of the application.  With that in mind, here is a picture of the stack.

 Automated UI Testing Framework... A Real Example

You can see in the diagram that I show a technology and how you interact with it.  IE uses HTML and CSS, Watij is built on top of IE and lets you work against the DOM.  My application framework is build on top of Watij and lets you work with domain specific concepts such as a specific screen, work flows and navigation.  The test scripts are written against the domain specific application framework.

Using this type of a framework, most tests can be written in as few a 5 lines of code.  Here is an example of a test:

     public void SetUp()
          this.caseNumber = CaseCreator.createCase( IBESDate.currentMonth(),
                                                    People.ADULT_MALE_1.onPrograms( Program.MEDICAID ) );

     public void taskCanBeSelected()
          Tasks.addTask( TaskType.RECEIVED_458, IBESDate.monthsInFuture( 3 ) );
                            IBESDate.monthsInFuture( 3 ),
                            IBESDate.monthsInFuture( 3 ) );

          Assert.assertTrue( WorkQueue.searchResultsHasCase( this.caseNumber ) );


Now this isn’t going to make much sense to most people who read it, and I actually consider that a good thing.  If you looked at the screens in the application it would make much more sense.  Someone writing these tests only needs to know about the screens and they can figure out how to do something.  There is actually a large amount of things going on in the background when you run this test.  It will create a complete case, add the person to the case, then click through a task workflow to add a specific task, then it will go through another workflow to search by the task to see that it was added.  There are probably about 20-25 different screen that this test ends up going through with these very few and simple lines of code.

The secret behind these kinds of a simple UI automation tests is abstraction.  The framework is designed to essentially be a domain specific language.  By abstracting away anything that has to do with the browser, html and even the dom, I am able to present a simple and stable interface for the tester to do exactly what they want and not know about all the other technology going on behind the wheel.  Think about how complex a motor vehicle is, yet how simple the interface to the user is.  I see many automation strategies that require the person writing the test scripts to know about all nuances of browsers and html.

What about change?

Most automation strategies break down when encountering UI changes.  I welcome UI changes.  Go ahead and change the UI, it won’t break all my tests, it will break one specific place in the framework that knows about what kind of things are on a particular screen.  With this framework it is very easy to fix a broken UI in one place and all the tests will work again.  Contrast this to many of the automation scenarios you have seen where a single renaming of a button requires 50 tests to be reworked.  All of the Watij interaction is in one namespace called “UI”.  All of the screens use UI to interact with Watij, so that I could even swap out Watij for another browser automation framework.

Back to the snake oil

I do have to say I agree with many of the points Michael made in his post, but I have seen them overcome.  A tool is never going to be a magic bullet.  Yet at the same time we should not run away from UI automation.   We just need to correctly apply the same principles we apply to create good abstractions in our source code to our automation code.