Category Archives: Automation

Test Automation Framework Architecture

Test automation framework architecture efforts are often complete failures.

It’s true. I’ve worked with many companies who have given up on creating a good test automation framework architecture, because after investing a large amount of time and money and resources in doing it the wrong way, they have incorrectly assumed the entire effort is not cost effective.

In this post, I’m going to simplify the process of creating a test automation framework architecture and show you that—if you do it right—it can be a very good investment.

Test automation framework architecture basics

Let’s start off by talking about what the goals of a successful test automation framework architecture should be.

There are two goals I am interested in when creating a test automation framework:

  1. Having the ability to easily create simple automated tests that use the framework.
  2. Decoupling cosmetic application changes from the tests so that when the application changes, the tests do not all have to be updated

A good test automation framework architecture should at the very least provide these two important services. We create a test automation framework to allow us to support the creation of tests and to keep those tests from being dependent on the actual UI of our application—as much as possible.

I’m going to break down these goals just a little more to make sure we are all on the same page.

test automation framework thumb Test Automation Framework Architecture

Creating simple automated tests

First, we’ll start with having the ability to easily create simple automated tests that use the framework. Why do we care about this goal? And why would this be the responsibility of the framework?

I have found that one of the biggest failure points of test automation efforts is test complexity. The more complex the tests are, the harder they are to maintain. If tests are difficult to maintain, guess what? They won’t be maintained.

So, we really want to make sure that when we create a test automation framework architecture, we focus on making sure that test automation framework makes it as easy as possible for someone to create tests using it.

It is always a better choice to put the complexity into the framework instead of into the tests. The framework is usually maintained by developers and does not contain repeated code. But, tests are often created by QA or even business folks and the code found in tests is usually repeated across many tests, so it is vital to reduce the complexity on the tests, even if it means a large increase of complexity in the framework itself.

When I create a test automation framework architecture, my goal is to make it so the tests can read as close as possible to plain English. This requires some effort to pull off, but it is well worth it.

Decoupling the UI from the tests

Next, let’s talk about what I mean by decoupling cosmetic application changes from the tests.

This is also a vital goal of any test automation framework architecture, because if you fail to decouple the UI of an application from the tests, every single UI change in an application will cause possibly hundreds or thousands of tests to have to be changed as well.

What we really want to do is to make it so that our test automation framework architecture creates an abstraction layer that insulates the tests from having to know about the actual UI of the application.

At first this seems a little strange, especially when I tell automation engineers or developers creating an automated testing framework not to put any Selenium code into their actual tests.

All of the Selenium examples show you creating tests that directly use a web browser driver like Selenium, but you don’t want to write your tests that way—trust me on this one.

(By the way, I haven’t found a good book on creating an actual test automation framework architecture—I’ll probably write one—but, for now if you are looking for a good book on Selenium that starts to go into the topics I discuss here, check out Selenium Testing Tools Cookbook.)

Instead, what you want to do is to make it so the test automation framework is the only code that directly interacts with the UI of the application. The tests use the framework for everything they want to do.

So, for example:

Suppose you are creating a simple test to check to see if a user can log into your application.

You could write a test that looks something like this: (C# and Selenium in this example)

var loginInput = driver.FindElement(By.Id("txtUsername"));
loginInput.SendKeys("Joe");

var passwordInput = driver.FindElement(By.Id("txtPassword"));
passwordInput.SendKeys("$ecret");

var loginButton = driver.FindElement(By.Id("btnLogin"));
loginButton.Click();

But, what is going to happen when you change the ID of your “User Name” field? Every single test that uses that field will break.

On the other hand, if you properly create a test automation framework architecture that abstracts the UI away from the tests themselves, you would end up with a much simpler and less fragile test, like this:

LoginPage.GoTo();
LoginPage.LoginAs("Joe").WithPassword("$ecret").Login();

Now, if the ID of your “User Name” field changes, you’ll just change the code in your automated testing framework in one place, instead of changing 100 tests that all depended on the specific implementation of the UI.

A simple test automation framework architecture

Once you understand those two very important goals and why they are important, it is easier to think about how you should design a test automation framework architecture.

I’ve created quite a few test automation framework architectures, so I’ll give you a basic design I use for most of them.

Take a look at this diagram:

test automation framework architecture thumb Test Automation Framework Architecture

Here you can see that there are four layers to my test automation framework architecture.

Frist, we have the browser layer or the web app itself. This just represents your actual application.

Next, we have the Selenium or web driver layer. This layer represents your browser automation tool. Selenium is basically just a framework for automating a browser. It doesn’t know anything about testing, it is an API that lets you interact with the browser programmatically. (By the way, you don’t have to use Selenium. I just use it as an example here, because it is the most popular browser automation framework.)

After that, we have the framework layer. This is the actual framework you create which uses Selenium, or whatever web driver you want, to actually automate your application. The framework is acting as an abstraction between your application and the tests. The framework knows about the UI of your application and how to interact with it using Selenium. It is your job to create this layer.

Finally, we have the actual tests. The tests use the framework to manipulate your application and check the state of the application. These tests should be simple to understand and should not have to know anything about the actual implementation of the UI of your application. These tests should rely on the framework to give them the ability to do whatever they need to do in your application.

Now what?

Obviously, creating a test automation framework architecture is more complicated than what I have shown you in this article, but with these basics you should be off to a good start.

If you want a more in-depth and detailed explanation as well as an example of how exactly to create a full test automation framework architecture, you might want to check out my Pluralsight video on the topic:

Creating an Automated Testing Framework With Selenium

I also frequently blog about test automation topics and other topics related to becoming a better and more profitable software developer.

Sign up here to make sure that you don’t miss any of my posts and get access to the content I only share with my email list. (Don’t worry I hate spam as much as you do.)

If you have any question, post them in the comments below.

Selenium With Node.js and Mocha: Automated Testing With JavaScript

I’ve always found it a bit strange that we’ve had to use a language like C# or Java to write automated tests using Selenium.

Not that I dislike either of these languages, but—especially in modern JavaScript times—it is a little bit strange to write a web application in primarily JavaScript and then write the automated test for it using C#, Java, Ruby or Python.

That is why I was pretty excited to see that there were Selenium bindings for JavaScript that would allow you to write your automated tests in JavaScript and run them from Node. You can finally use Selenium with Node.

In this post I am going to take you through the process of getting setup and running a simple JavaScript based test using Selenium with Node.js and Mocha.

(The instructions here are for Windows, but you could easily follow most of this post if you are planning on running your tests on Mac or Linux—in fact, it should be much easier.)

Installing Node.js

Before we can get started, we first need to get Node.js installed.

Just head over to the Node.js site’s download page to find downloads for Node.js for each platform.

2014 01 31 16 05 04 thumb Selenium With Node.js and Mocha: Automated Testing With JavaScript

Just select the appropriate version and install.  You’ll need Node.js to run the JavaScript tests because you need some way to execute your JavaScript locally.

(If you are interested in learning more about Node.js, I recommend the excellent Node.js in Action book.)

Getting Selenium with Node working

Once we have Node installed, we need to get Selenium webdriver so that we can actually have a way to automate a browser.

Not so fast though, before we can get that part working, we’ll need to make sure we have the latest chrome driver.  Go download the Chrome driver first and put it in your PATH.

2014 01 31 16 17 01 thumb Selenium With Node.js and Mocha: Automated Testing With JavaScript

2014 01 31 16 18 13 thumb Selenium With Node.js and Mocha: Automated Testing With JavaScript

Now, we are ready to get the Selenium webdriver bindings that we’ll use to automate the browser from JavaScript.  The bindings will just expose the Selenium API through JavaScript, which we can access by importing a module in Node.

Go to the location where you are going to create your test project and either open up a command line or terminal window there.

We’ll use Node’s built-in package manager, or NPM, to install the Selenium webdriver package.

Just run “npm install selenium-webdriver” to install Selenium webdriver in the current project directory.

2014 01 31 16 24 07 thumb Selenium With Node.js and Mocha: Automated Testing With JavaScript

Let’s do a quick test to make sure everything is working so far.

Create a file named first_test.js in your project with the following code:

var webdriver = require('selenium-webdriver');

var driver = new webdriver.Builder().
   withCapabilities(webdriver.Capabilities.chrome()).
   build();

driver.get('http://www.google.com');
driver.findElement(webdriver.By.name('q')).sendKeys('simple programmer');
driver.findElement(webdriver.By.name('btnG')).click();
driver.quit();

Then, execute the code by running:

node first_test.js

You should see the Chrome browser popup and it should search for “Simple Programmer”, then quit.  (If it goes too fast, take out the driver.quit line.)

(Also, a good book in Selenium is Selenium Testing Tools Cookbook.  You can also check out my courses on Pluralsight.)

Adding some Mocha

Mocha is a test framework commonly used to write unit tests for JavaScript code, but we can also use it as a driver to drive our Selenium tests when using Selenium with Node.

We can get Mocha using the Node package manager as well, but for Mocha we want to use the –g option which will install the package globally instead of just locally to our project.  This will allow us to run Mocha on Windows without using Cygwin.

Go ahead and install Mocha by typing:

npm install –g mocha

2014 01 31 20 55 35 thumb Selenium With Node.js and Mocha: Automated Testing With JavaScript

Writing your first test

Now we are ready to write our first test using Mocha.

Create a new file in your directory called mocha_test.js with the following contents:

var assert = require('assert'),
test = require('selenium-webdriver/testing'),
webdriver = require('selenium-webdriver');

test.describe('Google Search', function() {
  test.it('should work', function() {
    var driver = new webdriver.Builder().
    withCapabilities(webdriver.Capabilities.chrome()).
    build();
driver.get('http://www.google.com');
    var searchBox = driver.findElement(webdriver.By.name('q'));
    searchBox.sendKeys('simple programmer');
    searchBox.getAttribute('value').then(function(value) {
      assert.equal(value, 'simple programmer');
    });
	driver.quit();
  });
});

This simple test will test that we can type “simple programmer” into Google’s search box and verify that the text is there.

A pretty simple test, but it demonstrates the basic skeleton of creating an automated test with Mocha using Selenium with Node.js.

You can execute the test by typing:

mocha mocha_test.js

If the test executes too quickly, you can comment out the driver.quit to keep the browser window open.

2014 01 31 21 03 11 thumb Selenium With Node.js and Mocha: Automated Testing With JavaScript

Going further…

Well, that should get you started with using JavaScript and Node to run a Selenium test driven by Mocha.

You can use this basic setup to create much more robust automated tests.  I call these kinds of tests blackbox automated tests or BATs(Check out my series if you want to learn more about creating a automated testing framework.)

And if you liked this post, go ahead and sign up here to make sure you don’t miss other content like this.

Creating an Automated Testing Framework With Selenium

I have another new course on Pluralsight, check out:

Creating an Automated Testing Framework With Selenium

2013 09 28 16 31 58 Creating an Automated Testing Framework With Selenium

I am very excited to finally get this course out.  Many viewers have been asking me to make a comprehensive course that actually shows you how to build a real automation framework, and I finally did it in this course.

I reveal all my tricks and tips here and tell you exactly what I have done in the past to build successful automation frameworks.  It was a lot of fun to create this course and it brought me back to the glory days of creating automated tests.  (This is some serious fun!)

Anyway, check out the course and let me know what you think.

Here is the official course description:

Learning how to use a tool like Selenium to create automated tests is not enough to be successful with an automation effort. You also need to know how to build an automation framework that can support creating tests that are not so fragile that they constantly break. This is the real key to success in any automation effort.

In this course, I will reveal every secret I know from creating several successful automation frameworks and consulting on the creation of others. I will show you exactly, step-by-step how to create your own automation framework and I will explain to you the reasoning behind everything we are doing, so you can apply what you learn to your own framework.

We’ll start off this course by going over the basics of automation and talking about why it is so important as well as discuss some of the common reasons for success and failure.

Then, I’ll take you into the architecture of an automation framework and show you why you need to pay careful attention to the structure of any framework you build and give you some of the underlying design principles I use when creating an automation framework.

After that we’ll be ready to start creating a framework. In the next few modules, I’ll show you how to create a real automation framework capable of automating the WordPress blogging platform administrative console. We’ll start off by creating smoke tests and using those smoke tests to build out our initial framework.

Then, we’ll expand the capabilities of our framework as we create more tests and learn how to use techniques like dummy data generators to make our tests as simple and easy to read as possible.

Finally, I’ll take you through some best practices and tips that cover topics like scaling out, working in Agile environments and other important issues you are likely to face. If you are responsible for an automation project for a web application or you want to start using automation, you’ll definitely want to check this course out.

Getting up to BAT: Building a True DSL

If you’ve made it this far with your BAT implementation, you have finally arrived.

Not to say that you’ll ever be done expanding your automation framework and building and improving your BATs, but you are at the point of having a mature usable BAT framework and you can be proud of that accomplishment.

If you truly want to see where the rabbit hole goes though… Read on

Taking it to 11

spinal tap but it goes to eleven thumb Getting up to BAT: Building a True DSL

If you have designed a good automation framework for writing your BATs, you will most likely have created an internal domain specific language (DSL).

What is an internal DSL?

Basically it is a set of APIs and rules built in another language that function as their own language.

You might have some code that looks like this:

Pages.Login.Goto();
Pages.Login.LoginWithDefaultUser();
Pages.Customers.Goto();
Pages.Customers.AddNewCustomer();

 

You can see how this code is really looking like a language itself.  It is hard to tell if this code is even C# or Java.

This is a good example of an internal DSL.

If we were to take this same section of a BAT test and write it as an external DSL, it might look something like this:

LOGIN WITH DEFAULT-USER
ADD-CUSTOMER WITH DEFAULT-CUSTOMER

 

With a true DSL we can simplify even further to get right down to the bare minimum set of commands we need to execute what we want in our application.

How to implement a DSL?

There are a number of tools and techniques that can be used to create a DSL.  I’ll hit on some of those a bit later on, but first we need to design the DSL itself.

Designing a DSL can be challenging, but I have found the best approach is to use a similar approach to what we used to design our automation framework.

It is best to work backwards from the syntax that would make your BAT the easiest to write and the most clear.  Once you have done that you can worry later about implementing the DSL to support the syntax you have come up with.

You should focus on creating a limited vocabulary for your DSL so that it will be easy for someone to learn and know what their options are.

The more restrictions we put into a DSL the easier that DSL becomes for doing a specific purpose.

We want to give the user of a DSL no more options than required for what they want to do and no less.

This isn’t always easy to do.  It often requires some careful thought and refinement.

Your DSL should also use terminology and phrases that are familiar to the business domain of your application.  By doing this you will make it so someone understanding your application will understand BATs written in that DSL.

I also always try to make a BAT DSL read as much as plain English as possible.

Once you have designed the DSL, you will need a way to implement it. 

I’ve written about using ANTLR to create an external DSL.  Also I have written a review on a book that I recommend for learning ANTLR.

Microsoft also has a DSL SDK you can use in VS2010, although I haven’t used it myself.

The basic idea behind implementing a DSL is that you have to take some sort of grammar you design for your DSL, parse it into a syntax tree, then use that tree to generate code that will actually perform the actions.

It is by no means a trivial task, but using a tool like ANTLR, makes it much simpler than trying to completely generate a parser and execution engine yourself.

Why even bother?

Creating a DSL for your BATs is completely optional, but there are some significant benefits to doing so.

The biggest one I have found is that you are much more likely to get business people to write BATs if you have a simple DSL they can use to create them.  Once you get everyone using BATs to define behavior and requirements, communication becomes much more simple.

Another great thing about creating a DSL is that you can create a runner that can interpret the DSL BAT scripts which can be run on any machine.  Typically, to run BATs created in Java or C#, you would need a C# or Java compiler and a test runner, etc.  By creating a DSL and a stand alone executor, you can very easily allow anyone to run BATs with little to no setup.

Writing your BATs in a DSL will also make writing the BATs and understanding them much easier.  A good DSL can get rid of all the little language and API nuances that don’t really add information to the specific domain you are working in.  A good DSL strips information down to the minimum amount needed and is able to do so, because it assumes the context is your application’s business domain.

You will also find that you are better able to separate framework from tests when you have a DSL in place.  One issue that many teams face when creating a BAT framework is that some people come along and write lower level code in the higher level BATs.  This creates a problem because it is hard to force someone to use the higher level API when the framework and the BAT code are the same language.

Wrapping it up

So that completes my series on Getting up to BAT.

Hopefully you have learned enough to be able to implement your own BAT framework and get BATs running for your application.

I tried to cover most of the important steps here without getting too deep into the details.

If you have any questions or are interested in getting more in depth training or consulting on getting BATs setup for your environment, send me an email or leave a comment.

It can be a little difficult to get started and get a BAT framework up and running, but the rewards are well worth the effort!

As always, you can subscribe to this RSS feed to follow my posts on Making the Complex Simple.  Feel free to check out ElegantCode.com where I post about the topic of writing elegant code about once a week.  Also, you can follow me on twitter here.

Getting up to BAT: Scaling Out

If you haven’t been following, it has been a while since my last post on this topic.  I had a bit of distraction the last few weeks, but I am back and ready to go!

When we last left off we had just gotten our BATs as part of the acceptance criteria for any new backlogs that are worked on.  This puts us at a point where we could really say that we have successfully implemented BAT testing.

You don’t want to get too comfortable just yet though, because the next hurdle you will most likely face will be the problem of not having enough time to execute all your tests.

You want to think about this ahead of time

Nothing worse than getting everything going and then not being able to execute the entire test suite, because you didn’t plan ahead.

You don’t want to get to the point where confidence in your BAT suite it lost because you are not able to get all the tests executed in a reasonable amount of time.

The more frequently your tests are run the more value they have.

By reducing cycle time from the time a breaking change is made in the system and the time it is discovered, you greatly reduce the risk it imposes on your software and you decrease the scope of the code changes in which it could have occurred.

To put it simply, the faster you can find out you broke something, the more likely you can fix it before it does damage, and the more likely you will be to know what thing you did caused the breakage.

clarkson face thumb Getting up to BAT: Scaling Out

How can we reduce cycle time?

There are a few different strategies we can employ and we can mix and match some of these strategies.

Straight forward parallelization

The best and most effective thing you can do is to take a given test run, split up the execution of those tests on multiple machines and execute them in parallel.

This approach is going to give you the best bang for your buck.  You should really try to get some amount of parallelization going before attempting any other solution, since it is going to make a huge impact on the total execution time for your tests without making any sacrifices.

There are even many ways you can mix and match to do parallelization:

  • Use multiple physical machines
  • Run many virtual machines on a single host
  • Put an executor program on every team member’s machines that will execute tests when that machine is idle or put into test execution mode (perhaps at night)
  • Use a cloud based computing platform to execute your tests
  • Run multiple browser instances on a single machine

Data preloading

With this approach, you would preload some of the test data that your tests might generate by manually clicking through screens.

This is fairly hard to explain, so let me give you an example:

Suppose you had a set of tests that all involved creating customers, but each customer you create in the system takes about 3 minutes to create by clicking through the screens to get them into the system.

We don’t need to have 500 tests all executing the same exact logic in the system 500 times for 3 minutes just to generate all the customer data that will be used in the tests.

Instead, we can leave a few tests that are exercising the customer creation functionality, and we can execute a SQL script to push all the other customer data into the test database for the other tests to use.

Using this technique we might be able to reduce our total execution time by 3 minutes * each test, or about 25 hours for 500 tests.

This can be a huge savings in time, and it doesn’t come at that high of a cost.  The sanctity of our tests is slightly compromised, but we are taking a calculated risk here knowing that we already have covered the area of execution which we are preloading data for.

Consider this technique when you notice certain things in the system taking a very long time to do.

Test runs by area

With this technique, we can reduce the total execution time in a given period by splitting up test areas and running them either at different times or in response to changes in certain areas.

You have to be very careful with this approach, because if you don’t do it correctly, you can start to erode some of the safety your BAT tests are providing you.

I would only do something like this as a last resort, because it is so easy to virtualize today, and hardware is so cheap.

I’d much rather run too many tests than too few.

Test randomization

With test randomization, we are going to take our total desired execution time, and divide it by the average time for running a test.  We then can use that number of tests to run to randomize the execution of our tests each time we run them and only run the number or tests that will fit in the desired execution time.

This choice is also a compromise that I typically don’t like to take.

It can be very useful though, combined with other techniques when you still don’t have enough time to execute your entire test suite.

The basic idea here is that you are going to randomly run tests each time to fill up the time you have to run tests.

Test reduction

This one seems fairly obvious, but can be very helpful.

Often I will see teams starting out with automation, trying to write way too many BAT tests for a given feature.  Sure, with automated tests it is possible for run a test for every single possible combination of values in your 5 drop downs, but will it really benefit you?

In many cases you have to think about what you are trying to protect against with your BATs.  Sometimes running every combination of data selection choices is going to be important, but other times you are only going to need to write tests to test a few of the happy path scenarios.

It is important to find a balance between test coverage and test volume and not just for execution time.  There is a logistical overhead to having a large volume of mostly redundant tests.

So even though this technique might seem dangerous and counter-productive, I will almost always employ it to some degree.

Common pitfalls

Here are some of the things you might want to watch out for as you are scaling out and streamlining your execution of BATs:

  • Parallelization issues.  If you are using shared state, you can run into big trouble when your tests are executing in parallel.  There are many manifestations of this.  You could have issues at the database level or at the local machine memory level.  The best way to avoid this kind of problem is to use separate virtual machines for each test execution, and not reuse data setup between test cases.
  • Ineffective error reporting.  If you run a huge volume of tests in parallel, you better have a good way to sort through the results.  It is much harder to figure out why things failed when they are run across multiple machines.
  • Test order dependencies.  Make sure tests don’t rely on being run in a certain order or you will have lots of pain when you disrupt that order.
  • Environment setup.  Make sure all your test execution environments are exactly the same unless you are specifically testing different environments for execution.  You don’t want tests failing on one machine but passing on another.

As always, you can subscribe to this RSS feed to follow my posts on Making the Complex Simple.  Feel free to check out ElegantCode.com where I post about the topic of writing elegant code about once a week.  Also, you can follow me on twitter here.

Getting up to BAT: Adding BATs to Your Acceptance Criteria

So now that we have built our automation framework, and got our smoke tests running as part of the build, we now need to make sure new BATs are being created for new backlogs.

This is definitely one of the more challenging tasks you will face, and perhaps the most critical.

All of your efforts will be in vain if you can’t get to the point of creating new BATs for new backlogs that are finished, because if you don’t you’ll still be doing manual testing, and you’ll be playing catch-up all the time.

Start off small

The goal here is to eventually get to the point where every single backlog requires BATs to be written and passing in order for that backlog to be called “done.”

If you try to drop this lofty goal onto a team in one iteration, you are likely to get beaten and slapped with much vigor.

Instead, start with a simple goal of requiring that at least one backlog have BATs for all of the functionality of that backlog.  Start with this small goal, but enforce it with rigor.

By starting out small like this, you will get the team accustomed to the idea of building automated tests at the same time as building the functionality and you will give them a chance to have a small success without sacrificing much velocity.

The goal at this point should be to get people excited about writing BATs along with the backlog.

It’s actually quite fun

I have found that developers tend to really enjoy writing automated tests.  Make sure you let everyone who wants to share in the fun, not just QA and not just developers.

It is really very exciting to see the “magic” of an automated test clicking through buttons on your web page and doing all kinds of things that seemed so tedious before.

I have found that this is so much fun, that you might have the problem of developers not wanting to work on the real backlog, but rather to write the automated tests for it.

This is good, you really want to foster this attitude, it will greatly help when you finally…

Drop the bomb!

What bomb?  The bomb that says on it:

All backlogs will require BATs as acceptance criteria.  It’s not done unless it has automated tests that prove its functionality.

You should be getting to this point within 3 to 4 iterations on average.  Don’t go too fast to get here, but don’t go too slowly either.

I know this seems like an impossible task, but it really isn’t.  I’ve been here before and had much success with it on several different teams.  The key is making sure everyone is comfortable with writing automated tests and understands the value.

Whatever you do though, once you draw this line in the sand, Do not back down! I mean it.

You might start getting death threats.  People might directly laugh in your face all the while spitting skittle colored loogies in your coffee, daring you to “taste the rainbow,” but you have to hold your ground.

skittlesny6 thumb Getting up to BAT: Adding BATs to Your Acceptance Criteria

It will be worth it in the long run, because you will have built an awesome safety net for proving the functionality of your software over time.

It’s not all rainbows and butterflies

You are going to sometimes run into scenarios where automation will be too high of a price for a particular backlog.

This will especially happen in cases where you have some kind of process that spans multiple domains or software systems.

When you run into those situations, give it a solid effort to at least automate as much as possible, but don’t die on the hill and kill the whole automation project.  We have to be pragmatic or risk having our credibility called into question.

Don’t expect everyone to embrace writing BATs overnight, or to even be good at it.  When you first introduce this to the team, you are going to have a learning curve and your automation framework is still going to have some pretty big holes in it.

But, that is OK.  Over time you will build out that framework and produce some fierce battled scarred veterans able to automate web pages while scowling at you from afar, just don’t lose hope and keep beating that drum!

As always, you can subscribe to this RSS feed to follow my posts on Making the Complex Simple. Feel free to check out ElegantCode.com where I post about the topic of writing elegant code about once a week. Also, you can follow me on twitter here.

Getting up to BAT: Adding Smoke Tests to Your Build

Once you’ve built some smoke tests with your shiny new automation framework, you are going to want to get those smoke tests up and running as soon as possible…

But!  You might want to consider holding off for a second and reading this post!

It is worth taking a bit of time and thinking a bit about the strategy and psychology of adding the smoke tests to your regular build process.

There are several things we are going to want to discuss and think about before adding automated tests to a build.

build a site2 thumb Getting up to BAT: Adding Smoke Tests to Your Build

Why are we adding the smoke tests to the build?

I think it is always important that before we do something we ask ourselves why we are doing it.  Doing things, because someone said it is good, or because it seems right is not a very good reason.

The primary benefit we are going to get by adding smoke tests to the build now instead of waiting for a larger portion of tests to be written is that we are going to be able to immediately tell if a basic functionality of the system is broken instead of finding out after we have deployed the build to different environments and used it.

We want to be able to get this value as soon as possible.  We don’t want to have to wait until we have a huge amount of tests, because even a few tests will give us some value in this area.

The other reason we are adding the tests to the build is so that we can notify developers when a test fails, so that they know to fix whatever was broken.  By having the smoke tests run as part of the build, we are able to reduce the amount of time before a break is introduced and the break is discovered.

This concept of reducing the delta between introduction and discovery is very important, because it makes it much easier to identify what caused a break.

Perhaps a distant 3rd reason to add the smoke tests at this stage, is to prove out our framework and technologies.  Better to start small with a small number of tests and get everything working.

With those ideas in mind, we can move onto some of the more important considerations.

Don’t ever add failing tests to the build

When you add your smoke test suite to the automated build, all of the tests should pass!

I will say it again, because it is really important and ignoring this advice may doom your entire BAT project.

Do not ever add failing tests to the build!

It seems like an innocent enough thing, but it causes great harm, because it is very much like the boy who cried wolf.  Consider what happens when we introduce our new BAT system and we add our smoke tests to the build, but we have 3 failing tests.

First thing that will happen is that our tests will run during the next build and 3 will fail.  This should cause a build failure notification of some sort to go to the team, at which time you will be in the uncomfortable position of saying something to the effect of “ignore that fail, those are real defects, but they are known issues.”

Next, let’s assume a developer checks in some bad code and it causes a 4th test to fail.

Do you think most people will notice this 4th failing test?

Even if they do, do you think they will just assume it is another known issue?

Worse yet, they may think your BATs are meaningless and just fail randomly.

Do you see where I am going with this?  We want to start off with the correct expectations and we want test failures to be meaningful.  To ALWAYS be meaningful.  The second you have test failures that are not meaningful, you lose credibility for any future meaningful failures.

This is such a critical psychological battle with build systems and automated tests that I have often implemented a policy where no one is allowed to work on anything else but fixing the build or test failure when a build fails.  When you have 20 developers either sitting idle or working on one issue for 2 hours, it sends a pretty clear message that we take build failures seriously.

Make sure your tests never undermine this message.

What do I do with failing tests then?

If they are real defects, put them in your defect tracking system or create backlogs for them and make sure the tests are disabled until those defects are fixed.

Your automated tests should be acting as regression tests for things that are already working.  It is absurd to think that it is even close to cost effective to write automated tests to actually functionally test new features or code!

It is the process of writing the automated test that should uncover defects.  Running of automated tests should uncover new defects where functionality has regressed from working to not working.

Other considerations

Here are a list of some other considerations you will want to think about before embarking on this journey.

  • How will you actually run the tests? (Will you use the build script to kick it off, some 3rd party plugin, etc)
  • Where will you run the tests? (Will you use the build server itself, or another machine.  You probably don’t need to worry about scaling out to multiple machines right now, but you should at least keep it in the back of your mind.)
  • How long will it take to run the smoke tests? (Fast is better, because feedback cycle is reduced.  If you have a really long test suite, you might want to pare down the smoke tests and run some with every build, and all of them every night for now.)
  • Will developers be able to run the tests locally? (Make sure you consider how to do this before putting the tests into the build.  Nothing more frustrating than being measured against a bar you can’t yourself see.)
  • How will you notify and report results from the test failures? (We don’t care about passes, only failures, but you need a good way to report the result that prompts action.)
  • What is the policy for test failures? (Make sure there is some agreement on what kind of action will be taken when tests fail.  A great place to start, if you can get management to buy in on it, is all work stops until all tests pass.  Extreme, but extremely effective.)
  • How will you prevent false fails and identify them? (Don’t get stuck with egg on your face.  If your framework fails and the problem is not a real defect, you need to be able to quickly identify it and fix the framework problem.  There are a few strategies for doing this, but that is for another post.)
  • How do new tests get added and commissioned? (You are going to be constantly growing your test suite, so you will need a good method for adding tests to the smoke test suite.)

Remember, you only get one chance at a first impression!

As always, you can subscribe to this RSS feed to follow my posts on Making the Complex Simple.  Feel free to check out ElegantCode.com where I post about the topic of writing elegant code about once a week.  Also, you can follow me on twitter here.

Getting up to BAT: Creating Your First Smoke Tests

If you have been following my posts so far on Becoming Bat Man, you should already have hired an automation lead, figured out what browser driver you are going to use and come up with a design plan for your automation framework.

smoking computer thumb Getting up to BAT: Creating Your First Smoke Tests

Now you are probably faced with the difficult decision of…

Where to begin with your tests

I don’t suggest writing your automation framework first and then writing tests to use it.  If you go this route you’ll have two problems

  1. It will be a long time and large investment before you see any fruits from your automation effort.
  2. Your framework will likely be designed incorrectly or not test-writer friendly.  (Which is basically the same as wrong.)

What you really want to do is to write some basic tests that let the development of those tests drive the development of your framework.

The best tests to drive your framework are some basic “smoke tests.”  If you are not familiar with the term “smoke test,” it basically means some tests that make sure the application you are testing is at least somewhat working and not completely broken. 

There are a few different opinions of what exactly constitutes a smoke test, but in general, if you have any tests that you regularly run on a new build to make sure it is at least OK, those would be the tests we are interested in here.

You are going to want to start by taking the easiest and most simple tests you have and writing them in a concise syntax that you use to drive the development of your framework.

It is very important at this stage to not worry about what you think might be possible or how it would be implemented in the framework, but to instead focus and making the tests as terse and explanative as possible.

The basic idea is that you will write the test out in the syntax you would like to be able to use, then implement the framework features required to make that test pass.  This whole process is very similar to doing Test Driven Development.

An example case

The easiest way to demonstrate this is with an example.

Let’s say you have a smoke test like the following:

Create a new account, pick an on-sale item, add it to your cart and check out using PayPal.  Verify that we hit the order success page.

We might write a test for this that looks something like:

Workflow.Users.CreateNewUser();
Pages.Login.LoginWithLastCreatedUser();
Pages.SaleItems.AddAnyToCart();
Workflow.Cart.CheckoutUsingPayPal();
Assert.True(Pages.Checkout.SuccessPage.IsAt());


What I just wrote above is astonishingly simple.  You are likely saying to yourself something along the lines of “What if… blah.”

Forget it!  We’ll handle “what if” later.  Right now we are going to write the tests as terse as possible and not handle any other conditions.

What I wrote above is the style that I like to write the tests in, which is highly subject to opinion, but the content of what I wrote is not as subjective.

There are a few important points to note:

  • I used methods like CreateNewUser, LoginWithLasteCreatedUser and CheckoutUsingPayPal that do a huge amount of work and have no parameters.  Why?  Because for the purpose of this test, (and you’ll find for most of your tests), we don’t care about what the first name or last name or address of the user is, we don’t care how we get them logged in, and we only care that they checkout using PayPal, not what account is used, etc.
  • I didn’t declare any variables.  Tests should be simple.  Let the framework remember the state.
  • I over-simplified.  This is fine, you are working inside of a very specific domain.  Let your test framework methods do one large thing.  Don’t worry about oversimplifying for now.
  • I asserted something at the end.  All tests need to assert something.  If you can’t figure out what to assert, work on redesigning the test before automating it.
  • I didn’t pass parameters.  Avoid this also.  This goes along with the first point, but is so important that I feel it is OK to state it twice.  If you don’t care about variation, eliminate it.  You will constantly need to find places where a manual tester would have to enter specific information, but an automated test won’t care what that information is.
  • All my methods are static.  While this would be bad in normal production code, for a test API it is the easiest for test writers to use.  No variable declarations, no “newing” up objects, no constructors, etc.
  • It’s written in “business language.” As close to the language an end user or business person would use to describe the test.

So, this test is definitely doing a huge amount of stuff in a few lines of code, and because of this your inclination would be to say that the API level is too high. Don’t!

Instead, make the API level as high as you possibly can, until further tests you try to implement prove to you that it must be broken down further.

By working this way, you ensure a reduction in the total lines code and complexity of your tests by as much as possible.

You would NEVER work like this with normal code, but for automated tests this is essential.  The main reason is because you will be calling the same methods perhaps 1000s of times, where in normal code you usually only reuse a method in a few places.

Next steps

For my example above, the next step would be going through and implementing all the framework methods required in order to make that test runnable.  You will end up utilizing the general design you came up with for your automation framework to do this.

Then you will simply move on to the next test and repeat.  Each time you should have less and less effort required, as more of the basic functionality of your automation framework is implemented.

Automating your smoke tests should ensure that you get a fairly even distribution of the framework code written as the smoke tests should cover many different areas of your software.

But, don’t start adding these smoke tests to your build yet.  We’ll cover that in the next step.  Adding them to the build prematurely can cause the confidence in these tests to be forever diminished!  We don’t want to do that!

As always, you can subscribe to this RSS feed to follow my posts on Making the Complex Simple.  Feel free to check out ElegantCode.com where I post about the topic of writing elegant code about once a week.  Also, you can follow me on twitter here.

Guest Post: The Cross-Browser Compatibility Myth

This post is a special guest post from my wife, Heather Sonmez, who is an expert on the subject of blackbox automated tests and designing frameworks for them.  She has designed successful automation frameworks for several companies.  She deals with software automation and framework design issues on a daily basis, so I thought she might be able to offer some good insight on the topic.

John asked me to write a post based on my experience with writing automated tests.  I’ve been working as a QA Automation Engineer for the past three and a half years. During this time I have had the opportunity to work with Ruby/Watir, C#/Watin, and Java/Selenium RC. In thinking on a topic, I decided that there’s no time like the present to talk about what I’m presently facing, and would like to share some insight straight from the trench I currently find myself in today:

Automation tools that claim to support multiple browsers are LYING

Currently I am in the process of getting 400 automated tests that run in Firefox to run in Internet Explorer and Chrome. While the tests are using Selenium RC, a tool that claims to work on all three browser platforms, in actuality, many tests are failing and not for good reasons (bugs).

Here’s the rundown on what these overly-ambitious tools aren’t telling you:

  • Xpath in Browser A isn’t always supported in Browser B
  • The Dom in Browser A may vary from the Dom in Browser B
  • Browser B may have built-in security settings that interfere with testing
  • Bugs in the automation tool itself may cause issues in specific browsers
The XPath Issue

When it comes to XPath, I fully expected that any given expression was either valid or invalid. But it didn’t take more than one test run in Internet Explorer to suddenly find myself with tests that claimed certain elements didn’t exist. While Firefox accepted and successfully returned my element using the provided expression, IE claimed the expression itself was invalid. In this particular instance I discovered an issue with Selenium stripping off the end of my xpath statement, thus making it invalid.

But ultimately, whether it’s the browser or the tool doesn’t matter because the same expression is not working unequivocably in both browsers.

The Dom Issue

The next problem I immediately came across was the problem of Dom differences between browsers. In this particular case I had an element that occurred one time in Firefox but twice in IE. My test, expecting one instance of the element, failed. While I can’t tell you why this discrepancy exists (currently under investigation by some client-side engineers), I have a fair degree of confidence in saying this won’t be the last time I come across this issue.

In the quest to support many different browsers, it only follows that one is going to come across conditional browser logic, especially if a company is striving to support multiple browser versions (IE 6, 7, 8, 9). If a developer has to occasionally write browser-specific condition statements, it makes sense that we’d have to do the same to test specific browsers.

The Security Issue

If it seems that I’m picking on IE, it’s only because I can’t get very far in running our tests in Chrome. Chrome has special security settings that cause a Selenium exception if your test changes domains mid-run. This can easily be solved by passing a parameter to Chrome which will disable this security feature when the test runs, the trick is figuring out when/where/how to pass that parameter.

The Bug in the Tool Issue

Finally, the last obstacle I’ve come across in multi-browser testing: the tool itself. Any tool, regardless of how fabulous it is, is going to have bugs. Selenium RC has an issue where its select method (responsible for selecting a value in a dropdown menu) doesn’t trigger affiliated event handlers- in Internet Explorer. To work around this, one can manually call the event handler using Selenium’s runscript() but you will lose some degree of testing confidence in doing so (how do you know the event is really firing in response to a particular user action if you’re manually forcing it in automation?)

Another issue exists around multiple browser windows; if you need more than one open for your test, Selenium is unable to select the second window you open- another outstanding IE issue that has not been resolved.

Summary

In all fairness, some of these issues have more to do with the browsers themselves than with the automation tool. However, that makes it even more important to realize that when an automation tool says it supports multiple browsers, it’s not without some creativity, troubleshooting, legwork, and maintenance.

Knowing these kinds of issues exist is important in order to decide if and how much browser compatibility testing you want to automate. It’s definitely something to realize before deciding on a tool just because the tool claims to support other browsers.

Just as there is no silver bullet for writing a web application that works in every browser imaginable, there is no silver bullet for automation either.

Getting Up to BAT: Designing an Automation Framework

Now that you’ve gotten an automation lead and decided on the browser automation tool you are going to use, the next step is to design an actual automation framework.

This is one of the most critical components of the overall success of your automation strategy, so you will want to make sure you invest properly in this area.

I have seen a large number of automation projects go forth, but each time the critical component determining their success or failure was having a good automation framework.

Solid design and excellent technical resources are absolutely critical to success in this area!

What is an automation framework?

An automation framework is essentially an API that all of your BATs (Blackbox Automated Tests) are written against.

This API can be as simple as the API that is exposed by your browser automation tool (WatiN, Selenium, etc.), but I would highly recommend building your own layer on top of the browser automation tool’s API that will act as a DSL (Domain Specific Language) for automating your application.

Let me break this down a bit further by using an example.

Think about the task of making coffee.

There are several “API” levels we can interact with.

  1. We can go with a very low level API where we take whole coffee beans and grind them down.  Then we take some water, get it hot.  We take our filter put the ground beans in it, put it over a cup and pour the water into the filter.
  2. We can go with a higher level API where we use a traditional coffee maker.  In this case we load the coffee maker with a filter, ground coffee beans and water and push a “brew” button.  We could also set it to start at a certain time.
  3. We can go with a very high level API where we use a Keurig machine or similar device.  In this case we only make sure the machine has water in it, and we just insert a little pod and press brew.  We can make different kinds of coffee, cider or hot cocoa just by changing what pod we use.

unnamed thumb Getting Up to BAT: Designing an Automation Framework

Using the API provided by the browser driver is like making the coffee by hand.  It’s going to take a large amount of effort each time you do it.

We want our automation framework to be more like the Keurig machine.  We want to be able to compartmentalize our tests in little pods that are small and don’t require many hooks into the automation framework.

To rehash this one more time, basically our automation framework will be a framework we build on top of the browser driver framework, which is designed to make it easy to write tests which automate our application.

What makes a good automation framework design?

The true measure of an automation framework is the size of the tests that are written against them. The less lines of code in each test, the better the automation framework captures the domain of the application being automated.

In my earlier post about an example of an automation framework, I talked a bit about the strategy I used in a real implementation of an automation framework.

Here is a diagram showing the different layers.  You can see the framework layer is in green here.

automated testing framework 2 thumb Getting Up to BAT: Designing an Automation Framework

You can also see in this diagram that on the right hand side, I have screens, workflows and navigation.

This is one of the common design patterns you can use to build your automation framework, since it closely models that of most web applications.

A good way to design your framework is to use a pattern like this and create classes for each page in your application, create classes which might represent workflows in your application, and create some classes for navigating around the application.

The basic idea you are shooting for with your automation framework is to make it so the tests do not have to know anything about the browser.  Only your automation framework itself should be dealing with the DOM and HTML and CSS.  Your tests should be dealing with concepts which any user would be familiar with.  That is why I commonly use pages and workflows.

Let me give you an example.

Let’s say we ignored the advice of creating an automation framework and decided to program our tests directly against the browser driver layer.  In this case, let’s say that we want to automate a login page.  Our test might look something like this pseudo-code.

browser.goto("http://mywebsite/login.aspx");
Element userName = browser.getElementById("login_userName");
userNameTextField.Type("Joe");

Element userPassword = browser.getElementById("login_userPassword");
userPassword.TextField.Type("mypassword");

Element loginButton = browser.
    getElementByXPath("//button[class="loginButton"]");
loginButton.Click();

 

This is not very readable.  It is not something an average tester will be able to pick up and start writing, and it is extremely fragile.  If you have 500 tests like this and the id changes for one of the elements on the page, many tests will break.

Now, contrast it to this pseudo-code:

Pages.Login.Goto();
Pages.Login.LoginWithDefaultUser();

 

Where did all the code go?  You don’t need it!  Actually it went into the framework, where it can be reused.  Now your test code is simple, can be understood by just about anyone, and won’t break with changes to the UI.  (You will just have to change the framework method instead of 500 tests.)

If you want some more detailed examples take a look at my Boise Code Camp slides on the subject here.

Let me offer up some general guidelines for creating a good design for an automation framework.

  1. NEVER require the tests to declare variables.
  2. NEVER require the tests to use the new keyword or create new objects.
  3. NEVER require the tests to manage state on their own.
  4. ALWAYS reduce the number of parameters for API calls when possible.
  5. ALWAYS use default values instead of requiring a parameter when possible.
  6. PREFER to make the API easier to use over making the internals of the API less complex.
  7. PREFER using enumerations and constants to requiring the test to pass in primitive types.  (Make the input bounded when possible.)
  8. NEVER expose the browser or DOM to the tests or let them manipulate it directly.

You can see that these guidelines will almost force your API to be entirely static methods!  Don’t freak out.  It is perfectly fine.  There is nothing wrong with using static methods in this case.

Stop… take a breath, think about why you think static methods are bad, because I probably agree with you.

But, in this case we are HIGHLY valuing making the tests simple and easy to write over anything else, and because of this emphasis, static methods are going to be the way to go.

Some advice on getting started

Start with the tests in plain human English.

Take those tests and make them into code preserving as much English as possible.  Then work down from there implementing the methods needed to make those tests work.  Only add things as you need them.  Do not overdesign!

In my example above, we might have started with something like this:

1. Goto the login page.

2. Login with the default user.

The actual code looks just like that.  It should always be a 1 to 1 mapping.

This is where having someone that has some experience doing this is going to come in handy.

Just keep in mind the whole time your 2 big goals:

  1. Easy to write tests
  2. Tests are concise
As always, you can subscribe to this RSS feed to follow my posts on Making the Complex Simple.  Feel free to check out ElegantCode.com where I post about the topic of writing elegant code about once a week.  Also, you can follow me on twitter here.