Category Archives: BAT

Getting up to BAT: Building a True DSL

If you’ve made it this far with your BAT implementation, you have finally arrived.

Not to say that you’ll ever be done expanding your automation framework and building and improving your BATs, but you are at the point of having a mature usable BAT framework and you can be proud of that accomplishment.

If you truly want to see where the rabbit hole goes though… Read on

Taking it to 11

spinal tap but it goes to eleven thumb Getting up to BAT: Building a True DSL

If you have designed a good automation framework for writing your BATs, you will most likely have created an internal domain specific language (DSL).

What is an internal DSL?

Basically it is a set of APIs and rules built in another language that function as their own language.

You might have some code that looks like this:



You can see how this code is really looking like a language itself.  It is hard to tell if this code is even C# or Java.

This is a good example of an internal DSL.

If we were to take this same section of a BAT test and write it as an external DSL, it might look something like this:



With a true DSL we can simplify even further to get right down to the bare minimum set of commands we need to execute what we want in our application.

How to implement a DSL?

There are a number of tools and techniques that can be used to create a DSL.  I’ll hit on some of those a bit later on, but first we need to design the DSL itself.

Designing a DSL can be challenging, but I have found the best approach is to use a similar approach to what we used to design our automation framework.

It is best to work backwards from the syntax that would make your BAT the easiest to write and the most clear.  Once you have done that you can worry later about implementing the DSL to support the syntax you have come up with.

You should focus on creating a limited vocabulary for your DSL so that it will be easy for someone to learn and know what their options are.

The more restrictions we put into a DSL the easier that DSL becomes for doing a specific purpose.

We want to give the user of a DSL no more options than required for what they want to do and no less.

This isn’t always easy to do.  It often requires some careful thought and refinement.

Your DSL should also use terminology and phrases that are familiar to the business domain of your application.  By doing this you will make it so someone understanding your application will understand BATs written in that DSL.

I also always try to make a BAT DSL read as much as plain English as possible.

Once you have designed the DSL, you will need a way to implement it. 

I’ve written about using ANTLR to create an external DSL.  Also I have written a review on a book that I recommend for learning ANTLR.

Microsoft also has a DSL SDK you can use in VS2010, although I haven’t used it myself.

The basic idea behind implementing a DSL is that you have to take some sort of grammar you design for your DSL, parse it into a syntax tree, then use that tree to generate code that will actually perform the actions.

It is by no means a trivial task, but using a tool like ANTLR, makes it much simpler than trying to completely generate a parser and execution engine yourself.

Why even bother?

Creating a DSL for your BATs is completely optional, but there are some significant benefits to doing so.

The biggest one I have found is that you are much more likely to get business people to write BATs if you have a simple DSL they can use to create them.  Once you get everyone using BATs to define behavior and requirements, communication becomes much more simple.

Another great thing about creating a DSL is that you can create a runner that can interpret the DSL BAT scripts which can be run on any machine.  Typically, to run BATs created in Java or C#, you would need a C# or Java compiler and a test runner, etc.  By creating a DSL and a stand alone executor, you can very easily allow anyone to run BATs with little to no setup.

Writing your BATs in a DSL will also make writing the BATs and understanding them much easier.  A good DSL can get rid of all the little language and API nuances that don’t really add information to the specific domain you are working in.  A good DSL strips information down to the minimum amount needed and is able to do so, because it assumes the context is your application’s business domain.

You will also find that you are better able to separate framework from tests when you have a DSL in place.  One issue that many teams face when creating a BAT framework is that some people come along and write lower level code in the higher level BATs.  This creates a problem because it is hard to force someone to use the higher level API when the framework and the BAT code are the same language.

Wrapping it up

So that completes my series on Getting up to BAT.

Hopefully you have learned enough to be able to implement your own BAT framework and get BATs running for your application.

I tried to cover most of the important steps here without getting too deep into the details.

If you have any questions or are interested in getting more in depth training or consulting on getting BATs setup for your environment, send me an email or leave a comment.

It can be a little difficult to get started and get a BAT framework up and running, but the rewards are well worth the effort!

As always, you can subscribe to this RSS feed to follow my posts on Making the Complex Simple.  Feel free to check out where I post about the topic of writing elegant code about once a week.  Also, you can follow me on twitter here.

Getting up to BAT: Scaling Out

If you haven’t been following, it has been a while since my last post on this topic.  I had a bit of distraction the last few weeks, but I am back and ready to go!

When we last left off we had just gotten our BATs as part of the acceptance criteria for any new backlogs that are worked on.  This puts us at a point where we could really say that we have successfully implemented BAT testing.

You don’t want to get too comfortable just yet though, because the next hurdle you will most likely face will be the problem of not having enough time to execute all your tests.

You want to think about this ahead of time

Nothing worse than getting everything going and then not being able to execute the entire test suite, because you didn’t plan ahead.

You don’t want to get to the point where confidence in your BAT suite it lost because you are not able to get all the tests executed in a reasonable amount of time.

The more frequently your tests are run the more value they have.

By reducing cycle time from the time a breaking change is made in the system and the time it is discovered, you greatly reduce the risk it imposes on your software and you decrease the scope of the code changes in which it could have occurred.

To put it simply, the faster you can find out you broke something, the more likely you can fix it before it does damage, and the more likely you will be to know what thing you did caused the breakage.

clarkson face thumb Getting up to BAT: Scaling Out

How can we reduce cycle time?

There are a few different strategies we can employ and we can mix and match some of these strategies.

Straight forward parallelization

The best and most effective thing you can do is to take a given test run, split up the execution of those tests on multiple machines and execute them in parallel.

This approach is going to give you the best bang for your buck.  You should really try to get some amount of parallelization going before attempting any other solution, since it is going to make a huge impact on the total execution time for your tests without making any sacrifices.

There are even many ways you can mix and match to do parallelization:

  • Use multiple physical machines
  • Run many virtual machines on a single host
  • Put an executor program on every team member’s machines that will execute tests when that machine is idle or put into test execution mode (perhaps at night)
  • Use a cloud based computing platform to execute your tests
  • Run multiple browser instances on a single machine

Data preloading

With this approach, you would preload some of the test data that your tests might generate by manually clicking through screens.

This is fairly hard to explain, so let me give you an example:

Suppose you had a set of tests that all involved creating customers, but each customer you create in the system takes about 3 minutes to create by clicking through the screens to get them into the system.

We don’t need to have 500 tests all executing the same exact logic in the system 500 times for 3 minutes just to generate all the customer data that will be used in the tests.

Instead, we can leave a few tests that are exercising the customer creation functionality, and we can execute a SQL script to push all the other customer data into the test database for the other tests to use.

Using this technique we might be able to reduce our total execution time by 3 minutes * each test, or about 25 hours for 500 tests.

This can be a huge savings in time, and it doesn’t come at that high of a cost.  The sanctity of our tests is slightly compromised, but we are taking a calculated risk here knowing that we already have covered the area of execution which we are preloading data for.

Consider this technique when you notice certain things in the system taking a very long time to do.

Test runs by area

With this technique, we can reduce the total execution time in a given period by splitting up test areas and running them either at different times or in response to changes in certain areas.

You have to be very careful with this approach, because if you don’t do it correctly, you can start to erode some of the safety your BAT tests are providing you.

I would only do something like this as a last resort, because it is so easy to virtualize today, and hardware is so cheap.

I’d much rather run too many tests than too few.

Test randomization

With test randomization, we are going to take our total desired execution time, and divide it by the average time for running a test.  We then can use that number of tests to run to randomize the execution of our tests each time we run them and only run the number or tests that will fit in the desired execution time.

This choice is also a compromise that I typically don’t like to take.

It can be very useful though, combined with other techniques when you still don’t have enough time to execute your entire test suite.

The basic idea here is that you are going to randomly run tests each time to fill up the time you have to run tests.

Test reduction

This one seems fairly obvious, but can be very helpful.

Often I will see teams starting out with automation, trying to write way too many BAT tests for a given feature.  Sure, with automated tests it is possible for run a test for every single possible combination of values in your 5 drop downs, but will it really benefit you?

In many cases you have to think about what you are trying to protect against with your BATs.  Sometimes running every combination of data selection choices is going to be important, but other times you are only going to need to write tests to test a few of the happy path scenarios.

It is important to find a balance between test coverage and test volume and not just for execution time.  There is a logistical overhead to having a large volume of mostly redundant tests.

So even though this technique might seem dangerous and counter-productive, I will almost always employ it to some degree.

Common pitfalls

Here are some of the things you might want to watch out for as you are scaling out and streamlining your execution of BATs:

  • Parallelization issues.  If you are using shared state, you can run into big trouble when your tests are executing in parallel.  There are many manifestations of this.  You could have issues at the database level or at the local machine memory level.  The best way to avoid this kind of problem is to use separate virtual machines for each test execution, and not reuse data setup between test cases.
  • Ineffective error reporting.  If you run a huge volume of tests in parallel, you better have a good way to sort through the results.  It is much harder to figure out why things failed when they are run across multiple machines.
  • Test order dependencies.  Make sure tests don’t rely on being run in a certain order or you will have lots of pain when you disrupt that order.
  • Environment setup.  Make sure all your test execution environments are exactly the same unless you are specifically testing different environments for execution.  You don’t want tests failing on one machine but passing on another.

As always, you can subscribe to this RSS feed to follow my posts on Making the Complex Simple.  Feel free to check out where I post about the topic of writing elegant code about once a week.  Also, you can follow me on twitter here.

Getting up to BAT: Adding BATs to Your Acceptance Criteria

So now that we have built our automation framework, and got our smoke tests running as part of the build, we now need to make sure new BATs are being created for new backlogs.

This is definitely one of the more challenging tasks you will face, and perhaps the most critical.

All of your efforts will be in vain if you can’t get to the point of creating new BATs for new backlogs that are finished, because if you don’t you’ll still be doing manual testing, and you’ll be playing catch-up all the time.

Start off small

The goal here is to eventually get to the point where every single backlog requires BATs to be written and passing in order for that backlog to be called “done.”

If you try to drop this lofty goal onto a team in one iteration, you are likely to get beaten and slapped with much vigor.

Instead, start with a simple goal of requiring that at least one backlog have BATs for all of the functionality of that backlog.  Start with this small goal, but enforce it with rigor.

By starting out small like this, you will get the team accustomed to the idea of building automated tests at the same time as building the functionality and you will give them a chance to have a small success without sacrificing much velocity.

The goal at this point should be to get people excited about writing BATs along with the backlog.

It’s actually quite fun

I have found that developers tend to really enjoy writing automated tests.  Make sure you let everyone who wants to share in the fun, not just QA and not just developers.

It is really very exciting to see the “magic” of an automated test clicking through buttons on your web page and doing all kinds of things that seemed so tedious before.

I have found that this is so much fun, that you might have the problem of developers not wanting to work on the real backlog, but rather to write the automated tests for it.

This is good, you really want to foster this attitude, it will greatly help when you finally…

Drop the bomb!

What bomb?  The bomb that says on it:

All backlogs will require BATs as acceptance criteria.  It’s not done unless it has automated tests that prove its functionality.

You should be getting to this point within 3 to 4 iterations on average.  Don’t go too fast to get here, but don’t go too slowly either.

I know this seems like an impossible task, but it really isn’t.  I’ve been here before and had much success with it on several different teams.  The key is making sure everyone is comfortable with writing automated tests and understands the value.

Whatever you do though, once you draw this line in the sand, Do not back down! I mean it.

You might start getting death threats.  People might directly laugh in your face all the while spitting skittle colored loogies in your coffee, daring you to “taste the rainbow,” but you have to hold your ground.

skittlesny6 thumb Getting up to BAT: Adding BATs to Your Acceptance Criteria

It will be worth it in the long run, because you will have built an awesome safety net for proving the functionality of your software over time.

It’s not all rainbows and butterflies

You are going to sometimes run into scenarios where automation will be too high of a price for a particular backlog.

This will especially happen in cases where you have some kind of process that spans multiple domains or software systems.

When you run into those situations, give it a solid effort to at least automate as much as possible, but don’t die on the hill and kill the whole automation project.  We have to be pragmatic or risk having our credibility called into question.

Don’t expect everyone to embrace writing BATs overnight, or to even be good at it.  When you first introduce this to the team, you are going to have a learning curve and your automation framework is still going to have some pretty big holes in it.

But, that is OK.  Over time you will build out that framework and produce some fierce battled scarred veterans able to automate web pages while scowling at you from afar, just don’t lose hope and keep beating that drum!

As always, you can subscribe to this RSS feed to follow my posts on Making the Complex Simple. Feel free to check out where I post about the topic of writing elegant code about once a week. Also, you can follow me on twitter here.

Getting up to BAT: Adding Smoke Tests to Your Build

Once you’ve built some smoke tests with your shiny new automation framework, you are going to want to get those smoke tests up and running as soon as possible…

But!  You might want to consider holding off for a second and reading this post!

It is worth taking a bit of time and thinking a bit about the strategy and psychology of adding the smoke tests to your regular build process.

There are several things we are going to want to discuss and think about before adding automated tests to a build.

build a site2 thumb Getting up to BAT: Adding Smoke Tests to Your Build

Why are we adding the smoke tests to the build?

I think it is always important that before we do something we ask ourselves why we are doing it.  Doing things, because someone said it is good, or because it seems right is not a very good reason.

The primary benefit we are going to get by adding smoke tests to the build now instead of waiting for a larger portion of tests to be written is that we are going to be able to immediately tell if a basic functionality of the system is broken instead of finding out after we have deployed the build to different environments and used it.

We want to be able to get this value as soon as possible.  We don’t want to have to wait until we have a huge amount of tests, because even a few tests will give us some value in this area.

The other reason we are adding the tests to the build is so that we can notify developers when a test fails, so that they know to fix whatever was broken.  By having the smoke tests run as part of the build, we are able to reduce the amount of time before a break is introduced and the break is discovered.

This concept of reducing the delta between introduction and discovery is very important, because it makes it much easier to identify what caused a break.

Perhaps a distant 3rd reason to add the smoke tests at this stage, is to prove out our framework and technologies.  Better to start small with a small number of tests and get everything working.

With those ideas in mind, we can move onto some of the more important considerations.

Don’t ever add failing tests to the build

When you add your smoke test suite to the automated build, all of the tests should pass!

I will say it again, because it is really important and ignoring this advice may doom your entire BAT project.

Do not ever add failing tests to the build!

It seems like an innocent enough thing, but it causes great harm, because it is very much like the boy who cried wolf.  Consider what happens when we introduce our new BAT system and we add our smoke tests to the build, but we have 3 failing tests.

First thing that will happen is that our tests will run during the next build and 3 will fail.  This should cause a build failure notification of some sort to go to the team, at which time you will be in the uncomfortable position of saying something to the effect of “ignore that fail, those are real defects, but they are known issues.”

Next, let’s assume a developer checks in some bad code and it causes a 4th test to fail.

Do you think most people will notice this 4th failing test?

Even if they do, do you think they will just assume it is another known issue?

Worse yet, they may think your BATs are meaningless and just fail randomly.

Do you see where I am going with this?  We want to start off with the correct expectations and we want test failures to be meaningful.  To ALWAYS be meaningful.  The second you have test failures that are not meaningful, you lose credibility for any future meaningful failures.

This is such a critical psychological battle with build systems and automated tests that I have often implemented a policy where no one is allowed to work on anything else but fixing the build or test failure when a build fails.  When you have 20 developers either sitting idle or working on one issue for 2 hours, it sends a pretty clear message that we take build failures seriously.

Make sure your tests never undermine this message.

What do I do with failing tests then?

If they are real defects, put them in your defect tracking system or create backlogs for them and make sure the tests are disabled until those defects are fixed.

Your automated tests should be acting as regression tests for things that are already working.  It is absurd to think that it is even close to cost effective to write automated tests to actually functionally test new features or code!

It is the process of writing the automated test that should uncover defects.  Running of automated tests should uncover new defects where functionality has regressed from working to not working.

Other considerations

Here are a list of some other considerations you will want to think about before embarking on this journey.

  • How will you actually run the tests? (Will you use the build script to kick it off, some 3rd party plugin, etc)
  • Where will you run the tests? (Will you use the build server itself, or another machine.  You probably don’t need to worry about scaling out to multiple machines right now, but you should at least keep it in the back of your mind.)
  • How long will it take to run the smoke tests? (Fast is better, because feedback cycle is reduced.  If you have a really long test suite, you might want to pare down the smoke tests and run some with every build, and all of them every night for now.)
  • Will developers be able to run the tests locally? (Make sure you consider how to do this before putting the tests into the build.  Nothing more frustrating than being measured against a bar you can’t yourself see.)
  • How will you notify and report results from the test failures? (We don’t care about passes, only failures, but you need a good way to report the result that prompts action.)
  • What is the policy for test failures? (Make sure there is some agreement on what kind of action will be taken when tests fail.  A great place to start, if you can get management to buy in on it, is all work stops until all tests pass.  Extreme, but extremely effective.)
  • How will you prevent false fails and identify them? (Don’t get stuck with egg on your face.  If your framework fails and the problem is not a real defect, you need to be able to quickly identify it and fix the framework problem.  There are a few strategies for doing this, but that is for another post.)
  • How do new tests get added and commissioned? (You are going to be constantly growing your test suite, so you will need a good method for adding tests to the smoke test suite.)

Remember, you only get one chance at a first impression!

As always, you can subscribe to this RSS feed to follow my posts on Making the Complex Simple.  Feel free to check out where I post about the topic of writing elegant code about once a week.  Also, you can follow me on twitter here.

Getting up to BAT: Creating Your First Smoke Tests

If you have been following my posts so far on Becoming Bat Man, you should already have hired an automation lead, figured out what browser driver you are going to use and come up with a design plan for your automation framework.

smoking computer thumb Getting up to BAT: Creating Your First Smoke Tests

Now you are probably faced with the difficult decision of…

Where to begin with your tests

I don’t suggest writing your automation framework first and then writing tests to use it.  If you go this route you’ll have two problems

  1. It will be a long time and large investment before you see any fruits from your automation effort.
  2. Your framework will likely be designed incorrectly or not test-writer friendly.  (Which is basically the same as wrong.)

What you really want to do is to write some basic tests that let the development of those tests drive the development of your framework.

The best tests to drive your framework are some basic “smoke tests.”  If you are not familiar with the term “smoke test,” it basically means some tests that make sure the application you are testing is at least somewhat working and not completely broken. 

There are a few different opinions of what exactly constitutes a smoke test, but in general, if you have any tests that you regularly run on a new build to make sure it is at least OK, those would be the tests we are interested in here.

You are going to want to start by taking the easiest and most simple tests you have and writing them in a concise syntax that you use to drive the development of your framework.

It is very important at this stage to not worry about what you think might be possible or how it would be implemented in the framework, but to instead focus and making the tests as terse and explanative as possible.

The basic idea is that you will write the test out in the syntax you would like to be able to use, then implement the framework features required to make that test pass.  This whole process is very similar to doing Test Driven Development.

An example case

The easiest way to demonstrate this is with an example.

Let’s say you have a smoke test like the following:

Create a new account, pick an on-sale item, add it to your cart and check out using PayPal.  Verify that we hit the order success page.

We might write a test for this that looks something like:


What I just wrote above is astonishingly simple.  You are likely saying to yourself something along the lines of “What if… blah.”

Forget it!  We’ll handle “what if” later.  Right now we are going to write the tests as terse as possible and not handle any other conditions.

What I wrote above is the style that I like to write the tests in, which is highly subject to opinion, but the content of what I wrote is not as subjective.

There are a few important points to note:

  • I used methods like CreateNewUser, LoginWithLasteCreatedUser and CheckoutUsingPayPal that do a huge amount of work and have no parameters.  Why?  Because for the purpose of this test, (and you’ll find for most of your tests), we don’t care about what the first name or last name or address of the user is, we don’t care how we get them logged in, and we only care that they checkout using PayPal, not what account is used, etc.
  • I didn’t declare any variables.  Tests should be simple.  Let the framework remember the state.
  • I over-simplified.  This is fine, you are working inside of a very specific domain.  Let your test framework methods do one large thing.  Don’t worry about oversimplifying for now.
  • I asserted something at the end.  All tests need to assert something.  If you can’t figure out what to assert, work on redesigning the test before automating it.
  • I didn’t pass parameters.  Avoid this also.  This goes along with the first point, but is so important that I feel it is OK to state it twice.  If you don’t care about variation, eliminate it.  You will constantly need to find places where a manual tester would have to enter specific information, but an automated test won’t care what that information is.
  • All my methods are static.  While this would be bad in normal production code, for a test API it is the easiest for test writers to use.  No variable declarations, no “newing” up objects, no constructors, etc.
  • It’s written in “business language.” As close to the language an end user or business person would use to describe the test.

So, this test is definitely doing a huge amount of stuff in a few lines of code, and because of this your inclination would be to say that the API level is too high. Don’t!

Instead, make the API level as high as you possibly can, until further tests you try to implement prove to you that it must be broken down further.

By working this way, you ensure a reduction in the total lines code and complexity of your tests by as much as possible.

You would NEVER work like this with normal code, but for automated tests this is essential.  The main reason is because you will be calling the same methods perhaps 1000s of times, where in normal code you usually only reuse a method in a few places.

Next steps

For my example above, the next step would be going through and implementing all the framework methods required in order to make that test runnable.  You will end up utilizing the general design you came up with for your automation framework to do this.

Then you will simply move on to the next test and repeat.  Each time you should have less and less effort required, as more of the basic functionality of your automation framework is implemented.

Automating your smoke tests should ensure that you get a fairly even distribution of the framework code written as the smoke tests should cover many different areas of your software.

But, don’t start adding these smoke tests to your build yet.  We’ll cover that in the next step.  Adding them to the build prematurely can cause the confidence in these tests to be forever diminished!  We don’t want to do that!

As always, you can subscribe to this RSS feed to follow my posts on Making the Complex Simple.  Feel free to check out where I post about the topic of writing elegant code about once a week.  Also, you can follow me on twitter here.

Guest Post: The Cross-Browser Compatibility Myth

This post is a special guest post from my wife, Heather Sonmez, who is an expert on the subject of blackbox automated tests and designing frameworks for them.  She has designed successful automation frameworks for several companies.  She deals with software automation and framework design issues on a daily basis, so I thought she might be able to offer some good insight on the topic.

John asked me to write a post based on my experience with writing automated tests.  I’ve been working as a QA Automation Engineer for the past three and a half years. During this time I have had the opportunity to work with Ruby/Watir, C#/Watin, and Java/Selenium RC. In thinking on a topic, I decided that there’s no time like the present to talk about what I’m presently facing, and would like to share some insight straight from the trench I currently find myself in today:

Automation tools that claim to support multiple browsers are LYING

Currently I am in the process of getting 400 automated tests that run in Firefox to run in Internet Explorer and Chrome. While the tests are using Selenium RC, a tool that claims to work on all three browser platforms, in actuality, many tests are failing and not for good reasons (bugs).

Here’s the rundown on what these overly-ambitious tools aren’t telling you:

  • Xpath in Browser A isn’t always supported in Browser B
  • The Dom in Browser A may vary from the Dom in Browser B
  • Browser B may have built-in security settings that interfere with testing
  • Bugs in the automation tool itself may cause issues in specific browsers
The XPath Issue

When it comes to XPath, I fully expected that any given expression was either valid or invalid. But it didn’t take more than one test run in Internet Explorer to suddenly find myself with tests that claimed certain elements didn’t exist. While Firefox accepted and successfully returned my element using the provided expression, IE claimed the expression itself was invalid. In this particular instance I discovered an issue with Selenium stripping off the end of my xpath statement, thus making it invalid.

But ultimately, whether it’s the browser or the tool doesn’t matter because the same expression is not working unequivocably in both browsers.

The Dom Issue

The next problem I immediately came across was the problem of Dom differences between browsers. In this particular case I had an element that occurred one time in Firefox but twice in IE. My test, expecting one instance of the element, failed. While I can’t tell you why this discrepancy exists (currently under investigation by some client-side engineers), I have a fair degree of confidence in saying this won’t be the last time I come across this issue.

In the quest to support many different browsers, it only follows that one is going to come across conditional browser logic, especially if a company is striving to support multiple browser versions (IE 6, 7, 8, 9). If a developer has to occasionally write browser-specific condition statements, it makes sense that we’d have to do the same to test specific browsers.

The Security Issue

If it seems that I’m picking on IE, it’s only because I can’t get very far in running our tests in Chrome. Chrome has special security settings that cause a Selenium exception if your test changes domains mid-run. This can easily be solved by passing a parameter to Chrome which will disable this security feature when the test runs, the trick is figuring out when/where/how to pass that parameter.

The Bug in the Tool Issue

Finally, the last obstacle I’ve come across in multi-browser testing: the tool itself. Any tool, regardless of how fabulous it is, is going to have bugs. Selenium RC has an issue where its select method (responsible for selecting a value in a dropdown menu) doesn’t trigger affiliated event handlers- in Internet Explorer. To work around this, one can manually call the event handler using Selenium’s runscript() but you will lose some degree of testing confidence in doing so (how do you know the event is really firing in response to a particular user action if you’re manually forcing it in automation?)

Another issue exists around multiple browser windows; if you need more than one open for your test, Selenium is unable to select the second window you open- another outstanding IE issue that has not been resolved.


In all fairness, some of these issues have more to do with the browsers themselves than with the automation tool. However, that makes it even more important to realize that when an automation tool says it supports multiple browsers, it’s not without some creativity, troubleshooting, legwork, and maintenance.

Knowing these kinds of issues exist is important in order to decide if and how much browser compatibility testing you want to automate. It’s definitely something to realize before deciding on a tool just because the tool claims to support other browsers.

Just as there is no silver bullet for writing a web application that works in every browser imaginable, there is no silver bullet for automation either.

Getting Up to BAT: Designing an Automation Framework

Now that you’ve gotten an automation lead and decided on the browser automation tool you are going to use, the next step is to design an actual automation framework.

This is one of the most critical components of the overall success of your automation strategy, so you will want to make sure you invest properly in this area.

I have seen a large number of automation projects go forth, but each time the critical component determining their success or failure was having a good automation framework.

Solid design and excellent technical resources are absolutely critical to success in this area!

What is an automation framework?

An automation framework is essentially an API that all of your BATs (Blackbox Automated Tests) are written against.

This API can be as simple as the API that is exposed by your browser automation tool (WatiN, Selenium, etc.), but I would highly recommend building your own layer on top of the browser automation tool’s API that will act as a DSL (Domain Specific Language) for automating your application.

Let me break this down a bit further by using an example.

Think about the task of making coffee.

There are several “API” levels we can interact with.

  1. We can go with a very low level API where we take whole coffee beans and grind them down.  Then we take some water, get it hot.  We take our filter put the ground beans in it, put it over a cup and pour the water into the filter.
  2. We can go with a higher level API where we use a traditional coffee maker.  In this case we load the coffee maker with a filter, ground coffee beans and water and push a “brew” button.  We could also set it to start at a certain time.
  3. We can go with a very high level API where we use a Keurig machine or similar device.  In this case we only make sure the machine has water in it, and we just insert a little pod and press brew.  We can make different kinds of coffee, cider or hot cocoa just by changing what pod we use.

unnamed thumb Getting Up to BAT: Designing an Automation Framework

Using the API provided by the browser driver is like making the coffee by hand.  It’s going to take a large amount of effort each time you do it.

We want our automation framework to be more like the Keurig machine.  We want to be able to compartmentalize our tests in little pods that are small and don’t require many hooks into the automation framework.

To rehash this one more time, basically our automation framework will be a framework we build on top of the browser driver framework, which is designed to make it easy to write tests which automate our application.

What makes a good automation framework design?

The true measure of an automation framework is the size of the tests that are written against them. The less lines of code in each test, the better the automation framework captures the domain of the application being automated.

In my earlier post about an example of an automation framework, I talked a bit about the strategy I used in a real implementation of an automation framework.

Here is a diagram showing the different layers.  You can see the framework layer is in green here.

automated testing framework 2 thumb Getting Up to BAT: Designing an Automation Framework

You can also see in this diagram that on the right hand side, I have screens, workflows and navigation.

This is one of the common design patterns you can use to build your automation framework, since it closely models that of most web applications.

A good way to design your framework is to use a pattern like this and create classes for each page in your application, create classes which might represent workflows in your application, and create some classes for navigating around the application.

The basic idea you are shooting for with your automation framework is to make it so the tests do not have to know anything about the browser.  Only your automation framework itself should be dealing with the DOM and HTML and CSS.  Your tests should be dealing with concepts which any user would be familiar with.  That is why I commonly use pages and workflows.

Let me give you an example.

Let’s say we ignored the advice of creating an automation framework and decided to program our tests directly against the browser driver layer.  In this case, let’s say that we want to automate a login page.  Our test might look something like this pseudo-code.

Element userName = browser.getElementById("login_userName");

Element userPassword = browser.getElementById("login_userPassword");

Element loginButton = browser.


This is not very readable.  It is not something an average tester will be able to pick up and start writing, and it is extremely fragile.  If you have 500 tests like this and the id changes for one of the elements on the page, many tests will break.

Now, contrast it to this pseudo-code:



Where did all the code go?  You don’t need it!  Actually it went into the framework, where it can be reused.  Now your test code is simple, can be understood by just about anyone, and won’t break with changes to the UI.  (You will just have to change the framework method instead of 500 tests.)

If you want some more detailed examples take a look at my Boise Code Camp slides on the subject here.

Let me offer up some general guidelines for creating a good design for an automation framework.

  1. NEVER require the tests to declare variables.
  2. NEVER require the tests to use the new keyword or create new objects.
  3. NEVER require the tests to manage state on their own.
  4. ALWAYS reduce the number of parameters for API calls when possible.
  5. ALWAYS use default values instead of requiring a parameter when possible.
  6. PREFER to make the API easier to use over making the internals of the API less complex.
  7. PREFER using enumerations and constants to requiring the test to pass in primitive types.  (Make the input bounded when possible.)
  8. NEVER expose the browser or DOM to the tests or let them manipulate it directly.

You can see that these guidelines will almost force your API to be entirely static methods!  Don’t freak out.  It is perfectly fine.  There is nothing wrong with using static methods in this case.

Stop… take a breath, think about why you think static methods are bad, because I probably agree with you.

But, in this case we are HIGHLY valuing making the tests simple and easy to write over anything else, and because of this emphasis, static methods are going to be the way to go.

Some advice on getting started

Start with the tests in plain human English.

Take those tests and make them into code preserving as much English as possible.  Then work down from there implementing the methods needed to make those tests work.  Only add things as you need them.  Do not overdesign!

In my example above, we might have started with something like this:

1. Goto the login page.

2. Login with the default user.

The actual code looks just like that.  It should always be a 1 to 1 mapping.

This is where having someone that has some experience doing this is going to come in handy.

Just keep in mind the whole time your 2 big goals:

  1. Easy to write tests
  2. Tests are concise
As always, you can subscribe to this RSS feed to follow my posts on Making the Complex Simple.  Feel free to check out where I post about the topic of writing elegant code about once a week.  Also, you can follow me on twitter here.

Getting Up to BAT: Picking a Browser Automation Tool

Now that you’ve gotten an “Automation Lead” for your BAT (Black-box Automated Testing,) it’s time to make a very important decision.

It is time to pick the browser automation tool you are going to use for driving your automation efforts.

Before we can build an automation framework, we need a basic set of tools to build it on.  We want our automation framework to be tailored to our application under test specifically.  Our automation framework will act like a Domain Specific Language (DSL) for testing our application, but we need to build that automation framework on top of something.

What is a browser automation tool?

It is essentially what we will use to drive the web browser.

older driver thumb Getting Up to BAT: Picking a Browser Automation Tool

We need to pick a tool or framework that will easily allow us to interact with the web browser so we don’t have to write that code ourselves.

We will need to be able to click buttons, enter text, select drop downs, and check values on web pages.

Some of the things we will want to consider when choosing a browser automation tool include:

  • Language we can use with the driver
  • Browsers it supports
  • How we interact with the browser (do we use XPath, JQuery, or some other mechanism)
  • If we can access and manipulate everything we need to be able to in the browser
  • Speed of execution
  • Ability to execute in parallel
  • Support and future development

Let’s take a quick look at each of these considerations.

What language can we use?

This is an important consideration because you will want to be able to utilize your other programming resources to help with building an automation framework and to eventually create automation tests.

If you pick a browser driver that supports a language your regular programmers don’t like, or don’t know, you may not end up with their support, and you are definitely going to want their support.

What browsers does it support?

This is a tricky topic to discuss.  It might seem that a browser automation tool would need to support all of the browsers your application supports, but that is not really true.

You really have to think about what your goal is with your BATs.  I could certainly devote a whole blog post to this topic, but I will try and summarize my main point here.

BATs should be designed to test functionality of the application, not to test compatibility.  (At least most of your BATs should be, although you might have some that are specifically for compatibility.)  With that goal in mind, a browser driver supporting each and every browser your application supports is not really necessary.

You do need to consider that the browser automation tool should support at least one of your supported browsers, preferably the main one.

How do we interact with the browser?

There are several different APIs which are used to communicate with the web browser that are employed by various different browser drivers.

Some tools have APIs that are very low level and allow you to very easily manipulate objects in the web browser while others are at a much higher level and rely more on your understanding of the brower’s DOM.

For example, one API might expose elements on a web page by their actual element name and let you interact with them directly.



In the first example, the API is aware of element types like buttons.  In the second example, the API is more generic and will send a “click” to an element with the ID value you specify.

You’ll have to decide how important the ease of use is to your project at this level.  There are some tradeoffs to consider here.

The lower level the API is, the easier it will be to use, but the more dependent you will be on that API and the more language specific that API will be.

The higher level the API is, the harder it will be to use, but it will free you from depending so much on the API, because you will be interacting more directly with the browser’s DOM.

I prefer lower level APIs because I find that it is much easier to write the framework code on top of these.

You will also want to consider here the skill levels of the programmers who will be creating the framework.  If your API is lower level, native language skills are more important.  If it is higher level, DOM and HTML, and perhaps Javascript skills, are more important.

Can we access what we need?

If your application makes use of Javascript pop-up dialogs and your browser automation tool doesn’t give you a way to interact with them, you don’t want to find this out when you have already invested significantly into your automation framework.

Different automation frameworks support different features of browser; not all of them will allow you to interact with every single browser feature that exists.  As new features are introduced, you could also get stuck with a tool that is no longer being maintained and doesn’t add support for the new features.

You will want to consider things like how much your application uses AJAX or JQuery.  You will want to select a browser automation tool which would make it easy to interact with AJAX calls if your application relies on them heavily.

Speed of execution

Not all browser automation tools are the same in execution speed either.

This may not be important to you, depending on how you address the issue of concurrency, but you should at least consider that if you have a large number of automated tests, a small difference in execution speed can result in a large difference in total time to run the tests.

The faster the automated tests can be run, the more valuable they are.  I won’t get into the details here but there are many reason why this is true.

Also influencing the speed in which you can run your tests will be if you automation tool choice requires you to put pauses into the test to wait for the browser instead of responding to events that occur in the browser.

Ability to execute in parallel

This consideration will depend greatly on the volume of and speed in which your tests can execute.

At some time (sooner than you think) you will likely get to the point where you can no longer run all of your automated tests in 24 hours.  Perhaps even before this point, you will want to consider running your BATs in parallel to reduce the total time to execute the tests.

Some automation tools have built-in support and others will require you to build your own way to do this.  It is good to at least have an idea of how you are going to achieve parallel execution when considering what browser automation tool to use.

Support and future development

We are in a pretty rapid pace of browser development.  Many things in the web browser world are changing much more rapidly than ever before.  Tools to automate the browser must also change or you could end up in a very bad place.

If your application is going to take advantage of the latest browsers and browser features, you should make sure the automation tool you choose is in active development.

There is nothing worse than investing into an open-source project and then having that project die, resulting in you having to rip it out to replace it with another library.

You can always design your automation framework to be abstracted away as much as possible from the underlying browser driver, but that will be extra work, so consider this point carefully.

Name names sir!

Nope, I’m not going to do it.

I’m not going to tell you to use Watin or Watij or Selenium or WebAii or any other browser driver.  I don’t want to put the focus on the tool, since the real focus will be building the automation framework on top of the tool you use to drive the browser.

I would suggest that you try writing a few simple tests in each of the major browser drivers so that you can get a good feel of what the API is like and how it will work with your application.

I will say that I have used most of the major choices out there and there really is no clear winner in my mind.  It really is going to depend on what your environment is like and what kind of application you are testing.

I would also recommend picking a browser driver and sticking to it.  I’ve tried in the past to abstract away the browser driver from the automation framework and while it is possible, it can become quite messy and add quite a bit of overhead to your project.

My only other hints would be to not put too much emphasis on supporting multiple browsers or on using recording tools.  Neither of these things will benefit you much in the long run because you will find that you will not want to try and run all of your BATs on each browser. Recording tools will not be nearly as effective as writing your own custom framework (which I will talk about in my next post.)

As always, you can subscribe to this RSS feed to follow my posts on Making the Complex Simple.  Feel free to check out where I post about the topic of writing elegant code about once a week.  Also, you can follow me on twitter here.

Getting Up to BAT: Hiring an Automation Lead

This is my first in a series of posts about getting to fully automated blackbox testing, or BAT, as I like to call it.

In my last post in the Back to Basics series, I mentioned that I would begin this series with the intent of showing you step by step how to go from no automation to a quality BAT infrastructure.

By the end of this series you should have the information you need to actually get automated testing going in your organization instead of just talking about it.

First stop on this train is hiring an automation lead.

Don’t doom yourself before you start

If you think you are going to create BATs, but you are not going to have at least 1 dedicated resource to doing the job of getting this all going, you are probably bound for failure.

doom4 thumb Getting Up to BAT: Hiring an Automation Lead

Listen up, because this is the single biggest mistake that most organizations make when trying to get BATs up.

This mistake is thinking that you can get your QA team to get this going and you can part time some developers onto the project, but not having a dedicated resource.

Why is this absolutely absurd and you should expect it to fail?  Because, in order to get an infrastructure to support your BATs, you are going to need to basically design and create a testing framework that is tailored to the business domain of your application.

Blah, what?  Ok, in simple terms.  Someone is going to need to design and build an automation framework.

If you think you are just going to buy some tool and use it to record test scripts, you are wrong.  That approach simply is not going to work.  I won’t get into why here, but I have written on it before in the past, and if you are interested, let me know and I’ll go much more in depth.

Think about this, do you want your QA team, as good as they may be at QA work, designing and building an automation framework?  Give them a few C# or Java classes and have their first real coding work be to design a complex system?

How about a developer who has “real” work and can only devote part time to designing what may end up becoming one of the most important parts of your infrastructure?

It is fine to pull a developer off of the main product and put them in this role until it is up and running, but make no mistake, you need a dedicated resource.

You are going to want to a well designed maintainable automation framework, because the ease of use of that framework will determine how easy or not it is to write automated tests that use that framework.

I recommend getting someone who has done it before

Why?  Because creating an automation framework is much different than developing your software and it is much different than writing tests.  It really requires a special skill set that is not so easily acquired without the experience of having done this before.

I also think it is fine to use a developer resource, but if you do, you want to try to get a more senior developer to take this role.

Here is a list of some of the skills that I think are important which may also make it much more clear why I am recommending a specialist in this case:

Development of APIs (In the end the automation framework will be an API that is used to write automated tests.  The better the API is designed, the easier it will be to write tests against it.)

Design of reusable base components (Much of the framework consists of repeated concepts with different variations for different screens in your application.  As a result, the core of the framework will be a basic design that allows for variation without too much repetition… reuse)

Ability to quickly acquire domain knowledge of the software (Someone writing an automation framework needs to really understand how the system under test works, so that they can model the framework after it.)

Pragmatism (Many “accepted” good software designs are actually bad automation framework designs.  Static methods can make an automation framework’s API much more simple to use, especially for non-programmers, but would not be a good design choice in your main product.  A person who designs an automation framework should be pragmatic enough to “break the rules” in order to make the framework easiest to use and maintain.)

Experience or understanding of QA (It is important to understand the users of the framework, the QA team.  Understanding what kind of tests they will likely write or how they will think will go a long way in coming up with a good framework design.)

Communication skills (Good BATs read like English.  It is critical to make sure that the person designing the framework facilitates this to make the BATs as useful and understandable as possible.)

HTML / XPath (This specific skill set is very important for web based products.  Creating an automation framework built on top of one of the web browser drivers will require intimate knowledge of HTML and probably XPath to be able to access attributes and elements on pages in your application.)

I’ve written a few automation frameworks and contributed to the design of several others, and I can tell you that my version 6.0 of an automation framework was much much better than version 1.0.

There are a huge number of patterns that emerge that are unique to this problem domain.  So, I do believe having experience in this area specially is a huge benefit.

What does this automation person do?

We’ll get to more specific in my later posts in this series, but the basic responsibility should be:

  • Figure out the technology and tools to be used
  • Create the initial automation framework
  • Create smoke tests for the application under test
  • Get those smoke tests into some kind of automated build
  • Train developers and QA in the use of the framework
  • Drive the adoption of the framework and automation for all features
  • Set up rules for consistency in writing automated tests
  • Design of hierarchy and structure of automated tests

You want your automation person to be responsible for pushing and driving forward the goal of getting to the point of being able to automate each feature being created with minimal effort.

This is one of those jobs that can actually be “done.”  What I mean by that is you may have the automation lead end up creating the framework, developing it enough to push it out to your development teams and train them to write automated tests along with QA, and you really don’t need an automation lead anymore.

This is great though, because now you have a developer that really knows your product and is a good bridge to QA, or you can keep them improving the framework and overseeing it if you want to go that route.

Bottom line is:

  1. Make sure you get a dedicated resource
  2. Make sure that resource can write code and design a framework
  3. Make sure they train everyone else, one person cannot automate your entire application
As always, you can subscribe to this RSS feed to follow my posts on Making the Complex Simple.  Feel free to check out where I post about the topic of writing elegant code about once a week.  Also, you can follow me on twitter here.