Not to say that you’ll ever be done expanding your automation framework and building and improving your BATs, but you are at the point of having a mature usable BAT framework and you can be proud of that accomplishment.

If you truly want to see where the rabbit hole goes though… Read on

Taking it to 11

If you have designed a good automation framework for writing your BATs, you will most likely have created an internal domain specific language (DSL).

What is an internal DSL?

Basically it is a set of APIs and rules built in another language that function as their own language.

You might have some code that looks like this:

Pages.Login.Goto();
Pages.Login.LoginWithDefaultUser();
Pages.Customers.Goto();
Pages.Customers.AddNewCustomer();

You can see how this code is really looking like a language itself.  It is hard to tell if this code is even C# or Java.

This is a good example of an internal DSL.

If we were to take this same section of a BAT test and write it as an external DSL, it might look something like this:

LOGIN WITH DEFAULT-USER
ADD-CUSTOMER WITH DEFAULT-CUSTOMER

With a true DSL we can simplify even further to get right down to the bare minimum set of commands we need to execute what we want in our application.

How to implement a DSL?

There are a number of tools and techniques that can be used to create a DSL.  I’ll hit on some of those a bit later on, but first we need to design the DSL itself.

Designing a DSL can be challenging, but I have found the best approach is to use a similar approach to what we used to design our automation framework.

It is best to work backwards from the syntax that would make your BAT the easiest to write and the most clear.  Once you have done that you can worry later about implementing the DSL to support the syntax you have come up with.

You should focus on creating a limited vocabulary for your DSL so that it will be easy for someone to learn and know what their options are.

The more restrictions we put into a DSL the easier that DSL becomes for doing a specific purpose.

We want to give the user of a DSL no more options than required for what they want to do and no less.

This isn’t always easy to do.  It often requires some careful thought and refinement.

Your DSL should also use terminology and phrases that are familiar to the business domain of your application.  By doing this you will make it so someone understanding your application will understand BATs written in that DSL.

I also always try to make a BAT DSL read as much as plain English as possible.

Once you have designed the DSL, you will need a way to implement it. 

I’ve written about using ANTLR to create an external DSL.  Also I have written a review on a book that I recommend for learning ANTLR.

Microsoft also has a DSL SDK you can use in VS2010, although I haven’t used it myself.

The basic idea behind implementing a DSL is that you have to take some sort of grammar you design for your DSL, parse it into a syntax tree, then use that tree to generate code that will actually perform the actions.

It is by no means a trivial task, but using a tool like ANTLR, makes it much simpler than trying to completely generate a parser and execution engine yourself.

Why even bother?

Creating a DSL for your BATs is completely optional, but there are some significant benefits to doing so.

The biggest one I have found is that you are much more likely to get business people to write BATs if you have a simple DSL they can use to create them.  Once you get everyone using BATs to define behavior and requirements, communication becomes much more simple.

Another great thing about creating a DSL is that you can create a runner that can interpret the DSL BAT scripts which can be run on any machine.  Typically, to run BATs created in Java or C#, you would need a C# or Java compiler and a test runner, etc.  By creating a DSL and a stand alone executor, you can very easily allow anyone to run BATs with little to no setup.

Writing your BATs in a DSL will also make writing the BATs and understanding them much easier.  A good DSL can get rid of all the little language and API nuances that don’t really add information to the specific domain you are working in.  A good DSL strips information down to the minimum amount needed and is able to do so, because it assumes the context is your application’s business domain.

You will also find that you are better able to separate framework from tests when you have a DSL in place.  One issue that many teams face when creating a BAT framework is that some people come along and write lower level code in the higher level BATs.  This creates a problem because it is hard to force someone to use the higher level API when the framework and the BAT code are the same language.

Wrapping it up

So that completes my series on Getting up to BAT.

Hopefully you have learned enough to be able to implement your own BAT framework and get BATs running for your application.

I tried to cover most of the important steps here without getting too deep into the details.

If you have any questions or are interested in getting more in depth training or consulting on getting BATs setup for your environment, send me an email or leave a comment.

It can be a little difficult to get started and get a BAT framework up and running, but the rewards are well worth the effort!

As always, you can subscribe to this RSS feed to follow my posts on Making the Complex Simple.  Feel free to check out ElegantCode.com where I post about the topic of writing elegant code about once a week.  Also, you can follow me on twitter here.

When we last left off we had just gotten our BATs as part of the acceptance criteria for any new backlogs that are worked on.  This puts us at a point where we could really say that we have successfully implemented BAT testing.

You don’t want to get too comfortable just yet though, because the next hurdle you will most likely face will be the problem of not having enough time to execute all your tests.

You want to think about this ahead of time

Nothing worse than getting everything going and then not being able to execute the entire test suite, because you didn’t plan ahead.

You don’t want to get to the point where confidence in your BAT suite it lost because you are not able to get all the tests executed in a reasonable amount of time.

The more frequently your tests are run the more value they have.

By reducing cycle time from the time a breaking change is made in the system and the time it is discovered, you greatly reduce the risk it imposes on your software and you decrease the scope of the code changes in which it could have occurred.

To put it simply, the faster you can find out you broke something, the more likely you can fix it before it does damage, and the more likely you will be to know what thing you did caused the breakage.

How can we reduce cycle time?

There are a few different strategies we can employ and we can mix and match some of these strategies.

Straight forward parallelization

The best and most effective thing you can do is to take a given test run, split up the execution of those tests on multiple machines and execute them in parallel.

This approach is going to give you the best bang for your buck.  You should really try to get some amount of parallelization going before attempting any other solution, since it is going to make a huge impact on the total execution time for your tests without making any sacrifices.

There are even many ways you can mix and match to do parallelization:

• Use multiple physical machines
• Run many virtual machines on a single host
• Put an executor program on every team member’s machines that will execute tests when that machine is idle or put into test execution mode (perhaps at night)
• Use a cloud based computing platform to execute your tests
• Run multiple browser instances on a single machine

Data preloading

With this approach, you would preload some of the test data that your tests might generate by manually clicking through screens.

This is fairly hard to explain, so let me give you an example:

Suppose you had a set of tests that all involved creating customers, but each customer you create in the system takes about 3 minutes to create by clicking through the screens to get them into the system.

We don’t need to have 500 tests all executing the same exact logic in the system 500 times for 3 minutes just to generate all the customer data that will be used in the tests.

Instead, we can leave a few tests that are exercising the customer creation functionality, and we can execute a SQL script to push all the other customer data into the test database for the other tests to use.

Using this technique we might be able to reduce our total execution time by 3 minutes * each test, or about 25 hours for 500 tests.

This can be a huge savings in time, and it doesn’t come at that high of a cost.  The sanctity of our tests is slightly compromised, but we are taking a calculated risk here knowing that we already have covered the area of execution which we are preloading data for.

Consider this technique when you notice certain things in the system taking a very long time to do.

Test runs by area

With this technique, we can reduce the total execution time in a given period by splitting up test areas and running them either at different times or in response to changes in certain areas.

You have to be very careful with this approach, because if you don’t do it correctly, you can start to erode some of the safety your BAT tests are providing you.

I would only do something like this as a last resort, because it is so easy to virtualize today, and hardware is so cheap.

I’d much rather run too many tests than too few.

Test randomization

With test randomization, we are going to take our total desired execution time, and divide it by the average time for running a test.  We then can use that number of tests to run to randomize the execution of our tests each time we run them and only run the number or tests that will fit in the desired execution time.

This choice is also a compromise that I typically don’t like to take.

It can be very useful though, combined with other techniques when you still don’t have enough time to execute your entire test suite.

The basic idea here is that you are going to randomly run tests each time to fill up the time you have to run tests.

Test reduction

This one seems fairly obvious, but can be very helpful.

Often I will see teams starting out with automation, trying to write way too many BAT tests for a given feature.  Sure, with automated tests it is possible for run a test for every single possible combination of values in your 5 drop downs, but will it really benefit you?

In many cases you have to think about what you are trying to protect against with your BATs.  Sometimes running every combination of data selection choices is going to be important, but other times you are only going to need to write tests to test a few of the happy path scenarios.

It is important to find a balance between test coverage and test volume and not just for execution time.  There is a logistical overhead to having a large volume of mostly redundant tests.

So even though this technique might seem dangerous and counter-productive, I will almost always employ it to some degree.

Common pitfalls

Here are some of the things you might want to watch out for as you are scaling out and streamlining your execution of BATs:

• Parallelization issues.  If you are using shared state, you can run into big trouble when your tests are executing in parallel.  There are many manifestations of this.  You could have issues at the database level or at the local machine memory level.  The best way to avoid this kind of problem is to use separate virtual machines for each test execution, and not reuse data setup between test cases.
• Ineffective error reporting.  If you run a huge volume of tests in parallel, you better have a good way to sort through the results.  It is much harder to figure out why things failed when they are run across multiple machines.
• Test order dependencies.  Make sure tests don’t rely on being run in a certain order or you will have lots of pain when you disrupt that order.
• Environment setup.  Make sure all your test execution environments are exactly the same unless you are specifically testing different environments for execution.  You don’t want tests failing on one machine but passing on another.

As always, you can subscribe to this RSS feed to follow my posts on Making the Complex Simple.  Feel free to check out ElegantCode.com where I post about the topic of writing elegant code about once a week.  Also, you can follow me on twitter here.

This is definitely one of the more challenging tasks you will face, and perhaps the most critical.

All of your efforts will be in vain if you can’t get to the point of creating new BATs for new backlogs that are finished, because if you don’t you’ll still be doing manual testing, and you’ll be playing catch-up all the time.

Start off small

The goal here is to eventually get to the point where every single backlog requires BATs to be written and passing in order for that backlog to be called “done.”

If you try to drop this lofty goal onto a team in one iteration, you are likely to get beaten and slapped with much vigor.

Instead, start with a simple goal of requiring that at least one backlog have BATs for all of the functionality of that backlog.  Start with this small goal, but enforce it with rigor.

By starting out small like this, you will get the team accustomed to the idea of building automated tests at the same time as building the functionality and you will give them a chance to have a small success without sacrificing much velocity.

The goal at this point should be to get people excited about writing BATs along with the backlog.

It’s actually quite fun

I have found that developers tend to really enjoy writing automated tests.  Make sure you let everyone who wants to share in the fun, not just QA and not just developers.

It is really very exciting to see the “magic” of an automated test clicking through buttons on your web page and doing all kinds of things that seemed so tedious before.

I have found that this is so much fun, that you might have the problem of developers not wanting to work on the real backlog, but rather to write the automated tests for it.

This is good, you really want to foster this attitude, it will greatly help when you finally…

Drop the bomb!

What bomb?  The bomb that says on it:

All backlogs will require BATs as acceptance criteria.  It’s not done unless it has automated tests that prove its functionality.

You should be getting to this point within 3 to 4 iterations on average.  Don’t go too fast to get here, but don’t go too slowly either.

I know this seems like an impossible task, but it really isn’t.  I’ve been here before and had much success with it on several different teams.  The key is making sure everyone is comfortable with writing automated tests and understands the value.

Whatever you do though, once you draw this line in the sand, Do not back down! I mean it.

You might start getting death threats.  People might directly laugh in your face all the while spitting skittle colored loogies in your coffee, daring you to “taste the rainbow,” but you have to hold your ground.

It will be worth it in the long run, because you will have built an awesome safety net for proving the functionality of your software over time.

It’s not all rainbows and butterflies

You are going to sometimes run into scenarios where automation will be too high of a price for a particular backlog.

This will especially happen in cases where you have some kind of process that spans multiple domains or software systems.

When you run into those situations, give it a solid effort to at least automate as much as possible, but don’t die on the hill and kill the whole automation project.  We have to be pragmatic or risk having our credibility called into question.

Don’t expect everyone to embrace writing BATs overnight, or to even be good at it.  When you first introduce this to the team, you are going to have a learning curve and your automation framework is still going to have some pretty big holes in it.

But, that is OK.  Over time you will build out that framework and produce some fierce battled scarred veterans able to automate web pages while scowling at you from afar, just don’t lose hope and keep beating that drum!

As always, you can subscribe to this RSS feed to follow my posts on Making the Complex Simple. Feel free to check out ElegantCode.com where I post about the topic of writing elegant code about once a week. Also, you can follow me on twitter here.

But!  You might want to consider holding off for a second and reading this post!

It is worth taking a bit of time and thinking a bit about the strategy and psychology of adding the smoke tests to your regular build process.

There are several things we are going to want to discuss and think about before adding automated tests to a build.

Why are we adding the smoke tests to the build?

I think it is always important that before we do something we ask ourselves why we are doing it.  Doing things, because someone said it is good, or because it seems right is not a very good reason.

The primary benefit we are going to get by adding smoke tests to the build now instead of waiting for a larger portion of tests to be written is that we are going to be able to immediately tell if a basic functionality of the system is broken instead of finding out after we have deployed the build to different environments and used it.

We want to be able to get this value as soon as possible.  We don’t want to have to wait until we have a huge amount of tests, because even a few tests will give us some value in this area.

The other reason we are adding the tests to the build is so that we can notify developers when a test fails, so that they know to fix whatever was broken.  By having the smoke tests run as part of the build, we are able to reduce the amount of time before a break is introduced and the break is discovered.

This concept of reducing the delta between introduction and discovery is very important, because it makes it much easier to identify what caused a break.

Perhaps a distant 3rd reason to add the smoke tests at this stage, is to prove out our framework and technologies.  Better to start small with a small number of tests and get everything working.

With those ideas in mind, we can move onto some of the more important considerations.

Don’t ever add failing tests to the build

When you add your smoke test suite to the automated build, all of the tests should pass!

I will say it again, because it is really important and ignoring this advice may doom your entire BAT project.

Do not ever add failing tests to the build!

It seems like an innocent enough thing, but it causes great harm, because it is very much like the boy who cried wolf.  Consider what happens when we introduce our new BAT system and we add our smoke tests to the build, but we have 3 failing tests.

First thing that will happen is that our tests will run during the next build and 3 will fail.  This should cause a build failure notification of some sort to go to the team, at which time you will be in the uncomfortable position of saying something to the effect of “ignore that fail, those are real defects, but they are known issues.”

Next, let’s assume a developer checks in some bad code and it causes a 4th test to fail.

Do you think most people will notice this 4th failing test?

Even if they do, do you think they will just assume it is another known issue?

Worse yet, they may think your BATs are meaningless and just fail randomly.

Do you see where I am going with this?  We want to start off with the correct expectations and we want test failures to be meaningful.  To ALWAYS be meaningful.  The second you have test failures that are not meaningful, you lose credibility for any future meaningful failures.

This is such a critical psychological battle with build systems and automated tests that I have often implemented a policy where no one is allowed to work on anything else but fixing the build or test failure when a build fails.  When you have 20 developers either sitting idle or working on one issue for 2 hours, it sends a pretty clear message that we take build failures seriously.

Make sure your tests never undermine this message.

What do I do with failing tests then?

If they are real defects, put them in your defect tracking system or create backlogs for them and make sure the tests are disabled until those defects are fixed.

Your automated tests should be acting as regression tests for things that are already working.  It is absurd to think that it is even close to cost effective to write automated tests to actually functionally test new features or code!

It is the process of writing the automated test that should uncover defects.  Running of automated tests should uncover new defects where functionality has regressed from working to not working.

Other considerations

Here are a list of some other considerations you will want to think about before embarking on this journey.

• How will you actually run the tests? (Will you use the build script to kick it off, some 3rd party plugin, etc)
• Where will you run the tests? (Will you use the build server itself, or another machine.  You probably don’t need to worry about scaling out to multiple machines right now, but you should at least keep it in the back of your mind.)
• How long will it take to run the smoke tests? (Fast is better, because feedback cycle is reduced.  If you have a really long test suite, you might want to pare down the smoke tests and run some with every build, and all of them every night for now.)
• Will developers be able to run the tests locally? (Make sure you consider how to do this before putting the tests into the build.  Nothing more frustrating than being measured against a bar you can’t yourself see.)
• How will you notify and report results from the test failures? (We don’t care about passes, only failures, but you need a good way to report the result that prompts action.)
• What is the policy for test failures? (Make sure there is some agreement on what kind of action will be taken when tests fail.  A great place to start, if you can get management to buy in on it, is all work stops until all tests pass.  Extreme, but extremely effective.)
• How will you prevent false fails and identify them? (Don’t get stuck with egg on your face.  If your framework fails and the problem is not a real defect, you need to be able to quickly identify it and fix the framework problem.  There are a few strategies for doing this, but that is for another post.)
• How do new tests get added and commissioned? (You are going to be constantly growing your test suite, so you will need a good method for adding tests to the smoke test suite.)

Remember, you only get one chance at a first impression!

As always, you can subscribe to this RSS feed to follow my posts on Making the Complex Simple.  Feel free to check out ElegantCode.com where I post about the topic of writing elegant code about once a week.  Also, you can follow me on twitter here.

Now you are probably faced with the difficult decision of…

Where to begin with your tests

I don’t suggest writing your automation framework first and then writing tests to use it.  If you go this route you’ll have two problems

1. It will be a long time and large investment before you see any fruits from your automation effort.
2. Your framework will likely be designed incorrectly or not test-writer friendly.  (Which is basically the same as wrong.)

What you really want to do is to write some basic tests that let the development of those tests drive the development of your framework.

The best tests to drive your framework are some basic “smoke tests.”  If you are not familiar with the term “smoke test,” it basically means some tests that make sure the application you are testing is at least somewhat working and not completely broken. 

There are a few different opinions of what exactly constitutes a smoke test, but in general, if you have any tests that you regularly run on a new build to make sure it is at least OK, those would be the tests we are interested in here.

You are going to want to start by taking the easiest and most simple tests you have and writing them in a concise syntax that you use to drive the development of your framework.

It is very important at this stage to not worry about what you think might be possible or how it would be implemented in the framework, but to instead focus and making the tests as terse and explanative as possible.

The basic idea is that you will write the test out in the syntax you would like to be able to use, then implement the framework features required to make that test pass.  This whole process is very similar to doing Test Driven Development.

An example case

The easiest way to demonstrate this is with an example.

Let’s say you have a smoke test like the following:

Create a new account, pick an on-sale item, add it to your cart and check out using PayPal.  Verify that we hit the order success page.

We might write a test for this that looks something like:

Workflow.Users.CreateNewUser();
Pages.Login.LoginWithLastCreatedUser();
Pages.SaleItems.AddAnyToCart();
Workflow.Cart.CheckoutUsingPayPal();
Assert.True(Pages.Checkout.SuccessPage.IsAt());

What I just wrote above is astonishingly simple.  You are likely saying to yourself something along the lines of “What if… blah.”

Forget it!  We’ll handle “what if” later.  Right now we are going to write the tests as terse as possible and not handle any other conditions.

What I wrote above is the style that I like to write the tests in, which is highly subject to opinion, but the content of what I wrote is not as subjective.

There are a few important points to note:

• I used methods like CreateNewUser, LoginWithLasteCreatedUser and CheckoutUsingPayPal that do a huge amount of work and have no parameters.  Why?  Because for the purpose of this test, (and you’ll find for most of your tests), we don’t care about what the first name or last name or address of the user is, we don’t care how we get them logged in, and we only care that they checkout using PayPal, not what account is used, etc.
• I didn’t declare any variables.  Tests should be simple.  Let the framework remember the state.
• I over-simplified.  This is fine, you are working inside of a very specific domain.  Let your test framework methods do one large thing.  Don’t worry about oversimplifying for now.
• I asserted something at the end.  All tests need to assert something.  If you can’t figure out what to assert, work on redesigning the test before automating it.
• I didn’t pass parameters.  Avoid this also.  This goes along with the first point, but is so important that I feel it is OK to state it twice.  If you don’t care about variation, eliminate it.  You will constantly need to find places where a manual tester would have to enter specific information, but an automated test won’t care what that information is.
• All my methods are static.  While this would be bad in normal production code, for a test API it is the easiest for test writers to use.  No variable declarations, no “newing” up objects, no constructors, etc.
• It’s written in “business language.” As close to the language an end user or business person would use to describe the test.

So, this test is definitely doing a huge amount of stuff in a few lines of code, and because of this your inclination would be to say that the API level is too high. Don’t!

Instead, make the API level as high as you possibly can, until further tests you try to implement prove to you that it must be broken down further.

By working this way, you ensure a reduction in the total lines code and complexity of your tests by as much as possible.

You would NEVER work like this with normal code, but for automated tests this is essential.  The main reason is because you will be calling the same methods perhaps 1000s of times, where in normal code you usually only reuse a method in a few places.

Next steps

For my example above, the next step would be going through and implementing all the framework methods required in order to make that test runnable.  You will end up utilizing the general design you came up with for your automation framework to do this.

Then you will simply move on to the next test and repeat.  Each time you should have less and less effort required, as more of the basic functionality of your automation framework is implemented.

Automating your smoke tests should ensure that you get a fairly even distribution of the framework code written as the smoke tests should cover many different areas of your software.

But, don’t start adding these smoke tests to your build yet.  We’ll cover that in the next step.  Adding them to the build prematurely can cause the confidence in these tests to be forever diminished!  We don’t want to do that!

As always, you can subscribe to this RSS feed to follow my posts on Making the Complex Simple.  Feel free to check out ElegantCode.com where I post about the topic of writing elegant code about once a week.  Also, you can follow me on twitter here.

Automation tools that claim to support multiple browsers are LYING

Currently I am in the process of getting 400 automated tests that run in Firefox to run in Internet Explorer and Chrome. While the tests are using Selenium RC, a tool that claims to work on all three browser platforms, in actuality, many tests are failing and not for good reasons (bugs).

Here’s the rundown on what these overly-ambitious tools aren’t telling you:

• Xpath in Browser A isn’t always supported in Browser B
• The Dom in Browser A may vary from the Dom in Browser B
• Browser B may have built-in security settings that interfere with testing
• Bugs in the automation tool itself may cause issues in specific browsers

When it comes to XPath, I fully expected that any given expression was either valid or invalid. But it didn’t take more than one test run in Internet Explorer to suddenly find myself with tests that claimed certain elements didn’t exist. While Firefox accepted and successfully returned my element using the provided expression, IE claimed the expression itself was invalid. In this particular instance I discovered an issue with Selenium stripping off the end of my xpath statement, thus making it invalid.

But ultimately, whether it’s the browser or the tool doesn’t matter because the same expression is not working unequivocably in both browsers.

The Dom Issue

The next problem I immediately came across was the problem of Dom differences between browsers. In this particular case I had an element that occurred one time in Firefox but twice in IE. My test, expecting one instance of the element, failed. While I can’t tell you why this discrepancy exists (currently under investigation by some client-side engineers), I have a fair degree of confidence in saying this won’t be the last time I come across this issue.

In the quest to support many different browsers, it only follows that one is going to come across conditional browser logic, especially if a company is striving to support multiple browser versions (IE 6, 7, 8, 9). If a developer has to occasionally write browser-specific condition statements, it makes sense that we’d have to do the same to test specific browsers.

The Security Issue

If it seems that I’m picking on IE, it’s only because I can’t get very far in running our tests in Chrome. Chrome has special security settings that cause a Selenium exception if your test changes domains mid-run. This can easily be solved by passing a parameter to Chrome which will disable this security feature when the test runs, the trick is figuring out when/where/how to pass that parameter.

The Bug in the Tool Issue

Finally, the last obstacle I’ve come across in multi-browser testing: the tool itself. Any tool, regardless of how fabulous it is, is going to have bugs. Selenium RC has an issue where its select method (responsible for selecting a value in a dropdown menu) doesn’t trigger affiliated event handlers- in Internet Explorer. To work around this, one can manually call the event handler using Selenium’s runscript() but you will lose some degree of testing confidence in doing so (how do you know the event is really firing in response to a particular user action if you’re manually forcing it in automation?)

Another issue exists around multiple browser windows; if you need more than one open for your test, Selenium is unable to select the second window you open- another outstanding IE issue that has not been resolved.

Summary

In all fairness, some of these issues have more to do with the browsers themselves than with the automation tool. However, that makes it even more important to realize that when an automation tool says it supports multiple browsers, it’s not without some creativity, troubleshooting, legwork, and maintenance.

Knowing these kinds of issues exist is important in order to decide if and how much browser compatibility testing you want to automate. It’s definitely something to realize before deciding on a tool just because the tool claims to support other browsers.

Just as there is no silver bullet for writing a web application that works in every browser imaginable, there is no silver bullet for automation either.

This is one of the most critical components of the overall success of your automation strategy, so you will want to make sure you invest properly in this area.

I have seen a large number of automation projects go forth, but each time the critical component determining their success or failure was having a good automation framework.

Solid design and excellent technical resources are absolutely critical to success in this area!

What is an automation framework?

An automation framework is essentially an API that all of your BATs (Blackbox Automated Tests) are written against.

This API can be as simple as the API that is exposed by your browser automation tool (WatiN, Selenium, etc.), but I would highly recommend building your own layer on top of the browser automation tool’s API that will act as a DSL (Domain Specific Language) for automating your application.

Let me break this down a bit further by using an example.

Think about the task of making coffee.

There are several “API” levels we can interact with.

1. We can go with a very low level API where we take whole coffee beans and grind them down.  Then we take some water, get it hot.  We take our filter put the ground beans in it, put it over a cup and pour the water into the filter.
2. We can go with a higher level API where we use a traditional coffee maker.  In this case we load the coffee maker with a filter, ground coffee beans and water and push a “brew” button.  We could also set it to start at a certain time.
3. We can go with a very high level API where we use a Keurig machine or similar device.  In this case we only make sure the machine has water in it, and we just insert a little pod and press brew.  We can make different kinds of coffee, cider or hot cocoa just by changing what pod we use.

Using the API provided by the browser driver is like making the coffee by hand.  It’s going to take a large amount of effort each time you do it.

We want our automation framework to be more like the Keurig machine.  We want to be able to compartmentalize our tests in little pods that are small and don’t require many hooks into the automation framework.

To rehash this one more time, basically our automation framework will be a framework we build on top of the browser driver framework, which is designed to make it easy to write tests which automate our application.

What makes a good automation framework design?

The true measure of an automation framework is the size of the tests that are written against them. The less lines of code in each test, the better the automation framework captures the domain of the application being automated.

In my earlier post about an example of an automation framework, I talked a bit about the strategy I used in a real implementation of an automation framework.

Here is a diagram showing the different layers.  You can see the framework layer is in green here.

You can also see in this diagram that on the right hand side, I have screens, workflows and navigation.

This is one of the common design patterns you can use to build your automation framework, since it closely models that of most web applications.

A good way to design your framework is to use a pattern like this and create classes for each page in your application, create classes which might represent workflows in your application, and create some classes for navigating around the application.

The basic idea you are shooting for with your automation framework is to make it so the tests do not have to know anything about the browser.  Only your automation framework itself should be dealing with the DOM and HTML and CSS.  Your tests should be dealing with concepts which any user would be familiar with.  That is why I commonly use pages and workflows.

Let me give you an example.

Let’s say we ignored the advice of creating an automation framework and decided to program our tests directly against the browser driver layer.  In this case, let’s say that we want to automate a login page.  Our test might look something like this pseudo-code.

browser.goto("http://mywebsite/login.aspx");
Element userName = browser.getElementById("login_userName");
userNameTextField.Type("Joe");

Element userPassword = browser.getElementById("login_userPassword");
userPassword.TextField.Type("mypassword");

Element loginButton = browser.
    getElementByXPath("//button[class="loginButton"]");
loginButton.Click();

This is not very readable.  It is not something an average tester will be able to pick up and start writing, and it is extremely fragile.  If you have 500 tests like this and the id changes for one of the elements on the page, many tests will break.

Now, contrast it to this pseudo-code:

Pages.Login.Goto();
Pages.Login.LoginWithDefaultUser();

Where did all the code go?  You don’t need it!  Actually it went into the framework, where it can be reused.  Now your test code is simple, can be understood by just about anyone, and won’t break with changes to the UI.  (You will just have to change the framework method instead of 500 tests.)

If you want some more detailed examples take a look at my Boise Code Camp slides on the subject here.

Let me offer up some general guidelines for creating a good design for an automation framework.

1. NEVER require the tests to declare variables.
2. NEVER require the tests to use the new keyword or create new objects.
3. NEVER require the tests to manage state on their own.
4. ALWAYS reduce the number of parameters for API calls when possible.
5. ALWAYS use default values instead of requiring a parameter when possible.
6. PREFER to make the API easier to use over making the internals of the API less complex.
7. PREFER using enumerations and constants to requiring the test to pass in primitive types.  (Make the input bounded when possible.)
8. NEVER expose the browser or DOM to the tests or let them manipulate it directly.

You can see that these guidelines will almost force your API to be entirely static methods!  Don’t freak out.  It is perfectly fine.  There is nothing wrong with using static methods in this case.

Stop… take a breath, think about why you think static methods are bad, because I probably agree with you.

But, in this case we are HIGHLY valuing making the tests simple and easy to write over anything else, and because of this emphasis, static methods are going to be the way to go.

Some advice on getting started

Start with the tests in plain human English.

Take those tests and make them into code preserving as much English as possible.  Then work down from there implementing the methods needed to make those tests work.  Only add things as you need them.  Do not overdesign!

In my example above, we might have started with something like this:

1. Goto the login page.

2. Login with the default user.

The actual code looks just like that.  It should always be a 1 to 1 mapping.

This is where having someone that has some experience doing this is going to come in handy.

Just keep in mind the whole time your 2 big goals:

1. Easy to write tests
2. Tests are concise

As always, you can subscribe to this RSS feed to follow my posts on Making the Complex Simple.  Feel free to check out ElegantCode.com where I post about the topic of writing elegant code about once a week.  Also, you can follow me on twitter here.

It is time to pick the browser automation tool you are going to use for driving your automation efforts.

Before we can build an automation framework, we need a basic set of tools to build it on.  We want our automation framework to be tailored to our application under test specifically.  Our automation framework will act like a Domain Specific Language (DSL) for testing our application, but we need to build that automation framework on top of something.

What is a browser automation tool?

It is essentially what we will use to drive the web browser.

We need to pick a tool or framework that will easily allow us to interact with the web browser so we don’t have to write that code ourselves.

We will need to be able to click buttons, enter text, select drop downs, and check values on web pages.

Some of the things we will want to consider when choosing a browser automation tool include:

• Language we can use with the driver
• Browsers it supports
• How we interact with the browser (do we use XPath, JQuery, or some other mechanism)
• If we can access and manipulate everything we need to be able to in the browser
• Speed of execution
• Ability to execute in parallel
• Support and future development

Let’s take a quick look at each of these considerations.

What language can we use?

This is an important consideration because you will want to be able to utilize your other programming resources to help with building an automation framework and to eventually create automation tests.

If you pick a browser driver that supports a language your regular programmers don’t like, or don’t know, you may not end up with their support, and you are definitely going to want their support.

What browsers does it support?

This is a tricky topic to discuss.  It might seem that a browser automation tool would need to support all of the browsers your application supports, but that is not really true.

You really have to think about what your goal is with your BATs.  I could certainly devote a whole blog post to this topic, but I will try and summarize my main point here.

BATs should be designed to test functionality of the application, not to test compatibility.  (At least most of your BATs should be, although you might have some that are specifically for compatibility.)  With that goal in mind, a browser driver supporting each and every browser your application supports is not really necessary.

You do need to consider that the browser automation tool should support at least one of your supported browsers, preferably the main one.

How do we interact with the browser?

There are several different APIs which are used to communicate with the web browser that are employed by various different browser drivers.

Some tools have APIs that are very low level and allow you to very easily manipulate objects in the web browser while others are at a much higher level and rely more on your understanding of the brower’s DOM.

For example, one API might expose elements on a web page by their actual element name and let you interact with them directly.

browser.Button(Find.ByName(“submitButton”)).Click();

vs

instance.click(“submitButton”);

In the first example, the API is aware of element types like buttons.  In the second example, the API is more generic and will send a “click” to an element with the ID value you specify.

You’ll have to decide how important the ease of use is to your project at this level.  There are some tradeoffs to consider here.

The lower level the API is, the easier it will be to use, but the more dependent you will be on that API and the more language specific that API will be.

The higher level the API is, the harder it will be to use, but it will free you from depending so much on the API, because you will be interacting more directly with the browser’s DOM.

I prefer lower level APIs because I find that it is much easier to write the framework code on top of these.

You will also want to consider here the skill levels of the programmers who will be creating the framework.  If your API is lower level, native language skills are more important.  If it is higher level, DOM and HTML, and perhaps Javascript skills, are more important.

Can we access what we need?

If your application makes use of Javascript pop-up dialogs and your browser automation tool doesn’t give you a way to interact with them, you don’t want to find this out when you have already invested significantly into your automation framework.

Different automation frameworks support different features of browser; not all of them will allow you to interact with every single browser feature that exists.  As new features are introduced, you could also get stuck with a tool that is no longer being maintained and doesn’t add support for the new features.

You will want to consider things like how much your application uses AJAX or JQuery.  You will want to select a browser automation tool which would make it easy to interact with AJAX calls if your application relies on them heavily.

Speed of execution

Not all browser automation tools are the same in execution speed either.

This may not be important to you, depending on how you address the issue of concurrency, but you should at least consider that if you have a large number of automated tests, a small difference in execution speed can result in a large difference in total time to run the tests.

The faster the automated tests can be run, the more valuable they are.  I won’t get into the details here but there are many reason why this is true.

Also influencing the speed in which you can run your tests will be if you automation tool choice requires you to put pauses into the test to wait for the browser instead of responding to events that occur in the browser.

Ability to execute in parallel

This consideration will depend greatly on the volume of and speed in which your tests can execute.

At some time (sooner than you think) you will likely get to the point where you can no longer run all of your automated tests in 24 hours.  Perhaps even before this point, you will want to consider running your BATs in parallel to reduce the total time to execute the tests.

Some automation tools have built-in support and others will require you to build your own way to do this.  It is good to at least have an idea of how you are going to achieve parallel execution when considering what browser automation tool to use.

Support and future development

We are in a pretty rapid pace of browser development.  Many things in the web browser world are changing much more rapidly than ever before.  Tools to automate the browser must also change or you could end up in a very bad place.

If your application is going to take advantage of the latest browsers and browser features, you should make sure the automation tool you choose is in active development.

There is nothing worse than investing into an open-source project and then having that project die, resulting in you having to rip it out to replace it with another library.

You can always design your automation framework to be abstracted away as much as possible from the underlying browser driver, but that will be extra work, so consider this point carefully.

Name names sir!

Nope, I’m not going to do it.

I’m not going to tell you to use Watin or Watij or Selenium or WebAii or any other browser driver.  I don’t want to put the focus on the tool, since the real focus will be building the automation framework on top of the tool you use to drive the browser.

I would suggest that you try writing a few simple tests in each of the major browser drivers so that you can get a good feel of what the API is like and how it will work with your application.

I will say that I have used most of the major choices out there and there really is no clear winner in my mind.  It really is going to depend on what your environment is like and what kind of application you are testing.

I would also recommend picking a browser driver and sticking to it.  I’ve tried in the past to abstract away the browser driver from the automation framework and while it is possible, it can become quite messy and add quite a bit of overhead to your project.

My only other hints would be to not put too much emphasis on supporting multiple browsers or on using recording tools.  Neither of these things will benefit you much in the long run because you will find that you will not want to try and run all of your BATs on each browser. Recording tools will not be nearly as effective as writing your own custom framework (which I will talk about in my next post.)

As always, you can subscribe to this RSS feed to follow my posts on Making the Complex Simple.  Feel free to check out ElegantCode.com where I post about the topic of writing elegant code about once a week.  Also, you can follow me on twitter here.

Spend a majority of your effort, all the time you would have spent writing unit tests, instead writing what I will call blackbox automated tests or BATs.

In this post, I am going to outline what I think it takes for a team to go from BAT-less to BAT-man (or BAT-woman) without going batty.

I want to take a practical approach to getting from 0 blackbox automated tests to building a sustainable process of developing BATs as a integral component of the acceptance criteria for calling a backlog item “done.”

Let me paint a picture for you

I want to paint the picture for you of the end goal that we are seeking to achieve, from start to finish, of a single backlog.

1. A backlog is selected for work.
2. Developers, QA and product owner work together to clearly define the acceptance criteria which will enable that backlog to be signed off or called “done.”
3. QA and developers work together to transform the acceptance criteria into a high level description of BATs which will be created to verify each aspect of the acceptance criteria.
4. Developer begins coding with this high level set of BATs in mind (BATs drive the development), QA begins developing BATs.
5. As developer completes enough functionality to pass a BAT and not break existing BATs that functionality is checked into trunk.
6. Each check-in causes the entire battery of BATs to be kicked off split across multiple VMs which run 100s of hours worth of automated tests in minutes.
7. Once all BATs for a backlog are passing, that backlog is done and those BATs become part of the battery of tests which are run with each continuous integration build.

We should keep this simple process in mind and strive to reach it, knowing that when we are able to reach this process we will be able to have a HUGE amount of confidence in the functionality and non-regression of our system.

Put aside for a second your notion that unit testing is the correct thing to do.  Put aside the idea that blackbox automated testing is too hard and too fragile, and imagine the world of software development that flows like the picture I painted above.

Now listen up, because this part is important…

Before you put forth the effort to write one more unit test, before you dip your test double pen in the ink well of “mock,” make sure you are first taking efforts to develop a process to create BATs.

The value proposition

We really need to take the effort to understand the value proposition being presented here.

I’m going to use some real fake numbers to try and really convey this important point that I think is likely to be overlooked.

Consider first the amount of time that you spend doing two things:

1. Creating infrastructure to allow your code to be unit tested in isolation.  (Dependency injection, interfaces, etc.)
2. Time spent writing unit tests.

Now, before we go any further, I just want to reiterate.  I am not advocating completely abolishing the writing of unit tests.  See my original conclusions post, or my post on unit testing without mocks for a better understanding of what I am advocating in regards to unit testing.

So back to my point.  Think about all the time it takes to do this.  To properly isolate and unit test code most developers would probably estimate that for every hour worth of work writing the code there is about another hour to two hours worth of unit testing and unit test prep time.

There is a reason why unit tests take longer to write than the code they test; it takes MORE lines of code to test a line of code in isolation.

BATs developed using a good automation framework are just the opposite.  While unit test code might have a ratio of 4 lines of unit test code to 1 line of production code or a 4:1 ratio, BAT code has a completely opposite ratio.  It is very likely that 1 line of BAT code can test 20 lines or more of production code, a 1:20 ratio.  (Where do I get these numbers?  Looking at some of my real production code being tested with unit tests and BATs.)

Even if unit tests and BATs were equally effective in preventing the regression of your software system, and equally effective at providing an absolute assurance of meeting the acceptance criteria (which I would argue that BATs are much more effective at both), you can see easily that you are going to get a much higher return on your investment by investing your time in BATs vs. investing your time in unit tests.

I don’t make these statements lightly or in a theoretical tone.  I have real world experience with successfully implementing automation frameworks for writing BATs that reinforce these conclusions.

How to get there

If I have done my job, you are now at least convinced that getting to the point of having BATs for all backlogs is a good goal, but if you are like most people I talk to, you are skeptical of the costs and feasibility of doing it.

What you need is a good guide.  A step by step guide on how to do it.

I am going to conclude the Back to Basics posts and segue into a new series with the goal of providing a step by step guide to getting to fully automated blackbox testing.

Let’s take a look at the steps that I will cover in the next series of posts.

1. Hiring an automation lead – it’s important to either find someone who’s done this before, or a developer resource to this role.
2. Deciding on the underlying browser driver or browser automation tool – assuming web apps here, there are several to choose from, WatiX, Selenium and more.
3. Designing an automation framework – building a framework tailored specifically to your application under test will create an effective domain specific language for testing your application.
4. Creating your first smoke tests – building some smoke tests will help you prove and build out your framework and provide you the highest value tests first.
5. Adding smoke tests to your build – with smoke tests being run as part of the build, you can immediately start seeing value.
6. Adding BATs to your acceptance criteria – we need to start out slowly here, but eventually make all backlogs require BATs for each acceptance criterion.
7. Scaling out – as you start to get a large amount of BATs you’ll need to figure out a way to virtualize more servers and parallelize the test runs to be able to run all the BATs in a short amount of time.
8. Building a true DSL – once you get this far, it may be time to start thinking about creating a domain specific language that even business analysts can write tests with.

As always, you can subscribe to this RSS feed to follow my posts on Making the Complex Simple.  Feel free to check out ElegantCode.com where I post about the topic of writing elegant code about once a week.  Also, you can follow me on twitter here.

We’ve talked about what an interface is, cohesion and coupling, and even went a little bit off track to talk about sorting for a bit.

One of the reoccurring themes that kept showing up in most of the posts was unit testing.  I talked about why unit testing is hard, and I defined three levels of unit testing.

• Level 1 – we have a single class with no external dependencies and no state.  We are just testing an algorithm.
• Level 2 – we have a single class with no external dependencies but it does have state.  We are setting up an object and testing it as a whole.
• Level 3 – we have a single class with at least one external dependency, but it does not depend on its own internal state.
• Level 4 – we have a single class with at least one external dependency and depends on its own internal state.

Throughout this series I ended up tearing down using interfaces with only single non-unit test implementation.  I criticized the overuse of dependency injection for the sole purpose of unit testing.  I attacked a large portion of best practices that I felt were only really being used in order to be able to unit test classes in isolation.

But, I never offered a solution.  I told you what was bad, but I never told you what was good.

I said don’t create all these extra interfaces, use IoC containers all over your app, and mocks everywhere just for the purpose of being able to isolate a class you want to unit test, but when you asked me what to do instead, I said “I don’t know, I just know what we are doing is wrong and we need to stop.”

Well, that is no answer, but I intend to give one now.  I’ve been thinking about this for months, researching the topic and experimenting on my own.

I finally have an answer

But, before I give you it, I want to give you a little background on my position on the subject matter.

I come from a pretty solid background of unit testing and test driven development.  I have been preaching both for at least the last 7 years.

I was on board from the beginning with dependency injection and IoC containers.  I had even rolled my own as a way to facilitate isolating dependencies for true unit tests.

I think unit testing and TDD are very good skills to have.  I think everyone should learn them.  TDD truly helps you write object oriented code with small concentrated areas of responsibility.

But, after all this time I have finally concluded, for the most part, that unit tests and practicing TDD in general do more good for the coder than the software.

What?  How can I speak such blasphemy?

The truth of the matter is that I have personally grown as a developer by learning and practicing TDD, which has lead me to build better software, but not because the unit tests themselves did much. 

What happened is that while I was feeling all that pain of creating mocks for dependencies and trying to unit test code after I had written it, I was learning to reduce dependencies and how to create proper abstractions. 

I feel like I learned the most when the IoC frameworks were the weakest, because I was forced to minimize dependencies for the pain of trying to create so many mocks or not being able to unit test a class in isolation at all.

I’ve gotten to the point now where two things have happened:

1. I don’t need the TDD training wheels anymore.  I don’t pretend to be a coding god or demi-god of some sort, but in general the code I write that is done in a TDD or BDD style is almost exactly the same as the code I write without it.
2. The IoC containers have made it so easy to pass 50 dependencies into my constructor that I am no longer feeling the pain that caused my unit tests to cause me to write better code.

What I find myself ending up with now when I write unit tests is 70% mocking code that verifies that my code calls certain methods in a certain order.

Many times I can’t even be sure if my unit test is actually testing what I think it is, because it is so complex.

Umm, did you say you had an answer, dude?

Yes, I do have an answer.  I just wanted to make sure you understand where I am coming from before I throw out all these years of practical knowledge and good practices.

I am not the enemy.

My answer to the problem of what to do if you shouldn’t be using IoC containers and interfaces all over your code base just for the purpose of unit testing, is to take a two pronged approach.

1. Mostly only write level 1 or level 2 unit tests.  Occasionally write level 3 unit tests if you only have 1 or possibly 2 dependencies.  (I’ll talk about more how to do this in my next post)
2. Spend a majority of your effort, all the time you would have spent writing unit tests, instead writing what I will call blackbox automated tests or BATs.  (I used to call this automated functional tests, but I think that name is too ambiguous.)

I intend to drill really deep into these approaches in some upcoming posts, but I want to briefly talk about why I am suggesting these two things in place of traditional BDD or TDD approaches.

What are the benefits?

The first obvious benefit is that you won’t be complicating your production code with complex frameworks for injecting dependencies and other clever things that really amount to making unit testing easier.

Again, I am not saying you shouldn’t ever use dependency injection, interfaces or IoC containers.  I am just saying you should use them when they provide a real tangible value (which most of the time is going to require alternate non-unit test implementations of an interface.)

Think about how much simpler your code would be if you just went ahead and new’d up a concrete class when you needed it.  If you didn’t create an extra interface for it, and then pass it in the constructor.  You just used it where you needed it and that was that.

The second benefit is that you won’t spend so much time writing hard unit tests.  I know that when I am writing code for a feature I usually spend at least half the amount of time writing unit tests.  This is mostly because I am writing level 3 and level 4 unit tests, which require a large number of mocks.

Mocks kill us.  Mocking has a negative ROI.  Not only is creating them expensive in terms of time, but it also strongly couples our test classes to the system and makes them very fragile.  Plus, mocking adds huge amounts of complexity to unit tests.  Mocking usually ends up causing our unit test code to become unreadable, which makes it almost worthless.

I’ve been writing mocks for years.  I know just about every trick in the book.  I can show you how to do it in Java, in C#, even in C++.  It is always painful, even with auto-mocking libraries.

By skipping the hard unit tests and finding smart ways to make more classes only require level 1 and level 2 unit tests, you are making your job a whole lot easier and maximizing on the activities that give you a high ROI.  Level 1 and level 2 unit tests, in my estimation, give very high ROIs.

The thirds benefit is that blackbox automated tests are the most valuable tests in your entire system and now you’ll be writing more of them.  There are many names for these tests, I am calling them BATs now, but basically this is what most companies call automation.  Unfortunately, most companies leave this job to a QA automation engineer instead of the development teams.  Don’t get me wrong, QA automation engineers are great, but there aren’t many of them, good ones are very expensive, and the responsibility shouldn’t lie squarely on their shoulders.

BATs test the whole system working together.  BATs are your automated regression tests for the entire system.  BATs are automated customer acceptance tests and the ROI for each line for code in a BAT can be much higher than the ROI of each line of production code.

Why?  How is this even possible?  It’s all about leverage baby.  Each line of code in a BAT may be exercising anywhere from 5 to 500 lines of production code, which is quite the opposite case of a unit test where each line of unit test code might only be testing a 1/8th or 1/16th a line of production code on average (depending on code coverage numbers being reached.)

I’ll save the detail for later posts, but it is my strong opinion that a majority of a development teams effort should be put in BATs, because BATs

• Have high value to the customer
• Regression test the entire system
• Have a huge ROI per line of code (if you create a proper BAT framework)

Imagine how much higher quality your software would be if you had a BAT for each backlog item in your system which you could run every single iteration of your development process.  Imagine how confident you would be in making changes to the system, knowing that you have an automated set of tests that will catch almost any break in functionality.

Don’t you think that is worth giving up writing level 3 and level 4 unit tests, which are already painful and not very fun to begin with to achieve?

In my future posts on the Back to Basics series, I will cover in-depth how to push more of your code into level 1 and level 2 unit tests by extracting logic out to separate classes that have no dependencies, and I will talk more about BATs, and how to get started and be successful using them.  (Hint: you need a good BAT framework.)

As always, you can subscribe to this RSS feed to follow my posts on Making the Complex Simple.  Feel free to check out ElegantCode.com where I post about the topic of writing elegant code about once a week.  Also, you can follow me on twitter here.