Category Archives: Testing

Test Automation Framework Architecture

Test automation framework architecture efforts are often complete failures.

It’s true. I’ve worked with many companies who have given up on creating a good test automation framework architecture, because after investing a large amount of time and money and resources in doing it the wrong way, they have incorrectly assumed the entire effort is not cost effective.

In this post, I’m going to simplify the process of creating a test automation framework architecture and show you that—if you do it right—it can be a very good investment.

Test automation framework architecture basics

Let’s start off by talking about what the goals of a successful test automation framework architecture should be.

There are two goals I am interested in when creating a test automation framework:

  1. Having the ability to easily create simple automated tests that use the framework.
  2. Decoupling cosmetic application changes from the tests so that when the application changes, the tests do not all have to be updated

A good test automation framework architecture should at the very least provide these two important services. We create a test automation framework to allow us to support the creation of tests and to keep those tests from being dependent on the actual UI of our application—as much as possible.

I’m going to break down these goals just a little more to make sure we are all on the same page.

test automation framework thumb Test Automation Framework Architecture

Creating simple automated tests

First, we’ll start with having the ability to easily create simple automated tests that use the framework. Why do we care about this goal? And why would this be the responsibility of the framework?

I have found that one of the biggest failure points of test automation efforts is test complexity. The more complex the tests are, the harder they are to maintain. If tests are difficult to maintain, guess what? They won’t be maintained.

So, we really want to make sure that when we create a test automation framework architecture, we focus on making sure that test automation framework makes it as easy as possible for someone to create tests using it.

It is always a better choice to put the complexity into the framework instead of into the tests. The framework is usually maintained by developers and does not contain repeated code. But, tests are often created by QA or even business folks and the code found in tests is usually repeated across many tests, so it is vital to reduce the complexity on the tests, even if it means a large increase of complexity in the framework itself.

When I create a test automation framework architecture, my goal is to make it so the tests can read as close as possible to plain English. This requires some effort to pull off, but it is well worth it.

Decoupling the UI from the tests

Next, let’s talk about what I mean by decoupling cosmetic application changes from the tests.

This is also a vital goal of any test automation framework architecture, because if you fail to decouple the UI of an application from the tests, every single UI change in an application will cause possibly hundreds or thousands of tests to have to be changed as well.

What we really want to do is to make it so that our test automation framework architecture creates an abstraction layer that insulates the tests from having to know about the actual UI of the application.

At first this seems a little strange, especially when I tell automation engineers or developers creating an automated testing framework not to put any Selenium code into their actual tests.

All of the Selenium examples show you creating tests that directly use a web browser driver like Selenium, but you don’t want to write your tests that way—trust me on this one.

(By the way, I haven’t found a good book on creating an actual test automation framework architecture—I’ll probably write one—but, for now if you are looking for a good book on Selenium that starts to go into the topics I discuss here, check out Selenium Testing Tools Cookbook.)

Instead, what you want to do is to make it so the test automation framework is the only code that directly interacts with the UI of the application. The tests use the framework for everything they want to do.

So, for example:

Suppose you are creating a simple test to check to see if a user can log into your application.

You could write a test that looks something like this: (C# and Selenium in this example)

var loginInput = driver.FindElement(By.Id("txtUsername"));
loginInput.SendKeys("Joe");

var passwordInput = driver.FindElement(By.Id("txtPassword"));
passwordInput.SendKeys("$ecret");

var loginButton = driver.FindElement(By.Id("btnLogin"));
loginButton.Click();

But, what is going to happen when you change the ID of your “User Name” field? Every single test that uses that field will break.

On the other hand, if you properly create a test automation framework architecture that abstracts the UI away from the tests themselves, you would end up with a much simpler and less fragile test, like this:

LoginPage.GoTo();
LoginPage.LoginAs("Joe").WithPassword("$ecret").Login();

Now, if the ID of your “User Name” field changes, you’ll just change the code in your automated testing framework in one place, instead of changing 100 tests that all depended on the specific implementation of the UI.

A simple test automation framework architecture

Once you understand those two very important goals and why they are important, it is easier to think about how you should design a test automation framework architecture.

I’ve created quite a few test automation framework architectures, so I’ll give you a basic design I use for most of them.

Take a look at this diagram:

test automation framework architecture thumb Test Automation Framework Architecture

Here you can see that there are four layers to my test automation framework architecture.

Frist, we have the browser layer or the web app itself. This just represents your actual application.

Next, we have the Selenium or web driver layer. This layer represents your browser automation tool. Selenium is basically just a framework for automating a browser. It doesn’t know anything about testing, it is an API that lets you interact with the browser programmatically. (By the way, you don’t have to use Selenium. I just use it as an example here, because it is the most popular browser automation framework.)

After that, we have the framework layer. This is the actual framework you create which uses Selenium, or whatever web driver you want, to actually automate your application. The framework is acting as an abstraction between your application and the tests. The framework knows about the UI of your application and how to interact with it using Selenium. It is your job to create this layer.

Finally, we have the actual tests. The tests use the framework to manipulate your application and check the state of the application. These tests should be simple to understand and should not have to know anything about the actual implementation of the UI of your application. These tests should rely on the framework to give them the ability to do whatever they need to do in your application.

Now what?

Obviously, creating a test automation framework architecture is more complicated than what I have shown you in this article, but with these basics you should be off to a good start.

If you want a more in-depth and detailed explanation as well as an example of how exactly to create a full test automation framework architecture, you might want to check out my Pluralsight video on the topic:

Creating an Automated Testing Framework With Selenium

I also frequently blog about test automation topics and other topics related to becoming a better and more profitable software developer.

Sign up here to make sure that you don’t miss any of my posts and get access to the content I only share with my email list. (Don’t worry I hate spam as much as you do.)

If you have any question, post them in the comments below.

Selenium With Node.js and Mocha: Automated Testing With JavaScript

I’ve always found it a bit strange that we’ve had to use a language like C# or Java to write automated tests using Selenium.

Not that I dislike either of these languages, but—especially in modern JavaScript times—it is a little bit strange to write a web application in primarily JavaScript and then write the automated test for it using C#, Java, Ruby or Python.

That is why I was pretty excited to see that there were Selenium bindings for JavaScript that would allow you to write your automated tests in JavaScript and run them from Node. You can finally use Selenium with Node.

In this post I am going to take you through the process of getting setup and running a simple JavaScript based test using Selenium with Node.js and Mocha.

(The instructions here are for Windows, but you could easily follow most of this post if you are planning on running your tests on Mac or Linux—in fact, it should be much easier.)

Installing Node.js

Before we can get started, we first need to get Node.js installed.

Just head over to the Node.js site’s download page to find downloads for Node.js for each platform.

2014 01 31 16 05 04 thumb Selenium With Node.js and Mocha: Automated Testing With JavaScript

Just select the appropriate version and install.  You’ll need Node.js to run the JavaScript tests because you need some way to execute your JavaScript locally.

(If you are interested in learning more about Node.js, I recommend the excellent Node.js in Action book.)

Getting Selenium with Node working

Once we have Node installed, we need to get Selenium webdriver so that we can actually have a way to automate a browser.

Not so fast though, before we can get that part working, we’ll need to make sure we have the latest chrome driver.  Go download the Chrome driver first and put it in your PATH.

2014 01 31 16 17 01 thumb Selenium With Node.js and Mocha: Automated Testing With JavaScript

2014 01 31 16 18 13 thumb Selenium With Node.js and Mocha: Automated Testing With JavaScript

Now, we are ready to get the Selenium webdriver bindings that we’ll use to automate the browser from JavaScript.  The bindings will just expose the Selenium API through JavaScript, which we can access by importing a module in Node.

Go to the location where you are going to create your test project and either open up a command line or terminal window there.

We’ll use Node’s built-in package manager, or NPM, to install the Selenium webdriver package.

Just run “npm install selenium-webdriver” to install Selenium webdriver in the current project directory.

2014 01 31 16 24 07 thumb Selenium With Node.js and Mocha: Automated Testing With JavaScript

Let’s do a quick test to make sure everything is working so far.

Create a file named first_test.js in your project with the following code:

var webdriver = require('selenium-webdriver');

var driver = new webdriver.Builder().
   withCapabilities(webdriver.Capabilities.chrome()).
   build();

driver.get('http://www.google.com');
driver.findElement(webdriver.By.name('q')).sendKeys('simple programmer');
driver.findElement(webdriver.By.name('btnG')).click();
driver.quit();

Then, execute the code by running:

node first_test.js

You should see the Chrome browser popup and it should search for “Simple Programmer”, then quit.  (If it goes too fast, take out the driver.quit line.)

(Also, a good book in Selenium is Selenium Testing Tools Cookbook.  You can also check out my courses on Pluralsight.)

Adding some Mocha

Mocha is a test framework commonly used to write unit tests for JavaScript code, but we can also use it as a driver to drive our Selenium tests when using Selenium with Node.

We can get Mocha using the Node package manager as well, but for Mocha we want to use the –g option which will install the package globally instead of just locally to our project.  This will allow us to run Mocha on Windows without using Cygwin.

Go ahead and install Mocha by typing:

npm install –g mocha

2014 01 31 20 55 35 thumb Selenium With Node.js and Mocha: Automated Testing With JavaScript

Writing your first test

Now we are ready to write our first test using Mocha.

Create a new file in your directory called mocha_test.js with the following contents:

var assert = require('assert'),
test = require('selenium-webdriver/testing'),
webdriver = require('selenium-webdriver');

test.describe('Google Search', function() {
  test.it('should work', function() {
    var driver = new webdriver.Builder().
    withCapabilities(webdriver.Capabilities.chrome()).
    build();
driver.get('http://www.google.com');
    var searchBox = driver.findElement(webdriver.By.name('q'));
    searchBox.sendKeys('simple programmer');
    searchBox.getAttribute('value').then(function(value) {
      assert.equal(value, 'simple programmer');
    });
	driver.quit();
  });
});

This simple test will test that we can type “simple programmer” into Google’s search box and verify that the text is there.

A pretty simple test, but it demonstrates the basic skeleton of creating an automated test with Mocha using Selenium with Node.js.

You can execute the test by typing:

mocha mocha_test.js

If the test executes too quickly, you can comment out the driver.quit to keep the browser window open.

2014 01 31 21 03 11 thumb Selenium With Node.js and Mocha: Automated Testing With JavaScript

Going further…

Well, that should get you started with using JavaScript and Node to run a Selenium test driven by Mocha.

You can use this basic setup to create much more robust automated tests.  I call these kinds of tests blackbox automated tests or BATs(Check out my series if you want to learn more about creating a automated testing framework.)

And if you liked this post, go ahead and sign up here to make sure you don’t miss other content like this.

Creating an Automated Testing Framework With Selenium

I have another new course on Pluralsight, check out:

Creating an Automated Testing Framework With Selenium

2013 09 28 16 31 58 Creating an Automated Testing Framework With Selenium

I am very excited to finally get this course out.  Many viewers have been asking me to make a comprehensive course that actually shows you how to build a real automation framework, and I finally did it in this course.

I reveal all my tricks and tips here and tell you exactly what I have done in the past to build successful automation frameworks.  It was a lot of fun to create this course and it brought me back to the glory days of creating automated tests.  (This is some serious fun!)

Anyway, check out the course and let me know what you think.

Here is the official course description:

Learning how to use a tool like Selenium to create automated tests is not enough to be successful with an automation effort. You also need to know how to build an automation framework that can support creating tests that are not so fragile that they constantly break. This is the real key to success in any automation effort.

In this course, I will reveal every secret I know from creating several successful automation frameworks and consulting on the creation of others. I will show you exactly, step-by-step how to create your own automation framework and I will explain to you the reasoning behind everything we are doing, so you can apply what you learn to your own framework.

We’ll start off this course by going over the basics of automation and talking about why it is so important as well as discuss some of the common reasons for success and failure.

Then, I’ll take you into the architecture of an automation framework and show you why you need to pay careful attention to the structure of any framework you build and give you some of the underlying design principles I use when creating an automation framework.

After that we’ll be ready to start creating a framework. In the next few modules, I’ll show you how to create a real automation framework capable of automating the WordPress blogging platform administrative console. We’ll start off by creating smoke tests and using those smoke tests to build out our initial framework.

Then, we’ll expand the capabilities of our framework as we create more tests and learn how to use techniques like dummy data generators to make our tests as simple and easy to read as possible.

Finally, I’ll take you through some best practices and tips that cover topics like scaling out, working in Agile environments and other important issues you are likely to face. If you are responsible for an automation project for a web application or you want to start using automation, you’ll definitely want to check this course out.

The More I Know, the Less I Know

I used to be very confident in my abilities as a software developer.

I used to be able to walk up to a group of software developers and tell them exactly what they were doing wrong and exactly what was the “right” way to do things.

I used to be sure of this myself.

confident thumb The More I Know, the Less I Know

It wasn’t even that long ago.  Heck, when I look at the blog posts I wrote 3 years ago I have to slap myself upside my head in realization of just how stupid I was.

Not only was my writing bad, but some of my thoughts seem so immature and uneducated that it feels like a completely different person wrote them.

And I wrote those posts back when I knew it all.

The more I learn, the less I know

Lately I’ve been running into situations more and more often where I don’t have a good answer for problems.

I’ve found myself much more often giving 3 pieces of advice attached with pros and cons rather than giving a single absolute—like I would have done perhaps 3 years ago.

I’ve been finding as I have been learning more and more (the past 3 years have been an accelerated pace of growth for me,) that I am becoming less and less sure of what I know and more and more convinced that there is no such thing as a set of best practices.

I’ve even spent some time postulating on whether or not commonly held beliefs of best practices would be thrown completely out the window given a significant enough motivation to succeed.

My point is that the more doors I open, the more aware I become of the multitude of doors that exist.

doors thumb The More I Know, the Less I Know

It is not just the realization of what I don’t know, but also the realization of weakness of the foundation I am already standing on.

Taking it out of the meta-physical

Let’s drop down out of the philosophical discussion for a bit and talk about a real example.

Perhaps the biggest quandary I struggle with is whether or not to unit test or practice TDD and its variants.

The 3 years ago know-it-all version of me would tell you emphatically “yes, it is a best practice and you should definitely do it all the time.”

The more pragmatic version of me today says, in a much more uncertain tone, “perhaps.”

I don’t want to delve into the topic in this post since I am sure I could write volumes on my ponderings in this area, but I’ve come to a conclusion that it makes sense to write unit tests for code that has few or no dependencies and that it does not make sense to do so for other code.

From that I’ve also derived that I should strive to write code that separates algorithms from coordinators.

I still even feel today that my advice is not wholly sound.  I am convinced it is a better approach than 100% TDD and units tests, or no TDD and unit tests, but I am not convinced there isn’t a deeper understanding and truth that supersedes my current thoughts on the matter.

As you can imagine this is quite frustrating and unsettling.

Silver bullets and best practices

What I am coming to realize more and more is that there are no silver bullets and more surprisingly there are not even any such things as best practices.

silverbullet thumb The More I Know, the Less I Know

Now I’ve heard the adage of there being no silver bullets so many times that it makes me physically sick when I hear someone say it, because it is so cliché.

But, I’ve had a hard time swallowing the no best practices pill.

I feel like when I abandon this ship then I am sailing on my life raft in the middle of a vast ocean with no sails and no sense of what direction to go.

A corner-stone of my development career has been in the learning, applying and teaching of best practices.  If these things don’t exist, have I just been pedaling snake oil and drinking it myself?

No.

Best practices are simply concrete applications of abstract principles in software development that we cannot directly grasp or see clearly enough to absolutely identify.

Breaking this down a bit, what I am saying is that best practices are not the things themselves to seek, but through the pursuit of best practices we can arrive at a better understanding of the principles that actually are unchanging and absolute.

Best practices are optimal strategies for dealing with the problems of software development based on a particular context.  That context is primarily defined by:

  • Language and technology choice
  • Meta-game (what other software developers and perceived best practices are generally in place and how software development is viewed and understood at a given time.)
  • Individual skill and growth (what keeps me straight might slow you down; depends on where you are in your journey.)

There is a gentle balance between process and pragmatism.

When you decide to make your cuts without the cutting guide, it can make you go faster, but only if you know exactly what you are doing.

Where I am now

Every time I open my mouth I feel like I am spewing a bunch of bull crap.

I don’t trust half of what I say, because I know so much of it is wrong.

Yet I have perhaps 10 times more knowledge and quite a bit more experience in regards to software development than I did just 3 years ago.

So what gives?

Overall, I think I am giving better advice based on more practical experience and knowledge, it is just that I am far more aware of my own short-comings and how stupid I am even today.

I have the curse and blessing of knowing that only half of what I am saying has any merit and the other half is utter crap.

Much of this stems from the realization that there are no absolute right ways to do things and best answers for many of the problems of software development.

I used to be under the impression that someone out there had the answer to the question of what is the right way to develop software.

clues thumb The More I Know, the Less I Know

I used to think that I was picking up bit of knowledge, clues, that were unraveling the mystery of software development.  That someday I would have all the pieces of understanding and tell others exactly how they should be developing software.

What I found instead was that not only does nobody know the “right” way to develop software, but that it is perhaps an unknowable truth.

The best we can do is try to learn from obvious mistakes we have made before, start with a process that has had some level of success, and modify what we do based on our observations.

We can’t even accurately measure anything about software development and to think we can is just foolishness.

From story points, to velocity, to lines of code per defect and so on and so forth, all of those things are not only impossible to accurately measure, but they don’t really tell us if we are doing better or not.

So, what is my point?

My point is simple.

I have learned that not only do I not have all the answers, but I never will.

What I have learned is always subject for debate and is very rarely absolute, so I should have strong convictions, but hold onto them loosely.

And most importantly, don’t be deceived into thinking there is a right way to develop software that can be known.  You can improve the way you develop software and your software development skills, but it will never be based on an absolute set of rules that come down from some magical process or technology.

If you like this post don’t forget to or subscribe to my RSS feed.

There Are Only Two Roles of Code

All code can be classified into two distinct roles; code that does work (algorithms) and code that coordinates work (coordinators).

The real complexity that gets introduced into a code bases is usually directly related to the creation of classes that group together both of these roles under one roof.

I’m guilty of it myself.  I would say that 90% of the code I have written does not nicely divide my classes into algorithms and coordinators.

Defining things a bit more clearly

Before I dive into why we should be dividing our code into clear algorithmic or coordinating classes, I want to take a moment to better define what I mean by algorithms and coordinators.

Most of us are familiar with common algorithms in Computer Science like a Bubble Sort or a Binary Search, but what we don’t often realize is that all of our code that does something useful contains within it an algorithm.

What I mean by this is that there is a clear distinct set of instructions or steps by which some problem is solved or some work is done.  That set of steps does not require external dependencies, it works solely on data, just like a Bubble Sort does not care what it is sorting.

Take a moment to wrap your head around this.  I had to double check myself a couple of times to make sure this conclusion was right, because it is so profound.

It is profound because it means that all the code we write is essentially just as testable, as provable and potentially as dependency free as a common sorting algorithm if only we can find the way to express it so.

What is left over in our program (if we extract out the algorithms) is just glue.

Think of it like a computer.  Computer electronics have two roles: doing work and binding together the stuff that does the work.  If you take out the CPU, the memory and all the other components that actually do some sort of work, you’ll be left with coordinators.  Wires and busses that bind together the components in the system.

Why dividing code into algorithms and coordinators is important.

So now that we understand that code could potentially be divided into two broad categories, the next question of course is why?  And can we even do it?

Let’s address why first.

The biggest benefit to pulling algorithmic code into separate classes from any coordinating code is that it allows the algorithmic code to be free of dependencies.  (Practically all dependencies.)

Once you free this algorithmic code of dependencies you’ll find 3 things immediately happen to that code:

  1. It becomes easier to unit test
  2. It becomes more reusable
  3. Its complexity is reduced

A long time ago before mocks were widely used and IoC containers were rarely used, TDD was hard.  It was really hard!

I remember when I was first standing on the street corners proclaiming that all code should be TDD with 100% code coverage.  I was thought pretty crazy at the time, because there really weren’t any mocking frameworks and no IoC containers, so if you wanted to write all your code using TDD approaches, you’d actually have to separate out your algorithms.  You’d have to write classes that had minimal dependencies if you wanted to be able to truly unit test them.

Then things got easier by getting harder.  Many developers started to realize that the reason why TDD was so hard was because in the real world we usually write code that has many dependencies.  The problem with dependencies is that we need a way to create fake versions of them.  The idea of mocking dependencies became so popular that entire architectures were based on the idea and IoC containers were brought forth.

mp900175522 thumb There Are Only Two Roles of CodeWe, as a development community, essentially swept the crumbs of difficult unit testing under the rug.  TDD and unit testing in general became ubiquitous with writing good code, but one of the most important values of TDD was left behind, the forced separation of algorithmic code from coordinating code.

TDD got easier, but only because we found a way to solve the problems of dependencies interfering with our class isolation by making it less painful to mock out and fake the dependencies rather than getting rid of them.

There is a better way!

We can still fix this problem, but we have to make a concerted effort to do so.  The current path of least resistance is to just use an IoC container and write unit tests full of mocks that break every time you do all but the most trivial refactoring on a piece of code.

Let me show you a pretty simple example, but one that I think clearly illustrates how code can be refactored to remove dependencies and clearly separate out logic.

Take a look at this simplified calculator class:

 public class Calculator
    {
        private readonly IStorageService storageService;
        private List<int> history = new List<int>();
        private int sessionNumber = 1;
        private bool newSession;

        public Calculator(IStorageService storageService)
        {
            this.storageService = storageService;
        }

        public int Add(int firstNumber, int secondNumber)
        {
            if(newSession)
            {
                sessionNumber++;
                newSession = false;
            }

            var result = firstNumber + secondNumber;
            history.Add(result);

            return result;
        }

        public List<int> GetHistory()
        {
            if (storageService.IsServiceOnline())
                return storageService.GetHistorySession(sessionNumber);

            return new List<int>();
        }

        public int Done()
        {
            if (storageService.IsServiceOnline())
            {
                foreach(var result in history)
                    storageService.Store(result, sessionNumber);
            }
            newSession = true;
            return sessionNumber;
        }
    }

 

This class does simple add calculations and stores the results in a storage service while keeping track of the adding session.

It’s not extremely complicated code, but it is more than just an algorithm.  The Calculator class here is requiring a dependency on a storage service.

But this code can be rewritten to extract out the logic into another calculator class that has no dependencies and a coordinator class that really has no logic.

 public class Calculator_Mockless
    {
        private readonly StorageService storageService;
        private readonly BasicCalculator basicCalculator;

        public Calculator_Mockless()
        {
            this.storageService = new StorageService();
            this.basicCalculator = new BasicCalculator();
        }

        public int Add(int firstNumber, int secondNumber)
        {
            return basicCalculator.Add(firstNumber, secondNumber);
        }

        public List<int> GetHistory()
        {
            return storageService.
                   GetHistorySession(basicCalculator.SessionNumber);
        }

        public void Done()
        {
            foreach(var result in basicCalculator.History)
                storageService
                     .Store(result, basicCalculator.SessionNumber);

            basicCalculator.Done();
        }
    }

    public class BasicCalculator
    {
        private bool newSession;

        public int SessionNumber { get; private set; }

        public IList<int> History { get; private set; }

        public BasicCalculator()
        {
            History = new List<int>();
            SessionNumber = 1;
        }
        public int Add(int firstNumber, int secondNumber)
        {
            if (newSession)
            {
                SessionNumber++;
                newSession = false;
            }

            var result = firstNumber + secondNumber;
            History.Add(result);

            return result; ;
        }

        public void Done()
        {
            newSession = true;
            History.Clear();
        }
    }

 

Now you can see that the BasicCalculator class has no external dependencies and thus can be easily unit tested.  It is also much easier to tell what it is doing because it contains all of the real logic, while the Calculator class has now become just a coordinator, coordinating calls between the two classes.

This is of course a very basic example, but it was not contrived.  What I mean by this is that even though this example is very simple, I didn’t purposely create this code so that I could easily extract out the logic into an algorithm class.

Parting advice

I’ve found that if you focus on eliminating mocks or even just having the mindset that you will not use mocks in your code, you can produce code from the get go that clearly separates algorithm from coordination.

I’m still working on mastering this skill myself, because it is quite difficult to do, but I believe the rewards are very high for those that can do it.  In code where I have been able to separate out algorithm from coordination, I have seen much better designs that were more maintainable and easier to understand.

I’ll be talking about and showing some more ways to do this in my talk at the Warm Crocodile conference next year.

Are You Really Sure You Want to Make a Cancel Button?

Are you really sure you want to create the cancel button for your application?

You know, I might click it.

Not only might I click it, I might click it at the most inopportune time.

I might click it right in the middle of  that large file that you are copying, right after you spun up that 2nd thread.

exploding earth thumb Are You Really Sure You Want to Make a Cancel Button?

Cancel is a commitment

The next time you consider creating a cancel button, I suggest you think of it as a commitment.

In my world the cancel button has two promises.

  1. Stop what is currently happening in a non-destructive way.
  2. Stop RIGHT the F NOW!

I’ve been encountering a good deal of cancel button which don’t obey the basic laws of cancelling that any user would expect.

I have actually been randomly clicking “cancel” whenever I see it, much to my coworker’s dismay.

I started doing this, because I wanted to see how widespread the cancel cancer is.

And it is pretty widespread.  A majority of cancel buttons I have been testing neither stop right away or prevent destruction.

I found instead that clicking the “cancel” button in most programs is sure to hang that program for an extended period of time, ultimately causing you to kill the process, and it tends to leave processes half done without rolling them back.

Clearing things up

Let me be a bit clear here.  I am not talking about cancel buttons you see in an “Ok / Cancel” dialog.  Most of the time those cancel buttons actually work, because they are really operating as “back” buttons, they aren’t actually cancelling a process that is happening live.

I am talking mainly about cancel buttons that cancel an active ongoing activity.  For example, cancel buttons in an installer or during an SVN update.

We could call these kinds of cancel buttons “asynchronous cancel buttons.”

But, I need to provide a way for the user to cancel

Good, but don’t lie about it.

There are certain things that just can’t be cancelled.

When I get on a plane, I can cancel my trip when I am sitting there waiting for the door to close.  I can even cancel my trip while the plane is taxing to the runway, if I yell “I have been RAPED by a BOMB that is on FIRE!

But, I can’t cancel my trip, once the plane has lifted off the ground.  If I try to cancel it then… well, bad things would happen.  Very bad things.

So how come when I am installing your software, or your software is updating its database, I have this shiny little cancel button I can click at any time during that process?

Surely I cannot cancel just any time!

Surely there are parts of the process that cancelling would be fatal or it is too late to rollback.

My point is, if the user truly can’t cancel, don’t present a button that says they can.  More specifically, if you can’t obey the laws of cancel 1 and 2 from above, don’t even show a button.  Just say “sorry, you can’t cancel this process at this time.”

I don’t even need to know why I can’t cancel.  I mean, it will make me feel better if you tell me that the Unicorn Glitter Engine is in a critical state and any disruptions could end life as we know it, but I’ll settle for you just greying out that cancel button or not displaying it at all.

cancel button thumb Are You Really Sure You Want to Make a Cancel Button?

Putting back on the developer hat

I’m guilty of it myself.  I know I have created cancel buttons in the past that have caused pain and anguish.

But what can we do about it as developers?

First off, we should be thinking carefully about breaking a long running process into steps.  At each step of the way we should consider if it is conceivable to cancel that step without destroying data and hanging the application.

In any long running process, we should be able to identify certain parts which are cancellable and those which do not make sense to cancel.

It is your job as the developer to ensure that if you decide to allow for cancelling that the cancel undoes the existing process and work immediately.

I cannot stress this enough!

This is the behavior that most users expect and the very meaning of the word cancel.

To do this might take extra work.  You might have to think a bit more about the threading situation of your application.  You might have to think about rollback situations.  But, if you don’t do it, your cancel button will become just like the boy who cried wolf and no one will believe them.

And if you’re not willing to go through this effort, at the very least, be kind enough to your users to just remove the cancel button, because you can bet I’ll be clicking it!

Getting up to BAT: Building a True DSL

If you’ve made it this far with your BAT implementation, you have finally arrived.

Not to say that you’ll ever be done expanding your automation framework and building and improving your BATs, but you are at the point of having a mature usable BAT framework and you can be proud of that accomplishment.

If you truly want to see where the rabbit hole goes though… Read on

Taking it to 11

spinal tap but it goes to eleven thumb Getting up to BAT: Building a True DSL

If you have designed a good automation framework for writing your BATs, you will most likely have created an internal domain specific language (DSL).

What is an internal DSL?

Basically it is a set of APIs and rules built in another language that function as their own language.

You might have some code that looks like this:

Pages.Login.Goto();
Pages.Login.LoginWithDefaultUser();
Pages.Customers.Goto();
Pages.Customers.AddNewCustomer();

 

You can see how this code is really looking like a language itself.  It is hard to tell if this code is even C# or Java.

This is a good example of an internal DSL.

If we were to take this same section of a BAT test and write it as an external DSL, it might look something like this:

LOGIN WITH DEFAULT-USER
ADD-CUSTOMER WITH DEFAULT-CUSTOMER

 

With a true DSL we can simplify even further to get right down to the bare minimum set of commands we need to execute what we want in our application.

How to implement a DSL?

There are a number of tools and techniques that can be used to create a DSL.  I’ll hit on some of those a bit later on, but first we need to design the DSL itself.

Designing a DSL can be challenging, but I have found the best approach is to use a similar approach to what we used to design our automation framework.

It is best to work backwards from the syntax that would make your BAT the easiest to write and the most clear.  Once you have done that you can worry later about implementing the DSL to support the syntax you have come up with.

You should focus on creating a limited vocabulary for your DSL so that it will be easy for someone to learn and know what their options are.

The more restrictions we put into a DSL the easier that DSL becomes for doing a specific purpose.

We want to give the user of a DSL no more options than required for what they want to do and no less.

This isn’t always easy to do.  It often requires some careful thought and refinement.

Your DSL should also use terminology and phrases that are familiar to the business domain of your application.  By doing this you will make it so someone understanding your application will understand BATs written in that DSL.

I also always try to make a BAT DSL read as much as plain English as possible.

Once you have designed the DSL, you will need a way to implement it. 

I’ve written about using ANTLR to create an external DSL.  Also I have written a review on a book that I recommend for learning ANTLR.

Microsoft also has a DSL SDK you can use in VS2010, although I haven’t used it myself.

The basic idea behind implementing a DSL is that you have to take some sort of grammar you design for your DSL, parse it into a syntax tree, then use that tree to generate code that will actually perform the actions.

It is by no means a trivial task, but using a tool like ANTLR, makes it much simpler than trying to completely generate a parser and execution engine yourself.

Why even bother?

Creating a DSL for your BATs is completely optional, but there are some significant benefits to doing so.

The biggest one I have found is that you are much more likely to get business people to write BATs if you have a simple DSL they can use to create them.  Once you get everyone using BATs to define behavior and requirements, communication becomes much more simple.

Another great thing about creating a DSL is that you can create a runner that can interpret the DSL BAT scripts which can be run on any machine.  Typically, to run BATs created in Java or C#, you would need a C# or Java compiler and a test runner, etc.  By creating a DSL and a stand alone executor, you can very easily allow anyone to run BATs with little to no setup.

Writing your BATs in a DSL will also make writing the BATs and understanding them much easier.  A good DSL can get rid of all the little language and API nuances that don’t really add information to the specific domain you are working in.  A good DSL strips information down to the minimum amount needed and is able to do so, because it assumes the context is your application’s business domain.

You will also find that you are better able to separate framework from tests when you have a DSL in place.  One issue that many teams face when creating a BAT framework is that some people come along and write lower level code in the higher level BATs.  This creates a problem because it is hard to force someone to use the higher level API when the framework and the BAT code are the same language.

Wrapping it up

So that completes my series on Getting up to BAT.

Hopefully you have learned enough to be able to implement your own BAT framework and get BATs running for your application.

I tried to cover most of the important steps here without getting too deep into the details.

If you have any questions or are interested in getting more in depth training or consulting on getting BATs setup for your environment, send me an email or leave a comment.

It can be a little difficult to get started and get a BAT framework up and running, but the rewards are well worth the effort!

As always, you can subscribe to this RSS feed to follow my posts on Making the Complex Simple.  Feel free to check out ElegantCode.com where I post about the topic of writing elegant code about once a week.  Also, you can follow me on twitter here.

Getting up to BAT: Scaling Out

If you haven’t been following, it has been a while since my last post on this topic.  I had a bit of distraction the last few weeks, but I am back and ready to go!

When we last left off we had just gotten our BATs as part of the acceptance criteria for any new backlogs that are worked on.  This puts us at a point where we could really say that we have successfully implemented BAT testing.

You don’t want to get too comfortable just yet though, because the next hurdle you will most likely face will be the problem of not having enough time to execute all your tests.

You want to think about this ahead of time

Nothing worse than getting everything going and then not being able to execute the entire test suite, because you didn’t plan ahead.

You don’t want to get to the point where confidence in your BAT suite it lost because you are not able to get all the tests executed in a reasonable amount of time.

The more frequently your tests are run the more value they have.

By reducing cycle time from the time a breaking change is made in the system and the time it is discovered, you greatly reduce the risk it imposes on your software and you decrease the scope of the code changes in which it could have occurred.

To put it simply, the faster you can find out you broke something, the more likely you can fix it before it does damage, and the more likely you will be to know what thing you did caused the breakage.

clarkson face thumb Getting up to BAT: Scaling Out

How can we reduce cycle time?

There are a few different strategies we can employ and we can mix and match some of these strategies.

Straight forward parallelization

The best and most effective thing you can do is to take a given test run, split up the execution of those tests on multiple machines and execute them in parallel.

This approach is going to give you the best bang for your buck.  You should really try to get some amount of parallelization going before attempting any other solution, since it is going to make a huge impact on the total execution time for your tests without making any sacrifices.

There are even many ways you can mix and match to do parallelization:

  • Use multiple physical machines
  • Run many virtual machines on a single host
  • Put an executor program on every team member’s machines that will execute tests when that machine is idle or put into test execution mode (perhaps at night)
  • Use a cloud based computing platform to execute your tests
  • Run multiple browser instances on a single machine

Data preloading

With this approach, you would preload some of the test data that your tests might generate by manually clicking through screens.

This is fairly hard to explain, so let me give you an example:

Suppose you had a set of tests that all involved creating customers, but each customer you create in the system takes about 3 minutes to create by clicking through the screens to get them into the system.

We don’t need to have 500 tests all executing the same exact logic in the system 500 times for 3 minutes just to generate all the customer data that will be used in the tests.

Instead, we can leave a few tests that are exercising the customer creation functionality, and we can execute a SQL script to push all the other customer data into the test database for the other tests to use.

Using this technique we might be able to reduce our total execution time by 3 minutes * each test, or about 25 hours for 500 tests.

This can be a huge savings in time, and it doesn’t come at that high of a cost.  The sanctity of our tests is slightly compromised, but we are taking a calculated risk here knowing that we already have covered the area of execution which we are preloading data for.

Consider this technique when you notice certain things in the system taking a very long time to do.

Test runs by area

With this technique, we can reduce the total execution time in a given period by splitting up test areas and running them either at different times or in response to changes in certain areas.

You have to be very careful with this approach, because if you don’t do it correctly, you can start to erode some of the safety your BAT tests are providing you.

I would only do something like this as a last resort, because it is so easy to virtualize today, and hardware is so cheap.

I’d much rather run too many tests than too few.

Test randomization

With test randomization, we are going to take our total desired execution time, and divide it by the average time for running a test.  We then can use that number of tests to run to randomize the execution of our tests each time we run them and only run the number or tests that will fit in the desired execution time.

This choice is also a compromise that I typically don’t like to take.

It can be very useful though, combined with other techniques when you still don’t have enough time to execute your entire test suite.

The basic idea here is that you are going to randomly run tests each time to fill up the time you have to run tests.

Test reduction

This one seems fairly obvious, but can be very helpful.

Often I will see teams starting out with automation, trying to write way too many BAT tests for a given feature.  Sure, with automated tests it is possible for run a test for every single possible combination of values in your 5 drop downs, but will it really benefit you?

In many cases you have to think about what you are trying to protect against with your BATs.  Sometimes running every combination of data selection choices is going to be important, but other times you are only going to need to write tests to test a few of the happy path scenarios.

It is important to find a balance between test coverage and test volume and not just for execution time.  There is a logistical overhead to having a large volume of mostly redundant tests.

So even though this technique might seem dangerous and counter-productive, I will almost always employ it to some degree.

Common pitfalls

Here are some of the things you might want to watch out for as you are scaling out and streamlining your execution of BATs:

  • Parallelization issues.  If you are using shared state, you can run into big trouble when your tests are executing in parallel.  There are many manifestations of this.  You could have issues at the database level or at the local machine memory level.  The best way to avoid this kind of problem is to use separate virtual machines for each test execution, and not reuse data setup between test cases.
  • Ineffective error reporting.  If you run a huge volume of tests in parallel, you better have a good way to sort through the results.  It is much harder to figure out why things failed when they are run across multiple machines.
  • Test order dependencies.  Make sure tests don’t rely on being run in a certain order or you will have lots of pain when you disrupt that order.
  • Environment setup.  Make sure all your test execution environments are exactly the same unless you are specifically testing different environments for execution.  You don’t want tests failing on one machine but passing on another.

As always, you can subscribe to this RSS feed to follow my posts on Making the Complex Simple.  Feel free to check out ElegantCode.com where I post about the topic of writing elegant code about once a week.  Also, you can follow me on twitter here.

Getting up to BAT: Adding BATs to Your Acceptance Criteria

So now that we have built our automation framework, and got our smoke tests running as part of the build, we now need to make sure new BATs are being created for new backlogs.

This is definitely one of the more challenging tasks you will face, and perhaps the most critical.

All of your efforts will be in vain if you can’t get to the point of creating new BATs for new backlogs that are finished, because if you don’t you’ll still be doing manual testing, and you’ll be playing catch-up all the time.

Start off small

The goal here is to eventually get to the point where every single backlog requires BATs to be written and passing in order for that backlog to be called “done.”

If you try to drop this lofty goal onto a team in one iteration, you are likely to get beaten and slapped with much vigor.

Instead, start with a simple goal of requiring that at least one backlog have BATs for all of the functionality of that backlog.  Start with this small goal, but enforce it with rigor.

By starting out small like this, you will get the team accustomed to the idea of building automated tests at the same time as building the functionality and you will give them a chance to have a small success without sacrificing much velocity.

The goal at this point should be to get people excited about writing BATs along with the backlog.

It’s actually quite fun

I have found that developers tend to really enjoy writing automated tests.  Make sure you let everyone who wants to share in the fun, not just QA and not just developers.

It is really very exciting to see the “magic” of an automated test clicking through buttons on your web page and doing all kinds of things that seemed so tedious before.

I have found that this is so much fun, that you might have the problem of developers not wanting to work on the real backlog, but rather to write the automated tests for it.

This is good, you really want to foster this attitude, it will greatly help when you finally…

Drop the bomb!

What bomb?  The bomb that says on it:

All backlogs will require BATs as acceptance criteria.  It’s not done unless it has automated tests that prove its functionality.

You should be getting to this point within 3 to 4 iterations on average.  Don’t go too fast to get here, but don’t go too slowly either.

I know this seems like an impossible task, but it really isn’t.  I’ve been here before and had much success with it on several different teams.  The key is making sure everyone is comfortable with writing automated tests and understands the value.

Whatever you do though, once you draw this line in the sand, Do not back down! I mean it.

You might start getting death threats.  People might directly laugh in your face all the while spitting skittle colored loogies in your coffee, daring you to “taste the rainbow,” but you have to hold your ground.

skittlesny6 thumb Getting up to BAT: Adding BATs to Your Acceptance Criteria

It will be worth it in the long run, because you will have built an awesome safety net for proving the functionality of your software over time.

It’s not all rainbows and butterflies

You are going to sometimes run into scenarios where automation will be too high of a price for a particular backlog.

This will especially happen in cases where you have some kind of process that spans multiple domains or software systems.

When you run into those situations, give it a solid effort to at least automate as much as possible, but don’t die on the hill and kill the whole automation project.  We have to be pragmatic or risk having our credibility called into question.

Don’t expect everyone to embrace writing BATs overnight, or to even be good at it.  When you first introduce this to the team, you are going to have a learning curve and your automation framework is still going to have some pretty big holes in it.

But, that is OK.  Over time you will build out that framework and produce some fierce battled scarred veterans able to automate web pages while scowling at you from afar, just don’t lose hope and keep beating that drum!

As always, you can subscribe to this RSS feed to follow my posts on Making the Complex Simple. Feel free to check out ElegantCode.com where I post about the topic of writing elegant code about once a week. Also, you can follow me on twitter here.

Getting up to BAT: Adding Smoke Tests to Your Build

Once you’ve built some smoke tests with your shiny new automation framework, you are going to want to get those smoke tests up and running as soon as possible…

But!  You might want to consider holding off for a second and reading this post!

It is worth taking a bit of time and thinking a bit about the strategy and psychology of adding the smoke tests to your regular build process.

There are several things we are going to want to discuss and think about before adding automated tests to a build.

build a site2 thumb Getting up to BAT: Adding Smoke Tests to Your Build

Why are we adding the smoke tests to the build?

I think it is always important that before we do something we ask ourselves why we are doing it.  Doing things, because someone said it is good, or because it seems right is not a very good reason.

The primary benefit we are going to get by adding smoke tests to the build now instead of waiting for a larger portion of tests to be written is that we are going to be able to immediately tell if a basic functionality of the system is broken instead of finding out after we have deployed the build to different environments and used it.

We want to be able to get this value as soon as possible.  We don’t want to have to wait until we have a huge amount of tests, because even a few tests will give us some value in this area.

The other reason we are adding the tests to the build is so that we can notify developers when a test fails, so that they know to fix whatever was broken.  By having the smoke tests run as part of the build, we are able to reduce the amount of time before a break is introduced and the break is discovered.

This concept of reducing the delta between introduction and discovery is very important, because it makes it much easier to identify what caused a break.

Perhaps a distant 3rd reason to add the smoke tests at this stage, is to prove out our framework and technologies.  Better to start small with a small number of tests and get everything working.

With those ideas in mind, we can move onto some of the more important considerations.

Don’t ever add failing tests to the build

When you add your smoke test suite to the automated build, all of the tests should pass!

I will say it again, because it is really important and ignoring this advice may doom your entire BAT project.

Do not ever add failing tests to the build!

It seems like an innocent enough thing, but it causes great harm, because it is very much like the boy who cried wolf.  Consider what happens when we introduce our new BAT system and we add our smoke tests to the build, but we have 3 failing tests.

First thing that will happen is that our tests will run during the next build and 3 will fail.  This should cause a build failure notification of some sort to go to the team, at which time you will be in the uncomfortable position of saying something to the effect of “ignore that fail, those are real defects, but they are known issues.”

Next, let’s assume a developer checks in some bad code and it causes a 4th test to fail.

Do you think most people will notice this 4th failing test?

Even if they do, do you think they will just assume it is another known issue?

Worse yet, they may think your BATs are meaningless and just fail randomly.

Do you see where I am going with this?  We want to start off with the correct expectations and we want test failures to be meaningful.  To ALWAYS be meaningful.  The second you have test failures that are not meaningful, you lose credibility for any future meaningful failures.

This is such a critical psychological battle with build systems and automated tests that I have often implemented a policy where no one is allowed to work on anything else but fixing the build or test failure when a build fails.  When you have 20 developers either sitting idle or working on one issue for 2 hours, it sends a pretty clear message that we take build failures seriously.

Make sure your tests never undermine this message.

What do I do with failing tests then?

If they are real defects, put them in your defect tracking system or create backlogs for them and make sure the tests are disabled until those defects are fixed.

Your automated tests should be acting as regression tests for things that are already working.  It is absurd to think that it is even close to cost effective to write automated tests to actually functionally test new features or code!

It is the process of writing the automated test that should uncover defects.  Running of automated tests should uncover new defects where functionality has regressed from working to not working.

Other considerations

Here are a list of some other considerations you will want to think about before embarking on this journey.

  • How will you actually run the tests? (Will you use the build script to kick it off, some 3rd party plugin, etc)
  • Where will you run the tests? (Will you use the build server itself, or another machine.  You probably don’t need to worry about scaling out to multiple machines right now, but you should at least keep it in the back of your mind.)
  • How long will it take to run the smoke tests? (Fast is better, because feedback cycle is reduced.  If you have a really long test suite, you might want to pare down the smoke tests and run some with every build, and all of them every night for now.)
  • Will developers be able to run the tests locally? (Make sure you consider how to do this before putting the tests into the build.  Nothing more frustrating than being measured against a bar you can’t yourself see.)
  • How will you notify and report results from the test failures? (We don’t care about passes, only failures, but you need a good way to report the result that prompts action.)
  • What is the policy for test failures? (Make sure there is some agreement on what kind of action will be taken when tests fail.  A great place to start, if you can get management to buy in on it, is all work stops until all tests pass.  Extreme, but extremely effective.)
  • How will you prevent false fails and identify them? (Don’t get stuck with egg on your face.  If your framework fails and the problem is not a real defect, you need to be able to quickly identify it and fix the framework problem.  There are a few strategies for doing this, but that is for another post.)
  • How do new tests get added and commissioned? (You are going to be constantly growing your test suite, so you will need a good method for adding tests to the smoke test suite.)

Remember, you only get one chance at a first impression!

As always, you can subscribe to this RSS feed to follow my posts on Making the Complex Simple.  Feel free to check out ElegantCode.com where I post about the topic of writing elegant code about once a week.  Also, you can follow me on twitter here.