Category Archives: Functional

Are You Really Sure You Want to Make a Cancel Button?

Are you really sure you want to create the cancel button for your application?

You know, I might click it.

Not only might I click it, I might click it at the most inopportune time.

I might click it right in the middle of  that large file that you are copying, right after you spun up that 2nd thread.

exploding earth thumb Are You Really Sure You Want to Make a Cancel Button?

Cancel is a commitment

The next time you consider creating a cancel button, I suggest you think of it as a commitment.

In my world the cancel button has two promises.

  1. Stop what is currently happening in a non-destructive way.
  2. Stop RIGHT the F NOW!

I’ve been encountering a good deal of cancel button which don’t obey the basic laws of cancelling that any user would expect.

I have actually been randomly clicking “cancel” whenever I see it, much to my coworker’s dismay.

I started doing this, because I wanted to see how widespread the cancel cancer is.

And it is pretty widespread.  A majority of cancel buttons I have been testing neither stop right away or prevent destruction.

I found instead that clicking the “cancel” button in most programs is sure to hang that program for an extended period of time, ultimately causing you to kill the process, and it tends to leave processes half done without rolling them back.

Clearing things up

Let me be a bit clear here.  I am not talking about cancel buttons you see in an “Ok / Cancel” dialog.  Most of the time those cancel buttons actually work, because they are really operating as “back” buttons, they aren’t actually cancelling a process that is happening live.

I am talking mainly about cancel buttons that cancel an active ongoing activity.  For example, cancel buttons in an installer or during an SVN update.

We could call these kinds of cancel buttons “asynchronous cancel buttons.”

But, I need to provide a way for the user to cancel

Good, but don’t lie about it.

There are certain things that just can’t be cancelled.

When I get on a plane, I can cancel my trip when I am sitting there waiting for the door to close.  I can even cancel my trip while the plane is taxing to the runway, if I yell “I have been RAPED by a BOMB that is on FIRE!

But, I can’t cancel my trip, once the plane has lifted off the ground.  If I try to cancel it then… well, bad things would happen.  Very bad things.

So how come when I am installing your software, or your software is updating its database, I have this shiny little cancel button I can click at any time during that process?

Surely I cannot cancel just any time!

Surely there are parts of the process that cancelling would be fatal or it is too late to rollback.

My point is, if the user truly can’t cancel, don’t present a button that says they can.  More specifically, if you can’t obey the laws of cancel 1 and 2 from above, don’t even show a button.  Just say “sorry, you can’t cancel this process at this time.”

I don’t even need to know why I can’t cancel.  I mean, it will make me feel better if you tell me that the Unicorn Glitter Engine is in a critical state and any disruptions could end life as we know it, but I’ll settle for you just greying out that cancel button or not displaying it at all.

cancel button thumb Are You Really Sure You Want to Make a Cancel Button?

Putting back on the developer hat

I’m guilty of it myself.  I know I have created cancel buttons in the past that have caused pain and anguish.

But what can we do about it as developers?

First off, we should be thinking carefully about breaking a long running process into steps.  At each step of the way we should consider if it is conceivable to cancel that step without destroying data and hanging the application.

In any long running process, we should be able to identify certain parts which are cancellable and those which do not make sense to cancel.

It is your job as the developer to ensure that if you decide to allow for cancelling that the cancel undoes the existing process and work immediately.

I cannot stress this enough!

This is the behavior that most users expect and the very meaning of the word cancel.

To do this might take extra work.  You might have to think a bit more about the threading situation of your application.  You might have to think about rollback situations.  But, if you don’t do it, your cancel button will become just like the boy who cried wolf and no one will believe them.

And if you’re not willing to go through this effort, at the very least, be kind enough to your users to just remove the cancel button, because you can bet I’ll be clicking it!

Getting Up to BAT: Designing an Automation Framework

Now that you’ve gotten an automation lead and decided on the browser automation tool you are going to use, the next step is to design an actual automation framework.

This is one of the most critical components of the overall success of your automation strategy, so you will want to make sure you invest properly in this area.

I have seen a large number of automation projects go forth, but each time the critical component determining their success or failure was having a good automation framework.

Solid design and excellent technical resources are absolutely critical to success in this area!

What is an automation framework?

An automation framework is essentially an API that all of your BATs (Blackbox Automated Tests) are written against.

This API can be as simple as the API that is exposed by your browser automation tool (WatiN, Selenium, etc.), but I would highly recommend building your own layer on top of the browser automation tool’s API that will act as a DSL (Domain Specific Language) for automating your application.

Let me break this down a bit further by using an example.

Think about the task of making coffee.

There are several “API” levels we can interact with.

  1. We can go with a very low level API where we take whole coffee beans and grind them down.  Then we take some water, get it hot.  We take our filter put the ground beans in it, put it over a cup and pour the water into the filter.
  2. We can go with a higher level API where we use a traditional coffee maker.  In this case we load the coffee maker with a filter, ground coffee beans and water and push a “brew” button.  We could also set it to start at a certain time.
  3. We can go with a very high level API where we use a Keurig machine or similar device.  In this case we only make sure the machine has water in it, and we just insert a little pod and press brew.  We can make different kinds of coffee, cider or hot cocoa just by changing what pod we use.

unnamed thumb Getting Up to BAT: Designing an Automation Framework

Using the API provided by the browser driver is like making the coffee by hand.  It’s going to take a large amount of effort each time you do it.

We want our automation framework to be more like the Keurig machine.  We want to be able to compartmentalize our tests in little pods that are small and don’t require many hooks into the automation framework.

To rehash this one more time, basically our automation framework will be a framework we build on top of the browser driver framework, which is designed to make it easy to write tests which automate our application.

What makes a good automation framework design?

The true measure of an automation framework is the size of the tests that are written against them. The less lines of code in each test, the better the automation framework captures the domain of the application being automated.

In my earlier post about an example of an automation framework, I talked a bit about the strategy I used in a real implementation of an automation framework.

Here is a diagram showing the different layers.  You can see the framework layer is in green here.

automated testing framework 2 thumb Getting Up to BAT: Designing an Automation Framework

You can also see in this diagram that on the right hand side, I have screens, workflows and navigation.

This is one of the common design patterns you can use to build your automation framework, since it closely models that of most web applications.

A good way to design your framework is to use a pattern like this and create classes for each page in your application, create classes which might represent workflows in your application, and create some classes for navigating around the application.

The basic idea you are shooting for with your automation framework is to make it so the tests do not have to know anything about the browser.  Only your automation framework itself should be dealing with the DOM and HTML and CSS.  Your tests should be dealing with concepts which any user would be familiar with.  That is why I commonly use pages and workflows.

Let me give you an example.

Let’s say we ignored the advice of creating an automation framework and decided to program our tests directly against the browser driver layer.  In this case, let’s say that we want to automate a login page.  Our test might look something like this pseudo-code.

Element userName = browser.getElementById("login_userName");

Element userPassword = browser.getElementById("login_userPassword");

Element loginButton = browser.


This is not very readable.  It is not something an average tester will be able to pick up and start writing, and it is extremely fragile.  If you have 500 tests like this and the id changes for one of the elements on the page, many tests will break.

Now, contrast it to this pseudo-code:



Where did all the code go?  You don’t need it!  Actually it went into the framework, where it can be reused.  Now your test code is simple, can be understood by just about anyone, and won’t break with changes to the UI.  (You will just have to change the framework method instead of 500 tests.)

If you want some more detailed examples take a look at my Boise Code Camp slides on the subject here.

Let me offer up some general guidelines for creating a good design for an automation framework.

  1. NEVER require the tests to declare variables.
  2. NEVER require the tests to use the new keyword or create new objects.
  3. NEVER require the tests to manage state on their own.
  4. ALWAYS reduce the number of parameters for API calls when possible.
  5. ALWAYS use default values instead of requiring a parameter when possible.
  6. PREFER to make the API easier to use over making the internals of the API less complex.
  7. PREFER using enumerations and constants to requiring the test to pass in primitive types.  (Make the input bounded when possible.)
  8. NEVER expose the browser or DOM to the tests or let them manipulate it directly.

You can see that these guidelines will almost force your API to be entirely static methods!  Don’t freak out.  It is perfectly fine.  There is nothing wrong with using static methods in this case.

Stop… take a breath, think about why you think static methods are bad, because I probably agree with you.

But, in this case we are HIGHLY valuing making the tests simple and easy to write over anything else, and because of this emphasis, static methods are going to be the way to go.

Some advice on getting started

Start with the tests in plain human English.

Take those tests and make them into code preserving as much English as possible.  Then work down from there implementing the methods needed to make those tests work.  Only add things as you need them.  Do not overdesign!

In my example above, we might have started with something like this:

1. Goto the login page.

2. Login with the default user.

The actual code looks just like that.  It should always be a 1 to 1 mapping.

This is where having someone that has some experience doing this is going to come in handy.

Just keep in mind the whole time your 2 big goals:

  1. Easy to write tests
  2. Tests are concise
As always, you can subscribe to this RSS feed to follow my posts on Making the Complex Simple.  Feel free to check out where I post about the topic of writing elegant code about once a week.  Also, you can follow me on twitter here.

Getting Up to BAT: Picking a Browser Automation Tool

Now that you’ve gotten an “Automation Lead” for your BAT (Black-box Automated Testing,) it’s time to make a very important decision.

It is time to pick the browser automation tool you are going to use for driving your automation efforts.

Before we can build an automation framework, we need a basic set of tools to build it on.  We want our automation framework to be tailored to our application under test specifically.  Our automation framework will act like a Domain Specific Language (DSL) for testing our application, but we need to build that automation framework on top of something.

What is a browser automation tool?

It is essentially what we will use to drive the web browser.

older driver thumb Getting Up to BAT: Picking a Browser Automation Tool

We need to pick a tool or framework that will easily allow us to interact with the web browser so we don’t have to write that code ourselves.

We will need to be able to click buttons, enter text, select drop downs, and check values on web pages.

Some of the things we will want to consider when choosing a browser automation tool include:

  • Language we can use with the driver
  • Browsers it supports
  • How we interact with the browser (do we use XPath, JQuery, or some other mechanism)
  • If we can access and manipulate everything we need to be able to in the browser
  • Speed of execution
  • Ability to execute in parallel
  • Support and future development

Let’s take a quick look at each of these considerations.

What language can we use?

This is an important consideration because you will want to be able to utilize your other programming resources to help with building an automation framework and to eventually create automation tests.

If you pick a browser driver that supports a language your regular programmers don’t like, or don’t know, you may not end up with their support, and you are definitely going to want their support.

What browsers does it support?

This is a tricky topic to discuss.  It might seem that a browser automation tool would need to support all of the browsers your application supports, but that is not really true.

You really have to think about what your goal is with your BATs.  I could certainly devote a whole blog post to this topic, but I will try and summarize my main point here.

BATs should be designed to test functionality of the application, not to test compatibility.  (At least most of your BATs should be, although you might have some that are specifically for compatibility.)  With that goal in mind, a browser driver supporting each and every browser your application supports is not really necessary.

You do need to consider that the browser automation tool should support at least one of your supported browsers, preferably the main one.

How do we interact with the browser?

There are several different APIs which are used to communicate with the web browser that are employed by various different browser drivers.

Some tools have APIs that are very low level and allow you to very easily manipulate objects in the web browser while others are at a much higher level and rely more on your understanding of the brower’s DOM.

For example, one API might expose elements on a web page by their actual element name and let you interact with them directly.



In the first example, the API is aware of element types like buttons.  In the second example, the API is more generic and will send a “click” to an element with the ID value you specify.

You’ll have to decide how important the ease of use is to your project at this level.  There are some tradeoffs to consider here.

The lower level the API is, the easier it will be to use, but the more dependent you will be on that API and the more language specific that API will be.

The higher level the API is, the harder it will be to use, but it will free you from depending so much on the API, because you will be interacting more directly with the browser’s DOM.

I prefer lower level APIs because I find that it is much easier to write the framework code on top of these.

You will also want to consider here the skill levels of the programmers who will be creating the framework.  If your API is lower level, native language skills are more important.  If it is higher level, DOM and HTML, and perhaps Javascript skills, are more important.

Can we access what we need?

If your application makes use of Javascript pop-up dialogs and your browser automation tool doesn’t give you a way to interact with them, you don’t want to find this out when you have already invested significantly into your automation framework.

Different automation frameworks support different features of browser; not all of them will allow you to interact with every single browser feature that exists.  As new features are introduced, you could also get stuck with a tool that is no longer being maintained and doesn’t add support for the new features.

You will want to consider things like how much your application uses AJAX or JQuery.  You will want to select a browser automation tool which would make it easy to interact with AJAX calls if your application relies on them heavily.

Speed of execution

Not all browser automation tools are the same in execution speed either.

This may not be important to you, depending on how you address the issue of concurrency, but you should at least consider that if you have a large number of automated tests, a small difference in execution speed can result in a large difference in total time to run the tests.

The faster the automated tests can be run, the more valuable they are.  I won’t get into the details here but there are many reason why this is true.

Also influencing the speed in which you can run your tests will be if you automation tool choice requires you to put pauses into the test to wait for the browser instead of responding to events that occur in the browser.

Ability to execute in parallel

This consideration will depend greatly on the volume of and speed in which your tests can execute.

At some time (sooner than you think) you will likely get to the point where you can no longer run all of your automated tests in 24 hours.  Perhaps even before this point, you will want to consider running your BATs in parallel to reduce the total time to execute the tests.

Some automation tools have built-in support and others will require you to build your own way to do this.  It is good to at least have an idea of how you are going to achieve parallel execution when considering what browser automation tool to use.

Support and future development

We are in a pretty rapid pace of browser development.  Many things in the web browser world are changing much more rapidly than ever before.  Tools to automate the browser must also change or you could end up in a very bad place.

If your application is going to take advantage of the latest browsers and browser features, you should make sure the automation tool you choose is in active development.

There is nothing worse than investing into an open-source project and then having that project die, resulting in you having to rip it out to replace it with another library.

You can always design your automation framework to be abstracted away as much as possible from the underlying browser driver, but that will be extra work, so consider this point carefully.

Name names sir!

Nope, I’m not going to do it.

I’m not going to tell you to use Watin or Watij or Selenium or WebAii or any other browser driver.  I don’t want to put the focus on the tool, since the real focus will be building the automation framework on top of the tool you use to drive the browser.

I would suggest that you try writing a few simple tests in each of the major browser drivers so that you can get a good feel of what the API is like and how it will work with your application.

I will say that I have used most of the major choices out there and there really is no clear winner in my mind.  It really is going to depend on what your environment is like and what kind of application you are testing.

I would also recommend picking a browser driver and sticking to it.  I’ve tried in the past to abstract away the browser driver from the automation framework and while it is possible, it can become quite messy and add quite a bit of overhead to your project.

My only other hints would be to not put too much emphasis on supporting multiple browsers or on using recording tools.  Neither of these things will benefit you much in the long run because you will find that you will not want to try and run all of your BATs on each browser. Recording tools will not be nearly as effective as writing your own custom framework (which I will talk about in my next post.)

As always, you can subscribe to this RSS feed to follow my posts on Making the Complex Simple.  Feel free to check out where I post about the topic of writing elegant code about once a week.  Also, you can follow me on twitter here.

Back to Basics: Becoming BAT Man

In my Back to Basics post on my conclusions about blackbox automated tests (BATs) and unit testing, I said that we should:

Spend a majority of your effort, all the time you would have spent writing unit tests, instead writing what I will call blackbox automated tests or BATs.

In this post, I am going to outline what I think it takes for a team to go from BAT-less to BAT-man (or BAT-woman) without going batty.

I want to take a practical approach to getting from 0 blackbox automated tests to building a sustainable process of developing BATs as a integral component of the acceptance criteria for calling a backlog item “done.”

Let me paint a picture for you

human paint brush thumb Back to Basics: Becoming BAT Man

I want to paint the picture for you of the end goal that we are seeking to achieve, from start to finish, of a single backlog.

  1. A backlog is selected for work.
  2. Developers, QA and product owner work together to clearly define the acceptance criteria which will enable that backlog to be signed off or called “done.”
  3. QA and developers work together to transform the acceptance criteria into a high level description of BATs which will be created to verify each aspect of the acceptance criteria.
  4. Developer begins coding with this high level set of BATs in mind (BATs drive the development), QA begins developing BATs.
  5. As developer completes enough functionality to pass a BAT and not break existing BATs that functionality is checked into trunk.
  6. Each check-in causes the entire battery of BATs to be kicked off split across multiple VMs which run 100s of hours worth of automated tests in minutes.
  7. Once all BATs for a backlog are passing, that backlog is done and those BATs become part of the battery of tests which are run with each continuous integration build.

We should keep this simple process in mind and strive to reach it, knowing that when we are able to reach this process we will be able to have a HUGE amount of confidence in the functionality and non-regression of our system.

Put aside for a second your notion that unit testing is the correct thing to do.  Put aside the idea that blackbox automated testing is too hard and too fragile, and imagine the world of software development that flows like the picture I painted above.

Now listen up, because this part is important…

Before you put forth the effort to write one more unit test, before you dip your test double pen in the ink well of “mock,” make sure you are first taking efforts to develop a process to create BATs.

The value proposition

We really need to take the effort to understand the value proposition being presented here.

I’m going to use some real fake numbers to try and really convey this important point that I think is likely to be overlooked.

Consider first the amount of time that you spend doing two things:

  1. Creating infrastructure to allow your code to be unit tested in isolation.  (Dependency injection, interfaces, etc.)
  2. Time spent writing unit tests.

Now, before we go any further, I just want to reiterate.  I am not advocating completely abolishing the writing of unit tests.  See my original conclusions post, or my post on unit testing without mocks for a better understanding of what I am advocating in regards to unit testing.

So back to my point.  Think about all the time it takes to do this.  To properly isolate and unit test code most developers would probably estimate that for every hour worth of work writing the code there is about another hour to two hours worth of unit testing and unit test prep time.

There is a reason why unit tests take longer to write than the code they test; it takes MORE lines of code to test a line of code in isolation.

BATs developed using a good automation framework are just the opposite.  While unit test code might have a ratio of 4 lines of unit test code to 1 line of production code or a 4:1 ratio, BAT code has a completely opposite ratio.  It is very likely that 1 line of BAT code can test 20 lines or more of production code, a 1:20 ratio.  (Where do I get these numbers?  Looking at some of my real production code being tested with unit tests and BATs.)

Even if unit tests and BATs were equally effective in preventing the regression of your software system, and equally effective at providing an absolute assurance of meeting the acceptance criteria (which I would argue that BATs are much more effective at both), you can see easily that you are going to get a much higher return on your investment by investing your time in BATs vs. investing your time in unit tests.

I don’t make these statements lightly or in a theoretical tone.  I have real world experience with successfully implementing automation frameworks for writing BATs that reinforce these conclusions.

How to get there

If I have done my job, you are now at least convinced that getting to the point of having BATs for all backlogs is a good goal, but if you are like most people I talk to, you are skeptical of the costs and feasibility of doing it.

What you need is a good guide.  A step by step guide on how to do it.

I am going to conclude the Back to Basics posts and segue into a new series with the goal of providing a step by step guide to getting to fully automated blackbox testing.

Let’s take a look at the steps that I will cover in the next series of posts.

  1. Hiring an automation lead – it’s important to either find someone who’s done this before, or a developer resource to this role.
  2. Deciding on the underlying browser driver or browser automation tool – assuming web apps here, there are several to choose from, WatiX, Selenium and more.
  3. Designing an automation framework – building a framework tailored specifically to your application under test will create an effective domain specific language for testing your application.
  4. Creating your first smoke tests – building some smoke tests will help you prove and build out your framework and provide you the highest value tests first.
  5. Adding smoke tests to your build – with smoke tests being run as part of the build, you can immediately start seeing value.
  6. Adding BATs to your acceptance criteria – we need to start out slowly here, but eventually make all backlogs require BATs for each acceptance criterion.
  7. Scaling out – as you start to get a large amount of BATs you’ll need to figure out a way to virtualize more servers and parallelize the test runs to be able to run all the BATs in a short amount of time.
  8. Building a true DSL – once you get this far, it may be time to start thinking about creating a domain specific language that even business analysts can write tests with.
As always, you can subscribe to this RSS feed to follow my posts on Making the Complex Simple.  Feel free to check out where I post about the topic of writing elegant code about once a week.  Also, you can follow me on twitter here.

Back to Basics: Unit Testing, Automated Blackbox Testing, and Conclusions!

If you’ve been following me from the beginning of the Back to Basics series, you’ll know that I set out to reevaluate some of the commonly held truths of what best practices are, especially in regards to unit testing, dependency injection and inversion of control containers.

We’ve talked about what an interface is, cohesion and coupling, and even went a little bit off track to talk about sorting for a bit.

One of the reoccurring themes that kept showing up in most of the posts was unit testing.  I talked about why unit testing is hard, and I defined three levels of unit testing.

  • Level 1 – we have a single class with no external dependencies and no state.  We are just testing an algorithm.
  • Level 2 – we have a single class with no external dependencies but it does have state.  We are setting up an object and testing it as a whole.
  • Level 3 – we have a single class with at least one external dependency, but it does not depend on its own internal state.
  • Level 4 – we have a single class with at least one external dependency and depends on its own internal state.

Throughout this series I ended up tearing down using interfaces with only single non-unit test implementation.  I criticized the overuse of dependency injection for the sole purpose of unit testing.  I attacked a large portion of best practices that I felt were only really being used in order to be able to unit test classes in isolation.

But, I never offered a solution.  I told you what was bad, but I never told you what was good.

I said don’t create all these extra interfaces, use IoC containers all over your app, and mocks everywhere just for the purpose of being able to isolate a class you want to unit test, but when you asked me what to do instead, I said “I don’t know, I just know what we are doing is wrong and we need to stop.”

Well, that is no answer, but I intend to give one now.  I’ve been thinking about this for months, researching the topic and experimenting on my own.

cool experiment thumb Back to Basics: Unit Testing, Automated Blackbox Testing, and Conclusions!

I finally have an answer

But, before I give you it, I want to give you a little background on my position on the subject matter.

I come from a pretty solid background of unit testing and test driven development.  I have been preaching both for at least the last 7 years.

I was on board from the beginning with dependency injection and IoC containers.  I had even rolled my own as a way to facilitate isolating dependencies for true unit tests.

I think unit testing and TDD are very good skills to have.  I think everyone should learn them.  TDD truly helps you write object oriented code with small concentrated areas of responsibility.

But, after all this time I have finally concluded, for the most part, that unit tests and practicing TDD in general do more good for the coder than the software.

What?  How can I speak such blasphemy?

The truth of the matter is that I have personally grown as a developer by learning and practicing TDD, which has lead me to build better software, but not because the unit tests themselves did much. 

What happened is that while I was feeling all that pain of creating mocks for dependencies and trying to unit test code after I had written it, I was learning to reduce dependencies and how to create proper abstractions. 

I feel like I learned the most when the IoC frameworks were the weakest, because I was forced to minimize dependencies for the pain of trying to create so many mocks or not being able to unit test a class in isolation at all.

I’ve gotten to the point now where two things have happened:

  1. I don’t need the TDD training wheels anymore.  I don’t pretend to be a coding god or demi-god of some sort, but in general the code I write that is done in a TDD or BDD style is almost exactly the same as the code I write without it.
  2. The IoC containers have made it so easy to pass 50 dependencies into my constructor that I am no longer feeling the pain that caused my unit tests to cause me to write better code.

What I find myself ending up with now when I write unit tests is 70% mocking code that verifies that my code calls certain methods in a certain order.

Many times I can’t even be sure if my unit test is actually testing what I think it is, because it is so complex.

Umm, did you say you had an answer, dude?

Yes, I do have an answer.  I just wanted to make sure you understand where I am coming from before I throw out all these years of practical knowledge and good practices.

I am not the enemy.

My answer to the problem of what to do if you shouldn’t be using IoC containers and interfaces all over your code base just for the purpose of unit testing, is to take a two pronged approach.

2prong thumb Back to Basics: Unit Testing, Automated Blackbox Testing, and Conclusions!

  1. Mostly only write level 1 or level 2 unit tests.  Occasionally write level 3 unit tests if you only have 1 or possibly 2 dependencies.  (I’ll talk about more how to do this in my next post)
  2. Spend a majority of your effort, all the time you would have spent writing unit tests, instead writing what I will call blackbox automated tests or BATs.  (I used to call this automated functional tests, but I think that name is too ambiguous.)

I intend to drill really deep into these approaches in some upcoming posts, but I want to briefly talk about why I am suggesting these two things in place of traditional BDD or TDD approaches.

What are the benefits?

The first obvious benefit is that you won’t be complicating your production code with complex frameworks for injecting dependencies and other clever things that really amount to making unit testing easier.

Again, I am not saying you shouldn’t ever use dependency injection, interfaces or IoC containers.  I am just saying you should use them when they provide a real tangible value (which most of the time is going to require alternate non-unit test implementations of an interface.)

Think about how much simpler your code would be if you just went ahead and new’d up a concrete class when you needed it.  If you didn’t create an extra interface for it, and then pass it in the constructor.  You just used it where you needed it and that was that.

The second benefit is that you won’t spend so much time writing hard unit tests.  I know that when I am writing code for a feature I usually spend at least half the amount of time writing unit tests.  This is mostly because I am writing level 3 and level 4 unit tests, which require a large number of mocks.

Mocks kill us.  Mocking has a negative ROI.  Not only is creating them expensive in terms of time, but it also strongly couples our test classes to the system and makes them very fragile.  Plus, mocking adds huge amounts of complexity to unit tests.  Mocking usually ends up causing our unit test code to become unreadable, which makes it almost worthless.

I’ve been writing mocks for years.  I know just about every trick in the book.  I can show you how to do it in Java, in C#, even in C++.  It is always painful, even with auto-mocking libraries.

By skipping the hard unit tests and finding smart ways to make more classes only require level 1 and level 2 unit tests, you are making your job a whole lot easier and maximizing on the activities that give you a high ROI.  Level 1 and level 2 unit tests, in my estimation, give very high ROIs.

The thirds benefit is that blackbox automated tests are the most valuable tests in your entire system and now you’ll be writing more of them.  There are many names for these tests, I am calling them BATs now, but basically this is what most companies call automation.  Unfortunately, most companies leave this job to a QA automation engineer instead of the development teams.  Don’t get me wrong, QA automation engineers are great, but there aren’t many of them, good ones are very expensive, and the responsibility shouldn’t lie squarely on their shoulders.

BATs test the whole system working together.  BATs are your automated regression tests for the entire system.  BATs are automated customer acceptance tests and the ROI for each line for code in a BAT can be much higher than the ROI of each line of production code.

Why?  How is this even possible?  It’s all about leverage baby.  Each line of code in a BAT may be exercising anywhere from 5 to 500 lines of production code, which is quite the opposite case of a unit test where each line of unit test code might only be testing a 1/8th or 1/16th a line of production code on average (depending on code coverage numbers being reached.)

I’ll save the detail for later posts, but it is my strong opinion that a majority of a development teams effort should be put in BATs, because BATs

  • Have high value to the customer
  • Regression test the entire system
  • Have a huge ROI per line of code (if you create a proper BAT framework)

Imagine how much higher quality your software would be if you had a BAT for each backlog item in your system which you could run every single iteration of your development process.  Imagine how confident you would be in making changes to the system, knowing that you have an automated set of tests that will catch almost any break in functionality.

Don’t you think that is worth giving up writing level 3 and level 4 unit tests, which are already painful and not very fun to begin with to achieve?

In my future posts on the Back to Basics series, I will cover in-depth how to push more of your code into level 1 and level 2 unit tests by extracting logic out to separate classes that have no dependencies, and I will talk more about BATs, and how to get started and be successful using them.  (Hint: you need a good BAT framework.)

As always, you can subscribe to this RSS feed to follow my posts on Making the Complex Simple.  Feel free to check out where I post about the topic of writing elegant code about once a week.  Also, you can follow me on twitter here.

Agile Testing is Different

I come from a QA background.

I started off my career doing testing.  I learned about testing the same way I learned about development… reading lots of books and applying what I learned.

This is how you are supposed to do QA:

  1. Write a test plan for the project.
  2. Gather requirements.  Put them into a repository and give them cold hard labels like REQ1.2.5.
  3. Create test cases from the requirements.  Store those test cases into a tracking system which can map them back to individual requirements.
  4. Execute test cases, each test step should either pass or fail.  Record the results.  File any bugs found into the bug tracking system which maps the bug to the test that failed and the associated requirement.

There is more to it, but that is the basic outline.

But it doesn’t work in Agile!

Heck, it doesn’t work anywhere.  At least not the requirements part.  That is the part that is always missing, or woefully inadequate.

Don’t panic though.  It makes sense.  Let’s think this through.  The reason we all started jumping off the waterfall ship and into Agile is because we realized that it is almost impossible and certainly worthless to try and define requirements up front.

With that idea in mind, how do we test?  How can we test in an Agile environment?

First goal is automation

Let me explain to you why this is the most important part.  It may seem a little out of order, but bear with me.

When you are doing iterative development, you are building things little by little.  You’re not waiting until the end to test things, so there is a really good chance something you do in any iteration will break something you did earlier.

How do you deal with this?  It’s called “regression.”  Yes, you probably need it in waterfall, but in Agile it is an absolute must, otherwise you’ll find that for every step forward you take, you take a step backward as you fix something that you broke.

checkingforleaks thumb Agile Testing is Different

Running an Agile project without automation is like walking the tight-rope without a safety net, except the tight-rope is on fire, someone is strumming it like a banjo, and you’re wearing two left shoes.

People groan when I say this, but I firmly believe it and I have done it several times.

Part of every backlog’s done criteria should be that it has automated tests!

Unless you have automated tests for each piece of functionality you build (or at least the ones you care if they stay working), or an army of manual testers running your regression suite every iteration, your software will break in your customer’s hands.

How to build an Internal DSL for Automated Functional Testing

Automated UI Testing Framework… A Real Example

Next goal is change the focus

Testing is traditionally an activity that happens at the end of development and is the gate-keeping for software.  It has to go through QA before it gets released.

That is not a good model for Agile, because what happens when there is a bug?  If the testing happens at the end of the iteration and there is a bug found, there is no time to fix it.

What if we change the order of things, change the focus?

When you write your code, you write unit tests first, right?  (Or if you like BDD, you write the specifications, or behaviors.)  And you run them continuously as you’re working on your code to make sure you don’t break them, right?

We can apply the same thing at the higher level.

  • We can focus on creating a failing automated test before implementing the software.
  • We can iteratively build more failing automated tests and make each one pass by building the software.
  • We can run those tests as we build the software.

How many times have you built a feature only to have it tested and find out the test cases were not at all testing what you built? (I’ll raise my hand.)

That happens when we try to independently create the test cases from the development of the software.

What we want to do is blur the lines between testing and development.  We want to get close to continuous testing.  Just like we do when we are running unit tests as we are writing code.  Short feedback cycles are the key.

Sounds good, but how do we do it?

Here is a walk through of what your workflow might be when doing Agile testing.

  1. Backlog item is pulled into iteration.
  2. Team members talk to backlog owner about the backlog.  The goal of this conversation is not to get all the details of the backlog, but to get enough information to understand the bigger picture of the backlog and get enough information to get started.
  3. Team members talk about what is the basic, first, automated test to write for this backlog.
  4. Team starts writing the automated test.
  5. Team starts writing the code to make the automated test pass.
  6. Team goes back to backlog owner to demonstrate completed parts, get feedback, and ask for more clarification.
  7. Team continues to write automated tests, make them pass, and have conversations and demonstrations with backlog owner.
  8. When backlog owner is happy and backlog is complete, and everyone considers it well-tested, the backlog is done.

Observations about that process:

  • Steps can happen parallel or out of order, steps can overlap.
  • Steps can be split among team members.
  • I erased the words “QA person” and “Developer” and replaced them with “Team,” we don’t care so much about roles, we want to blur those lines as much as possible.
  • There aren’t really phases, for development, QA, etc.  Development involves testing as you go, when the backlog is done being “developed,” it is done.
  • Focus is on making customer happy, not passing an arbitrary set of requirements the customer gave you upfront, or you tried to interpret.  Take a look at my post on comparing user stories requirements gathering to hanging a picture for more on this.
  • There is no useless documentation.  All documentation is executable.
  • There is no “gate-keeper,” quality is being built as we go, we are not trying to “test it in” at the end of the iteration.

What is the role of the QA person?

It might seem that we have eliminated that role by this process, or diminished the value, but nothing could be further from the truth.

What we have actually done here is elevated the role from writing documentation and manually executing tests to being the expert on quality and the representative of the customer on the team.

You can think of the QA person as the “quality coach” for the team.  They might not always be the one creating the automated tests or running them, but they are one of the main forces guiding the direction of the creation of those tests, and making sure the customer’s interests are represented.

The QA team members will also occasionally manually test things that cannot be automated, and exploratory test as needed, but the focus should shift towards being the “quality coach” for the team instead of doing all of the quality work.

I found this excellent presentation on Agile Testing by Elisabeth Hendrickson from Quality Tree Software if you are interested in further reading on the topic.

Boise Code Camp 2010 Presentations

Went to Boise Code Camp this weekend and it was great!

There were some really good sessions and good discussions on Agile, Scrum and domain specific languages.

I think events like these are important because they help us to zoom out a little and see what kinds of things other people are doing in our field and what companies around us are interested in high quality software.

Code camps are also a great place to get some experience doing presentations and share what you are working on.  Networking opportunities abound.

I presented two talks this weekend.  I am posting the links to those here.

The first presentation I did was on Pair Programming.

You can find the presentation on Prezi here.

We also did a live pair programming session, which I thought went pretty well.  We ran into some of the issues that often come up with pair programming, like being afraid to tell someone you don’t like their method name.

The second presentation I did was on Creating an Internal DSL for Automated Functional Testing.

You can find the presentation on Slideshare here.

We went over how to go about creating a good automation framework which fills in the gaps between the business language and technical language.

Thanks to everyone who attended.

Automated Functional Tests: Record or Program?

Recently, I’ve been devoting some time to building out a framework using Watij with the goal to be able to easily write automated functional tests for backlog items for the application I am working on.  There are many factors in the technology choice, but one of the most prominent of questions about doing automation seems to be around whether or not to use a tool that records test scripts vs using a WatiX or similar library and rolling your own framework.  Let me put a little caveat here, since I know this will come up.  If you use a recoding tool and then you use its internal language to build a framework that is actually reusable and makes it so you don’t have to re-record every time you create a new test, I would count that as programming vs recording.  When I say recording, I mean mostly just recording and plugging in variables.

Let’s looks at the pro’s and con’s:


+ Very fast to get started, you can have a working test right from the get go.

+ Anyone can do it.  Just requires a little training in the tool, but not much technical knowledge needed.

+ Easy execution of tests, usually integration with test case management systems, defect systems, etc.

- Speed of test creation doesn’t really increase as test library increases.

- Lots of pointing and clicking to create the tests each time.

- Fragile: if a screen changes it can break many tests which each have to be changed.

- Expensive, most of the recoding tools are very expensive.

- Heavy installation, most of the tools require thick clients to use, installed on any machine recording or running them.


+ As your library increases in size, test case creation becomes very rapid.

+ Developers are more likely to participate since it is a programming type activity.

+ Can abstract screen changes from functions it order to protect against change.  (If one screen changes you are more likely to only need to update the framework in one spot, not change every single test.)

+ Low cost, as in free, solution.  (Besides development cost)

+ Can run on any machine that has Ruby, .NET Framework, or Java depending on framework used.

– Takes at least some development expertise to build the initial framework.

– Requires more training to understand how to use.

– Initial creation of test cases takes a long time.


The patterns of pro’s and con’s here closely resembles the pattern of pro’s and con’s for doing unit testing or test driven development.  Higher cost initially, but over time large amount of savings.  Programming seems to be a clear winner over recording, in my opinion.

Next time I’ll take a look at how to implement a good framework for automated functional testing using a tool like Watij, Watin, or Watir.

Updated, here is the post that looks at the action implementation of the framework.