Back to Basics: Becoming BAT Man

In my Back to Basics post on my conclusions about blackbox automated tests (BATs) and unit testing, I said that we should:

Spend a majority of your effort, all the time you would have spent writing unit tests, instead writing what I will call blackbox automated tests or BATs.

In this post, I am going to outline what I think it takes for a team to go from BAT-less to BAT-man (or BAT-woman) without going batty.

I want to take a practical approach to getting from 0 blackbox automated tests to building a sustainable process of developing BATs as a integral component of the acceptance criteria for calling a backlog item “done.”

Let me paint a picture for you

I want to paint the picture for you of the end goal that we are seeking to achieve, from start to finish, of a single backlog.

1. A backlog is selected for work.
2. Developers, QA and product owner work together to clearly define the acceptance criteria which will enable that backlog to be signed off or called “done.”
3. QA and developers work together to transform the acceptance criteria into a high level description of BATs which will be created to verify each aspect of the acceptance criteria.
4. Developer begins coding with this high level set of BATs in mind (BATs drive the development), QA begins developing BATs.
5. As developer completes enough functionality to pass a BAT and not break existing BATs that functionality is checked into trunk.
6. Each check-in causes the entire battery of BATs to be kicked off split across multiple VMs which run 100s of hours worth of automated tests in minutes.
7. Once all BATs for a backlog are passing, that backlog is done and those BATs become part of the battery of tests which are run with each continuous integration build.

We should keep this simple process in mind and strive to reach it, knowing that when we are able to reach this process we will be able to have a HUGE amount of confidence in the functionality and non-regression of our system.

Put aside for a second your notion that unit testing is the correct thing to do.  Put aside the idea that blackbox automated testing is too hard and too fragile, and imagine the world of software development that flows like the picture I painted above.

Now listen up, because this part is important…

Before you put forth the effort to write one more unit test, before you dip your test double pen in the ink well of “mock,” make sure you are first taking efforts to develop a process to create BATs.

The value proposition

We really need to take the effort to understand the value proposition being presented here.

I’m going to use some real fake numbers to try and really convey this important point that I think is likely to be overlooked.

Consider first the amount of time that you spend doing two things:

1. Creating infrastructure to allow your code to be unit tested in isolation.  (Dependency injection, interfaces, etc.)
2. Time spent writing unit tests.

Now, before we go any further, I just want to reiterate.  I am not advocating completely abolishing the writing of unit tests.  See my original conclusions post, or my post on unit testing without mocks for a better understanding of what I am advocating in regards to unit testing.

So back to my point.  Think about all the time it takes to do this.  To properly isolate and unit test code most developers would probably estimate that for every hour worth of work writing the code there is about another hour to two hours worth of unit testing and unit test prep time.

There is a reason why unit tests take longer to write than the code they test; it takes MORE lines of code to test a line of code in isolation.

BATs developed using a good automation framework are just the opposite.  While unit test code might have a ratio of 4 lines of unit test code to 1 line of production code or a 4:1 ratio, BAT code has a completely opposite ratio.  It is very likely that 1 line of BAT code can test 20 lines or more of production code, a 1:20 ratio.  (Where do I get these numbers?  Looking at some of my real production code being tested with unit tests and BATs.)

Even if unit tests and BATs were equally effective in preventing the regression of your software system, and equally effective at providing an absolute assurance of meeting the acceptance criteria (which I would argue that BATs are much more effective at both), you can see easily that you are going to get a much higher return on your investment by investing your time in BATs vs. investing your time in unit tests.

I don’t make these statements lightly or in a theoretical tone.  I have real world experience with successfully implementing automation frameworks for writing BATs that reinforce these conclusions.

How to get there

If I have done my job, you are now at least convinced that getting to the point of having BATs for all backlogs is a good goal, but if you are like most people I talk to, you are skeptical of the costs and feasibility of doing it.

What you need is a good guide.  A step by step guide on how to do it.

I am going to conclude the Back to Basics posts and segue into a new series with the goal of providing a step by step guide to getting to fully automated blackbox testing.

Let’s take a look at the steps that I will cover in the next series of posts.

1. Hiring an automation lead – it’s important to either find someone who’s done this before, or a developer resource to this role.
2. Deciding on the underlying browser driver or browser automation tool – assuming web apps here, there are several to choose from, WatiX, Selenium and more.
3. Designing an automation framework – building a framework tailored specifically to your application under test will create an effective domain specific language for testing your application.
4. Creating your first smoke tests – building some smoke tests will help you prove and build out your framework and provide you the highest value tests first.
5. Adding smoke tests to your build – with smoke tests being run as part of the build, you can immediately start seeing value.
6. Adding BATs to your acceptance criteria – we need to start out slowly here, but eventually make all backlogs require BATs for each acceptance criterion.
7. Scaling out – as you start to get a large amount of BATs you’ll need to figure out a way to virtualize more servers and parallelize the test runs to be able to run all the BATs in a short amount of time.
8. Building a true DSL – once you get this far, it may be time to start thinking about creating a domain specific language that even business analysts can write tests with.