Not to say that you’ll ever be done expanding your automation framework and building and improving your BATs, but you are at the point of having a mature usable BAT framework and you can be proud of that accomplishment.

If you truly want to see where the rabbit hole goes though… Read on

Taking it to 11

If you have designed a good automation framework for writing your BATs, you will most likely have created an internal domain specific language (DSL).

What is an internal DSL?

Basically it is a set of APIs and rules built in another language that function as their own language.

You might have some code that looks like this:

Pages.Login.Goto();
Pages.Login.LoginWithDefaultUser();
Pages.Customers.Goto();
Pages.Customers.AddNewCustomer();

You can see how this code is really looking like a language itself.  It is hard to tell if this code is even C# or Java.

This is a good example of an internal DSL.

If we were to take this same section of a BAT test and write it as an external DSL, it might look something like this:

LOGIN WITH DEFAULT-USER
ADD-CUSTOMER WITH DEFAULT-CUSTOMER

With a true DSL we can simplify even further to get right down to the bare minimum set of commands we need to execute what we want in our application.

How to implement a DSL?

There are a number of tools and techniques that can be used to create a DSL.  I’ll hit on some of those a bit later on, but first we need to design the DSL itself.

Designing a DSL can be challenging, but I have found the best approach is to use a similar approach to what we used to design our automation framework.

It is best to work backwards from the syntax that would make your BAT the easiest to write and the most clear.  Once you have done that you can worry later about implementing the DSL to support the syntax you have come up with.

You should focus on creating a limited vocabulary for your DSL so that it will be easy for someone to learn and know what their options are.

The more restrictions we put into a DSL the easier that DSL becomes for doing a specific purpose.

We want to give the user of a DSL no more options than required for what they want to do and no less.

This isn’t always easy to do.  It often requires some careful thought and refinement.

Your DSL should also use terminology and phrases that are familiar to the business domain of your application.  By doing this you will make it so someone understanding your application will understand BATs written in that DSL.

I also always try to make a BAT DSL read as much as plain English as possible.

Once you have designed the DSL, you will need a way to implement it. 

I’ve written about using ANTLR to create an external DSL.  Also I have written a review on a book that I recommend for learning ANTLR.

Microsoft also has a DSL SDK you can use in VS2010, although I haven’t used it myself.

The basic idea behind implementing a DSL is that you have to take some sort of grammar you design for your DSL, parse it into a syntax tree, then use that tree to generate code that will actually perform the actions.

It is by no means a trivial task, but using a tool like ANTLR, makes it much simpler than trying to completely generate a parser and execution engine yourself.

Why even bother?

Creating a DSL for your BATs is completely optional, but there are some significant benefits to doing so.

The biggest one I have found is that you are much more likely to get business people to write BATs if you have a simple DSL they can use to create them.  Once you get everyone using BATs to define behavior and requirements, communication becomes much more simple.

Another great thing about creating a DSL is that you can create a runner that can interpret the DSL BAT scripts which can be run on any machine.  Typically, to run BATs created in Java or C#, you would need a C# or Java compiler and a test runner, etc.  By creating a DSL and a stand alone executor, you can very easily allow anyone to run BATs with little to no setup.

Writing your BATs in a DSL will also make writing the BATs and understanding them much easier.  A good DSL can get rid of all the little language and API nuances that don’t really add information to the specific domain you are working in.  A good DSL strips information down to the minimum amount needed and is able to do so, because it assumes the context is your application’s business domain.

You will also find that you are better able to separate framework from tests when you have a DSL in place.  One issue that many teams face when creating a BAT framework is that some people come along and write lower level code in the higher level BATs.  This creates a problem because it is hard to force someone to use the higher level API when the framework and the BAT code are the same language.

Wrapping it up

So that completes my series on Getting up to BAT.

Hopefully you have learned enough to be able to implement your own BAT framework and get BATs running for your application.

I tried to cover most of the important steps here without getting too deep into the details.

If you have any questions or are interested in getting more in depth training or consulting on getting BATs setup for your environment, send me an email or leave a comment.

It can be a little difficult to get started and get a BAT framework up and running, but the rewards are well worth the effort!

As always, you can subscribe to this RSS feed to follow my posts on Making the Complex Simple.  Feel free to check out ElegantCode.com where I post about the topic of writing elegant code about once a week.  Also, you can follow me on twitter here.

When we last left off we had just gotten our BATs as part of the acceptance criteria for any new backlogs that are worked on.  This puts us at a point where we could really say that we have successfully implemented BAT testing.

You don’t want to get too comfortable just yet though, because the next hurdle you will most likely face will be the problem of not having enough time to execute all your tests.

You want to think about this ahead of time

Nothing worse than getting everything going and then not being able to execute the entire test suite, because you didn’t plan ahead.

You don’t want to get to the point where confidence in your BAT suite it lost because you are not able to get all the tests executed in a reasonable amount of time.

The more frequently your tests are run the more value they have.

By reducing cycle time from the time a breaking change is made in the system and the time it is discovered, you greatly reduce the risk it imposes on your software and you decrease the scope of the code changes in which it could have occurred.

To put it simply, the faster you can find out you broke something, the more likely you can fix it before it does damage, and the more likely you will be to know what thing you did caused the breakage.

How can we reduce cycle time?

There are a few different strategies we can employ and we can mix and match some of these strategies.

Straight forward parallelization

The best and most effective thing you can do is to take a given test run, split up the execution of those tests on multiple machines and execute them in parallel.

This approach is going to give you the best bang for your buck.  You should really try to get some amount of parallelization going before attempting any other solution, since it is going to make a huge impact on the total execution time for your tests without making any sacrifices.

There are even many ways you can mix and match to do parallelization:

• Use multiple physical machines
• Run many virtual machines on a single host
• Put an executor program on every team member’s machines that will execute tests when that machine is idle or put into test execution mode (perhaps at night)
• Use a cloud based computing platform to execute your tests
• Run multiple browser instances on a single machine

Data preloading

With this approach, you would preload some of the test data that your tests might generate by manually clicking through screens.

This is fairly hard to explain, so let me give you an example:

Suppose you had a set of tests that all involved creating customers, but each customer you create in the system takes about 3 minutes to create by clicking through the screens to get them into the system.

We don’t need to have 500 tests all executing the same exact logic in the system 500 times for 3 minutes just to generate all the customer data that will be used in the tests.

Instead, we can leave a few tests that are exercising the customer creation functionality, and we can execute a SQL script to push all the other customer data into the test database for the other tests to use.

Using this technique we might be able to reduce our total execution time by 3 minutes * each test, or about 25 hours for 500 tests.

This can be a huge savings in time, and it doesn’t come at that high of a cost.  The sanctity of our tests is slightly compromised, but we are taking a calculated risk here knowing that we already have covered the area of execution which we are preloading data for.

Consider this technique when you notice certain things in the system taking a very long time to do.

Test runs by area

With this technique, we can reduce the total execution time in a given period by splitting up test areas and running them either at different times or in response to changes in certain areas.

You have to be very careful with this approach, because if you don’t do it correctly, you can start to erode some of the safety your BAT tests are providing you.

I would only do something like this as a last resort, because it is so easy to virtualize today, and hardware is so cheap.

I’d much rather run too many tests than too few.

Test randomization

With test randomization, we are going to take our total desired execution time, and divide it by the average time for running a test.  We then can use that number of tests to run to randomize the execution of our tests each time we run them and only run the number or tests that will fit in the desired execution time.

This choice is also a compromise that I typically don’t like to take.

It can be very useful though, combined with other techniques when you still don’t have enough time to execute your entire test suite.

The basic idea here is that you are going to randomly run tests each time to fill up the time you have to run tests.

Test reduction

This one seems fairly obvious, but can be very helpful.

Often I will see teams starting out with automation, trying to write way too many BAT tests for a given feature.  Sure, with automated tests it is possible for run a test for every single possible combination of values in your 5 drop downs, but will it really benefit you?

In many cases you have to think about what you are trying to protect against with your BATs.  Sometimes running every combination of data selection choices is going to be important, but other times you are only going to need to write tests to test a few of the happy path scenarios.

It is important to find a balance between test coverage and test volume and not just for execution time.  There is a logistical overhead to having a large volume of mostly redundant tests.

So even though this technique might seem dangerous and counter-productive, I will almost always employ it to some degree.

Common pitfalls

Here are some of the things you might want to watch out for as you are scaling out and streamlining your execution of BATs:

• Parallelization issues.  If you are using shared state, you can run into big trouble when your tests are executing in parallel.  There are many manifestations of this.  You could have issues at the database level or at the local machine memory level.  The best way to avoid this kind of problem is to use separate virtual machines for each test execution, and not reuse data setup between test cases.
• Ineffective error reporting.  If you run a huge volume of tests in parallel, you better have a good way to sort through the results.  It is much harder to figure out why things failed when they are run across multiple machines.
• Test order dependencies.  Make sure tests don’t rely on being run in a certain order or you will have lots of pain when you disrupt that order.
• Environment setup.  Make sure all your test execution environments are exactly the same unless you are specifically testing different environments for execution.  You don’t want tests failing on one machine but passing on another.

As always, you can subscribe to this RSS feed to follow my posts on Making the Complex Simple.  Feel free to check out ElegantCode.com where I post about the topic of writing elegant code about once a week.  Also, you can follow me on twitter here.

Give me your most complex program and I will trump it by presenting you with the program that was responsible for creating the human who created that program.

If you haven’t figured it out by now that program is human DNA and that masterful programmer is the Creator Himself.

Doesn’t matter what religion you are

It is pretty difficult to fail to recognize that a program as complex as a human being’s genetic code was created, not randomly assimilated.

As programmers we have a unique perspective in life.  We are able to truly understand God’s masterwork in the creation of human life.  We are able to grasp loosely the ungraspable magnificence of a program so complicated, yet executed so simply.

We also have the unique position of fundamentally understanding that complex programs could never be created by random chance, regardless of time.  We can look at a piece of code or functioning computer program and we can understand that its creation required an intelligent creator.  Other people and professionals do not have this insight that so many of us are fortunate to have.

Qualities of the code

We’ve been treating data like code recently in C# with the introduction of Lambda expressions and other languages have been doing it for much longer, but the ultimate example of this practice is right there in our own DNA.

DNA is data, a message, and code all together at once.  Just like a data structure in the memory of a computer executing a program, it lacks meaning without the context of the cell that it is in.  Amazingly though, this data is recombined in different ways that are dictated by two separate and different instances of itself to produce a third separate instance of itself that still makes sense.

Just to give you an idea of how insanely complex that idea is (despite the simple execution of it), it would be like if you wrote a webserver that could spawn new instances of itself by contacting another, but different copy of your webserver and getting 1/2 the source code payload from itself and the contacted server, then actually combining that code to create a new copy that is different, yet still passes all of your unit and integration tests.

One of the qualities we often look at for our code is the size of the code in relation to what it can accomplish.

A decent estimate is that the human genome is about .35 gigabytes of data.  That data is stored on such a small space that we can’t even see it with the naked eye.  There is no more compact form of data in the universe that we know of.

Not only is this data compact, but consider how much functionality is built into this small .35 gigabyte package?  If you had to write acceptance tests for the human body, you might spend several lifetimes doing it.  If you had to write acceptance tests for just a single cell in your body, that feat itself could take years and still not be complete.  Imagine trying to write a program that took up .35 gigabytes of data, yet contained millions of acceptance tests worth of functionality.

We certainly could go on and on about the qualities of the human code, but I want to point out what perhaps is one of the most unique and amazing… the ability for the code to create its own hardware.

In actuality the code and that hardware are much the same.  DNA replicates by a chemical reaction that causes the entire molecule to unwind and split.  RNA inside a cell is created from specific portions of the DNA in order to create different protein chains which are used to construct little machines that will replicate the DNA and cells and in turn create machines capable of doing the same.

The process is very complex and not wholly understood, but in effect the end result is that DNA contains the instructions inside of a cell to create the machinery to host another copy of itself and execute that code.

Easter eggs

Let’s not forget the little Easter eggs that have been thrown in by the designer of our codebase.

How about a hiccup or a tear?

Sadness or laughter?

There are some “evolutionary theories” as to why these things exist, but in reality they serve no specific purpose for either our survival or well being.

In my mind these are clearly Easter eggs left by the designer to show His handiwork.

Why am I writing this post now?

I didn’t want to go completely off subject and boast about my wonderful daughter that was born on April 17th 2011 at 2:33 PM, since this is a professional blog. 

But, there you have it.

The amazing birth of my first child reminded my of the incredibly complex code that keeps us all breathing, procreating and living life that we seem to take for granted each and every day.

Welcome to the world Sophia Grace!

As always, you can subscribe to this RSS feed to follow my posts on Making the Complex Simple.  Feel free to check out ElegantCode.com where I post about the topic of writing elegant code about once a week.  Also, you can follow me on twitter here.

This is one of the most critical components of the overall success of your automation strategy, so you will want to make sure you invest properly in this area.

I have seen a large number of automation projects go forth, but each time the critical component determining their success or failure was having a good automation framework.

Solid design and excellent technical resources are absolutely critical to success in this area!

What is an automation framework?

An automation framework is essentially an API that all of your BATs (Blackbox Automated Tests) are written against.

This API can be as simple as the API that is exposed by your browser automation tool (WatiN, Selenium, etc.), but I would highly recommend building your own layer on top of the browser automation tool’s API that will act as a DSL (Domain Specific Language) for automating your application.

Let me break this down a bit further by using an example.

Think about the task of making coffee.

There are several “API” levels we can interact with.

1. We can go with a very low level API where we take whole coffee beans and grind them down.  Then we take some water, get it hot.  We take our filter put the ground beans in it, put it over a cup and pour the water into the filter.
2. We can go with a higher level API where we use a traditional coffee maker.  In this case we load the coffee maker with a filter, ground coffee beans and water and push a “brew” button.  We could also set it to start at a certain time.
3. We can go with a very high level API where we use a Keurig machine or similar device.  In this case we only make sure the machine has water in it, and we just insert a little pod and press brew.  We can make different kinds of coffee, cider or hot cocoa just by changing what pod we use.

Using the API provided by the browser driver is like making the coffee by hand.  It’s going to take a large amount of effort each time you do it.

We want our automation framework to be more like the Keurig machine.  We want to be able to compartmentalize our tests in little pods that are small and don’t require many hooks into the automation framework.

To rehash this one more time, basically our automation framework will be a framework we build on top of the browser driver framework, which is designed to make it easy to write tests which automate our application.

What makes a good automation framework design?

The true measure of an automation framework is the size of the tests that are written against them. The less lines of code in each test, the better the automation framework captures the domain of the application being automated.

In my earlier post about an example of an automation framework, I talked a bit about the strategy I used in a real implementation of an automation framework.

Here is a diagram showing the different layers.  You can see the framework layer is in green here.

You can also see in this diagram that on the right hand side, I have screens, workflows and navigation.

This is one of the common design patterns you can use to build your automation framework, since it closely models that of most web applications.

A good way to design your framework is to use a pattern like this and create classes for each page in your application, create classes which might represent workflows in your application, and create some classes for navigating around the application.

The basic idea you are shooting for with your automation framework is to make it so the tests do not have to know anything about the browser.  Only your automation framework itself should be dealing with the DOM and HTML and CSS.  Your tests should be dealing with concepts which any user would be familiar with.  That is why I commonly use pages and workflows.

Let me give you an example.

Let’s say we ignored the advice of creating an automation framework and decided to program our tests directly against the browser driver layer.  In this case, let’s say that we want to automate a login page.  Our test might look something like this pseudo-code.

browser.goto("http://mywebsite/login.aspx");
Element userName = browser.getElementById("login_userName");
userNameTextField.Type("Joe");

Element userPassword = browser.getElementById("login_userPassword");
userPassword.TextField.Type("mypassword");

Element loginButton = browser.
    getElementByXPath("//button[class="loginButton"]");
loginButton.Click();

This is not very readable.  It is not something an average tester will be able to pick up and start writing, and it is extremely fragile.  If you have 500 tests like this and the id changes for one of the elements on the page, many tests will break.

Now, contrast it to this pseudo-code:

Pages.Login.Goto();
Pages.Login.LoginWithDefaultUser();

Where did all the code go?  You don’t need it!  Actually it went into the framework, where it can be reused.  Now your test code is simple, can be understood by just about anyone, and won’t break with changes to the UI.  (You will just have to change the framework method instead of 500 tests.)

If you want some more detailed examples take a look at my Boise Code Camp slides on the subject here.

Let me offer up some general guidelines for creating a good design for an automation framework.

1. NEVER require the tests to declare variables.
2. NEVER require the tests to use the new keyword or create new objects.
3. NEVER require the tests to manage state on their own.
4. ALWAYS reduce the number of parameters for API calls when possible.
5. ALWAYS use default values instead of requiring a parameter when possible.
6. PREFER to make the API easier to use over making the internals of the API less complex.
7. PREFER using enumerations and constants to requiring the test to pass in primitive types.  (Make the input bounded when possible.)
8. NEVER expose the browser or DOM to the tests or let them manipulate it directly.

You can see that these guidelines will almost force your API to be entirely static methods!  Don’t freak out.  It is perfectly fine.  There is nothing wrong with using static methods in this case.

Stop… take a breath, think about why you think static methods are bad, because I probably agree with you.

But, in this case we are HIGHLY valuing making the tests simple and easy to write over anything else, and because of this emphasis, static methods are going to be the way to go.

Some advice on getting started

Start with the tests in plain human English.

Take those tests and make them into code preserving as much English as possible.  Then work down from there implementing the methods needed to make those tests work.  Only add things as you need them.  Do not overdesign!

In my example above, we might have started with something like this:

1. Goto the login page.

2. Login with the default user.

The actual code looks just like that.  It should always be a 1 to 1 mapping.

This is where having someone that has some experience doing this is going to come in handy.

Just keep in mind the whole time your 2 big goals:

1. Easy to write tests
2. Tests are concise

As always, you can subscribe to this RSS feed to follow my posts on Making the Complex Simple.  Feel free to check out ElegantCode.com where I post about the topic of writing elegant code about once a week.  Also, you can follow me on twitter here.

In my last post in the Back to Basics series, I mentioned that I would begin this series with the intent of showing you step by step how to go from no automation to a quality BAT infrastructure.

By the end of this series you should have the information you need to actually get automated testing going in your organization instead of just talking about it.

First stop on this train is hiring an automation lead.

Don’t doom yourself before you start

If you think you are going to create BATs, but you are not going to have at least 1 dedicated resource to doing the job of getting this all going, you are probably bound for failure.

Listen up, because this is the single biggest mistake that most organizations make when trying to get BATs up.

This mistake is thinking that you can get your QA team to get this going and you can part time some developers onto the project, but not having a dedicated resource.

Why is this absolutely absurd and you should expect it to fail?  Because, in order to get an infrastructure to support your BATs, you are going to need to basically design and create a testing framework that is tailored to the business domain of your application.

Blah, what?  Ok, in simple terms.  Someone is going to need to design and build an automation framework.

If you think you are just going to buy some tool and use it to record test scripts, you are wrong.  That approach simply is not going to work.  I won’t get into why here, but I have written on it before in the past, and if you are interested, let me know and I’ll go much more in depth.

Think about this, do you want your QA team, as good as they may be at QA work, designing and building an automation framework?  Give them a few C# or Java classes and have their first real coding work be to design a complex system?

How about a developer who has “real” work and can only devote part time to designing what may end up becoming one of the most important parts of your infrastructure?

It is fine to pull a developer off of the main product and put them in this role until it is up and running, but make no mistake, you need a dedicated resource.

You are going to want to a well designed maintainable automation framework, because the ease of use of that framework will determine how easy or not it is to write automated tests that use that framework.

I recommend getting someone who has done it before

Why?  Because creating an automation framework is much different than developing your software and it is much different than writing tests.  It really requires a special skill set that is not so easily acquired without the experience of having done this before.

I also think it is fine to use a developer resource, but if you do, you want to try to get a more senior developer to take this role.

Here is a list of some of the skills that I think are important which may also make it much more clear why I am recommending a specialist in this case:

Development of APIs (In the end the automation framework will be an API that is used to write automated tests.  The better the API is designed, the easier it will be to write tests against it.)

Design of reusable base components (Much of the framework consists of repeated concepts with different variations for different screens in your application.  As a result, the core of the framework will be a basic design that allows for variation without too much repetition… reuse)

Ability to quickly acquire domain knowledge of the software (Someone writing an automation framework needs to really understand how the system under test works, so that they can model the framework after it.)

Pragmatism (Many “accepted” good software designs are actually bad automation framework designs.  Static methods can make an automation framework’s API much more simple to use, especially for non-programmers, but would not be a good design choice in your main product.  A person who designs an automation framework should be pragmatic enough to “break the rules” in order to make the framework easiest to use and maintain.)

Experience or understanding of QA (It is important to understand the users of the framework, the QA team.  Understanding what kind of tests they will likely write or how they will think will go a long way in coming up with a good framework design.)

Communication skills (Good BATs read like English.  It is critical to make sure that the person designing the framework facilitates this to make the BATs as useful and understandable as possible.)

HTML / XPath (This specific skill set is very important for web based products.  Creating an automation framework built on top of one of the web browser drivers will require intimate knowledge of HTML and probably XPath to be able to access attributes and elements on pages in your application.)

I’ve written a few automation frameworks and contributed to the design of several others, and I can tell you that my version 6.0 of an automation framework was much much better than version 1.0.

There are a huge number of patterns that emerge that are unique to this problem domain.  So, I do believe having experience in this area specially is a huge benefit.

What does this automation person do?

We’ll get to more specific in my later posts in this series, but the basic responsibility should be:

• Figure out the technology and tools to be used
• Create the initial automation framework
• Create smoke tests for the application under test
• Get those smoke tests into some kind of automated build
• Train developers and QA in the use of the framework
• Drive the adoption of the framework and automation for all features
• Set up rules for consistency in writing automated tests
• Design of hierarchy and structure of automated tests

You want your automation person to be responsible for pushing and driving forward the goal of getting to the point of being able to automate each feature being created with minimal effort.

This is one of those jobs that can actually be “done.”  What I mean by that is you may have the automation lead end up creating the framework, developing it enough to push it out to your development teams and train them to write automated tests along with QA, and you really don’t need an automation lead anymore.

This is great though, because now you have a developer that really knows your product and is a good bridge to QA, or you can keep them improving the framework and overseeing it if you want to go that route.

Bottom line is:

1. Make sure you get a dedicated resource
2. Make sure that resource can write code and design a framework
3. Make sure they train everyone else, one person cannot automate your entire application

As always, you can subscribe to this RSS feed to follow my posts on Making the Complex Simple.  Feel free to check out ElegantCode.com where I post about the topic of writing elegant code about once a week.  Also, you can follow me on twitter here.

But, I think it is worth taking a moment to talk about one of the biggest blunders professional programmers make in their job search, careers.

The mistake is not having a resume that is of the same caliber in both presentation and content as the set of skills you are trying to sell.

I’m no marketing whiz

But, over the years I have come to understand the value of marketing.  As a professional, you should always be trying to sell yourself.

I’ll be honest, part of the reason I write this blog is because I know it is a valuable tool for marketing myself.

For most professional programmers your resume is your only real marketing method. When it looks worse than a 3rd grade book report and I see it, I immediately think you suck.

Sorry for the boldness, but no need to mince words here. 

Unless you have some awesome open source project you built, or a blog, or some other kind of flashy marketing tool, your resume IS your ONLY marketing and if that sucks, people are going to assume you do also. 

To be completely honest again, in some part you have to admit that you do suck. Perhaps you are a great programmer, but you suck at common sense.  I mean going through all the hard work of becoming a great programmer and not having the common sense to do the easy thing that will make it all worth it…

Trent: You know what you are? You’re like a big bear with claws and with fangs…
Sue: …big &!%* teeth, man.
Trent: Yeah… big &!%*teeth on ya’. And she’s just like this little bunny, who’s just kinda cowering in the corner.
Sue: Shivering.
Trent: Yeah, man just kinda… you know, you got these claws and you’re staring at these claws and you’re thinking to yourself, and with these claws you’re thinking, "How am I supposed to kill this bunny, how am I supposed to kill this bunny?"
Sue: And you’re poking at it, you’re poking at it…
Trent: Yeah, you’re not hurting it. You’re just kinda gently batting the bunny around, you know what I mean? And the bunny’s scared Mike, the bunny’s scared of you, shivering.
Sue: And you got these &!%* claws and these fangs…
Trent: And you got these &!%* claws and these fangs, man! And you’re looking at your claws and you’re looking at your fangs. And you’re thinking to yourself, you don’t know what to do, man. "I don’t know how to kill the bunny." With *this* you don’t know how to kill the bunny, do you know what I mean?
Sue: You’re like a big bear, man.
Mike: So you’re not just like &!%* with me?
Trent: No I’m not &!%* with you.
Sue: Honestly, man.

I know you might be thinking I am being harsh here, but really it is for your own good.

Let me give you an example.

Lets pretend that Pepsi let me design their new logo and ad copy for a magazine ad.

PEPSI!  It eez so freakin good it will nock you’re socks off!

I drink it all the time.  I have spearheaded drinking it at parties with 100-1000 people.

PEPSI! YES!  TOTALLY AWESOME AND QUITE DRINKABLE!

Next time you are at the store and you see a 12 pack of Pepsi with my hand drawn logo on it, and a big white board with no color that says "DRINK PEPSI! TOTALLY AWESOME AND QUITE DRINKABLE" are you going to buy it or are you going to think whoever made that piece of crap is a doofus?

The situation is far worse than what I described.  Pepsi has many marketing venues and advertising campaigns.  If all you have is your resume, then you just have one. There is nothing wrong with that, but it better be damn good!

Your resume is like a Superbowl ad for you and your programming skills.

I would love to make a nice resume, but I do suck at writing and design

It is good that you admit that, because if you are a programmer and your name isn’t Hugo, you probably do suck at writing and design and that is Ok. 

You should not be the person writing your resume or designing it.  You should provide the information, you should review the design, you should even proof-read and edit it, but you should be letting someone else do most of the work.

Why?  Because there are professionals that know how to write and design a resume. Perhaps with many hours of work and research you could create a resume as good or better than a professional resume writer, but chances are you can’t.

Just like a professional resume writer shouldn’t try and write their own software, they should hire you to do it, you shouldn’t try and write your own resume.

This might seem like strange advice.  Perhaps it seems like a cop-out or uber-lame. You are welcome to think that if you want.

Perhaps you think, well if everyone did that then all resumes would be nicely formatted, not contain spelling errors and typos and be full of marketing hype. 

That is true, but the good news for you is that HARDLY ANYONE IS DOING IT and by being one of the few who do, your resume will stand out and make the others look like something your cat puked up.

If you can get an edge, take the edge!

People should judge my skills not my design and formatting on my resume

I completely disagree.  Having an ugly resume tells me a large amount about you… shall I list a few prejudice things I assume about you when I see your double spaced 6 page monstrosity?

1. Doesn’t care about appearances – hmm too bad my customers do.
2. Doesn’t have self-respect – those who don’t respect themselves don’t respect others usually.
3. No sense of pride – if you can’t take pride in your own work, why do I expect you to take pride in the software you are building for me?
4. Doesn’t value the opinion of others.
5. Is limited by own abilities – if you can’t reach out to others for help in your area of weakness, I expect when you can’t figure something out, you’ll just flounder.
6. Doesn’t care about quality.
7. Is not detail oriented.
8. Lacks ambition, will settle for less.
9. Your code reflects all the above statements.

Unfair?  Perhaps.  Untrue?  Perhaps, but given no other information and a stack of 100 more resumes to go through, I’m going to go ahead and be prejudice.

But it costs too much money

It is called an investment. If you are not willing to put some money into your career, then why should someone else be willing to take a chance and make an investment in you?

A good professional resume writer might charge around $500.  That is chump change compared to the potential earnings increase of having a professionally written and designed resume.  You may increase your negotiating power by a huge margin by appearing to be more professional and you will certainly increase the number of possible opportunities that are available to you.

Let’s use some real fake numbers like I always like to do.

Suppose that you have skills that on average would be worth about 60k a year in the market and your resume is getting around a 10% response rate from jobs you submit to.

Now of the jobs you submit to, most of the 10% responses you are getting are for jobs that pay right at your market rate, some are above, very few are way above, and some are below.

If you invest $500 in getting a really well done professional resume, you could reasonably expect to increase the base value of your market rate to say 65k and improve your response rate to 20%.

That is a pretty modest improvement, which only requires an investment of money to achieve.  The immediate return on that investment is HUGE, and the lifelong return on that investment could be literally 10s of 1000s of percent.

Think about it.  If you increase your starting wage at a company by just 5k, for a $500 investment, and you work for that company for 10 years, if you got no raises at all, you would make 50k return on your $500 investment.

Raises, by the way, are usually based on percentages of base pay, so it is realistically going to be even more.

Two more points, then I promise to stop beating this horse.

• It is likely that someone who had a very bad resume and expected to find jobs making 60k, could get a very good resume and find that one company willing to pay them 80k for that same set of skills.
• It is very much more likely that the dream job you are after will have a much wider pool of candidates to choose from, even if they are paying the same market rate.  So by having the better resume, you are more likely to get seen and get that dream job.

Final advice

• Don’t hire some cheap resume writing service. You get what you pay for.  Spend the money to get it done by a real professional who will do it right.  This is the only hard part you have to do, you need to do a large amount of research here.  I have hired several resume writers for myself and my wife, and I have had to fire a few, because they were basically just rip-offs.  If you try to be cheap and you end up getting ripped off, don’t come crying to me.
• Review the resume.  You will need to go through more than 1 revision, probably at least 5.  If you want something changed, ask for it.  They will do it, you are paying for it.
• You are still going to do most of the work of gathering the information.  Figure out all the things you want to have on your resume and put it all in a mind-map to give to the resume writer.  Garbage in, garbage out.
• Whether you take this advice or not, before you ever send your resume to an employer, send it to your mom, your girlfriend, your English major buddy, anyone who will look at it and tell you why it is ugly and what you need to do to fix it.

As always, you can subscribe to this RSS feed to follow my posts on Making the Complex Simple.  Feel free to check out ElegantCode.com where I post about the topic of writing elegant code about once a week.  Also, you can follow me on twitter here.

This time I want to take a look at some of the common patterns we can use to extract parts of our code into dependency-lite or dependency-less classes that we can unit test without needing mocks.

I’ll have to admit, I haven’t really be practicing this approach for a long time, so my list of patterns is probably lacking, but these are some of the ones I kept seeing as I was working through the code.

If you have some ideas for other patterns or know of some already or have better names than what I have chosen, please let me know.

Pattern 1: Pull out state

In many cases we can have complex objects that contain their own state often split across member variables.

If you find that a class has many methods that check some member variable and do one thing or another based on what value that member variable is set to, you might be able to benefit by extracting all the state changing logic into a separate class and creating events to notify the original class when the state changes.

We can then keep the new class as the single source of state for the class it came from and easily write dependency free (level 2) unit tests for the state class.

From the perspective of the class you are pulling the state out of, we might turn:

isSpecial = false;
timeToLive = 50;
timeToLove = 1
penguinsHaveAttacked = true;

public ApocalypseScenario CreateNew()
{
    if(isSpecial && timeToLive < timeToLove || penguinsHaveAttacked)
        return new ApocalypseScenario("Penguin Apocalypse, Code Blue");
}

Into:

private apocalypseStateMachine = new ApocalypseStateMachine();

public ApocalypseScenario CreateNew()
{
    if(apocalypseStateMachine.State == States.PENGUIN_SLAUGHTER)
        return new ApocalypseScenario("Penguin Apocalypse, Code Blue");
}

Obviously in this case there would be many ways the states get changed.  I haven’t included those here, but you can imagine how those would become part of the new state class.

Pattern 2: Method to class

Often I have found that I can’t find a way to extract all of the logic out of a class into one cohesive class, because of multiple interactions inside the class.

When I encounter this problem, I have found that I can usually identify a single large method that is using data from dependencies in the class to perform its logic.

These methods can usually be extracted to be their own new class.  We can pass in the data from the dependencies the method was originally using instead of using the dependencies directly.

Good candidates for this kind of pattern are methods which use multiple dependencies to get data in order to compute a single result.

When applying this pattern we may end up with a very small class, containing only 1 or 2 methods, but we are able to easily unit test this class with state and dependency free unit tests (level 1.)

Pattern 3: Pull data sources up

It is often the case that we have dependencies in a class that exist only to provide some sort of data to the actual logic in the class.

In these situations it is often possible to pull the source of the data up and out of the methods that use the data and instead pass in just the used data.

Performing this refactoring allows us to have methods in our class that do not use the dependency in the class directly, but instead have the data that dependency class provided passed in.

When we do this, it opens up the ability to write clean and simple unit tests without mocks for those methods in the class or to apply the “Method to class” pattern to extract the method into its own testable class.

As a first step to refactoring my code to make it easily unit testable, I will often scan the methods in the class to find methods that are only using data from dependencies, but not manipulating the state or sending commands to the dependencies.  As I am able to identify these methods, I can apply this pattern to remove the dependencies from the methods.

Pattern 4: Gather then use

Often we find that we cannot pull a dependency out of a method because that dependency is being manipulated inside of a loop or in the middle of some process.

In cases like these, we can often gather all the data that will be used to manipulate the dependency in some form of a collection and then reiterate through that collection to perform the manipulation on the dependency.

A good way to envision this is to think about two people mailing letters.  In one scenario we might have the first guy stuffing the envelope and the 2nd guy sealing the envelope.

We could carry on this process of 1st guy stuffs, 2nd guy seals for each letter we want to mail, but the process is dependent on both guys being present and working.

If we change it so that the first guy stuffs all the envelopes first and gives that stack of stuffed envelopes to the 2nd guy, they can do their jobs independently.

By applying the same idea to algorithms in our code, we can take a single method and break off the parts that don’t rely on dependencies to test independently or even move to their own classes.

As always, you can subscribe to this RSS feed to follow my posts on Making the Complex Simple.  Feel free to check out ElegantCode.com where I post about the topic of writing elegant code about once a week.  Also, you can follow me on twitter here.

One of the approaches I suggested was to focus on writing level 1 or level 2 unit tests and occasionally level 3 unit tests with 1 or possibly 2 dependencies at most.

I want to focus on how this is possible, using real code where I can.

First let’s talk about the approach

When I first talk about writing mainly level 1 or level 2 unit tests, most people assume that I mean to cherry pick the few classes in your code base that already qualify and only write unit tests for those classes.

That is not at all what I am advocating.

Instead, the approach I am suggesting is to find ways to make most of the actual logic in your code become encapsulated into classes that depend mostly on data.

What I mean by this is that our goal should be to refactor or write our code in such a way that logic is grouped into classes that only depend on primitive types and data classes, not other classes that contain logic.

This of course is not fully achievable, because something will have to tie all of these logic containing classes together.  We we need these tie-together classes, but if we can make their job to simply execute commands and tie other classes together, we can feel pretty confident in not unit testing them, and we make our job a whole lot easier.

So to summarize, the basic strategy we are going to employ here is to eliminate the need for mocks by designing our classes to be much smaller, having much tighter roles and responsibilities, and operating on data that is passed in rather than manipulating methods on other classes.

There are many patterns we can use to achieve this goal.  I’ll show you an example, then I’ll try to cover some of the major patterns I have discovered so far.

A real world example

I recently released a Java based Android application called PaceMaker.  When I had started out building this application, I set out with the high and mighty goal of using Google Guice framework for dependency injection and BDD style unit tests using JMock to mock passed in dependencies.  It wasn’t a horrible approach, I wrote about it here.

What I found with this approach though was that I was spending a large amount of time creating mocks, and I wasn’t getting much benefit from it.  So, I had abandoned writing unit tests for the code all together, and I pulled out the now almost useless Guice.

The past couple of nights, I decided to use this project as a test project to demonstrate some of the ideas I have been talking about and thinking about.

I wanted to take my real code, refactor the logic out of it into small classes with little or no dependencies, and write BDD style unit tests that would be simple and easy to understand.

The big challenge here was trying to find a small enough piece of code to use an as example.  For this example, I am going to use a bit of code that I was using to generate the name of a file that I write to disk for saving the data from a run.

This code was originally inside of a presenter class that handled generating a file name to pass to a serializer class in order to serialize the run.

Here is the original private method that existed in the presenter.

private String getRunFileName()
{
     String completeFileName = StorageManager.getDataStorageDirectory();
     Location firstLocation = locationTracker
          .getLocations().iterator().next();
     Date firstPointTime = new Date(firstLocation.getTime());

     SimpleDateFormat dateFormat = new
         SimpleDateFormat("MMddyyyy_HHmmss");
     dateFormat.setTimeZone(TimeZone.getTimeZone("GMT"));
     String fileName = dateFormat.format(firstPointTime) + ".gpx";

     completeFileName = completeFileName + fileName;
     return completeFileName;
}

This method was a perfect candidate for some easily testable logic that could be put into its own class, but there are a few problems we should notice here.

• We are dependent on StorageManager to get the data storage directory used as the base directory.
• We are dependent on the locationTracker object to get the time of the first location.
• There is some real logic here in the form of a transformation.  (It is important to note that we are dealing with logic, not just commands, because testing execution of commands is not as important as testing logic.)

My approach to this refactor is actually pretty simple.  The first thing we need to do is see the dependencies for what they are.  It looks like our logic is dependent on StorageManager and locationTracker, but in reality the logic is dependent on the string which is the base directory for the file and the time to use for the file name.

We can change this code to reflect that pretty easily.

private String getRunFileName(String baseDirectory, Calendar time)
{
     String completeFileName = baseDirectory;
     SimpleDateFormat dateFormat = new
         SimpleDateFormat("MMddyyyy_HHmmss");
     dateFormat.setTimeZone(TimeZone.getTimeZone("GMT"));
     String fileName = dateFormat.format(time.getTime()) + ".gpx";

     completeFileName = completeFileName + fileName;
     return completeFileName;
}

What we have done here is small, but it is critical.  We have eliminated dependencies that would otherwise have to be mocked to test this logic.  Sure, we will still need to use those dependencies to get the data to pass into this method, but we can leave that code in the presenter class and move this code into its own class.  The class will be small for now, but it will be easily testable with a level 1 unit test (our favorite kind.)

More examples

I didn’t cherry pick this example from the source code in PaceMaker, but I did cherry pick it for this blog post, because it was one of the shorter examples I could use.

I have several other areas of code in my presenter class in PaceMaker where I used a similar approach to extract out the logic, pull it into its own class with little or no dependencies and write unit tests for.

Here are two other examples:

• Pulled the state logic for starting, stopping, pausing, resuming and calculating length of time paused during a run into its own class.  I elected to add one dependency (the presenter itself as an observer) to the refactored class in order to allow the class to notify the presenter when the state changed.  In C# I would have just used an event to do this, but in Java we use the observer pattern.
• Pulled out the logic that created the GPX data files into a GPXDataModelBuilder which instead of depending on the LocationTracker class depended only on the data from that class.

The result ended up being very clear, easy to write, level 1 and level 2 unit tests with no mocking.  In addition, my class structure is now much more tightly cohesive, with a much tighter and clearer responsibility.  Before my presenter was doing several things, but now many of those things are broken up into very small testable classes.

In my next post, I’ll go into the patterns you can use to create classes that are able to be tested by level 1 and level 2 unit tests.

As always, you can subscribe to this RSS feed to follow my posts on Making the Complex Simple.  Feel free to check out ElegantCode.com where I post about the topic of writing elegant code about once a week.  Also, you can follow me on twitter here.

We’ve talked about what an interface is, cohesion and coupling, and even went a little bit off track to talk about sorting for a bit.

One of the reoccurring themes that kept showing up in most of the posts was unit testing.  I talked about why unit testing is hard, and I defined three levels of unit testing.

• Level 1 – we have a single class with no external dependencies and no state.  We are just testing an algorithm.
• Level 2 – we have a single class with no external dependencies but it does have state.  We are setting up an object and testing it as a whole.
• Level 3 – we have a single class with at least one external dependency, but it does not depend on its own internal state.
• Level 4 – we have a single class with at least one external dependency and depends on its own internal state.

Throughout this series I ended up tearing down using interfaces with only single non-unit test implementation.  I criticized the overuse of dependency injection for the sole purpose of unit testing.  I attacked a large portion of best practices that I felt were only really being used in order to be able to unit test classes in isolation.

But, I never offered a solution.  I told you what was bad, but I never told you what was good.

I said don’t create all these extra interfaces, use IoC containers all over your app, and mocks everywhere just for the purpose of being able to isolate a class you want to unit test, but when you asked me what to do instead, I said “I don’t know, I just know what we are doing is wrong and we need to stop.”

Well, that is no answer, but I intend to give one now.  I’ve been thinking about this for months, researching the topic and experimenting on my own.

I finally have an answer

But, before I give you it, I want to give you a little background on my position on the subject matter.

I come from a pretty solid background of unit testing and test driven development.  I have been preaching both for at least the last 7 years.

I was on board from the beginning with dependency injection and IoC containers.  I had even rolled my own as a way to facilitate isolating dependencies for true unit tests.

I think unit testing and TDD are very good skills to have.  I think everyone should learn them.  TDD truly helps you write object oriented code with small concentrated areas of responsibility.

But, after all this time I have finally concluded, for the most part, that unit tests and practicing TDD in general do more good for the coder than the software.

What?  How can I speak such blasphemy?

The truth of the matter is that I have personally grown as a developer by learning and practicing TDD, which has lead me to build better software, but not because the unit tests themselves did much. 

What happened is that while I was feeling all that pain of creating mocks for dependencies and trying to unit test code after I had written it, I was learning to reduce dependencies and how to create proper abstractions. 

I feel like I learned the most when the IoC frameworks were the weakest, because I was forced to minimize dependencies for the pain of trying to create so many mocks or not being able to unit test a class in isolation at all.

I’ve gotten to the point now where two things have happened:

1. I don’t need the TDD training wheels anymore.  I don’t pretend to be a coding god or demi-god of some sort, but in general the code I write that is done in a TDD or BDD style is almost exactly the same as the code I write without it.
2. The IoC containers have made it so easy to pass 50 dependencies into my constructor that I am no longer feeling the pain that caused my unit tests to cause me to write better code.

What I find myself ending up with now when I write unit tests is 70% mocking code that verifies that my code calls certain methods in a certain order.

Many times I can’t even be sure if my unit test is actually testing what I think it is, because it is so complex.

Umm, did you say you had an answer, dude?

Yes, I do have an answer.  I just wanted to make sure you understand where I am coming from before I throw out all these years of practical knowledge and good practices.

I am not the enemy.

My answer to the problem of what to do if you shouldn’t be using IoC containers and interfaces all over your code base just for the purpose of unit testing, is to take a two pronged approach.

1. Mostly only write level 1 or level 2 unit tests.  Occasionally write level 3 unit tests if you only have 1 or possibly 2 dependencies.  (I’ll talk about more how to do this in my next post)
2. Spend a majority of your effort, all the time you would have spent writing unit tests, instead writing what I will call blackbox automated tests or BATs.  (I used to call this automated functional tests, but I think that name is too ambiguous.)

I intend to drill really deep into these approaches in some upcoming posts, but I want to briefly talk about why I am suggesting these two things in place of traditional BDD or TDD approaches.

What are the benefits?

The first obvious benefit is that you won’t be complicating your production code with complex frameworks for injecting dependencies and other clever things that really amount to making unit testing easier.

Again, I am not saying you shouldn’t ever use dependency injection, interfaces or IoC containers.  I am just saying you should use them when they provide a real tangible value (which most of the time is going to require alternate non-unit test implementations of an interface.)

Think about how much simpler your code would be if you just went ahead and new’d up a concrete class when you needed it.  If you didn’t create an extra interface for it, and then pass it in the constructor.  You just used it where you needed it and that was that.

The second benefit is that you won’t spend so much time writing hard unit tests.  I know that when I am writing code for a feature I usually spend at least half the amount of time writing unit tests.  This is mostly because I am writing level 3 and level 4 unit tests, which require a large number of mocks.

Mocks kill us.  Mocking has a negative ROI.  Not only is creating them expensive in terms of time, but it also strongly couples our test classes to the system and makes them very fragile.  Plus, mocking adds huge amounts of complexity to unit tests.  Mocking usually ends up causing our unit test code to become unreadable, which makes it almost worthless.

I’ve been writing mocks for years.  I know just about every trick in the book.  I can show you how to do it in Java, in C#, even in C++.  It is always painful, even with auto-mocking libraries.

By skipping the hard unit tests and finding smart ways to make more classes only require level 1 and level 2 unit tests, you are making your job a whole lot easier and maximizing on the activities that give you a high ROI.  Level 1 and level 2 unit tests, in my estimation, give very high ROIs.

The thirds benefit is that blackbox automated tests are the most valuable tests in your entire system and now you’ll be writing more of them.  There are many names for these tests, I am calling them BATs now, but basically this is what most companies call automation.  Unfortunately, most companies leave this job to a QA automation engineer instead of the development teams.  Don’t get me wrong, QA automation engineers are great, but there aren’t many of them, good ones are very expensive, and the responsibility shouldn’t lie squarely on their shoulders.

BATs test the whole system working together.  BATs are your automated regression tests for the entire system.  BATs are automated customer acceptance tests and the ROI for each line for code in a BAT can be much higher than the ROI of each line of production code.

Why?  How is this even possible?  It’s all about leverage baby.  Each line of code in a BAT may be exercising anywhere from 5 to 500 lines of production code, which is quite the opposite case of a unit test where each line of unit test code might only be testing a 1/8th or 1/16th a line of production code on average (depending on code coverage numbers being reached.)

I’ll save the detail for later posts, but it is my strong opinion that a majority of a development teams effort should be put in BATs, because BATs

• Have high value to the customer
• Regression test the entire system
• Have a huge ROI per line of code (if you create a proper BAT framework)

Imagine how much higher quality your software would be if you had a BAT for each backlog item in your system which you could run every single iteration of your development process.  Imagine how confident you would be in making changes to the system, knowing that you have an automated set of tests that will catch almost any break in functionality.

Don’t you think that is worth giving up writing level 3 and level 4 unit tests, which are already painful and not very fun to begin with to achieve?

In my future posts on the Back to Basics series, I will cover in-depth how to push more of your code into level 1 and level 2 unit tests by extracting logic out to separate classes that have no dependencies, and I will talk more about BATs, and how to get started and be successful using them.  (Hint: you need a good BAT framework.)

As always, you can subscribe to this RSS feed to follow my posts on Making the Complex Simple.  Feel free to check out ElegantCode.com where I post about the topic of writing elegant code about once a week.  Also, you can follow me on twitter here.

I’m not going to talk about that subject yet.  Instead, I want to look at some of the costs of unit testing and ask the question “why is unit testing hard?”

After all, if unit testing weren’t hard, we wouldn’t have to question whether or not it was worth it.  It makes sense then to look at first why it is hard and what makes it hard.

The ideal scenario

Unit testing itself is rather easy once you understand how to do it.  Even test driven or behavior driven development is easy one mastered… at least for the ideal scenario.

What is the ideal scenario then?

It is a unit test where the class under test has no external dependencies.

When a class we are writing unit tests for doesn’t have any external dependencies, we don’t need mocks or stubs or anything else.  We can just write code that tests our code.

Let’s look at an example of this.  Suppose, I had a class called Calculator.  This Calculator class has some very simple methods.  Specifically, let us talk about testing a method Add.  Add takes two single digit integers and returns the result.  If either integer passed in is more than a single digit, it throws an exception.

It is a pretty stupid method, with little use, but it will serve the point well here.

We can TDD or BDD this baby with minimal effort.

Let’s start thinking of test cases:

• When I add 0 and a single digit number it should return the single digit number
• When I add 0 and 0 it should return 0
• When I add two single digit numbers it should return the sum of those numbers
• When I add one two digit number it should throw an exception

Pretty easy to come up with test cases, just as easy to implement them:

[Test]
public void ZeroAndANumber_IsANumber()
{
   var calculator = new Calculator();
   var result =  calculator.Add(0, 5);
   
   Assert.AreEqual(5, result);
}

We can then implement the code that will make this test pass pretty easily.  I won’t show it here since it is so trivial.

This is the ideal scenario, or what I will call Level 1 Unit Testing.

Level 1 Unit Testing is where we have a single class with no external dependencies and no state.  We are just testing an algorithm.

Taking it up a notch

The next level of unit testing is reached if we add state to the class under test.

Level 2 Unit Testing is where we have a single class with no external dependencies but it does have state.  We are setting up an object and testing it as a whole.

If we take our existing example, and now we want to add a new method called GetHistory, it is still not difficult to implement the tests, but it gets harder, because we have to make sure we are setting up some state for our object as part of the test.

Let’s look at one of the test cases we might implement for this functionality:

[Test]
public void When3AddOperationsThenGetHistory_ShouldReturnThose3Results()
{
   var calculator = new Calculator();

   // Arrange   
   calculator.Add(1, 3);
   calculator.Add(2, 5);
   calculator.Add(3, 6);

   // Act
   var result = calculator.GetHistory();

   // Assert
   Assert.Equal(4, result[0]);
   Assert.Equal(7, resut[1]);
   Assert.Equal(9, result[2]);
   Assert.Equal(3, result.Count);
}

Again, not too hard.  But, state does make this a bit more difficult.  Here the value of a Behavior Driven Development (BDD) style of unit testing can be seen as it helps us to clearly divide the test up into the different parts we now have.

The real complexity we have added here is that we now have to deal with a setup step before we can execute our test.  BDD deals with this by having a special step for defining the context in which the actual test takes place.  It is called a few different things in different BDD circles, but let’s stick with AAA for this post, since it is easy to remember.

The major difference between Level 1 Unit Testing and Level 2 Unit Testing is that in Level 1, we were really testing only one method.  In Level 2 we are testing at the class level.  Really we could call Level 1 Unit Testing method testing, since the unit we are testing is the method.  The class that method existed in didn’t matter.

Enter dependencies

Let’s see what happens when we throw dependencies into the Calculator class.

Imagine that our Calculator class has to keep an audit trail of our calculations.  We have a service that we can use to put calculations from the Add method into a storage location, like a database and our GetHistory method can query the storage location for the history.

As I was thinking about this, an important point occurred to me.  Were this an integration test, our example test method above wouldn’t change at all.

But, as it turns out we are talking about unit tests here, we need to isolate the testing down to the class level.

So let’s think about what our test should do now.  Here are some possible tests we might have.

• When I add two number the result is returned and the Store method is called on the StorageService with that result.
• When I get the history, the RetrieveHistory method is called on the StorageService and it’s results are returned back.

Let’s see what one of these tests might look like:

[Test]
public void WhenAdding2NumbersAndServiceOnline_SumIsReturnedAndStored()
{
   // Arrange
   IStorageService storageServiceMock = Mocker.Mock<IStorageService>();
   storageServiceMock.Stub(service => service.IsServiceOnline())
          .Return(true);

    var calculator = new Calculator(storageServiceMock);   

   // Act
   var result = calculator.Add(3, 4);

   // Assert
   storageServiceMock.AssertWasCalled(service => service.Store(7);
   Assert.Equals(7, result);
}

I call this Level 3 Unit Testing.

Level 3 Unit Testing is when we have a single class with at least one external dependency, but it does not depend on its own internal state.

Things really start to get complicated here, because we have to start thinking not just about inputs and outputs and sequences, but now have to think about interactions.

It really starts to get blurry here about what the expectations of our unit tests should be.  In the example code above, do we need to check to make sure IsServiceOnline is called on the StorageService or do we only check that Store was called?

You’ll also notice here that we had to use a mock and pass our dependency into our class so that we could change its behavior.  Along with that came the burden of creating an interface, so that we could have a mock implementation.

If you’re paying attention right now, you may be thinking to yourself that the example is bad.  You may be thinking that the Calculator class now has two responsibilities.

• It calculates things and return the result
• It stores calculation results

Right you are, but we can’t wish away this problem.  Let’s suppose we refactor and move the StorageService dependency out of the Calculator class.  We have several options.  We could make a decorator and use it like this:

var calculator = new Calculator();
var StoringCalculator = new StoringCalculator(calculator);

Or we could do something more like a mediator pattern like so:

var calcMediator = new CalculatorMediator(calculator, storageService);

However we attempt to solve this problem, we are still going to have to have some class that will have to have a mock in its unit test. 

There is a simple fact that we cannot get around. If we are going to use the StorageService to store calculations, either Calculator will depend on it, or something else will depend on calculator and it.  There is no alternative to those two options.

There is another simple fact we can’t get around also.  If we are going to depend on another class in our unit test, we either need an interface that we can use for the mock class, or we need a mocking framework that will support mocking concrete classes.

So with Level 3 Unit Testing we are stuck with needing to mock at least one dependency and either creating a bogus interface, or using a mocking library that will let us mock concrete classes.

It gets worse

It only gets worse from here.  At Level 3 we didn’t worry about state inside our calculator class, we worried about an external dependency that pretty much handled state for us.  In many cases though we will have to worry about state and dependencies.

Level 4 Unit Testing is when we have a single class with at least one external dependency and depends on its own internal state.

In our calculator example, we can simply add the requirement that we only want to get the history for a particular session of calculations.  We need to keep track of the calculations so that we can ask the StorageService for the history for our session.

Let’s look at a possible test for that scenario:

[Test]
public void When3AddsThenGetHistory_ShouldReturnOnlyThose3Results()
{
   // Arrange   
   IStorageService storageServiceMock = Mocker.Mock<IStorageService>();
   storageServiceMock.Stub(service => service.IsServiceOnline())
          .Return(true);
   storageServiceMock.Stub(service => service.GetHistorySession(1))
          .Return(new List<int>{4, 7, 9});
   var calculator = new Calculator(storageServiceMock);


   calculator.Add(1, 3);
   calculator.Add(2, 5);
   calculator.Add(3, 6);

   // Act
   var result = calculator.GetHistory();

   // Assert
   storageServiceMock.AssertWasCalled(service => 
              service.GetHistorySession(1);
   Assert.Equal(4, result[0]);
   Assert.Equal(7, resut[1]);
   Assert.Equal(9, result[2]);
   Assert.Equal(3, result.Count);
}

Consider for a moment how fragile and complex this unit testing code is.  Consider how simple the functionality of our class is.

We have a major problem here.  Our unit testing code is more complex than the code it is testing!  It’s ok, if the unit testing code is more lines of code than the code it is testing, that is usually the case.  But, I consider it a big problem when our unit testing code is more complex, because you have to ask yourself the very real question.

Where is there more likely to be a bug?

I’m not saying anything yet

My point is not to make a point, at least not yet.  My real goal here is to help us to change the way we think about unit testing.

We need to stop asking the general question of whether not unit testing is worth the cost and instead ask the more specific question of what level of unit testing is worth the cost.

Level 3+ has a very steep cost as mocking is unavoidable and adds considerable complexity to even the most trivial of implementations.

From that we can draw a bit of wisdom.  If we are going to unit test we should strive to encapsulate as much of our pure logic into classes without dependencies and if possible without state.

The other thing to consider is that as the difficulty and complexity of the unit tests are increasing each level, the goal of the test and value starts to become lost also.

What I mean by this, is that when we start testing that our class properly calls another class with certain parameters, we are crossing over into testing the implementation details of the class.

If I say a class should be able to add 2 numbers and return the result.  I am not talking about how it has to do it.  As long as the result is correct, how doesn’t matter.

When I add a mock and say a class needs to add 2 numbers and store the result using a StorageService by calling the method Store on it, I have now tied how into the test.  Changing how breaks the test.

That’s all we are going to look at for now.  If you’ve read some of my other back to basics posts, you can see the progression up to this point.  I’ve been discounting using interfaces and dependency injection for the sake of unit testing, but I have yet to offer an alternative.  I still don’t.  To be honest, I don’t have one yet.  But, I do believe by breaking down this problem to its roots we can evaluate what we are doing and determine what our true problems are.

By the end of this series I hope to have a solution and a recommendation for tackling these kinds of problems.

As always, you can subscribe to this RSS feed to follow my posts on Making the Complex Simple.  Feel free to check out ElegantCode.com where I post about the topic of writing elegant code about once a week.  Also, you can follow me on twitter here.