Back to Basics: Unit Testing, Automated Blackbox Testing, and Conclusions!

If you’ve been following me from the beginning of the Back to Basics series, you’ll know that I set out to reevaluate some of the commonly held truths of what best practices are, especially in regards to unit testing, dependency injection and inversion of control containers.

We’ve talked about what an interface is, cohesion and coupling, and even went a little bit off track to talk about sorting for a bit.

One of the reoccurring themes that kept showing up in most of the posts was unit testing.  I talked about why unit testing is hard, and I defined three levels of unit testing.

  • Level 1 – we have a single class with no external dependencies and no state.  We are just testing an algorithm.
  • Level 2 – we have a single class with no external dependencies but it does have state.  We are setting up an object and testing it as a whole.
  • Level 3 – we have a single class with at least one external dependency, but it does not depend on its own internal state.
  • Level 4 – we have a single class with at least one external dependency and depends on its own internal state.

Throughout this series I ended up tearing down using interfaces with only single non-unit test implementation.  I criticized the overuse of dependency injection for the sole purpose of unit testing.  I attacked a large portion of best practices that I felt were only really being used in order to be able to unit test classes in isolation.

But, I never offered a solution.  I told you what was bad, but I never told you what was good.

I said don’t create all these extra interfaces, use IoC containers all over your app, and mocks everywhere just for the purpose of being able to isolate a class you want to unit test, but when you asked me what to do instead, I said “I don’t know, I just know what we are doing is wrong and we need to stop.”

Well, that is no answer, but I intend to give one now.  I’ve been thinking about this for months, researching the topic and experimenting on my own.

cool experiment thumb Back to Basics: Unit Testing, Automated Blackbox Testing, and Conclusions!

I finally have an answer

But, before I give you it, I want to give you a little background on my position on the subject matter.

I come from a pretty solid background of unit testing and test driven development.  I have been preaching both for at least the last 7 years.

I was on board from the beginning with dependency injection and IoC containers.  I had even rolled my own as a way to facilitate isolating dependencies for true unit tests.

I think unit testing and TDD are very good skills to have.  I think everyone should learn them.  TDD truly helps you write object oriented code with small concentrated areas of responsibility.

But, after all this time I have finally concluded, for the most part, that unit tests and practicing TDD in general do more good for the coder than the software.

What?  How can I speak such blasphemy?

The truth of the matter is that I have personally grown as a developer by learning and practicing TDD, which has lead me to build better software, but not because the unit tests themselves did much. 

What happened is that while I was feeling all that pain of creating mocks for dependencies and trying to unit test code after I had written it, I was learning to reduce dependencies and how to create proper abstractions. 

I feel like I learned the most when the IoC frameworks were the weakest, because I was forced to minimize dependencies for the pain of trying to create so many mocks or not being able to unit test a class in isolation at all.

I’ve gotten to the point now where two things have happened:

  1. I don’t need the TDD training wheels anymore.  I don’t pretend to be a coding god or demi-god of some sort, but in general the code I write that is done in a TDD or BDD style is almost exactly the same as the code I write without it.
  2. The IoC containers have made it so easy to pass 50 dependencies into my constructor that I am no longer feeling the pain that caused my unit tests to cause me to write better code.

What I find myself ending up with now when I write unit tests is 70% mocking code that verifies that my code calls certain methods in a certain order.

Many times I can’t even be sure if my unit test is actually testing what I think it is, because it is so complex.

Umm, did you say you had an answer, dude?

Yes, I do have an answer.  I just wanted to make sure you understand where I am coming from before I throw out all these years of practical knowledge and good practices.

I am not the enemy.

My answer to the problem of what to do if you shouldn’t be using IoC containers and interfaces all over your code base just for the purpose of unit testing, is to take a two pronged approach.

2prong thumb Back to Basics: Unit Testing, Automated Blackbox Testing, and Conclusions!

  1. Mostly only write level 1 or level 2 unit tests.  Occasionally write level 3 unit tests if you only have 1 or possibly 2 dependencies.  (I’ll talk about more how to do this in my next post)
  2. Spend a majority of your effort, all the time you would have spent writing unit tests, instead writing what I will call blackbox automated tests or BATs.  (I used to call this automated functional tests, but I think that name is too ambiguous.)

I intend to drill really deep into these approaches in some upcoming posts, but I want to briefly talk about why I am suggesting these two things in place of traditional BDD or TDD approaches.

What are the benefits?

The first obvious benefit is that you won’t be complicating your production code with complex frameworks for injecting dependencies and other clever things that really amount to making unit testing easier.

Again, I am not saying you shouldn’t ever use dependency injection, interfaces or IoC containers.  I am just saying you should use them when they provide a real tangible value (which most of the time is going to require alternate non-unit test implementations of an interface.)

Think about how much simpler your code would be if you just went ahead and new’d up a concrete class when you needed it.  If you didn’t create an extra interface for it, and then pass it in the constructor.  You just used it where you needed it and that was that.

The second benefit is that you won’t spend so much time writing hard unit tests.  I know that when I am writing code for a feature I usually spend at least half the amount of time writing unit tests.  This is mostly because I am writing level 3 and level 4 unit tests, which require a large number of mocks.

Mocks kill us.  Mocking has a negative ROI.  Not only is creating them expensive in terms of time, but it also strongly couples our test classes to the system and makes them very fragile.  Plus, mocking adds huge amounts of complexity to unit tests.  Mocking usually ends up causing our unit test code to become unreadable, which makes it almost worthless.

I’ve been writing mocks for years.  I know just about every trick in the book.  I can show you how to do it in Java, in C#, even in C++.  It is always painful, even with auto-mocking libraries.

By skipping the hard unit tests and finding smart ways to make more classes only require level 1 and level 2 unit tests, you are making your job a whole lot easier and maximizing on the activities that give you a high ROI.  Level 1 and level 2 unit tests, in my estimation, give very high ROIs.

The thirds benefit is that blackbox automated tests are the most valuable tests in your entire system and now you’ll be writing more of them.  There are many names for these tests, I am calling them BATs now, but basically this is what most companies call automation.  Unfortunately, most companies leave this job to a QA automation engineer instead of the development teams.  Don’t get me wrong, QA automation engineers are great, but there aren’t many of them, good ones are very expensive, and the responsibility shouldn’t lie squarely on their shoulders.

BATs test the whole system working together.  BATs are your automated regression tests for the entire system.  BATs are automated customer acceptance tests and the ROI for each line for code in a BAT can be much higher than the ROI of each line of production code.

Why?  How is this even possible?  It’s all about leverage baby.  Each line of code in a BAT may be exercising anywhere from 5 to 500 lines of production code, which is quite the opposite case of a unit test where each line of unit test code might only be testing a 1/8th or 1/16th a line of production code on average (depending on code coverage numbers being reached.)

I’ll save the detail for later posts, but it is my strong opinion that a majority of a development teams effort should be put in BATs, because BATs

  • Have high value to the customer
  • Regression test the entire system
  • Have a huge ROI per line of code (if you create a proper BAT framework)

Imagine how much higher quality your software would be if you had a BAT for each backlog item in your system which you could run every single iteration of your development process.  Imagine how confident you would be in making changes to the system, knowing that you have an automated set of tests that will catch almost any break in functionality.

Don’t you think that is worth giving up writing level 3 and level 4 unit tests, which are already painful and not very fun to begin with to achieve?

In my future posts on the Back to Basics series, I will cover in-depth how to push more of your code into level 1 and level 2 unit tests by extracting logic out to separate classes that have no dependencies, and I will talk more about BATs, and how to get started and be successful using them.  (Hint: you need a good BAT framework.)

As always, you can subscribe to this RSS feed to follow my posts on Making the Complex Simple.  Feel free to check out ElegantCode.com where I post about the topic of writing elegant code about once a week.  Also, you can follow me on twitter here.
  • Jason Meckley

    Jimmy Bogard (http://www.lostechies.com/blogs/derickbailey/default.aspx) and Derick Bailey (
    http://www.lostechies.com/blogs/jimmy_bogard/default.aspx) are discussing this same topic with similar conclusions.

    • http://simpleprogrammer.com jsonmez

      Yes, thank you. Been following those also :)

  • Pingback: Back to Basics: Unit Testing, Automated Blackbox Testing, and Conclusions!()

  • http://invalidcast.com Martin Rue

    My thinking on this fits pretty much exactly with yours. I’m finding it more and more important to write really simple unit tests that are just assertions of some expected output state.

    And as you’ve noted, I’ve also found that any moderate use of mocks, in the end, just couples the test to the implementation and certainly increases the friction of any future refactoring. I’m now avoiding those where I can completely.

    The way I see it: If I go into my calculator class and completely change the way my internal add method works, so long as when 5 and 5 are input I get back 10, that’s all I care about. I certainly don’t want my new refactoring to break a test because my internal add method is no longer calling some other internal method that some mock had an expectation on just so the unit test could be “pure” and without dependencies.

    Recently I’ve taken this same thinking (simple observations of state) to the BAT (to use your terminology here) level and started creating automated end-to-end tests that map pretty much directly to a feature. They are essentially my virtual customer who, multiple times per day, checks that every feature they have ever asked for still work in the way they expect. I think the most value comes from these tests.

    Above everything else, I want to know that the software works in the way a user sitting in front it will expect. The only way to do that is to automate the sitting in front of it and checking.

    Of course, the safety net argument still applies for unit tests and we need to be able to refactor code and have quick feedback about any broken behaviour (where a large set of constantly running BATs might not provide that quickly enough) – so a decent set of unit tests is still really important.

    But focusing on the value that each and every unit test will give you is where I’m at now. To me, a unit test is more code – a form of debt, and I am going to evaluate the need for each one to make damn sure it’s paying for itself it terms of its reliability and usefulness. If it doesn’t negate its own debt, I don’t want it. I’m not going to test that my view has been passed the correct type of model any more.

    Rant over. Great most and very thought provoking :)

    • http://simpleprogrammer.com jsonmez

      Thanks for the comment. Sorry for the late reply. I believe you get exactly what I am saying. Glad to hear that someone else has come to some of the same conclusions.

      Your calculator example is spot on. That is exactly why I don’t like mocks anymore. Very good point.

  • ntcoding

    I think you and Jimmy Boggard should start showing us a few examples. The theory is sound, but then you could say that about the jurassic park movie :)

    Come one John – examples P-lease.

  • Pingback: The Morning Brew - Chris Alcock » The Morning Brew #771()

  • John

    What about the fact that BAT tests are usually slower compared to other tests? Usually they have to launch browser (e.g. Watin) and the system needs to connect to some kind of database making the whole process slower.

    Another question: what would be your advice to someone new to TDD? Should they avoid mocks and DI containers? It’s very easy to use Watin even if you have very bad quality code which is difficult to test. Advocating usage of BAT tests might cause teams not caring about OOP and other best practices.

    • http://simpleprogrammer.com jsonmez

      As for the slowness of BAT tests, there are many solutions available today that weren’t as readily available before that can make running BAT tests in parallel reduce to total time by a great deal. Obviously you wouldn’t be able to run these kinds of tests as quickly as traditional unit tests, but running them on a daily or even 2x a day schedule will provide an huge benefit.

      As for someone new to TDD. I would suggest avoiding mocks and DI containers, but I would not suggest at all avoiding unit tests or TDD in general. In my next post in this series I planned to talk about how to refactor code in a such a way that a large portion of the code requires level 1 or level 2 unit tests. Using a TDD approach without allowing yourself mocks or DI naturally leads to this kind of code.

      I agree that some teams may write BAT tests instead of caring about OOP and other best practices, but those teams are likely to have been teams that weren’t writing unit tests or any quality either. In the long run, if you have a complete end to end regression suite that you can run daily though, (as much as it pains me as a software craftsman to say this), some of the principles we hold dear are not nearly as important, since failures in functionality are found so quickly.

      • John

        Thanks for reply.

        In my opinion unit tests is one of the tangible assets that good quality code has. Other ones such as easy maintenance, easy to read code and so on are too hypothetical for management to convince them that writing high quality code is important. By using BAT tests it is more difficult to get someone believe that code quality matters.

  • http://LooseCouplings.blogspot.com Loose Couplings

    Good post and good series. While I don’t agree with every point you make, it’s a good read.

    I find your Level 1-Level 4 class categorization interesting but not really useful for the purposes of testability. I tend to mentally categorize in terms of newables and injectables (and then sub-categorize in DDD terms). I tried to explain this categorization in this post http://loosecouplings.blogspot.com/2011/01/how-to-write-testable-code-overview.html

    I also find the practice of ‘using IoC containers all over your app’ alarming. I touched on a similar point [trying to preach the virtues of manual DI] in http://loosecouplings.blogspot.com/2011/01/dependency-injection-using-di-container.html

    I think we (and others discussing similar issues in the blogosphere now) are all working toward the same goal here from different angles. If you figure it out, let me know! :)

  • Clement

    Awesome article, I’m with you a 100%.
    The title of this serie is spot on. Once the dev community have played enough with their new toys (DI, unit testing), it’s time to go back to what really works.

  • Pingback: Back To Basics: Unit Testing Without Mocks « Making the Complex Simple()

  • Pingback: Back to Basics: Becoming BAT Man « Making the Complex Simple()

  • http://www.parasoft.com Matt

    I’ve found that writing interfaces and doing inversion of control just for the sake of unit testing makes the underlying code harder to understand and troubleshoot. I completely agree that these interfaces should only be used when the software design may need to support multiple implementations all from production code. The black box automated tests that verify a requirement or usage scenario are great for restoring the confidence lost from not unit testing those classes with external dependencies. However, those tests also take a lot longer to run and may be limited to only running nightly. Commercial tools, like http://tinyurl.com/4dofj4p, can automatically generate the missing unit tests with mocks for external dependencies. These synthetic tests do not instill the same confidence as a QA test script for a specific requirement. Those BAT are still valuable. Instead, automatically generated tests are useful for quick regression testing. When a class calls external methods in a certain order and then those calls change, the automatic unit tests will detect it. These tests require more maintenance work during changes from code development, but that work is mitigated by automated test generation.

  • Pingback: The More I Know, the Less I Know « Making the Complex Simple()

  • Pingback: Principles Are Timeless Best Practices Are Fads « Making the Complex Simple()