Category Archives: Architecture

Test Automation Framework Architecture

Test automation framework architecture efforts are often complete failures.

It’s true. I’ve worked with many companies who have given up on creating a good test automation framework architecture, because after investing a large amount of time and money and resources in doing it the wrong way, they have incorrectly assumed the entire effort is not cost effective.

In this post, I’m going to simplify the process of creating a test automation framework architecture and show you that—if you do it right—it can be a very good investment.

Test automation framework architecture basics

Let’s start off by talking about what the goals of a successful test automation framework architecture should be.

There are two goals I am interested in when creating a test automation framework:

  1. Having the ability to easily create simple automated tests that use the framework.
  2. Decoupling cosmetic application changes from the tests so that when the application changes, the tests do not all have to be updated

A good test automation framework architecture should at the very least provide these two important services. We create a test automation framework to allow us to support the creation of tests and to keep those tests from being dependent on the actual UI of our application—as much as possible.

I’m going to break down these goals just a little more to make sure we are all on the same page.

test automation framework thumb Test Automation Framework Architecture

Creating simple automated tests

First, we’ll start with having the ability to easily create simple automated tests that use the framework. Why do we care about this goal? And why would this be the responsibility of the framework?

I have found that one of the biggest failure points of test automation efforts is test complexity. The more complex the tests are, the harder they are to maintain. If tests are difficult to maintain, guess what? They won’t be maintained.

So, we really want to make sure that when we create a test automation framework architecture, we focus on making sure that test automation framework makes it as easy as possible for someone to create tests using it.

It is always a better choice to put the complexity into the framework instead of into the tests. The framework is usually maintained by developers and does not contain repeated code. But, tests are often created by QA or even business folks and the code found in tests is usually repeated across many tests, so it is vital to reduce the complexity on the tests, even if it means a large increase of complexity in the framework itself.

When I create a test automation framework architecture, my goal is to make it so the tests can read as close as possible to plain English. This requires some effort to pull off, but it is well worth it.

Decoupling the UI from the tests

Next, let’s talk about what I mean by decoupling cosmetic application changes from the tests.

This is also a vital goal of any test automation framework architecture, because if you fail to decouple the UI of an application from the tests, every single UI change in an application will cause possibly hundreds or thousands of tests to have to be changed as well.

What we really want to do is to make it so that our test automation framework architecture creates an abstraction layer that insulates the tests from having to know about the actual UI of the application.

At first this seems a little strange, especially when I tell automation engineers or developers creating an automated testing framework not to put any Selenium code into their actual tests.

All of the Selenium examples show you creating tests that directly use a web browser driver like Selenium, but you don’t want to write your tests that way—trust me on this one.

(By the way, I haven’t found a good book on creating an actual test automation framework architecture—I’ll probably write one—but, for now if you are looking for a good book on Selenium that starts to go into the topics I discuss here, check out Selenium Testing Tools Cookbook.)

Instead, what you want to do is to make it so the test automation framework is the only code that directly interacts with the UI of the application. The tests use the framework for everything they want to do.

So, for example:

Suppose you are creating a simple test to check to see if a user can log into your application.

You could write a test that looks something like this: (C# and Selenium in this example)

var loginInput = driver.FindElement(By.Id("txtUsername"));
loginInput.SendKeys("Joe");

var passwordInput = driver.FindElement(By.Id("txtPassword"));
passwordInput.SendKeys("$ecret");

var loginButton = driver.FindElement(By.Id("btnLogin"));
loginButton.Click();

But, what is going to happen when you change the ID of your “User Name” field? Every single test that uses that field will break.

On the other hand, if you properly create a test automation framework architecture that abstracts the UI away from the tests themselves, you would end up with a much simpler and less fragile test, like this:

LoginPage.GoTo();
LoginPage.LoginAs("Joe").WithPassword("$ecret").Login();

Now, if the ID of your “User Name” field changes, you’ll just change the code in your automated testing framework in one place, instead of changing 100 tests that all depended on the specific implementation of the UI.

A simple test automation framework architecture

Once you understand those two very important goals and why they are important, it is easier to think about how you should design a test automation framework architecture.

I’ve created quite a few test automation framework architectures, so I’ll give you a basic design I use for most of them.

Take a look at this diagram:

test automation framework architecture thumb Test Automation Framework Architecture

Here you can see that there are four layers to my test automation framework architecture.

Frist, we have the browser layer or the web app itself. This just represents your actual application.

Next, we have the Selenium or web driver layer. This layer represents your browser automation tool. Selenium is basically just a framework for automating a browser. It doesn’t know anything about testing, it is an API that lets you interact with the browser programmatically. (By the way, you don’t have to use Selenium. I just use it as an example here, because it is the most popular browser automation framework.)

After that, we have the framework layer. This is the actual framework you create which uses Selenium, or whatever web driver you want, to actually automate your application. The framework is acting as an abstraction between your application and the tests. The framework knows about the UI of your application and how to interact with it using Selenium. It is your job to create this layer.

Finally, we have the actual tests. The tests use the framework to manipulate your application and check the state of the application. These tests should be simple to understand and should not have to know anything about the actual implementation of the UI of your application. These tests should rely on the framework to give them the ability to do whatever they need to do in your application.

Now what?

Obviously, creating a test automation framework architecture is more complicated than what I have shown you in this article, but with these basics you should be off to a good start.

If you want a more in-depth and detailed explanation as well as an example of how exactly to create a full test automation framework architecture, you might want to check out my Pluralsight video on the topic:

Creating an Automated Testing Framework With Selenium

I also frequently blog about test automation topics and other topics related to becoming a better and more profitable software developer.

Sign up here to make sure that you don’t miss any of my posts and get access to the content I only share with my email list. (Don’t worry I hate spam as much as you do.)

If you have any question, post them in the comments below.

Leaky Abstractions Are Holding Us Back

Let’s just get right into it, shall we?

What is an abstraction?

Before we can talk about leaky abstractions, and why they are bad, let’s define what exactly an abstraction is.

An abstraction is when we take a complicated thing and we make it simpler by generalizing it based on the way we are using or talking about that thing.

We do this all the time. Our brains work by creating abstractions so that we can deal with vast amounts of information.

abstraction thumb Leaky Abstractions Are Holding Us Back

We don’t say “my Dell Inspiron 2676 with 2 GB of RAM and a 512 GB hard drive, serial number 12398ASD.” Instead we say simply “my computer.”

In programming, abstractions let us think at higher levels. We create abstractions when we give a bunch of lines of code that get the username and password out of a textbox and log the user into our system, a name; login. And we use abstractions when we utilize APIs or write to the “file system.”

Without abstractions we’d have a really hard time managing the complexity of any software system. Without abstractions, it would be pretty difficult to write any kind of non-trivial application.

If you want to feel what it is like to program with fewer abstractions, try serving up a web page with assembly code. If you want to feel what it is like to program with almost no abstractions, do the same exercise, but write your code in machine code… 0s and 1s.

How do abstractions “leak?”

Abstractions leak when you have to understand the lower level concept to use the higher level concept.leaking thumb Leaky Abstractions Are Holding Us Back

This means the details of the lower level concept that are being abstracted away are leaking up through the higher level concept.

Ever get a cryptic error message when an application you are using crashes? That is a leaky abstraction.

“Memory location 00FFE133 could not be written to” is just bad news. It means that in order to understand what is going wrong with your application, you now have to understand the inner workings of the application. Your abstraction has effectively “leaked.”

Abstractions leak whenever you are forced to pull back the curtains and see the gnomes running in hamster wheels spinning the cogs.

Why “leaking” is bad

I’m often surprised when I hear someone say “I like how you didn’t hide the details of what is really going on” about some code or some technology.

Not being able to hide the details of what is really going on points to an inability to create the proper abstraction, which in my opinion, should be water tight.

It is not a good thing that we can see what is really going on. No, it is actually a very bad thing, because it forces us to think at the higher level some of the time, but have to switch gears to drop down to the lower level some of the time.

Leaky abstractions prevent us from being able to comfortably drive the kids to school; we have to constantly keep looking in our rear view mirror and checking to see if those rowdy kids who insist on sitting at the back of the bus are causing trouble again.

A good abstraction hides the details so well that we don’t ever have to think about them. A good abstraction is like a sturdy step on a ladder, it allows us to build another step above us and climb up it. A leaky abstraction is like a wobbly rotted wood rung; put too much weight on it and you’ll fall through—it traps you at the current step—you can’t go any higher.

Shouldn’t we understand how things really work?

No! Stop already! Enough with the guilt trips and unnecessary burdens. You are defeating the purpose of abstractions and stunting further growth.

You only “need” to know the details if you want or need to know the details.

I know that many people want to know how things really work, myself included for many things, but in many cases I don’t need to understand how things really work.

For example, I can buy and use a watch without understanding how the watch works. It really is better that way. I don’t need those details floating around in my head. I get no benefit from knowing what is going on inside my watch and frankly, I don’t want to know.

Personally, I have a similar view on cars, although I know that many other people don’t, so that is why I didn’t use it as my first example.car engine thumb Leaky Abstractions Are Holding Us Back

When I buy a car, I just want to drive it. I don’t want to remember when to rotate tires, check fluids, etc. Waste of my time. I don’t want to know how the engine works or anything about it besides I have to put gas in, turn a key, and press a few pedals to make the thing go. I know I could save money servicing my own car, but you know what? I’d rather pay money than have to do it myself. And since that is the case, as long as I can find someone I trust to work on my car, I don’t want that abstraction to leak.

The same with machine code and compilers. Yes, it is interesting to understand how memory mapping and CPUs and logic gates work and how my C# code gets compiled into MSIL which is interpreted by the CLR and so forth and so on, until it ends up as electrons operating on registers in my CPU. And I learned about how much of that works, because I was interested, but let’s be completely honest here. That abstraction is really good. It is so good and so air-tight, that I don’t need to know all of that crap in order to write a web app. If I did, I’d probably find another profession, because my brain would explode every time I tried to write the simplest piece of code.

By the way, if you do want to understand what is happening behind many of those abstractions, I highly recommend Charles Petzold’s book: “Code: The Hidden Language of Computer Hardware and Software”.  (This is the kind of book you read for “fun,” not because you need to know this information.)

Leaking abstractions in technology

Too many technologies and frameworks today are leaky, and this is bad– really bad.

Want an example? Whatever happened to ORM (Object Relational Mapping?)

Nothing. Nothing happened. It was invented, oh, like thousands of years ago and while lots of other technologies and frameworks grew up and got jobs and left the nest, ORM is still a fat unemployed balding 30 year old man playing WOW all day in his mum and dad’s basement.

Don’t get me wrong. I use ORMS; I sometimes like to think I like ORMS. But, I can’t use them well because I have to understand SQL and what is going on in my relational databases anyway. What good is that?!

Seriously. This is a completely failed abstraction. The leakiness of it makes it actually more work to learn an ORM framework, because in learning one you must also learn hairy details of SQL and understand what N+1 means and so on. You can’t just fire up your ORM and put in the Konami code, you have to blow the cartridge out and fiddle with the reset button half a dozen times. Useless.

The other major example that is holding us back today is MV* in JavaScript. Look, I’m not trying to bash JavaScript this time. I’m actually starting to like JavaScript, but it is total bullcrap that in order to write a single page web application today, you have to understand 500 other different technologies and how JavaScript is working under the covers. If we are going to do MV* in JavaScript– and I am actually starting to become a believer in this truth—then we gotta seal up the abstraction and put what we need into the language or the DOM or the technology stack itself. Bolting things on to create abstractions that are ultimately leaky works for now, but it is like that rotten creaky step in the ladder, eventually we are going to fall through it and we aren’t going to be able to climb any higher until we replace that step.

I used to be convinced replacing that step meant getting rid of JavaScript and using something else superior that does what we want, but maybe it just means making JavaScript and browser APIs themselves evolve. I don’t care how we do it, but we aren’t going to advance until we do.

I could go on and on with examples of leaky abstractions in technology and how they are holding us back, but I think you get the point by now.

Leaky abstractions are bad. They don’t do anything good for us besides require us to know more and add more points where things can go wrong. We’ve got to seal up these abstractions, if we want to be able to keep climbing the software development ladder. You can’t keep building a tower if you have to worry about every single block beneath you. You eventually get to a point where you can’t keep it all straight and your tower comes crashing to the floor.

Leaky abstractions and you

So, what can you do about it?

Not much as far as ORMs and JavaScript. But, next time you write some code, seriously think about whether or not you are creating a leaky abstraction.

You really have to make a choice. Am I going to hide the details so that no one needs to know about them, or am I going to expose the whole thing? Either make a really good abstraction that doesn’t require someone using your code to understand what is going on underneath that abstraction or don’t attempt to make an abstraction at all.

It takes more effort to create good abstractions. Sometimes the effort required is orders of magnitude greater. But, if other people are going to use your abstraction, to save the mental burden of understanding what is happening beneath it is going to go a long way in making up for your extra effort.

10 Steps to learn anything quickly course

By the way, I just released by 100% 10 Steps to Learn Anything course.  If you want free access to it, just signup to my newsletter here.  It is a totally free video course and optional workbook that reveals the process I use to quickly learn technologies.

Creating an Automated Testing Framework With Selenium

I have another new course on Pluralsight, check out:

Creating an Automated Testing Framework With Selenium

2013 09 28 16 31 58 Creating an Automated Testing Framework With Selenium

I am very excited to finally get this course out.  Many viewers have been asking me to make a comprehensive course that actually shows you how to build a real automation framework, and I finally did it in this course.

I reveal all my tricks and tips here and tell you exactly what I have done in the past to build successful automation frameworks.  It was a lot of fun to create this course and it brought me back to the glory days of creating automated tests.  (This is some serious fun!)

Anyway, check out the course and let me know what you think.

Here is the official course description:

Learning how to use a tool like Selenium to create automated tests is not enough to be successful with an automation effort. You also need to know how to build an automation framework that can support creating tests that are not so fragile that they constantly break. This is the real key to success in any automation effort.

In this course, I will reveal every secret I know from creating several successful automation frameworks and consulting on the creation of others. I will show you exactly, step-by-step how to create your own automation framework and I will explain to you the reasoning behind everything we are doing, so you can apply what you learn to your own framework.

We’ll start off this course by going over the basics of automation and talking about why it is so important as well as discuss some of the common reasons for success and failure.

Then, I’ll take you into the architecture of an automation framework and show you why you need to pay careful attention to the structure of any framework you build and give you some of the underlying design principles I use when creating an automation framework.

After that we’ll be ready to start creating a framework. In the next few modules, I’ll show you how to create a real automation framework capable of automating the WordPress blogging platform administrative console. We’ll start off by creating smoke tests and using those smoke tests to build out our initial framework.

Then, we’ll expand the capabilities of our framework as we create more tests and learn how to use techniques like dummy data generators to make our tests as simple and easy to read as possible.

Finally, I’ll take you through some best practices and tips that cover topics like scaling out, working in Agile environments and other important issues you are likely to face. If you are responsible for an automation project for a web application or you want to start using automation, you’ll definitely want to check this course out.

There Are Only Two Roles of Code

All code can be classified into two distinct roles; code that does work (algorithms) and code that coordinates work (coordinators).

The real complexity that gets introduced into a code bases is usually directly related to the creation of classes that group together both of these roles under one roof.

I’m guilty of it myself.  I would say that 90% of the code I have written does not nicely divide my classes into algorithms and coordinators.

Defining things a bit more clearly

Before I dive into why we should be dividing our code into clear algorithmic or coordinating classes, I want to take a moment to better define what I mean by algorithms and coordinators.

Most of us are familiar with common algorithms in Computer Science like a Bubble Sort or a Binary Search, but what we don’t often realize is that all of our code that does something useful contains within it an algorithm.

What I mean by this is that there is a clear distinct set of instructions or steps by which some problem is solved or some work is done.  That set of steps does not require external dependencies, it works solely on data, just like a Bubble Sort does not care what it is sorting.

Take a moment to wrap your head around this.  I had to double check myself a couple of times to make sure this conclusion was right, because it is so profound.

It is profound because it means that all the code we write is essentially just as testable, as provable and potentially as dependency free as a common sorting algorithm if only we can find the way to express it so.

What is left over in our program (if we extract out the algorithms) is just glue.

Think of it like a computer.  Computer electronics have two roles: doing work and binding together the stuff that does the work.  If you take out the CPU, the memory and all the other components that actually do some sort of work, you’ll be left with coordinators.  Wires and busses that bind together the components in the system.

Why dividing code into algorithms and coordinators is important.

So now that we understand that code could potentially be divided into two broad categories, the next question of course is why?  And can we even do it?

Let’s address why first.

The biggest benefit to pulling algorithmic code into separate classes from any coordinating code is that it allows the algorithmic code to be free of dependencies.  (Practically all dependencies.)

Once you free this algorithmic code of dependencies you’ll find 3 things immediately happen to that code:

  1. It becomes easier to unit test
  2. It becomes more reusable
  3. Its complexity is reduced

A long time ago before mocks were widely used and IoC containers were rarely used, TDD was hard.  It was really hard!

I remember when I was first standing on the street corners proclaiming that all code should be TDD with 100% code coverage.  I was thought pretty crazy at the time, because there really weren’t any mocking frameworks and no IoC containers, so if you wanted to write all your code using TDD approaches, you’d actually have to separate out your algorithms.  You’d have to write classes that had minimal dependencies if you wanted to be able to truly unit test them.

Then things got easier by getting harder.  Many developers started to realize that the reason why TDD was so hard was because in the real world we usually write code that has many dependencies.  The problem with dependencies is that we need a way to create fake versions of them.  The idea of mocking dependencies became so popular that entire architectures were based on the idea and IoC containers were brought forth.

mp900175522 thumb There Are Only Two Roles of CodeWe, as a development community, essentially swept the crumbs of difficult unit testing under the rug.  TDD and unit testing in general became ubiquitous with writing good code, but one of the most important values of TDD was left behind, the forced separation of algorithmic code from coordinating code.

TDD got easier, but only because we found a way to solve the problems of dependencies interfering with our class isolation by making it less painful to mock out and fake the dependencies rather than getting rid of them.

There is a better way!

We can still fix this problem, but we have to make a concerted effort to do so.  The current path of least resistance is to just use an IoC container and write unit tests full of mocks that break every time you do all but the most trivial refactoring on a piece of code.

Let me show you a pretty simple example, but one that I think clearly illustrates how code can be refactored to remove dependencies and clearly separate out logic.

Take a look at this simplified calculator class:

 public class Calculator
    {
        private readonly IStorageService storageService;
        private List<int> history = new List<int>();
        private int sessionNumber = 1;
        private bool newSession;

        public Calculator(IStorageService storageService)
        {
            this.storageService = storageService;
        }

        public int Add(int firstNumber, int secondNumber)
        {
            if(newSession)
            {
                sessionNumber++;
                newSession = false;
            }

            var result = firstNumber + secondNumber;
            history.Add(result);

            return result;
        }

        public List<int> GetHistory()
        {
            if (storageService.IsServiceOnline())
                return storageService.GetHistorySession(sessionNumber);

            return new List<int>();
        }

        public int Done()
        {
            if (storageService.IsServiceOnline())
            {
                foreach(var result in history)
                    storageService.Store(result, sessionNumber);
            }
            newSession = true;
            return sessionNumber;
        }
    }

 

This class does simple add calculations and stores the results in a storage service while keeping track of the adding session.

It’s not extremely complicated code, but it is more than just an algorithm.  The Calculator class here is requiring a dependency on a storage service.

But this code can be rewritten to extract out the logic into another calculator class that has no dependencies and a coordinator class that really has no logic.

 public class Calculator_Mockless
    {
        private readonly StorageService storageService;
        private readonly BasicCalculator basicCalculator;

        public Calculator_Mockless()
        {
            this.storageService = new StorageService();
            this.basicCalculator = new BasicCalculator();
        }

        public int Add(int firstNumber, int secondNumber)
        {
            return basicCalculator.Add(firstNumber, secondNumber);
        }

        public List<int> GetHistory()
        {
            return storageService.
                   GetHistorySession(basicCalculator.SessionNumber);
        }

        public void Done()
        {
            foreach(var result in basicCalculator.History)
                storageService
                     .Store(result, basicCalculator.SessionNumber);

            basicCalculator.Done();
        }
    }

    public class BasicCalculator
    {
        private bool newSession;

        public int SessionNumber { get; private set; }

        public IList<int> History { get; private set; }

        public BasicCalculator()
        {
            History = new List<int>();
            SessionNumber = 1;
        }
        public int Add(int firstNumber, int secondNumber)
        {
            if (newSession)
            {
                SessionNumber++;
                newSession = false;
            }

            var result = firstNumber + secondNumber;
            History.Add(result);

            return result; ;
        }

        public void Done()
        {
            newSession = true;
            History.Clear();
        }
    }

 

Now you can see that the BasicCalculator class has no external dependencies and thus can be easily unit tested.  It is also much easier to tell what it is doing because it contains all of the real logic, while the Calculator class has now become just a coordinator, coordinating calls between the two classes.

This is of course a very basic example, but it was not contrived.  What I mean by this is that even though this example is very simple, I didn’t purposely create this code so that I could easily extract out the logic into an algorithm class.

Parting advice

I’ve found that if you focus on eliminating mocks or even just having the mindset that you will not use mocks in your code, you can produce code from the get go that clearly separates algorithm from coordination.

I’m still working on mastering this skill myself, because it is quite difficult to do, but I believe the rewards are very high for those that can do it.  In code where I have been able to separate out algorithm from coordination, I have seen much better designs that were more maintainable and easier to understand.

I’ll be talking about and showing some more ways to do this in my talk at the Warm Crocodile conference next year.

Tie Your Shoes and Pull Up Your Pants

What slows down the development of software?

Think about this question for a bit.  Why is it that as most software evolves it gets harder and harder to add features and improve its structure?

Why is it that tasks that would have at one point been simple are now difficult and complex?

Why is it that teams that should be doing better over time seem to get worse?

mp900430685 thumb Tie Your Shoes and Pull Up Your Pants

Seeking answers

Don’t feel bad if you don’t have an immediate answer to those questions.  Most software practitioners don’t.  They are hard questions after all.

If we knew all the answers, we wouldn’t really have these problems to begin with.

Regardless though, you’ll find many managers, business owners, customers and even software developers themselves looking for the answers to these questions, but often looking in the wrong place.

Process is almost always the first to be blamed. It stands to reason that a degradation of process or problems with the software development process are slowing things down.

Often there is some merit to this proposition, but I’ve found that it is often not the root cause. If your team is not sitting idle and the work that is important is being prioritized, chances are your process is not slowing you down.

Now don’t get me wrong here.  I am not saying that these are the only two important aspects to judge a software development process, but I am saying that if generally your team is working hard on important stuff most of the time, you can’t magically improve process to the point of increasing productivity to any considerable order of magnitude.  (In most cases.)

Often questions are asked like:

  • Should we pair program or not pair program?
  • Should we be using Scrum instead of Kanban?
  • Should we be changing the way we define a backlog?
  • Should we use t-shirt sizes or story points or make all backlogs the same size?
  • Do we need more developers or more business analysts?
  • Do we need to organize the team differently?

Now these are all great questions that every software project should constantly evaluate and ask themselves, but I’ve found over and over again that there is often a bigger problem staring us in the face that often gets ignored.

The code!

mp900289113 thumb Tie Your Shoes and Pull Up Your Pants

Let’s do a little experiment.

Forget about process.  Forget about Scrum and backlogs and story points and everything else for a moment.

You are a developer.  You have a task to implement some feature in the code base.  No one else is around, there is no process, you just need to get this work done.

It might help to think about a feature you recently implemented or one that you are working on now.  The important thing with this experiment is that I want to take away all the other “stuff” that isn’t related directly to designing and implementing that feature in the code base.

You will likely come to one of these conclusions:

1. The feature is easy to implement, you can do it quickly and know where to go and what to modify.

Good!  That means you don’t really have a problem.

2. It is unclear what to do.  You aren’t sure exactly what you are supposed to implement and how it fits into the way the system will be used.

In this case, you may actually have somewhat of a process problem.  Your work needs to be more clearly defined before you begin on it.  It may be that you just need to ask more questions.  It may be that half baked ideas are ending up in your pipeline and someone needs to do a bit more thinking and legwork, before asking a developer to work on them.

3. Its hard to change the code.  You’ve got to really dig into multiple areas and ask many questions about how things are working or are intended to work before you can make any changes.

This is the most likely case.  Actually usually a combination of 2 and 3.  And they both share a common problem—the code and system do not have a design or have departed from that design.

I find time and time again with most software systems experiencing a slow down in feature development turnaround that the problem is the code itself and the system has lost touch with its original design.

You only find this problem in successful companies though, because…

Sometimes you need to run with your shoelaces untied

I’ve consulted for several startups that eventually failed.  There was one thing in common with those startups and many other startups in general—they had a well maintained and cared for codebase.

I’ve seen the best designs and best code in failed startups.

This seems a bit contradictory, I know, but let me explain.

The problem is that often these startups with pristine and well maintained code don’t make it to market fast enough.  They are basically making sure their shoes laces are nicely tied as they stroll down the block carefully judging each step before it is taken.

What happens is they have the best designed and most maintainable product, but it either doesn’t get out there fast enough and the competition comes in with some VB6 app that two caffeine fueled not-really-programmers-but-I-learned-a-bit-of-code developers wrote overnight or they don’t actually build what the customer wants, because they don’t iterate quick enough.

Now am I saying that you need to write crap code with no design and ship it or you will fail?

Am I saying that you can’t start a company with good software development practices and a clean well maintainable codebase and succeed?

No, but what I am saying is that a majority of companies that are successful are the ones that put the focus on customers and getting the product out there first and software second.

In other words if you look at 10 successful companies over 5 years old and look at their codebase, 9 of them might have some pretty crappy or non-existent architecture and a system that departed pretty far from the original design.

Didn’t you say something about pulling up your pants?

pants around ankles thumb Tie Your Shoes and Pull Up Your Pants

Ok, so where am I driving at with all this?

Time for an analogy.

So these companies that are winning and surviving past year 5, they are usually running.  They are running fast, but in the process of running their shoelaces come untied.

They might not even notice the shoelaces are untied until the first few times they step on one and trip.  Regardless they keep running.  And to some degree, this is good, this is what makes them succeed when some of their failed competitors do take the time to tie their shoelaces, but those competitors end up getting far behind in the race.

The problem comes pretty close to after that 5 year mark, when they want to take things to the next level.  All this time they have been running with those shoelaces untied and they have learned to do this kind of wobble run where they occasionally trip on a shoe lace, but they try to keep their legs far enough apart to not actually step on a shoelace.

It slows them down a bit, but they are still running.  Still adding those features fast and furious.

After some time though, their pants start to fall down.  They don’t really have time to stop running and pull up those pants, so as they are running those pants slip further down.

Now they are really running funny.  At this point they are putting forth the effort of running, but the shoelaces and pants are just too much, they are moving quite slow.  An old woman with ankle weights power walks past them, but they can’t stop now to tie the shoelaces and pull up those pants, because they have to make up for the time they lost earlier when the pants first fell down.

At this point they start looking for ways to fix the problem without slowing down and pulling up the pants.  At this point they try running different ways.  They try skipping.  Someone gets the idea that they need more legs.

I think you get the idea.

What they really need to do at this point though is…

Stop running, tie your shoes and pull up your pants!

Hopefully you’ve figured out that this analogy is what happens to a mature system’s code base and overall architecture.

Over time when you are running so fast, your system ends up getting its shoelaces undone, which slows you down a little.  Soon, your system’s pants start to fall down and then you really start to slow down.

It gets worse and worse until you are moving so slow you are actually moving backwards.

Unfortunately, I don’t have a magic answer.  If you’ve gotten the artificial speed boost you can gain from neglecting overall system design and architecture, you have to pay the piper and redesign that system and refactor it back into an architecture.

This might be a complete rewrite, it might be a concerted effort to get things back on track.  But, regardless it is going to require you to stop running.  (Have you ever tried to tie your shoelaces while running?)

Don’t feel bad, you didn’t do anything wrong.  You survived where others who were too careful failed.  Just don’t ignore the fact that your pants are at your ankles and you are tripping over every step, do something about it!

Going Backwards to Go Forwards

I worked on an interesting problem this week that might have looked like I was running around in circles if you just looked at my SVN commits.

The problem, and the eventual solution, reminded me of an important part of software development—of building anything really.

Sometimes you must tear it down!

No really, sometimes you build a structure only to tear it down the very next day.

It’s not a mistake.  It is intentional and productive and if you are not doing it, you very well might be making a real mistake.

Skeptical?

data and spock thumb Going Backwards to Go Forwards

That is highly illogical

Imagine for a moment that you are tasked with the job of repairing the outside walls of a 2 story building.

There are of course many ways you could go about doing something like this.

  • Use a ladder and just reach the part of the wall the ladder allows you to.  Then move that ladder, repeating the process as needed to complete the repair to the entire wall.
  • Try to lower yourself down from different windows to reach as much of the wall as possible.
  • Tear down the entire building and rebuild the building and walls.

I am sure there are plenty of other methods besides what I listed here.

Yet, a very simple approach would be to build a scaffolding.

A scaffolding is basically a temporary construction used just to help repair or initially build a building which is built for the very purpose of eventually being torn down.

Software is different, we don’t contend with physics!

You are absolutely right!

We contend with a much more powerful force…

LOGIC!

Conceptually anything you can create in software could be created without any kind of software scaffolding.  Unfortunately though, the complexities of the logic of a system and our abilities as humans to only contain so much of it in our brains impose a very real limitation on our ability to even see what the final structure of a metaphysical thing like software should be.

So what am I saying here then?

I’m just saying that it sometimes helps to remember that you can temporarily change some code or design in a way that you know will not be permanent to get you to a place in the codebase where your head is above the clouds and you can look down and see the problem a little better.

Just like there are physical ways to repair a wall on a 2 story building that don’t involve wasting any materials by building something that will be torn down, there are ways to do build software without ever building scaffoldings, but in either case it is not the most efficient use of time or effort.

Don’t be afraid to take a blind hop of faith

Source control is awesome!

Source control lets us make changes to a code base, track those changes and revert them if needed.

Working on a code base without source control is like crawling into a small crevice in an underground cave that has just enough room to fit your shoulders—you are pretty sure you can go forward, but you don’t know if you can crawl back out.

morrils cave crevice thumb Going Backwards to Go Forwards

But, with source control, it is like walking around a wide open cave charting your trail on a map as you go.  You can get back to where you want to any time.

So as long as you have source control on your side, and especially if you are using a distributed source control where you can make local commits all day long, you don’t have to be afraid to step out on a software ledge and see where things go.

Oftentimes I will encounter some problem in code that I know needs to be fixed and I know what is wrong with it, but I just don’t know how to fix it.

When I encounter a problem like this, I will often have an idea of at least one step I could take that should take me in the direction I believe the code needs to go.

A real example

To give you a real example, I was working recently on some code that involved creating a lot for a product.  (Think about the lot number you might see on a box of Aspirin that identifies what batch of ingredients it came from and what machine produced it when.)

kas 9012 thumb Going Backwards to Go Forwards

Up to the point in which I had been working with my teammate on refactoring this code, there had only been only one way to produce the lots.

We were adding some code that would add an additional way to create the lots and the labels for those lots.  We also knew that there would be more additional ways in the future that would need to be added.

Well, we knew we wanted to have a generic way of specifying how lots should be created and labeled, but we didn’t know what the code that would do that would look like.

We could have rewritten the entire lot handling code and made it generic, but it was quite complex code and that would be equivalent to tearing down a building to fix a wall.

It was apparent there were some parts of the existing code that were specific to the way that the current lots were being produced, so we knew those parts of the code had to be changed.

So what did we do?

We started with what we knew needed to be changed and what we thought would move us in the direction we wanted to go, and we built some structures to allow us to refactor those parts of the code, knowing that we would probably end up deleting those structures.

The scaffolds that we created allowed us to have a midway point in the long journey across the Pacific ocean of complexity in our refactor trip.

The point here is that we had to take a bit of hop blindly in a direction we thought things needed to go and part of that hop involved creating some scaffolding type structures to get that code to a place where it still worked and we could examine it again for the next refinement.

The final refinements ended up deleting those scaffolds and replacing them with a more elegant solution, but it is doubtful we would have been able to see the path to the elegant solution without building them in the first place.

Show me the code!?

You may wonder why I am talking about code in such an abstract way instead of just showing you a nice little “scaffold pattern” or a real code example.

I’m not going to show the code, because it isn’t relevant to my point and it is so situational that it would detract from what I am trying to say here.

The point is not what our problem or solution was, but how we got there.

There isn’t a “scaffolding” pattern that you can just apply to your code, rather it is a tool that you can use and shouldn’t be afraid to use in order to move your code forward to a better design.  (To its happy place.)

Wrapping Callbacks

I’ve recently had the problem of trying to display a progress dialog when executing an asynchronous operation and to dismiss that progress dialog when the operation completes.

I wanted to build a way to do this that is generic to my application, so that it would work with any asynchronous operation in order to reduce duplication of writing progress dialog logic everywhere in the code.

11 290x300 thumb Wrapping Callbacks

Looking at the problem

So here is some basic pseudo-code of the problem.

public void DoCall()
{
   ShowProgressDialog();
   RemoteService.DoAsync(CallBackMethod);
}

public void CallBackMethod(Result result)
{
    HideProgressDialog();
    // Process result;
}

Now the problem with this code is that it would have to be repeated everywhere I want to make an asynchronous call and display a progress dialog.

If I want to do this in my application every time that I make an asynchronous call, I have to put this kind of code in many places in the application.

You can also see a mixing of responsibilities here.  We are handling UI related showing of a progress dialog split between an initial call and the callback.  It just seems a bit dirty to do things this way.

Breaking it down

So how can we solve this problem?

Go ahead and think of some way that you might be able to solve this.

One of the best ways that I have found to solve any problem like this is to work backwards.

Let us assume that any syntax we can think of is possible, and then only change the ideal syntax if it proves to not be possible.

What do I mean by this?

Simple, let’s come up with the way we want this code to look and we will worry about implementing it later.

It would be nice to be able to execute an asynchronous method and display a progress dialog with a one-liner, like so:

UIServiceCaller.ExecuteAsync(RemoteService.DoAsync, CallBackMethod);

If we could just do this wherever we want to make an asynchronous call and automatically have the progress dialog shown and dismissed, life would be wonderful.

Solving the problem

In order to solve this problem, we can do something very similar to a decorator pattern.

We can basically just wrap our callback method with a new callback that adds the functionality of dismissing our progress dialog.

So the idea will be that we will do the following steps in our ExecuteAsync method:

  1. Show the progress dialog
  2. Wrap our callback with one that dismisses our dialog and then executes the callback.
  3. Execute our asynchronous operation passing the new wrapped callback as the parameter.

gift wrap nature lovers easy style fabric wrapping 1211 l thumb Wrapping Callbacks

Here is what this looks like in code:

public static void ExecuteAsync(Action<Action<Result>> asyncMethod, Action callback)
{
   ShowProgressDialog();

   Action wrappedCallback = (r =>
   {
      DismissProgressDialog();
      callback(r);
   }

   asyncMethod(wrappedCallback);
}

This code actually looks more complicated than it really is.

You will need to understand how Action works though.  If you don’t, check out this post I did explaining Action and Func.

Understanding the solution

It can be a bit hard to wrap your head around Action of type Action of type Result.

Let me translate this to make it a bit easier.

The first parameter to ExecuteAsync is a method that takes another method as a parameter (the callback,) which takes a Result object as a parameter.

Go ahead and read that over until you get it.

So, the idea here is that we are passing in the method that is going to do the callback as the first parameter to your ExecuteAsync method.

Then we are passing in our actual callback as the 2nd parameter to ExecuteAsync.

The idea is that we are splitting apart the method and the callback it takes, so that we can insert our code to handle the dialog where we need to.

Now we can just call ExecuteAsync and pass it our method and its callback and progress dialog code will automatically be handled for us.

Building on the idea

We can expand upon this concept in several ways.

One thing we could do is also apply error handling logic to the callback wrapper method.  We could preprocess the result in some way before passing it to the callback.  This could allow us to display an error message or even retry a number of times before executing the callback.

Another thing we could do is to change the signature of the ExecuteAsync method so that it takes a type T instead of Result; this would allow us to handle any kind of return type and method, and the ExecuteAsync method would work with just about any asynchronous method call.

No Class is an Island

One of the biggest challenges I’ve found with any framework is to make it self-discoverable. 

It is often difficult to build a framework or API in a way that users of that framework can easily know what exists and when to use what.

One of the main reasons why it is difficult to write a good framework is that many developers tend to create classes that are very loosely connected to other classes in that framework with which they are intended to interact.

island thumb No Class is an Island

Defining the island

When you create a series of classes, do you explicitly plan for those classes to depend on each other or do you try to design them in a way so that they are independent?

This question brings up the old topic of loose coupling and tight cohesion.

I consider a class to be an island if it is loosely coupled, but also loosely cohesive.

A class that is an island is a class that does its own thing with very little dependencies or very weak dependencies.

Imagine for a moment a CurrencyConverter class.

This class is designed to take an amount of one currency and convert it to another currency.

There are several ways we could create the API for this class.

Here is one of them:

public class CurrencyConverter
{
    CurrencyConverter(ICurrency destinationCurrency)
    { ... }

    ICurrency Convert(ICurrency sourceCurrency, decimal amount);
    { ... }
}

 

Now this seems like a pretty reasonable API, but there is a problem here.  The problem is if you have a class that implements ICurrency, you don’t have any good way to know that this class exists.

If we don’t do anything else, this class might as well not exist at all.  It is an island.

It can see classes it needs to use, but the classes that could benefit from CurrencyConverter don’t know it exists.

The developer that is using an ICurrency implementation has no good way to know that this convert exists.  Most likely a developer will end up writing their own implementation of a currency converter. 

Duplication! Bad!

The date and time link

You will often see this in code bases and frameworks.  You will have many different utility folders or classes that help do date and time operations.

The problem is that no one really knows these utility classes exist and so everyone ends up rewriting the functionality all over your code base and you have a mess.

Take a look at your own project, see if you can find all the utility classes that deal with date, time or currency.  Now check to see if those utility classes are being used everywhere they could be.  Chances are they are not.

Building bridges

We need to build a bridge to the island so that everyone can enjoy the nice beaches.

No point building your wonderful CurrencyConverter if no one knows it exists, right?

So how do we solve this problem?

It definitely is a tricky problem to solve.

The best way I have found is to tie the dependent class back to its dependency. 

Yes, I am advocating circular dependencies!

Don’t be alarmed, it is not that bad.  All you have to do is make sure that your ICurrency interface has a reference to CurrencyConverter so that someone using ICurrency will know it exists.

Now there are many ways of doing this.  Some involve using a base Currency class, others involve creating different static constructors for Currency classes.

I am going to show you a very simple example, just to make my point.

public interface ICurrency
{
    ConvertUsing(CurrencyConverter converter);
}

 

It really is that simple.  You have now built a bridge to the island that was CurrencyConverter.

Now when a developer types ‘.’ on an ICurrency implementation they will see a convert method that uses your converter and they will know it exists!  Joy!

Again, there are many ways you could build this bridge.  My intention is not to debate them here.

The point is… BUILD THE BRIDGE!

But it is a circular dependency, that is bad

Really?

It is worse than 5 implementations of currency conversion in your code base?

What we have really done here is build something that is extremely tightly cohesive.  This is not necessarily a bad thing.

What we don’t want to do is to tightly couple the currency conversion to our billing system code.  We don’t want to have some building system class being used by our CurrencyConverter.

Applying the idea further

The basic idea here is that every time you create a new class you should think about how someone will know it exists.

If the class is out there on its own and it is likely to be useful in the the future, you must do something to tether it back to its dependencies.

It is helpful to have the attitude that if a developer can’t discover your class by some other class through intelli-sense, your class might as well not exist.

Every time I create a class, I try to think of two things:

  1. How will someone use this class
  2. How will they know this class exists

Basic to Basics: Understanding IoC Part 2 (Creation)

In my last back to basics post we talked about what inversion of control (IoC) is in regards to inverting control of interfaces.

We looked at how we can benefit from changing the control of the interface from the service to the client of that service.

This time we are going to tackle the more common form of IoC that is quite popular these days, and I’m going to show you why dependency injection is only one way to invert the control of the creation of objects in our code.

What is creation inversion?

In a normal everyday code we usually create objects by doing something like:

MyClass myObject = new MyClass();

We usually do this creation of objects inside of the class that is going to use or needs that object.

If we used interface inversion, we would probably end up with a declaration like:

IMyInterface myImplementation = new MyImplementation();

You can see that even though the interface may have been inverted we are still controlling the creation of the object from inside the class that uses the object.

Let’s withhold our judgment of whether this is a bad thing or not until we fully understand the idea of creation inversion and the problem it is trying to solve.

So we can see clearly that the normal control is that objects that are used by a class are created by the class using the object.

If we wanted to invert this control, we would say that objects that are used by a class are created outside of the class using the object.

There, you understand IoC in regards to creation.  If you create the object you are going to use inside your class, you are not inverting control.  If you create the object somewhere else, you are inverting control.

take control pic thumb Basic to Basics: Understanding IoC Part 2 (Creation)

Before we talk about how, let’s talk about why

Why would you want to invert control of the creation of your objects?

If you are familiar with design patterns, factory pattern should have just popped into your head when you read that statement.  If you are not familiar with design patterns, let me suggest either reading the original gang of four Design Patterns book, or the highly recommended Head First Design Patterns.

I mention the factory pattern because you might want to implement inversion of control for the same types of reasons why you would want to use the factory pattern.

Shh… I’ll tell you a little secret if you promise not to tell the cool IoC container / DI kids.  The factory pattern is inversion of control!

Let’s use factory pattern to tell us why we would want to invert control.  The description of factory pattern is:

Centralize creation of an object of a specific type choosing one of several implementations.

From that description we can deduce that we might want to use the factory pattern or some other form of inversion of control anytime we have an object or interface that has several different implementations and we want to put the logic of choosing which implementation to use in a single location.

Let’s use the classic example of having a skinnable UI.

If we have a UI framework that includes classes like Button, DropDownList, TextBox, etc, and we want to make it so that we can have differently skinned implementations of each UI control, we don’t want to put a bunch of code into all of our screens in our application like:

Button button;
switch(UserSettings.UserSkinType)
{
   case UserSkinTypes.Normal:
        button = new Button();
        break;
   case UserSkinTypes.Fancy:
        button = new FancyButton();
        break;
   case UserSkinTypes.Gothic:
        button = new VampireButton();
}

 

If we can somehow move the creation of the right kind of object outside of our classes and put it in one place, then we can eliminate all this duplication.  This prize comes at a price.  The price is control.  For convenience sake we must rip the control of the creation of button from our UI screens and give them to a central authority who doles out buttons at this whim.

We need a mediator that says to our UI screen classes, “You want a button?  I got button for ya, and you don’t need to know what kind, cause it’s none of your business.  All you need to know is it’ll be a button.”

I’ll harp on this more later, but I want to bring up the point here before we move on.  Why would you use inversion of control when you don’t have a problem of needing to select from more than one implementation of an interface or class?

How can I gets me some of this IoC goodness?

So let’s talk about how to implement inversion of control.

I’m not going to cover every conceivable way that you could invert control in regards to creation, but I will focus on a few common ways to do so.

Factory pattern

Let’s start here since we already mentioned it.  In our case above we could easily create a button factory that would be able to hand us the correct kind of button.  We would put all the logic for determining which button to use inside of the factory and let it hand us the correct kind of button.  Our code from the above example would be simplified to:

Button button = ButtonFactory.CreateButton();

Service locator

A service locator is very much like a factory.  You can even use a service locator to get you the right kind of factory that can make your object.  The difference is a factory creates a specific kind of object and a service locator can create different kinds of objects.

Using a service locator pattern, we might rewrite our example as:

Button button = ServiceLocator.Create(Button.class)

 

(We could also use generics here and do Create<Button>())

Dependency injection

(Notice I’m not saying IoC container based dependency injection.  I’m talking about dependency injection in general here.  You can use an IoC container to do it, but it is not required.)

This is the one most people are most familiar with.  We can use a special class that maps interfaces to their implementations, and then injects those implementations into our class via one of several ways.

Some of the common ways to inject the implementation into the class are:

  • Constructor injection – pass the dependency into your class via the constructor.
  • Setter injection – pass the dependency into your class by setting it on your class.
  • Interface injection – implement an interface that allows another class to call your class to give it the dependencies.

Using constructor based dependency injection we could rewrite our example like:

Button button = GetTheRightDangButton();
OurScreen ourScreen = new OutScreen(button);

 

Notice in this example, we are showing how to create our class.  We don’t have to do anything special inside our class besides provide a constructor that take our dependency.

But dependency injection looks silly in this example

Right you are!

What if we have more than one button we are going to use in our screen?  Seems pretty silly to put a single button in the constructor.

If you already know about IoC and dependency injection, there are two important things you should be taking away from this look back to the basics of creational IoC patterns.

  1. IoC doesn’t even require interfaces, it just requires that an object be created outside of the class that uses it.
  2. IoC does not always mean IoC container and dependency injection.  Many times the best solution is to directly instantiate an object or use a factory or service locator.  (Especially when you are using multiple instances of a class.)

Before embarking on an IoC journey, it is important to understand what problem you are trying to solve by introducing IoC and to understand the differences between using IoC to invert interfaces, using IoC to invert flow control, and using IoC to invert creation control.

In almost all cases the correct application of creational IoC involves solving one of two possible problems.

  1. I have multiple implementations of an interface or class and I need to be able to have the logic to choose which one all in one place.
  2. I have only one implementation of an interface or a class, but I need to completely decouple the interface from the implementation for a really good reason.  (A really good reason is something like: “I am writing the code that will use the implementation of this interface, but another company is going to provide the implementation in another library.)

As always, you can subscribe to this RSS feed to follow my posts on Making the Complex Simple.  Feel free to check out ElegantCode.com where I post about the topic of writing elegant code about once a week.  Also, you can follow me on twitter here.

Back to Basics: Understanding IoC

In my last back to basics post, we talked about dependency inversion and how it is the underlying principle that Inversion of Control or IoC is based upon.

We also talked a little about IoC and the three main forms of control that can be inverted; interface, flow, and creation.

In this post I want to dive a little deeper into IoC as it relates to interfaces and creation, we’ll ignore flow inversion for now, because it is often used in a way that is completely unrelated to the other two.

We’ll focus more on creation since we already covered interface inversion a bit and creation inversion is the “hot thing” right now.

Interface inversion

Why would we want to invert interfaces?

We covered much of the reasoning in my post on dependency inversion, but there are some important points that specifically relate to interfaces in programming that I would like to address here.

First we must understand what non-inverted interfaces are.

Let’s take an example.  Suppose I have a Kangaroo class that has several methods, among them is Punch. Normally, Kangaroo defines the interface that has to be used if you want to use the Kangaroo class.

By default the constructor for the class and the method signatures all make up an interface that the user of the Kangaroo class must adhere to in order to use the Kangaroo class.  The module that contains the Kangaroo class usually also would include the interfaces if one is defined.  In this case the interface would be something like IKangaroo.

non inverted thumb Back to Basics: Understanding IoC

Why do I say this is normal and not inverted?  Because it makes sense that a class would define what it does and how to use it, not some other entity.

An inversion would be if some other class or module told Kangaroo what its interface needed to be.  Kangaroo would stop being an independent definer of a service it has to offer and instead become a provider of a service someone else wants.

From our narrow viewpoint of the Kangaroo class we don’t know what kind of interface to provide, so we have to assume it is something like IKangaroo.  The way we look at the Kangaroo class can completely change though, if we consider the BoxingMatch class.

If we create a BoxingMatch class, we really want that class to define the interface that all boxers must adhere to.  We don’t want to have to have the BoxingMatch class know how to make multiple classes with different interfaces box.  So within the module that the BoxingMatch class is defined, we define an interface called IBoxer.  Within the BoxingMatch class itself we use IBoxer instead of specific classes that we want to use in our boxing match.

Now we have given Kangaroo a suitable interface to implement.  IKangaroo hardly makes sense, but IBoxer gives our Kangaroo a purpose.  Our little Kangaroo class is providing a service.  He can box!

inverted thumb Back to Basics: Understanding IoC

In this example you can see that we have inverted the control of the interface.  Instead of letting the provider of the interface define that interface, we have made the user of the interface define it.

The value we gained here is that we have given control to the higher level module and reduced the complexity of its logic by preventing it from having to deal with every different interface defined by classes that can box.

Up next creation inversion

I’m going to end this post here, since creation inversion is going to be a longer topic, and I am on vacation right now.

In my next post we’ll cover creation inversion and talk about how it relates to Dependency Injection or DI.

As always, you can subscribe to this RSS feed to follow my posts on Making the Complex Simple.  Feel free to check out ElegantCode.com where I post about the topic of writing elegant code about once a week.  Also, you can follow me on twitter here.