test automation framework architecture post

Test Automation Framework Architecture

Test automation framework architecture efforts are often complete failures.

It’s true. I’ve worked with many companies who have given up on creating a good test automation framework architecture, because after investing a large amount of time and money and resources in doing it the wrong way, they have incorrectly assumed the entire effort is not cost effective.

In this post, I’m going to simplify the process of creating a test automation framework architecture and show you that—if you do it right—it can be a very good investment.

Test automation framework architecture basics

Let’s start off by talking about what the goals of a successful test automation framework architecture should be.

There are two goals I am interested in when creating a test automation framework:

  1. Having the ability to easily create simple automated tests that use the framework.
  2. Decoupling cosmetic application changes from the tests so that when the application changes, the tests do not all have to be updated

A good test automation framework architecture should at the very least provide these two important services. We create a test automation framework to allow us to support the creation of tests and to keep those tests from being dependent on the actual UI of our application—as much as possible.

I’m going to break down these goals just a little more to make sure we are all on the same page.

test automation framework thumb Test Automation Framework Architecture

Creating simple automated tests

First, we’ll start with having the ability to easily create simple automated tests that use the framework. Why do we care about this goal? And why would this be the responsibility of the framework?

I have found that one of the biggest failure points of test automation efforts is test complexity. The more complex the tests are, the harder they are to maintain. If tests are difficult to maintain, guess what? They won’t be maintained.

So, we really want to make sure that when we create a test automation framework architecture, we focus on making sure that test automation framework makes it as easy as possible for someone to create tests using it.

It is always a better choice to put the complexity into the framework instead of into the tests. The framework is usually maintained by developers and does not contain repeated code. But, tests are often created by QA or even business folks and the code found in tests is usually repeated across many tests, so it is vital to reduce the complexity on the tests, even if it means a large increase of complexity in the framework itself.

When I create a test automation framework architecture, my goal is to make it so the tests can read as close as possible to plain English. This requires some effort to pull off, but it is well worth it.

Decoupling the UI from the tests

Next, let’s talk about what I mean by decoupling cosmetic application changes from the tests.

This is also a vital goal of any test automation framework architecture, because if you fail to decouple the UI of an application from the tests, every single UI change in an application will cause possibly hundreds or thousands of tests to have to be changed as well.

What we really want to do is to make it so that our test automation framework architecture creates an abstraction layer that insulates the tests from having to know about the actual UI of the application.

At first this seems a little strange, especially when I tell automation engineers or developers creating an automated testing framework not to put any Selenium code into their actual tests.

All of the Selenium examples show you creating tests that directly use a web browser driver like Selenium, but you don’t want to write your tests that way—trust me on this one.

(By the way, I haven’t found a good book on creating an actual test automation framework architecture—I’ll probably write one—but, for now if you are looking for a good book on Selenium that starts to go into the topics I discuss here, check out Selenium Testing Tools Cookbook.)

Instead, what you want to do is to make it so the test automation framework is the only code that directly interacts with the UI of the application. The tests use the framework for everything they want to do.

So, for example:

Suppose you are creating a simple test to check to see if a user can log into your application.

You could write a test that looks something like this: (C# and Selenium in this example)

[sourcecode language=”csharp” padlinenumbers=”true”] var loginInput = driver.FindElement(By.Id("txtUsername"));
loginInput.SendKeys("Joe");

var passwordInput = driver.FindElement(By.Id("txtPassword"));
passwordInput.SendKeys("$ecret");

var loginButton = driver.FindElement(By.Id("btnLogin"));
loginButton.Click();

[/sourcecode]

But, what is going to happen when you change the ID of your “User Name” field? Every single test that uses that field will break.

On the other hand, if you properly create a test automation framework architecture that abstracts the UI away from the tests themselves, you would end up with a much simpler and less fragile test, like this:

[sourcecode language=”csharp”] LoginPage.GoTo();
LoginPage.LoginAs("Joe").WithPassword("$ecret").Login();
[/sourcecode]

Now, if the ID of your “User Name” field changes, you’ll just change the code in your automated testing framework in one place, instead of changing 100 tests that all depended on the specific implementation of the UI.

A simple test automation framework architecture

Once you understand those two very important goals and why they are important, it is easier to think about how you should design a test automation framework architecture.

I’ve created quite a few test automation framework architectures, so I’ll give you a basic design I use for most of them.

Take a look at this diagram:

test automation framework architecture thumb Test Automation Framework Architecture

Here you can see that there are four layers to my test automation framework architecture.

Frist, we have the browser layer or the web app itself. This just represents your actual application.

Next, we have the Selenium or web driver layer. This layer represents your browser automation tool. Selenium is basically just a framework for automating a browser. It doesn’t know anything about testing, it is an API that lets you interact with the browser programmatically. (By the way, you don’t have to use Selenium. I just use it as an example here, because it is the most popular browser automation framework.)

After that, we have the framework layer. This is the actual framework you create which uses Selenium, or whatever web driver you want, to actually automate your application. The framework is acting as an abstraction between your application and the tests. The framework knows about the UI of your application and how to interact with it using Selenium. It is your job to create this layer.

Finally, we have the actual tests. The tests use the framework to manipulate your application and check the state of the application. These tests should be simple to understand and should not have to know anything about the actual implementation of the UI of your application. These tests should rely on the framework to give them the ability to do whatever they need to do in your application.

Now what?

Obviously, creating a test automation framework architecture is more complicated than what I have shown you in this article, but with these basics you should be off to a good start.

If you want a more in-depth and detailed explanation as well as an example of how exactly to create a full test automation framework architecture, you might want to check out my Pluralsight video on the topic:

Creating an Automated Testing Framework With Selenium

I also frequently blog about test automation topics and other topics related to becoming a better and more profitable software developer.

Sign up here to make sure that you don’t miss any of my posts and get access to the content I only share with my email list. (Don’t worry I hate spam as much as you do.)

If you have any question, post them in the comments below.

  • Phil S

    Very useful post John…I was demonstrating a Selenium based framework only today and there was some discussion if this was the better approach compared to the quick recordings via the Firefox plugin. Very useful. Cheers.

    • jsonmez

      Great, glad you found this useful.

  • Pingback: Test Automation Framework Architecture - Simple...()

  • AdamL

    Completely agree with the content here. One thing I’d maybe add is to change the name of what you call ‘Framework.’ In the same way that Selenium has a ‘WebDriver’ component, I would call your ‘Framework’ layer the ‘ApplicationDriver’ layer.

    ‘Framework’ is a little too loaded of a term.

    • jsonmez

      Good point. I agree about framework being overloaded. Application driver is a good name.

  • http://trycatchfail.com/ Matt Honeycutt

    I agree, simplicity and insulation from UI refactorings (things like renaming a field are refactorings IMO) are essential for creating maintainable end-to-end tests. If your tests break for the wrong reasons, they become a barrier to change, not an enabler.

    For ASP.NET MVC apps, that’s why I created SpecsFor.Mvc: your tests look like normal unit tests, they’re refactor friendly and strongly-typed, and much of the complexity is swept under the covers. It isn’t something a non-technical person would use though…

    • jsonmez

      Yeah, that makes sense. I’ll have to check out your SpecsFor.Mvc. I haven’t had a chance to work with it yet.

  • David Kiss

    Interesting post. Building a test automation framework upfront instead of starting to write the actual tests can be a huge investment (depending on the potential number of test cases). I’m interested to hear how you measure ROI on that investment and how you find the balance between code that should go into the framework and code that should stay in individual tests?

    • jsonmez

      Almost all the code should go in the framework. The tests should be as bare as possible. This makes it easier to maintain the tests and makes their meaning much more clear.
      As for ROI, it is difficult to measure. The best way to measure it is to compare it against spending the same effort creating unit tests that cover the same amount of the application’s code. These kind of automated tests almost always come out way ahead. (Not that units tests are useless, it is just that these kinds of tests are generally more valuable.)

    • David Luu

      I think another interesting aspect for discussion is whether to fully build out the framework (in terms of interfacing “APIs” to all of the application’s UI and parts) or whether to build out the framework as you need it while writing each test. This is generally not a problem when you start from scratch for both the framework as well as application under test. But is a problem if you already have an existing app for which you are now playing catch up for automation or switching over frameworks/tools.

      I could be wrong, but it seems more people agree upon the build it as you need it approach for the framework in this latter case. Which makes sense to a point. But I’ve personally found it problematic as well when you do build out the structure then resume later to build out more when you get to it and also factor in distributed teamwork and how people code differently. Building it piece by piece as you need it won’t necessarily end up with the same (ideal) intended design as if you build it out fully in the first place (taking up more upfront investment) unless you happen to have a highly cohesive collaborating and skilled automation team or a very diligent automation lead/architect who follows up all the time to ensure the code worked on by different people at different times ends up fitting the intended design. Unfortunately, in practice, neither of those occur very often.

      • jsonmez

        I definitely am in the habit of doing a build it as you need it, but I usually like to build out smoke tests first so that we hit different areas of the application and can build a solid foundation for the framework. You have some interesting insights that go much deeper than most people on automation. What is your background?

        • David Luu

          I’ve worked across the spectrum within QA from working with automated tests, automation frameworks, custom test tools and scripts & glue code, load/performance testing, manual and exploratory testing, training others, and sometimes coming up with proof of concepts & defining specifications and architecture for others to then implement/complete. Though I’m in QA field, I specialize in working with test tools and frameworks, and integrating disparate platforms together for testing, and with a developer background. I’m also a hobbyist software developer. I tend to think like a test automation architect, though that’s not always my role.

          My insight is comprised of my experiences working on frameworks and tools while collaborating with others in QA. I’ve yet to work on an A-Team of QA members with solid automation & development skills that rival or surpass mine for me to think more optimistically. There are such people, I just don’t work with them enough, and these people also tend to do the same work I do leading projects. But to really have solid automation, you need a group of these people in a team to do the work rather than farm out to the juniors with much less skills. That or you need to dedicate some of these senior folks to really scrutinize the work of the junior members regularly to enforce the good framework coding standard. Or it kind of falls apart over time (though never completely of course, except in the worst cases). Code reviews would probably help. Sadly test automation code reviews are not valued as much as application code reviews for developers for it to be used and enforced on a regular basis.

          • jsonmez

            Awesome! Glad you found my blog. Looking forward to your insights.

  • Brian Burress

    Good information and perspective. I have always enjoyed creating automated tests as I view them to be just as challenging and fun as the project itself. While factoring in the need to test from the beginning of a project is ideal, even spending some time writing automated tests for existing code bases can be helpful. One of the failures that management makes in the arena of testing is a sort of “all or nothing” view — i.e. an entirely automated suite of tests is the required outcome to make an automated test effort useful. You comments and approach sort of forces a hybrid approach. Certainly for validating a UI, a human is much more adaptable and hence better than an automated test for the reasons you state unless the UI is fairly static. In addition, for tests that are for obscure scenarios difficult to simulate or just for lack of time to complete the automated tests then manual testing works fine. I have never been disappointed with having some set of automated tests that I can work with. One of the applications a currently support is a fairly mature transaction processing system. I began to work on some automated tests just to help have something in place to test some of the code base and provide some piece of mind that nothing gets broken when I make a change. I found it somewhat funny that despite being what I call a “mature” system that I found bugs in the code as a by-product of creating the tests. Whether or not the automated tests ever catch a bug that gets introduced, finding an existing problem was worth it for me.

    • jsonmez

      Yeah. I agree with you. You basically have to use common sense and be pragmatic. An all or nothing approach is never good.

  • David Luu

    Building a good test framework is one thing. It’s also another to keep maintaining it well. Unless the framework has built in safeguards that prevent misuse or deviation from intended design, which are highly unlikely, the framework’s efficiency can be derailed over time as regular oversight of the framework slips away, or as different people transition in and out of the team that work with the framework. This can potentially be a problem when you have more junior members (in terms of programming & design skill) of the team working with the framework, adding/modifying it without much oversight and if they also don’t consult the senior members about their work often. A good framework really requires solid “developers” to use it or otherwise constant guidance from one to junior members in order to preserve the ultimate goodness of the framework.

    • jsonmez

      Yes. Agree 100%. I always train the teams that I help develop the framework on how to maintain it properly and stress exactly what you have pointed out here. Sometimes I can put in rules that can do some static checking to help aid with that.

  • testmonger

    Would it be useful to include a parallel test path that perhaps hits the application but not the full UI?
    For example, I have written http request level tests for an application (using Visual Studio web test) that worked just fine day after day; but, if you ran the exact same test steps in the GUI they will fail two weeks after the initial creation. There was a java script filter in one of the drop down lists that exclude certain accounts more than two weeks old, enforcing a certain business rule. But the goal of my test was not functional testing, it was to detect if the system was up and responding and that the response time fell within a given tolerance threshold.
    There could be a similar class of tests written for web services.

    • jsonmez

      Yes and no. It is a different kind of testing when you don’t go through the full UI. It depends on what you are trying to test. I often combine the two when automation tests start to become to lengthy to run all at once and there is a lot of redundancy in what is being tested in the UI.
      Good point though. Sometimes you need to just test specific functionality and there is no need to involve the UI.