Once you’ve built some smoke tests with your shiny new automation framework, you are going to want to get those smoke tests up and running as soon as possible…
But! You might want to consider holding off for a second and reading this post!
It is worth taking a bit of time and thinking a bit about the strategy and psychology of adding the smoke tests to your regular build process.
There are several things we are going to want to discuss and think about before adding automated tests to a build.
Why are we adding the smoke tests to the build?
I think it is always important that before we do something we ask ourselves why we are doing it. Doing things, because someone said it is good, or because it seems right is not a very good reason.
The primary benefit we are going to get by adding smoke tests to the build now instead of waiting for a larger portion of tests to be written is that we are going to be able to immediately tell if a basic functionality of the system is broken instead of finding out after we have deployed the build to different environments and used it.
We want to be able to get this value as soon as possible. We don’t want to have to wait until we have a huge amount of tests, because even a few tests will give us some value in this area.
The other reason we are adding the tests to the build is so that we can notify developers when a test fails, so that they know to fix whatever was broken. By having the smoke tests run as part of the build, we are able to reduce the amount of time before a break is introduced and the break is discovered.
This concept of reducing the delta between introduction and discovery is very important, because it makes it much easier to identify what caused a break.
Perhaps a distant 3rd reason to add the smoke tests at this stage, is to prove out our framework and technologies. Better to start small with a small number of tests and get everything working.
With those ideas in mind, we can move onto some of the more important considerations.
Don’t ever add failing tests to the build
When you add your smoke test suite to the automated build, all of the tests should pass!
I will say it again, because it is really important and ignoring this advice may doom your entire BAT project.
Do not ever add failing tests to the build!
It seems like an innocent enough thing, but it causes great harm, because it is very much like the boy who cried wolf. Consider what happens when we introduce our new BAT system and we add our smoke tests to the build, but we have 3 failing tests.
First thing that will happen is that our tests will run during the next build and 3 will fail. This should cause a build failure notification of some sort to go to the team, at which time you will be in the uncomfortable position of saying something to the effect of “ignore that fail, those are real defects, but they are known issues.”
Next, let’s assume a developer checks in some bad code and it causes a 4th test to fail.
Do you think most people will notice this 4th failing test?
Even if they do, do you think they will just assume it is another known issue?
Worse yet, they may think your BATs are meaningless and just fail randomly.
Do you see where I am going with this? We want to start off with the correct expectations and we want test failures to be meaningful. To ALWAYS be meaningful. The second you have test failures that are not meaningful, you lose credibility for any future meaningful failures.
This is such a critical psychological battle with build systems and automated tests that I have often implemented a policy where no one is allowed to work on anything else but fixing the build or test failure when a build fails. When you have 20 developers either sitting idle or working on one issue for 2 hours, it sends a pretty clear message that we take build failures seriously.
Make sure your tests never undermine this message.
What do I do with failing tests then?
If they are real defects, put them in your defect tracking system or create backlogs for them and make sure the tests are disabled until those defects are fixed.
Your automated tests should be acting as regression tests for things that are already working. It is absurd to think that it is even close to cost effective to write automated tests to actually functionally test new features or code!
It is the process of writing the automated test that should uncover defects. Running of automated tests should uncover new defects where functionality has regressed from working to not working.
Other considerations
Here are a list of some other considerations you will want to think about before embarking on this journey.
- How will you actually run the tests? (Will you use the build script to kick it off, some 3rd party plugin, etc)
- Where will you run the tests? (Will you use the build server itself, or another machine. You probably don’t need to worry about scaling out to multiple machines right now, but you should at least keep it in the back of your mind.)
- How long will it take to run the smoke tests? (Fast is better, because feedback cycle is reduced. If you have a really long test suite, you might want to pare down the smoke tests and run some with every build, and all of them every night for now.)
- Will developers be able to run the tests locally? (Make sure you consider how to do this before putting the tests into the build. Nothing more frustrating than being measured against a bar you can’t yourself see.)
- How will you notify and report results from the test failures? (We don’t care about passes, only failures, but you need a good way to report the result that prompts action.)
- What is the policy for test failures? (Make sure there is some agreement on what kind of action will be taken when tests fail. A great place to start, if you can get management to buy in on it, is all work stops until all tests pass. Extreme, but extremely effective.)
- How will you prevent false fails and identify them? (Don’t get stuck with egg on your face. If your framework fails and the problem is not a real defect, you need to be able to quickly identify it and fix the framework problem. There are a few strategies for doing this, but that is for another post.)
- How do new tests get added and commissioned? (You are going to be constantly growing your test suite, so you will need a good method for adding tests to the smoke test suite.)
Remember, you only get one chance at a first impression!