By March 28, 2011

Getting up to BAT: Creating Your First Smoke Tests

If you have been following my posts so far on Becoming Bat Man, you should already have hired an automation lead, figured out what browser driver you are going to use and come up with a design plan for your automation framework.


Now you are probably faced with the difficult decision of…

Where to begin with your tests

I don’t suggest writing your automation framework first and then writing tests to use it.  If you go this route you’ll have two problems

  1. It will be a long time and large investment before you see any fruits from your automation effort.
  2. Your framework will likely be designed incorrectly or not test-writer friendly.  (Which is basically the same as wrong.)

What you really want to do is to write some basic tests that let the development of those tests drive the development of your framework.

The best tests to drive your framework are some basic “smoke tests.”  If you are not familiar with the term “smoke test,” it basically means some tests that make sure the application you are testing is at least somewhat working and not completely broken. 

There are a few different opinions of what exactly constitutes a smoke test, but in general, if you have any tests that you regularly run on a new build to make sure it is at least OK, those would be the tests we are interested in here.

You are going to want to start by taking the easiest and most simple tests you have and writing them in a concise syntax that you use to drive the development of your framework.

It is very important at this stage to not worry about what you think might be possible or how it would be implemented in the framework, but to instead focus and making the tests as terse and explanative as possible.

The basic idea is that you will write the test out in the syntax you would like to be able to use, then implement the framework features required to make that test pass.  This whole process is very similar to doing Test Driven Development.

An example case

The easiest way to demonstrate this is with an example.

Let’s say you have a smoke test like the following:

Create a new account, pick an on-sale item, add it to your cart and check out using PayPal.  Verify that we hit the order success page.

We might write a test for this that looks something like:

What I just wrote above is astonishingly simple.  You are likely saying to yourself something along the lines of “What if… blah.”

Forget it!  We’ll handle “what if” later.  Right now we are going to write the tests as terse as possible and not handle any other conditions.

What I wrote above is the style that I like to write the tests in, which is highly subject to opinion, but the content of what I wrote is not as subjective.

There are a few important points to note:

  • I used methods like CreateNewUser, LoginWithLasteCreatedUser and CheckoutUsingPayPal that do a huge amount of work and have no parameters.  Why?  Because for the purpose of this test, (and you’ll find for most of your tests), we don’t care about what the first name or last name or address of the user is, we don’t care how we get them logged in, and we only care that they checkout using PayPal, not what account is used, etc.
  • I didn’t declare any variables.  Tests should be simple.  Let the framework remember the state.
  • I over-simplified.  This is fine, you are working inside of a very specific domain.  Let your test framework methods do one large thing.  Don’t worry about oversimplifying for now.
  • I asserted something at the end.  All tests need to assert something.  If you can’t figure out what to assert, work on redesigning the test before automating it.
  • I didn’t pass parameters.  Avoid this also.  This goes along with the first point, but is so important that I feel it is OK to state it twice.  If you don’t care about variation, eliminate it.  You will constantly need to find places where a manual tester would have to enter specific information, but an automated test won’t care what that information is.
  • All my methods are static.  While this would be bad in normal production code, for a test API it is the easiest for test writers to use.  No variable declarations, no “newing” up objects, no constructors, etc.
  • It’s written in “business language.” As close to the language an end user or business person would use to describe the test.

So, this test is definitely doing a huge amount of stuff in a few lines of code, and because of this your inclination would be to say that the API level is too high. Don’t!

Instead, make the API level as high as you possibly can, until further tests you try to implement prove to you that it must be broken down further.

By working this way, you ensure a reduction in the total lines code and complexity of your tests by as much as possible.

You would NEVER work like this with normal code, but for automated tests this is essential.  The main reason is because you will be calling the same methods perhaps 1000s of times, where in normal code you usually only reuse a method in a few places.

Next steps

For my example above, the next step would be going through and implementing all the framework methods required in order to make that test runnable.  You will end up utilizing the general design you came up with for your automation framework to do this.

Then you will simply move on to the next test and repeat.  Each time you should have less and less effort required, as more of the basic functionality of your automation framework is implemented.

Automating your smoke tests should ensure that you get a fairly even distribution of the framework code written as the smoke tests should cover many different areas of your software.

But, don’t start adding these smoke tests to your build yet.  We’ll cover that in the next step.  Adding them to the build prematurely can cause the confidence in these tests to be forever diminished!  We don’t want to do that!

About the author

John Sonmez

John Sonmez is the founder of Simple Programmer and a life coach for software developers. He is the best selling author of the book "Soft Skills: The Software Developer's Life Manual."