Automated Functional Tests: Record or Program?
Recently, I've been devoting some time to building out a framework using Watij with the goal to be able to easily write automated functional tests for backlog items for the application I am working on. There are many factors in the technology choice, but one of the most prominent of questions about doing automation seems to be around whether or not to use a tool that records test scripts vs using a WatiX or similar library and rolling your own framework. Let me put a little caveat here, since I know this will come up. If you use a recoding tool and then you use its internal language to build a framework that is actually reusable and makes it so you don't have to re-record every time you create a new test, I would count that as programming vs recording. When I say recording, I mean mostly just recording and plugging in variables.
Let's looks at the pro's and con's:
+ Very fast to get started, you can have a working test right from the get go.
+ Anyone can do it. Just requires a little training in the tool, but not much technical knowledge needed.
+ Easy execution of tests, usually integration with test case management systems, defect systems, etc.
– Speed of test creation doesn't really increase as test library increases.
– Lots of pointing and clicking to create the tests each time.
– Fragile: if a screen changes it can break many tests which each have to be changed.
– Expensive, most of the recoding tools are very expensive.
– Heavy installation, most of the tools require thick clients to use, installed on any machine recording or running them.
+ As your library increases in size, test case creation becomes very rapid.
- Developers are more likely to participate since it is a programming type activity.
Can abstract screen changes from functions it order to protect against change. (If one screen changes you are more likely to only need to update the framework in one spot, not change every single test.)
Low cost, as in free, solution. (Besides development cost)
Can run on any machine that has Ruby, .NET Framework, or Java depending on framework used.
Takes at least some development expertise to build the initial framework.
Requires more training to understand how to use.
Initial creation of test cases takes a long time.
The patterns of pro's and con's here closely resembles the pattern of pro's and con's for doing unit testing or test driven development. Higher cost initially, but over time large amount of savings. Programming seems to be a clear winner over recording, in my opinion.
Next time I'll take a look at how to implement a good framework for automated functional testing using a tool like Watij, Watin, or Watir.