One of my first official jobs in the software development industry was that of a tester.
My job entailed looking at stacks of papers that were printed out by a new printer we were testing at HP, and comparing them to the “master” printouts produced by older printers.
I didn’t actually do the comparison of the pages myself; instead I would execute the tests, someone else would compare the printouts, and I’d look at the differences they flagged.
With each difference, I would review and decide, based on the test, whether the result was a true failure or defect. If it was the latter, I’d write up a defect report for a developer to look at—and possibly fix.
Later in my career, I took on a different testing role as a testing lead for a multi-function printer.
I’d decide what should be tested, how it should be tested, and then I’d come up with a testing plan and run the tests to verify the printer worked how it was supposed to.
It was through those experiences that I learned that most developers have no clue about how testing is actually done, and how valuable this understanding can be for developers who really want to excel in their careers.
I owe a large amount of the success I have had in my career as a software developer to my background in testing.
That background caused me to look at the code I was writing a little differently and to realize that my job as a software developer wasn’t just to implement features and fix bugs, but to make the software I was writing work correctly and as intended.
Seems like a simple and obvious idea, but if you don’t at least know the basics of testing, you probably aren’t going to have the best idea of what “working correctly and as intended” actually means.
The basic idea behind testing
Often new programmers don’t understand testing. They don’t think it necessary.
At the surface level it can seem a bit extraneous.
Do we really need to test that code? I ran it on my machine and it worked perfectly, so let’s just ship it.
Testing, at its core, is really about reducing risk.
The goal of testing software is not to find bugs or to make software better. It’s to reduce the risk by proactively finding and helping to eliminate problems which would most greatly impact the customer using the software.
Impact can happen with the frequency of an error or undesired functionality, or it can be because of the severity of the problem.
If you had a bug in your accounting software which caused it to freeze up for a second or two whenever a value higher than $1,000 was entered, it would not really have a huge impact; but that would be a high enough frequency to be very annoying to the customer.
On the other hand, if you had a bug in the accounting software that caused all of the data to become corrupted every 1,000th time the data was saved, that would be huge impact, but very low frequency.
The reason I define software testing in this way is because—as any tester will tell you—you can never find all the bugs or defects in a piece of software and you can never test every possible input into the software. (For any non-trivial application.)
So, the idea is not to find every single possible thing that is wrong, or even to verify the software against a spec—as some people like to define software testing—because both are impossible.
Oh, also if you ever find a complete spec of any application in your experience as a software developer, let me know.
Instead, the focus and main idea behind software testing is reducing the risk that the customer is greatly impacted in a negative manner using the software.
Typically this is achieved by first prioritizing what areas of the software are likely to have the biggest impact (i.e. risk), and then deciding on a set of tests to run which verify the desired functionality.
When the actual functionality deviates from the desired functionality, a defect is usually logged and those defects are prioritized based on severity.
Some defects get fixed, other defects are low enough impact that they are just noted and left in the system.
Common types of testing
The world of testing and quality assurance is huge.
Just like the development world has many concepts and methodologies for creating software, there are many ways to think about an executing testing and the field is changing all the time.
Even in name.
Early in my career it could be perceived as a slight or insult to call someone who worked in testing a tester, they preferred to be called QA (or quality assurance) professionals.
Just a year or two ago, I attended a testing conference and I made the mistake of calling someone a QA person. They corrected me and said that tester was the prefered term.
You can’t win them all.
Anyway, let’s talk about the different kinds of testing, so you can get a general idea of what someone is talking about when they throw around these terms—which you will hear plenty often in the software development world.
This is not an exhaustive list by any means.
Black box testing
One of the most common forms of testing—and really a way to describe a whole category of testing—is black box testing.
Black box testing is simply testing as if the software itself was a black box.
When you do black box testing, you are only concerned with inputs and outputs. You don’t care how the actual outputs are derived.
You don’t know anything about the code or how it works, just that for a given set of inputs into the software, a given set of outputs should be produced.
Most testing is done in this fashion, because it is largely unbiased. It either works or it doesn’t.
White box testing
White box testing is pretty much the opposite of black box testing.
With white box testing, you have at least some idea of what is going on inside the software.
Often unit testing is called white box testing, but I disagree. Unit testing is not testing at all—we’ll talk about that more in an upcoming chapter.
Instead real white box testing is when you understand some of the internals of the system and perhaps have access to the actual source code, which you use to inform your testing and what you target.
For example, if you looked at the code that did complex calculations for some accounting software, and you saw that there was a section of the code that did one set of calculations for values above a certain amount and another set of calculations for any other values, you’d be able to create tests that target both of those scenarios.
Acceptance testing
Acceptance testing goes by many different names.
Sometimes it’s called user acceptance testing.
Sometimes it’s called system testing.
The basic idea of acceptance testing though is that you have some tests which test the actual requirements or expectations of the customer, and other tests that run against the system as a whole.
What I mean by this is that you don’t just test one part of the software in isolation.
This kind of testing could be testing the functionality of the system or it could be testing the usability, or both.
The idea is that acceptance testing tests what is expected versus what actually happens.
Automated testing
This is another broad kind of testing which can take many forms and has many definitions, but I define automated testing as any testing in which the execution of the test and the verification of the results is automated.
So, you might automate the testing of a web application by running scripts which open up a web page, input some data, push some buttons and then check for some results on a page.
You could also automate the testing of an API by writing scripts which call out to the API with various data and then check the results that are returned.
More and more of testing is moving towards automated testing, because manually running through test cases over and over again can be tedious, error prone and costly–especially in an Agile environment where the same set of tests may need to be run every two weeks or so to verify nothing has broken.
Regression testing
That brings us to regression testing, which is basically testing which is done to verify that the system still works the way it did before.
The purpose of regression testing is to make sure the software doesn’t regress in functionality.
This is extremely important with Agile development methodologies–more on that in a future chapter–where software is developed incrementally and there is a constant potential that adding new features could break existing ones.
Most automated tests are regression tests.
In fact, you could really make the argument that all automated tests are regression tests, since the whole purpose of automating a test is so that it can be run multiple times.
Functional testing
Functional testing is another broad term used in the testing world to refer to testing activities where what is being tested is the actual functionality of the system.
This might seem obvious.
You might be thinking “duh, what else would you test if you didn’t test the functionality of the system.”
But, it turns out that you can test all kinds of things that aren’t related to functionality, like performance, usability, resilience, security, scalability–I could go on and on, believe me.
So, functional testing the kind of testing where you are really concerned with if the system does what it is supposed to do given from a functional perspective.
If I put in this input and push this button, do I get this expected output?
I don’t care how long it takes. I don’t care if the screen flashes bright read and the computer starts to smoke, do I get my result?
Exploratory testing
I like to make fun of exploratory testing and call it “lazy ass testing.”
It really pisses testers off when I do that.
But, there is definitely some legitimacy to the idea of exploratory testing and perhaps I am a bit too harsh and judgemental.
The idea behind exploratory testing–when done correctly–is that you have some guidelines and basic plan of areas of the application you are going to test and ways you are going to test it.
Then, you go about without actual test cases and explore the application, looking for things that might be wrong or behaviour that is unexpected.
Often exploratory testing sessions are recorded, so that if an error is found, the problem can be reproduced by retracing the steps taken by the exploratory tester.
While, I’m generally not a huge advocate of this kind of testing, I do have to acknowledge its merits as exploratory testing can often uncover bugs which no rational test case would have ever been designed to exploit.
Other forms of testing
Truly we’ve only scratched the surface of all the different types and classifications of testing there are.
Many other forms of testing exist including load testing, to see how an application performs under a heavy load, performance testing, to test performance of the application based on certain scenarios, recovery testing, to test recovery from error conditions or hardware issues, security testing, to test the security of the system, stress testing, usability testing… the list goes on and on.
I just wanted to cover some of the basics here which you’ll hear about and see in everyday conversations as a software developer.
The Testing Process
Different organizations are going to have very different ideas of how testing should be done and what process should be followed.
You’ll also see plenty of formal specification of what is the “testing process” out there put out by various testing organizations.
So, again, like a large amount of what I said about testing, the idea here is not to be prescriptive or to perfectly model the perfect testing process, but rather to give you an idea of what the testing process is like in general and what it entails.
I like the pragmatic approach to life.
Testing usually begins with the development of some kind of a test plan.
How will things be tested?
What is our strategy for testing?
What kind of testing are we going to do?
What features are we going to test?
What is the schedule?
These are all questions that are generally answered in the test plan or if the test plan is not a formal document the planning of testing for a project.
Next, the tests are usually designed at a high level based on the requirements or functionality of the system.
So, at this stage a tester might be coming up with a list of general test cases which will be run, what kinds of conditions will be tested, and coming up with what will be needed to perform the tests.
After that, the tests are usually actually created and executed.
Sometimes this occurs as a single step.
Sometimes tests are written in a test management software first and executed later.
The results from the test execution are recorded and evaluated and any bugs or defects are usually logged into some kind of bug tracking system.
Bugs are prioritized and sent to developers to fix.
Fixed bugs are retested and this cycle continues until the software meets the quality standards where it is decided it is shippable code.
And that’s basically it.
Plan how to test, design the tests, write the tests, execute the tests, find bugs, fix bugs, release software.
How Testing Works on Agile Teams
The standard process of testing tends to run into some problems on Agile teams where new features are being coded and implemented every couple of weeks or so.
Many teams try to either strictly follow the standard testing process or completely throw it out the window instead of working it into the Agile lifecycle of software development.
Both of these approaches are wrong.
Instead, the focus really has to change on developing the test cases and test scenarios up front, before any code is even written and simply shrinking the test process into a smaller iteration, just like we do when we develop software in an Agile way.
This just means that we have to chop things up into smaller pieces and have a bit of a tighter feedback loop.
Instead of spending a large amount of time upfront creating a testing plan for the project and intricately designing test cases, teams have to run the testing process at the feature level.
Each feature should be treated like a mini-project and should be tested by a miniature version of the testing process, which begins before any code is even written.
In fact, ideally, the test cases are created before the code is written at all–or at least the test design, then the development of both the code and the test cases can happen simultaneously.
Another major consideration with Agile testing is automation.
Since new software is released on very short iterations, regression testing becomes more and more important, thus automated testing becomes even more critical.
In my perfect world of Agile testing, automated tests are created before the code is actually written to implement the features–truly test driven development–but, this rarely happens in reality.
Testing and You, the Developer
What about you, the software developer, what is your role in all this testing stuff?
Do you even have one?
Yes. Definitely.
One of the big failings of software development teams is not getting developers involved enough or taking enough ownership for testing and the quality of their own code.
As a software developer, you should be concerned with quality more than anyone else.
You can not have the mindset that QA will find the bugs in your code.
Instead, you should absolutely make it your responsibility to find and fix the bugs before your code goes to testing.
The reason is fairly simple. The further along in the development of software a bug is found, the more expensive it is to fix.
Think about it this way.
If you test your own code thoroughly before you check it in and hand it over to QA and find a bug in that code, you can quickly fix that bug and perhaps it costs an extra hour of time.
If you take that same bug, and you don’t take the time to find it yourself and fix it, the process might go something like this:
A tester runs a test which finds the bug in your code.
The tester re-runs the test to make sure the bug is valid.
The tester files a defect in the bug tracking software.
A development manager decides that the bug is severe enough for you to work on and the bug is assigned to you.
You try to recreate the defect, but it seems to work on your machine.
Tester reproduces the bug and puts more detailed steps in the bug report.
You finally are able to reproduce the bug and you fix it.
You update the bug report with the fix.
The tester goes back and checks that the bug is actually fixed and marks the defect as resolved.
So, perhaps you should take that extra 10 minutes to test your own code before checking it in.
You won’t catch everything, but if you can even catch 10% of the bugs that would otherwise make it to QA, you’ll be saving quite a bit of time, don’t you think?
