You know those tests you like to run? You know what I mean, those test cases that always give you a green bar and never fail?
If you are having a bad day, and you feel like nothing is going right, you can just “right-click” run and you get a nice 100% passing green bar. It is easy to refactor the code those tests are testing, because you know in your heart that no matter how much you change the code, those tests will always pass. What a great feeling. Doesn’t that make you feel like the king of the world?
I like to call those tests Ego Tests, because they are the equivalent of code flattery. An Ego Test always validates you. It tells you your code is pretty when it really has a fat butt. The Ego Test says “you can do nothing wrong.” The Ego Test says that “everything is going to be ok.” That is until you roll the code out to production and everything breaks. Then when you look for the Ego Test, you realize his message of “GREEN IS GOOD” is still blaring at full volume in an endless repeat cycle, like a mantra, or one of those self-hypnosis, “you can quit smoking” tapes. He is not really your friend; he is just a recording stuck in an endless loop. He won’t help you fix your bugs, he won’t even tell you they exist.
He must be stopped!
Identifying the Ego Test
The Ego Test can be any kind of test from unit test to automated functional test and more.
The Ego Test is identifiable by one or more of the following characteristics:
- Does not contain asserts.
- Contains asserts which only assert on conditions which are always true such as: assertTrue(isJohnSonmezCool?true:true);
- Asserts on things which are not related to what you are testing.
- Catches exceptions and eats them so the test will not error out.
- Tests things through a mock, but then does not assert the mocking expectations are satisfied.
- Never fails, no matter what you do to the code.
You would think from this list it would be pretty easy to stop and catch an Ego Test. It is not always so easy. Many Ego Tests appear to be testing the right thing, or seem like they could fail even though they can’t.
One quick way to figure out if you have an Ego Test is to change the input to the test and see if it fails. If you think you are looking for the word “Blue” in a drop down, and you change the HTML being tested to not contain the word “Blue” in the drop down, and if the test still passes, you have a problem.
Avoiding the Ego Test
The Ego Test is fairly easy to avoid. The best way is to make sure your test will fail when you change the input. Most of the time you can do this by changing the source code temporarily to produce a result which should be a failure. Sometimes though, you don’t have the ability to change the source code or doing so would be more work than it is worth. In those cases, you can change the test instead, to test for a condition which should not be true. Be very careful here though, many people are tripped up trying to validate tests the second way because they simply invert an assert statement.
For example, if you have an assert that is[sourcecode language=”csharp”] assertTrue(dropDownContains("Blue"));
and you try to verify the test does not always pass by inverting the assert to produce[sourcecode language=”csharp”] assertTrue(!dropDownContains("Blue"));
Well, that is just plain dumb. But I have seen it before. Of course it will now fail.
What is valid is to change the assert to be something like:
When you know that “Green” should not be in the drop down. If the test still passes in that case, there is something wrong with the dropDownContains method, or some other assumption you are making about the contents of the data you are testing. Perhaps you are testing the wrong drop down control? Perhaps something else is wrong.
One more tip on avoiding the Ego Test: know what you are testing and test only one thing. If you are writing some code to increase your code coverage and the best name you can come with for the test is the name of the method itself followed by the word “test”, you have no idea what you are actually testing. 100% code coverage is pretty easy to get in unit tests when you aren’t actually testing anything.
By the way, the official name of this kind of testing is called Mutation Testing. I am not sure if it is completely necessary to do full mutation testing on your tests if you actually follow the advice I give to avoid the Ego Test in the first place, but there are actually some pretty cool mutation testing tools out there. One I know of for Java is called Jester. It will randomly change your source code and run all your unit tests. Jester then reports which tests still passed after changing your source. Looks like there is one for C# called Nester. If you know of a good updated one in either language, be sure to let me know.
Disclaimer: This post doesn’t apply to Chuck Norris. Chuck Norris’s tests look like ego tests in that they never fail, but they don’t fail because the code actually changes to conform to the perfect will of Chuck Norris as indicated in his tests, not because his tests are not testing the right thing.