A word of warning for people using code coverage, there is a difference between code coverage and testing coverage. I know that this shouldn’t be news to some people, but it keeps cropping up again and again. Currently the Wikipedia article on Code Coverage claims that it is with this gooey nugget of falsehood.
Code coverage is a measure used in software testing. It describes the degree to which the source code of a program has been tested. It is a form of testing that inspects the code directly and is therefore a form of white box testing.
In Wikipedia’s defence it does say that the article does not cite any references or sources. This is good a thing because nobody would want to be associated with that statement.
Now I like code coverage. I like statistics, so why wouldn’t I like code coverage. It’s a cool statistic that lets you know some information about the code you are writing and the unit tests that accompany your code. Code coverage can hint to you where additional unit tests should be written. This would be where one type of code coverage is low or even zero.
Code coverage can also give you a false sense of security. This is what worries me. I see people thinking that code coverage is test coverage. Test coverage is the percentage of tests that cover requirements. When you get into permutations and combinations test coverage itself can be a very unrealistic measure of how tested a system truly is.
This is where experienced testers come in to determine what percentage of test cases are on the critical path and what percentage are not. The same applies for unit tests and code coverage. By itself code coverage is a statistic that can easily be misread or abused, with an experienced developer the relevancy of the statistic is brought out and useful unit tests are written.
I will provide an example of where code coverage reports that the world is rosy, but in reality this code wouldn’t pass test. My example is obviously contrived to illustrate my point. In real life a moderately skilled developer would have built a unit test to sufficiently test this code.
Our Code (C++)
float reciprocal (const float ex)
{
return (1.0f / ex) ;
}
Our Unit Test
Assert.AreEqual (1.0f, reciprocal (1.0f)) ;
Our Code Coverage
- Statement Coverage = 100%
- Branch Coverage = 100%
Taking the statistics at face value it would be time to stop working on this code because it cannot be more perfect than this. For those that don’t know, and non-coders may be reading this, there needs to be at least one more unit test where the value of X (ex) is zero. This will cause a divide by zero exception. For completeness one would add the following tests for X (0.0, 1.0, -1.0, a positive and negative value that is less than +/-1.0f and a positive and negative value that is greater than +/-1.0f).
The important thing to take home from all of this is that code coverage, like many other tools, figures, statistics and adages, are designed to impart information. Making an educated decision based upon such information is where you come in.
Note: I have put a comment on the Wikipedia article stating that I suggesting changing the article to something like: Code Coverage reports on the percentage of lines of source code that are executed by one or more test cases. Ideally I would like them to cover the misuses of the statistic a little more, we shall see what happens.
2nd Note: This is a post about code coverage, not the standard of Wikipedia.
|
Ryan Boucher is a Software Inquisitor and is passionate about it. You can find a whole raft of articles and anecdotes about software testing and other topics he gets excited about. |
| Tags |
