Coverage testing is considered to be an essential part of development nowadays, but I don’t think many people reflect deeply enough about what it involves, and why they are doing it.
Coverage tools measure how much of the code is exercise by the tests that are run and, broadly, a higher number is better. But it doesn’t tell you how good the tests are. You can have problems such as high coverage with redundant tests, code being exercised without being tested properly, and unmaintainable test code. Sometimes coverage is used as a replacement for understanding unit testing.
The other important question is how high the coverage should be. 100% coverage is extremely hard to achieve, so much so that many people suggest it is a waste of time. Having said that, all other levels of coverage are somewhat arbitrary. Some places I’ve worked have aimed for an ‘appropriate level of coverage’, but never have time to review and enforce that.
Good coverage is difficult to add retrospectively; easily testable code needs to be written in a certain way. The Michael Feathers book Working Effectively with Legacy Code is still an excellent guide to salvaging untested code, despite being almost 12 years old. Actually implementing Feathers’ recommendation takes a care and diligence that few people bother with. It’s better to aim for excellent coverage from the start.
The good thing about having very little code in our new project is that we can get our coverage up to 100%. Adding jacoco is a simple matter of putting a single line into the build.gradle file:
apply plugin: 'jacoco'
The jacoco reports can then be produced locally with the command ./gradlew clean test jacocoTestReport
The test case covered the Greeter class’s greet method, but not the main method. Perhaps controversially, I’ve chosen to remove this method rather than add a test for it – this was originally used as a test harness and that behaviour is not needed when we have the unit tests. And less code means less to keep track of.
Something interesting happens when we add the jacoco reports to Jenkins. Initially I’ve added the Jacoco reports to the main job for the project. This doesn’t matter when the project is small, but will become a problem later on. We want to maintain a fast response to errors. The current run is taking about 30 seconds, which is on the outside edge of acceptability.
A problem comes with the results for the build. No coverage results are produced, and an error occurs on the Jenkins command line although not the build: java.io.IOException: Incompatible version 1006
The problem here is that the Jenkins gradle plugin doesn’t work with all versions of Jacoco. This appears to be a problem with gradle versions which can be solved by forcing version 0.75 of gradle. This is now working:
(The branch coverage is at zero because the code currently has no branches)
The problem with this adding this line to the build file is that it is a significant piece of technical debt. We have one tool’s version restricted due to compatability with one. This may cause problems if we introduce another tool requiring a specific version; and we have to keep track of when to remove the version. It doesn’t take long for the purity of a greenfield development to disappear.
The commit for the latest version is 2f7307d. Next it’s time to look at adding Spring Boot to the project.