The Problem with Emma…
Emma is a code coverage tool with great potential. I (sort of) like it and use it (well, until I got sick and tired of seeing huge numbers of “uncovered” sections that were, in fact, IMPOSSIBLE for me to cover). This comment from Tony makes a great point:
One of the ways that 100% code coverage really helps a project is you quickly pick out those classes that are not tested fully. The difference between 87% coverage and 88% coverage doesn’t show up as much of a difference.
But, the difference betweeen 99% and 100% shows up as a huge difference when you use tools like Clover.
This sounds an awful lot like unit testing. When 100% of your tests normally pass, you immediately recognize failures.
If your team habitually tolerates less than 100% test success, you may will fail to recognize new test failures. Test failures are lost in the noise. Over time, developers learn to expect failing tests, ignore the results, and stop running the test suite.
The best way to avoid test rot? Ensure your build FAILS if a unit test fails.
The Problem with Emma
Achieving 100% test success is easy if you start on day 1 and never let failures work their way into the code. Although this sounds similar to code coverage, it is not.
In fast, it is practically impossible to achieve 100% coverage with Emma. Some branches are impossible to reach, such as private constructors. The same is true for the default case in some switch statements, or some of the generated code in enums.
Thus, when you use Emma, you never get a “green bar”. Instead, you constantly have to wade through false “not covered” indicators to determine if you actually forgot to cover something important.
What Can Be Done?
I believe Emma is off to a great start, but it really needs to be smarter about data flow analysis. Private constructors come to mind first: they should not be reported as “uncovered”.
Secondly, I think it would be really nice we had a database of previous runs. It would be great if we could review the coverage report, and mark sections of code as “I know this is not covered, please stop warning me about this.”
If this tool were really smart, it would recognize when the code changes, and clear these markers for those sections.
re: confession. There’s two ways to feel dirty. One is to write dummy unit tests. But the other is to write the code in such a way that it bypasses Emma’s limitations. Has anyone done that, I wonder?
e.g. Has anyone written a class with a non-private constructor solely to maintain 100% code coverage? How did you cleanse your soul afterwards?
There was a note on the Emma mailing list about the possibility of Eclipse taking over the development.
I think the test analogy is apt. The biggest problems with Emma right now are
1) you can’t do a diff on two runs, so that you can ensure that your overall coverage doesn’t go down between code and test changes and, more importantly, that you still cover all of the code that you did before any changes
2) The metrics are too course-grained to be useful. I care less that my 100 line class has 80% coverage than that my 1000 line class has 85% coverage. It would be useful to rank classes by total lines not covered, and more importantly by contiguous lines not covered, which indicates you missed a branch or aren’t calling a method. Also, having a way to display in some sort of ranked “problem areas” format would be usefule, so you didn’t need to browse through the entire report.
I agree, annotations would be a great thing. However, most of the useful annotations would require Java 7 support for putting them anywhere in code, e.g., mark this catch IOException as “don’t care” because even if I could figure out how to trigger one here, I’m just going to bail out anyway. The “you didn’t call the private constructor on this class that only has static methods” is annoying, though.
If you haven’t tried EclEmma http://www.eclemma.org/ check it out. This is my favorite way to use Emma (or any coverage tool) while I’m writing unit tests and code to ensure the tests are covering what I expect.
I’ve had the similar problem with private constructors in Emma but then I’ve never cared about 100% coverage, so didn’t really bother me much. Personally, I think you get a lot more mileage out of using something like FindBugs than squeezing out that last 15-20% of coverage.
Clover does a bunch of the reporting you mention in terms of finding coverage holes and such. Clover 2 is really slick if you haven’t tried it out.
@Phil I wouldn’t want actual annotations in the code. The “markers” I’m speculating about would be in some external database or data file.
@Alex I’m not concerned about 100% coverage, either. I’m concerned about all of the false “NOT COVERED” noise that bury all of the important data.
Here is an extreme way that would enforce 100% coverage, a little evil perhaps, but IMO interesting enough to try on a toy project: Imagine a tool that (say, every night) deletes all untested pieces of code from your current HEAD or trunk. With untested pieces of code, I mean methods that are not called, branches that were never executed, etc. If the tool screwed up, you could easily roll back to a previous version from CVS/SVN. Would you be able to use such a tool to your advantage, or would it just be a nuisance?
@Boris that tool is called Guantanamo. Good luck with that. Seems like with Emma, at least, a tool like that would delete valid, but allegedly untested, code, such as private constructors.
@Eric – thanks for the pointer!
Atlassian’s continuous integration server supports various metrics-over-time, not sure if test coverage is one of them though.