Comments on: The Problem with Emma… http://stuffthathappens.com/blog/2007/11/26/the-problem-with-emma/ Technology and Geek Stuff by Eric Burke Mon, 26 Jul 2010 14:36:32 -0400 http://wordpress.org/?v=2.8.6 hourly 1 By: Barend http://stuffthathappens.com/blog/2007/11/26/the-problem-with-emma/comment-page-1/#comment-2262 Barend Sat, 01 Dec 2007 18:18:13 +0000 http://stuffthathappens.com/blog/2007/11/26/the-problem-with-emma/#comment-2262 Atlassian's continuous integration server supports various metrics-over-time, not sure if test coverage is one of them though. Atlassian’s continuous integration server supports various metrics-over-time, not sure if test coverage is one of them though.

]]> By: Boris Bokowski http://stuffthathappens.com/blog/2007/11/26/the-problem-with-emma/comment-page-1/#comment-2057 Boris Bokowski Tue, 27 Nov 2007 16:49:49 +0000 http://stuffthathappens.com/blog/2007/11/26/the-problem-with-emma/#comment-2057 @Eric - thanks for the pointer! @Eric – thanks for the pointer!

]]> By: Eric Burke http://stuffthathappens.com/blog/2007/11/26/the-problem-with-emma/comment-page-1/#comment-2045 Eric Burke Tue, 27 Nov 2007 14:57:09 +0000 http://stuffthathappens.com/blog/2007/11/26/the-problem-with-emma/#comment-2045 @Boris that tool is called <a href="http://docs.codehaus.org/display/ASH/Guantanamo" rel="nofollow">Guantanamo</a>. Good luck with that. Seems like with Emma, at least, a tool like that would delete valid, but allegedly untested, code, such as private constructors. @Boris that tool is called Guantanamo. Good luck with that. Seems like with Emma, at least, a tool like that would delete valid, but allegedly untested, code, such as private constructors.

]]> By: Boris Bokowski http://stuffthathappens.com/blog/2007/11/26/the-problem-with-emma/comment-page-1/#comment-2044 Boris Bokowski Tue, 27 Nov 2007 14:53:16 +0000 http://stuffthathappens.com/blog/2007/11/26/the-problem-with-emma/#comment-2044 Here is an extreme way that would enforce 100% coverage, a little evil perhaps, but IMO interesting enough to try on a toy project: Imagine a tool that (say, every night) deletes all untested pieces of code from your current HEAD or trunk. With untested pieces of code, I mean methods that are not called, branches that were never executed, etc. If the tool screwed up, you could easily roll back to a previous version from CVS/SVN. Would you be able to use such a tool to your advantage, or would it just be a nuisance? Here is an extreme way that would enforce 100% coverage, a little evil perhaps, but IMO interesting enough to try on a toy project: Imagine a tool that (say, every night) deletes all untested pieces of code from your current HEAD or trunk. With untested pieces of code, I mean methods that are not called, branches that were never executed, etc. If the tool screwed up, you could easily roll back to a previous version from CVS/SVN. Would you be able to use such a tool to your advantage, or would it just be a nuisance?

]]> By: Eric Burke http://stuffthathappens.com/blog/2007/11/26/the-problem-with-emma/comment-page-1/#comment-2038 Eric Burke Tue, 27 Nov 2007 13:18:34 +0000 http://stuffthathappens.com/blog/2007/11/26/the-problem-with-emma/#comment-2038 @Phil I wouldn't want actual annotations in the code. The "markers" I'm speculating about would be in some external database or data file. @Alex I'm not concerned about 100% coverage, either. I'm concerned about all of the false "NOT COVERED" noise that bury all of the important data. @Phil I wouldn’t want actual annotations in the code. The “markers” I’m speculating about would be in some external database or data file.

@Alex I’m not concerned about 100% coverage, either. I’m concerned about all of the false “NOT COVERED” noise that bury all of the important data.

]]> By: Alex Miler http://stuffthathappens.com/blog/2007/11/26/the-problem-with-emma/comment-page-1/#comment-2029 Alex Miler Tue, 27 Nov 2007 06:10:47 +0000 http://stuffthathappens.com/blog/2007/11/26/the-problem-with-emma/#comment-2029 If you haven't tried EclEmma http://www.eclemma.org/ check it out. This is my favorite way to use Emma (or any coverage tool) while I'm writing unit tests and code to ensure the tests are covering what I expect. I've had the similar problem with private constructors in Emma but then I've never cared about 100% coverage, so didn't really bother me much. Personally, I think you get a lot more mileage out of using something like FindBugs than squeezing out that last 15-20% of coverage. Clover does a bunch of the reporting you mention in terms of finding coverage holes and such. Clover 2 is really slick if you haven't tried it out. If you haven’t tried EclEmma http://www.eclemma.org/ check it out. This is my favorite way to use Emma (or any coverage tool) while I’m writing unit tests and code to ensure the tests are covering what I expect.

I’ve had the similar problem with private constructors in Emma but then I’ve never cared about 100% coverage, so didn’t really bother me much. Personally, I think you get a lot more mileage out of using something like FindBugs than squeezing out that last 15-20% of coverage.

Clover does a bunch of the reporting you mention in terms of finding coverage holes and such. Clover 2 is really slick if you haven’t tried it out.

]]> By: Phil Varner http://stuffthathappens.com/blog/2007/11/26/the-problem-with-emma/comment-page-1/#comment-2026 Phil Varner Tue, 27 Nov 2007 05:23:48 +0000 http://stuffthathappens.com/blog/2007/11/26/the-problem-with-emma/#comment-2026 There was a note on the Emma mailing list about the possibility of Eclipse taking over the development. I think the test analogy is apt. The biggest problems with Emma right now are 1) you can't do a diff on two runs, so that you can ensure that your overall coverage doesn't go down between code and test changes and, more importantly, that you still cover all of the code that you did before any changes 2) The metrics are too course-grained to be useful. I care less that my 100 line class has 80% coverage than that my 1000 line class has 85% coverage. It would be useful to rank classes by total lines not covered, and more importantly by contiguous lines not covered, which indicates you missed a branch or aren't calling a method. Also, having a way to display in some sort of ranked "problem areas" format would be usefule, so you didn't need to browse through the entire report. I agree, annotations would be a great thing. However, most of the useful annotations would require Java 7 support for putting them anywhere in code, e.g., mark this catch IOException as "don't care" because even if I could figure out how to trigger one here, I'm just going to bail out anyway. The "you didn't call the private constructor on this class that only has static methods" is annoying, though. There was a note on the Emma mailing list about the possibility of Eclipse taking over the development.

I think the test analogy is apt. The biggest problems with Emma right now are
1) you can’t do a diff on two runs, so that you can ensure that your overall coverage doesn’t go down between code and test changes and, more importantly, that you still cover all of the code that you did before any changes
2) The metrics are too course-grained to be useful. I care less that my 100 line class has 80% coverage than that my 1000 line class has 85% coverage. It would be useful to rank classes by total lines not covered, and more importantly by contiguous lines not covered, which indicates you missed a branch or aren’t calling a method. Also, having a way to display in some sort of ranked “problem areas” format would be usefule, so you didn’t need to browse through the entire report.

I agree, annotations would be a great thing. However, most of the useful annotations would require Java 7 support for putting them anywhere in code, e.g., mark this catch IOException as “don’t care” because even if I could figure out how to trigger one here, I’m just going to bail out anyway. The “you didn’t call the private constructor on this class that only has static methods” is annoying, though.

]]> By: Michael Easter http://stuffthathappens.com/blog/2007/11/26/the-problem-with-emma/comment-page-1/#comment-2024 Michael Easter Tue, 27 Nov 2007 04:31:30 +0000 http://stuffthathappens.com/blog/2007/11/26/the-problem-with-emma/#comment-2024 re: confession. There's two ways to feel dirty. One is to write dummy unit tests. But the other is to write the code in such a way that it bypasses Emma's limitations. Has anyone done that, I wonder? e.g. Has anyone written a class with a non-private constructor solely to maintain 100% code coverage? How did you cleanse your soul afterwards? re: confession. There’s two ways to feel dirty. One is to write dummy unit tests. But the other is to write the code in such a way that it bypasses Emma’s limitations. Has anyone done that, I wonder?

e.g. Has anyone written a class with a non-private constructor solely to maintain 100% code coverage? How did you cleanse your soul afterwards?

]]>