Maybe Tests Don’t Belong in Your Project
Maybe unit tests don’t belong in our projects. Perhaps by moving tests to a standalone project, separated from application code, we can improve backwards compatibility and provide better regression testing. Read on….
First, Some Background
Some people put tests right next to source code:
[proj]
+--src
+--com/acme/killerapp/
Customer.java
CustomerTest.java
etc...
I do not like this because it makes it harder to separate test code from application code. The “*Test.java” sources are easy to filter out, but sometimes you have helper classes that are used by tests, but are not tests themselves. These helper classes may not follow an easily filtered naming convention.
Separate Test Directories
I prefer to structure my projects like this:
[proj]
+--src
+--main
| +--com/acme/killerapp/
| Customer.java
+--unittests
| +--com/acme/killerapp/
| CustomerTest.java
| TestUtils.java
| etc...
+--integrationtests (optional)
+--com/acme/killerapp/
With any IDE, multiple source trees are virtually transparent to developers. Isolated source directories make it really easy to create Ant buildfiles that keep test code and application code separate, and even let you define a boundary between unit tests and integration tests.
Is More Separation Valuable?
Let’s start with a question:
Is version 2.2 of your API compatible with version 2.1?
Do your tests help answer this question? I bet not. You see, most of us store our unit tests in the same project as our application code. We may break out tests into their own source tree, but in the end, it’s all lumped together in the same project.
As we add features and fix bugs for version 2.2, we hope to continually maintain the test suite in unison. We know that version 2.2 compiles against itself, but what about customers who have application code using version 2.1, and now wish to upgrade to version 2.2 of your product?
A New Proposal
What if we put application code into one project and all tests into a completely different project? To compile tests, you need to first build the application code. The tests should then compile and run against a JAR file from the core application.
Because of this extreme separation, we can now easily compile and run the version 2.2 unit tests against version 2.1 of the application. Or, we can compile and run version 2.1 unit tests against version 2.2 of the application. Mixing application code and test code into the same project makes this kind of cross-version testing harder.
How Might This Help?
I suppose I originally thought of this because of a recent experience upgrading an open source library. This upgrade was supposedly a “minor bug fix” release, but my application code would not compile against the new release.
If they had attempted compiling their old tests against their new software, they would have discovered this problem and could have taken one of these actions:
- Inform customers of the issue
- Back out the incompatible change
- Choose to call this a “feature release” instead of a “bug fix release”
- …
But I’ll bet they didn’t even realize they were breaking code.
What do you think?
I like it!
You can do that easily with Maven 2. It uses different directory trees for app and test code, and it supports the automatic creation of a appname-tests.jar artifact that can be stored in the repository. I’m not sure if it would be all that easy to really perform the unit-tests automatically with earlier versions, though.
I see the benefits to the approach, but what was your take on how tests get versioned? I assume that you’d still version your tests as you version your source base, but have the ability to run the source for 2.2 against the test sources of 2.2, 2.1, 2.0, etc to see if compatibility still exists.
JR, yes, that’s how I envisioned it. Each version of the app would have a corresponding test release. Although I suppose if you release a “bug fix only” release of your app, I wonder if you really need a whole new version of the tests? Maybe you only need new test versions when you break APIs in the core app.
I actually use this method in my latest self-designated project, a web based photo album (original idea, eh?). In some ways it seems a lot better for deployment; for example, all of your jars needed for testing are in the test project (jmock, junit, spring-mock, etc.) so there’s less crud to filter out when making the war file; similar to the problem of filtering out the test classes as you pointed out. And I have a separate log4j.xml set up for my test project. But the downside is if you’re working in Eclipse, it’s not too savvy about this. It does have this thing where one project can depend on the other, so my test project depends on the main project, but the Spring xml config files aren’t found correctly. I forget the details but I have my applicationContext.xml and the two xml files it imports in my WEB-INF directory in the test project. Other than that I think it’s really great having them entirely separated.