A few months ago, I came across this article about 'Enemy Unit Testing'. It sounds like a strange term, but when you think about it, the concept makes sense.
If you write good Unit tests, the confirm that certain functionality works exactly as expected. If you have better Unit Testing, you confirm that certain functionality fails where it is supposed to fail. Therefore, you check the inside, outside, and on the boundary conditions, all special cases, and a reasonable subset of any other set of conditions that are considered likely within the expected usage. Generally, you only write these tests on your own code, but stop and ask yourself “Why?”
Developers who create other libraries and API's are just as fallible as you are. Sure, there is potentially some form of Quality Assurance involved in their release, but we all know the incompleteness of any form of testing. In addition to this, some things are merely design decisions that have changed over time and not “bugs”. Anyone who's worked in the Windows API understands this one. In comes a partial solution: Enemy Unit Tests.
Enemy Unit Tests work on the simple premise that anything coming from outside your immediate project needs to be held to certain standards to be included in the project. You don't necessarily have to test all the standard functionality included in your language, but checking other organizations' libraries, is a good starting point. Add in the functionality that has bitten you before or was known to be broken/sensitive before and you should be in good shape. Now you'll be able to detect when things are broken, which areas are broken, and when they became broken. After all, you don't check in your own code unless it passes your own Unit Tests, right? Right?
This article came to mind because I lived through it first hand this week. On CaseySoftware's biggest project, we are using a standard Apache-Tomcat-MySql configuration. The previous version of code had been developed and deployed by hand in Tomcat 4.1.x and it was decided to update to Tomcat 5.5.x and deploy via war (a slightly glorified zip file). This seemed like a simple proposition, so we began. I wrote the Ant build file to handle all of the repeatable actions including building and deploying the war to a test environment. Seeing a glimpse of how the code was going to be used, we decided to incorporate some simple Struts functionality into limited areas of the system. The trouble happened when we went to deploy and portions didn't work.
Our Tomcat had changed (4.1.x -> 5.5.x); the deployment method had changed (jsp -> war); the Apache config was different between development, staging, and production; and the supporting libraries had changed (simple jsp -> Struts). In the words of the customer's Technical Lead: “Too many variables.” Our previous baseline was in no way related to what we had. As a result, it created a nightmare for any and all debugging attempts. We ended up having a late night and ripping out the Struts pages. It cost us an entire day, but it could have been significantly worse. Although this scenario would not have been prevented by Unit Testing, the concept remains the same.
Remember, according to Martin Fowler:
Software that isn't testable is not un-testable… it's Detestable.