Lessons learned on the road to 100% code coverage
What is 100% code coverage
Code coverage is a metric that tells you how much of the source code for an application is run when the unit tests for the application are run. If 100% of the lines of source code are run when the unit tests are run, then you have 100% code coverage.
100% code coverage is typically a result of using the Test Driven Development methodology.
How important is 100% code coverage?
That depends on your perspective. The purpose of writing unit tests is to ensure that an application works as expected. In theory, the higher the percentage of code coverage, the higher the functional quality of the code. In practice, while 100% code coverage does not guarantee that an application has no bugs, your confidence in the quality of your code increases significantly the higher your code coverage is.
This topic has generated a fair amount of debate in the industry. Test Infected and Don't be fooled by coverage report.
An initiative worth looking at is the open quality initiative at Agitar. Their open quality dashboard shows code coverage for a number of their products as well as some of the open source projects like Apache Jakarta Commons. I encourage teams to download a tool like cobertura and compare with their code coverage with that shown on the open quality initiative site.
One of the biggest advantages to high levels of code coverage is the agility you gain. Agility means you have a thorough suite of tests to prove your code works as expected. When a new requirement comes in that means you have to change some core classes within your application, suddenly you can do such changes with courage and not fear the consequences because your tests tell you the exact consequences of the changes you make. You can make new releases more often and with a high level of confidence in the quality of the code in each release.
I found the ability to release often and with confidence quite a mind shift for me. As I worked through numerous iterations of a log analysis application called toaster, and released them for production use, I kept thinking I hadn't tested it enough and would cringe about not going through a more formal test cycle. Occasionally I would manually test the production results just to confirm. And occasionally I would encounter a bug that wasn't picked up in the unit tests even though I had 100% coverage. But, this was far far far less than the number of bugs I was used to with more traditional / heavyweight methodologies.
Background
My personal goal is to get as close to 100% code coverage as possible. I spent a number of years as a development manager when I first got involved in the agile development methodology. One of my frustrations in this role was how difficult it was to get anywhere near 100% code coverage. This was a problem in several areas.
As a result, in 2005 I made it a personal goal of mine to see how close to 100% code coverage I could get to. I've applied this learning exercise to two different projects, the first was a MVC (Model-View-Controller) web application framework, the second was a log analysis tool called toaster. With the web application framework I was able to achieve 99% code coverage, with toaster I was able to achieve and sustain 100% code coverage. Both of these were written targetting 100% coverage from the first line of code and were written test first: the unit test was almost always written before any of the production code was written.
I learnt some interesting lessons along the way...Make your goal 100% coverage, nothing less
95% code coverage is a great target but anything less than 100% is a slippery slope. If you test 95%, how do you know if the most complicated 5% of your code is the 5% that is not tested. I personally believe the effort to get to 100% coverage is well worth the effort.
The other problem with moving away from 100% is that once you allow exceptions to the rule, it becomes an easy way out. If your application has less than 5 exceptions to the 100% rule, it is pretty easy to track. The code coverage tools provide a progress bar which is green for 100% and red for anything less. With 5 or less classes it is easy to remember the exception and the reason behind it. But when you allow the number of exceptions to creep higher, it becomes much harder to track. One code inspection tool I've worked with provides for "waivers" to track exceptions to coding standards. If such a system becomes available for code coverage tools, it may be easier to move away from the 100% goal. But still inadvisable in my opinion.
At times, my enthusiasm has faded because it is hard work. But this was brought into perspective when discussing how difficult I found it with another project manager who stated words to the effect that if you think it is hard work being pedantic about fully unit testing all code, he is in the situation where none of the code for his project is unit tested and he would gladly swap.
The task is much much harder trying to apply unit testing rigor to a legacy code base. Difficult for me to say if that is worth the effort, I'd encourage teams to gauge that based on what their challenges are. 100% test code coverage provides advantages not only in reducing the bug count but also in making it easier to make significant architectural refactorings to the existing code base, gives you much more agility!
I spent some time discussing and debating the merits of 100% coverage with several like minded development managers. The conclusion was that 100% coverage was the answer because moving away from 100% to anything less is a slippery slope and once you start down that slope it is very difficult to recover.
This means that even simple getters and setters are tested. This is often a point of contention with developers but my perspective is that we often design a class to use getters and setters because the power of object oriented languages means that we can then change the implementation as needed. So, my argument to those who don't want to test simple getters and setters is that the implementation could easily change and even simple tests make the ability to embrace change much more achievable.
Test first most important principle to get to 100
The biggest single difference for me was switching to writing my unit tests before writing my "real" code. This was a significant retraining for me. The best resource in learning how to change was Kent Beck's book, Test Driven Development. Try it Kent's way and you'll see the difference.
Traditionally we have built our logic in an iterative approach, writing, compiling, debugging, back to writing, then, if we had time, the unit tests would be written. Many developers still work this way. Writing test first means the debugging is thought about and done up front and never thrown away. All the intellectual property put into debugging code is saved. If you aren't test driven, you are saying to the world that you are happy to throw away a lot of your intellectual property cause you are just too lazy to save it.
Before I switched to test first development, I'd write the code that I thought solved the problem and then proved that it did what I thought it should, whether that mean writing details to a log file, stepping through a debugger, or visually watching the results of running the code. All valid approaches to debugging and testing but when it came time to revisit the code, or fix a bug, all that effort had to be duplicated again. With test first you don't make that an option.
Mock objects second most important principle
Using mock objects was most valuable in taking my code from somewhere between 90 to 95% coverage right up to 100%. Before I started using mock objects, I'd invariably have a small amount of code that just wasn't testable. In the case of developing web applications using servlets, I couldn't get anywhere near even 90% without mock objects.
Moving to mock objects also helped me to get around a mind set that is clearly a result of my years working for a database company. Typically, when testing a database application, you need to set up a database (often not a trivial task), load seed data, load test data and finally run the unit tests. I found the ant sql task very useful for this. However....
When I started to write some code to validate ldap objects, I started thinking the same way, I've got to set up an ldap server, initialize it, load it with seed data, run tests, it all became a bit daunting. Was discussing this with one of the XP proponents in Brisbane, Greg Davis, and he told me I shouldn't do any of that, he said just mock the ldap object. (Check out Greg's boost project, One of his non-negotiable rules is 100% test coverage!).
So.... I heeded his advice and was amazed at how much better I was able to write and test the code. When I get more time, I'll add details about this code so you can review. Makes me want to do the same with database mocking. Speaking of database (jdbc) mocking, check out mockrunner, I like the way these tests are coded...
public void testTransferOk() throws SQLException
{
prepareResultSet();
Bank bank = new Bank();
bank.connect();
bank.transfer(1, 2, 5000);
bank.disconnect();
verifySQLStatementExecuted("select balance");
verifySQLStatementExecuted("update account");
verifySQLStatementParameter("update account", 0, 1, new Integer(-5000));
verifySQLStatementParameter("update account", 0, 2, new Integer(1));
verifySQLStatementParameter("update account", 1, 1, new Integer(5000));
verifySQLStatementParameter("update account", 1, 2, new Integer(2));
verifyCommitted();
verifyNotRolledBack();
verifyAllResultSetsClosed();
verifyAllStatementsClosed();
verifyConnectionClosed();
}
}
Mock objects also allow easier testing of your object in isolation. This concept is covered very well in the book Junit in Action, sample chapter 7.
The techniques covered in the Junit in Action book allow you to test scenarios like having an exception returned in the middle of reading a file or fetching from a database query. Also check out using a mock as a trojan horse. This technique allows you to ensure, for example, that all usages of a database prepared statement close the prepared statement once and only once.
Here is an example of using a mock to emulate behavior in the java virtual machine that would be impossible to test otherwise.
It is also worth taking a look at some of the dynamic mock object frameworks. Jmock is one such framework I've looked at. Jmock is useful for testing APIs that you have no control over. One down side, in my opinion, is that you assert, and very specifically at times, the method calls that will happen in the API you are testing. This goes against the idea of encapsulation within object oriented development in that you should not care how an object implements a particular piece of functionality. With Jmock sometimes you have to care about this in order to test fully. Check out this jmock quick reference.
Testing Singletons
A "singleton" is a design pattern that specifies a particular object will only have a single instantiation. As a result, singletons can be challenging to test to 100% coverage. This is because the code that creates the object ensures one, and only one, instantiation of the object occurs and instantiating with different constructors or parameters is often impossible.
The spring framework has a lot to offer in this regard. The "core" component within Spring looks after the creation of what I call "managed" objects. Instead of coding an object to be a singleton, you code it like any other object (any number of instantiations allowed) which makes it much easier to test. Then you ask the "core" component within Spring to create the object for you. In a configuration file (applicationContext.xml) you specify properties for the object including whether it is instantiated one or multiple times. The creation of the object is "managed" by the core component within Spring.
The only change to your code is instead of creating object as a new object, you make a call to spring to get you an instantiation of a named bean.
While on the subject of Spring, as a framework it has much to offer in the area of limiting object dependencies through a technique called inversion of control (IOC). It is common in java / object oriented applications to have a "Context" object that gets passed around to most objects in the system. This "Context" object holds references to objects that are used / needed by most objects in the application. Database connections are commonly passed around using such a "Context" object.
The problem with passing around such a "Context" object is that these objects can get complicated to create and affect testability as a result. Additionally, it is not uncommon for an object to require references to a number of different "Context" objects as required by various APIs. Can get quite complicated and confusing quickly.
With Springs inversion of control (IOC), these kind of dependencies get "injected" into each object based on the specification for the object in the applicationContext.xml configuration file. Objects then, instead of passing around a "Context" object, have properties for the dependencies they would normally get from the "Context" object. For example, a setConnection method would exist if the object needed a database connection. It is then the responsibility of the IOC container (Spring) to create the object and supply the required properties like the database connection.
Testing also becomes easier with IOC because you can create the object normally for testing (using "new" constructor rather than getting a managed object via the IOC container) and set the properties as required. This also makes it easier to subsitute test / mock objects to test and model specific behaviours.
If you like these concepts but cannot use Spring, consider creating your own object manager / register that can be responsible for instantiating managed objects. By its nature, such an object manager is likely to be a singleton but will be the only hard coded singleton in your application. (Except of course for the Oracle Database which would have to be the biggest singleton around!).
One drawback with the Spring approach is that logic which used to be in code and thereby theoretically unit testable, now resides in a configuration file (applicationContext.xml) and harder to test. While integration / system testing should determine whether the configuration file is correct, I'd like to be more confident of the configuration from a unit test perspective. I haven't taken the time to determine how to go about this and welcome any feedback.
Logging API possibly less important
Historically, I've written code with a significant amount of logging. I remember chosing logging frameworks carefully based on their features. This gives you an idea of how important I considered logging to be. With test driven / test first development, suddenly I didn't need to write log details as copiously as I used to. In fact, for the MVC (model view controller) framework I wrote no logging. This seemed quite weird when I thought about it but was very natural as I went through the development process.
While I don't suggest that writing a production enterprise type of application without logging is appropriate, being freed from the need to log thanks to test first development gave me a much different perspective on logging. A favourite book of mine, Agile Java, by Jeff Langr, talks about whether the creation of log messages should be unit tested...
'Sometimes pain is a good thing. The pain here will make you think about each and every logging message going into the system. "Do I really need to log something here? What will this buy me? Does it fit into our team's logging strategy?" If you take the care to log only with appropriate answers to these questions, you will avoid the serious problem of overlogging.'
Hopefully this gives you some food for thought on logging.
Sometimes count how many lines of test code you have
I attended an excellent course several years back on Agile development by Martin Fowler. He provided an interesting statistic. He stated that the "best practice" projects he has seen where the bug count is extremely low by industry standards (no more than a few bugs every few months), the percentage of test code is between 40 to 50 percent of the total code base. A number of times I've counted the total number of lines (either in a class or a project) and the number of lines of test code is pretty darn close to 50%. See what your count is!
Refactor
Another topic that generates a lot of debate is when to refactor. Is this a business decision? Is it a technical decision?
Refactoring is a much larger topic and is deliberately not given much coverage in this page since we are talking about code coverage lessons. Having said that, please don't treat refactoring lightly in your project.
Consider introducing coding standards that limit the size of a method to a small number of lines. How many lines should you limit it to? I'd be concerned about code that regularly had methods with more than 25 lines. This is very subjective.... I also know of projects where the maximum number of lines allowed per method is 7. That sounds a bit extreme to me. In practice I've found that generally there is room to refactor and simplify the code if a method is longer than 15 lines.
Refactor your unit test code
Early in my test driven efforts, I treated my tests as just another artifact of development. I didn't take much care how they were written as long as they tested as much of the application as possible. This led to some pretty sloppy test code and lots of cut and paste inheritance.
One of the main advantages a full suite of tests cases is supposed to give you is the ability to refactor your code because you have a wealth of tests to prove that your refactoring didn't break existing functionality. This provides the agility projects yearn for. However, when the unit tests are a mess and treated as an artifact, rather than the most important client of the code, agility can suffer.
Now, I treat the unit test code with the same level of importance as the production code. Another test I'm carrying out is commenting my production code far less than I used to and treating my unit tests as the documentation for my production code. So far i'm sold on this approach but it hasn't passed the test of time. I'll keep you posted on this.
Write test utilities to make it easier to unit test your code, this can lead to a significant increase in TDD productivity.
Design for testability
How you design your classes can have a huge impact on how testable they are. Moving to test first development is one of the best ways to force yourself to design for testability.
Don't be resistant to adding methods just to make it easier to test your class. Look under hood / bonnet of your car some time and notice the number of places you can inspect how your engine is doing. You have dipsticks for your oil and transmission. You have the ability to see how much brake and clutch fluid you have. You also have the ability to check the level of coolant in your radiator. And the ability to check how much water is in your windscreen washer.
Treat your classes with a similar level of inspectability. Some of this might break the principles of encapsulation but add comments to the javadoc for such methods stating use of this method for non-testing purposes is not supported. Also don't make such methods public. Maybe you'll want to create your own standard by using the prefix of "inspect" instead of the standard "get" for such methods.
Don't overlook simple changes
Some times simple changes to your classes can make it easier to test. One example was an object I created called a "FileSet". At the core of the fileset was a directory but the calling classes read the name of the directory from a configuration file. So the constructor looked like this...
public FileSet(String dirname)
{
this.directory = new File(dirname);
}
As the class changed and more functionality was added, it became harder to test until I changed the constructor as follows...
public FileSet(File dir)
{
this.directory = dir;
}
Now it was much easier to pass a Mock file to the FileSet constructor. In this case, changing the signature of the constructor didn't cause any problems but I could have just as easily overloaded the constructor if required.
HomeCopyright (c) 2006. Tony Obermeit. All rights reserved worldwide.