Archive for November, 2006

Unit Testing tools

November 9, 2006

Most of developers know how great and useful is unit testing. What is even greater is the bunch of libraries you can find out there for almost any language around (C, C++, Objective-C, Java, Pyhton, Perl…).

All these tools are great (I have used some of the ones I have mentioned), but what happens when you start working in a big and commercial project and your bosses have decided to buy a fantastic tool (I’m not going to mention the one I will talk about) which costs a lot of money and they have never try it? What happens is that you, the developer, start working in this fantastic tool and start feeling sort of depressing.

We started our project a year ago and by now, I have had to send so many e-mails to the unit testing software company that I can’t remember. The tool looked promising at first: unit testing, static analysis, user interfaces to speed up the process, cross platform (we use a cross compiler) and many other things. I started to use the GUI without much success, I don’t even remember how it looked like. After a couple of days we decided (we had it already in mind) to create the tests by hand (with help of some elisp code). After a month or so, we had our first problem: the parser for static analysis did not support some of the C99 syntax, first e-mail and first example to show the problem. It took like two months or more to fix the problem, this means that we had to code without using some of the syntax we wanted to use, and of course we had to remember it each time. After some months of tranquility, second problem: a header file provided by the unit testing software had some errors, so second e-mail with a fix suggestion. This time (or even before) I started thinking: “why have I to use this tool? I’m only using the basic macros all the free libraries already give me and no one in the company is looking at the metrics generated by the software”. Nevertheless, I kept on using the tool. A month ago (and after 6000 € paid for support), third problem: I got linking errors using the static analysis again. I prepared another test and sent it to the company again. They started by sending me an update that did not fix the C99 issue I mentioned above, and of course didn’t fix the current one. Then, I recognize it, I made a mistake and I saw a new issue when there really was no one, despite that the current problem was not fixed. This morning, I received a new version of the executables which were supposed to fix the problem. Again, no luck, no problem resolved. May be I made a mistake again, but no one has complained by now. Is any one trying the fix before they send it? Am I too exigent?

My conclusion is: whenever you can, avoid the tools you don’t know and use the ones you do. Commercial tools may not worth the money.

A co-worker told me that one can write its own code to perform unit testing, and his is totally right. If the problem is that the free libraries are not certified or whatever, write your own code and it will be certified as it will be the rest of the code.

Happy testing!