Unit Tests II

Our current project is approaching delivery. As you would expect when the word "release" is uttered, the bugs start wriggling to the surface. I just wasted some time trying to figure out a page that is supposed to display current and future data, but not data that is historical.

The issue was that an item with a date in the year 2009 was getting filtered out as historical. The isHistory method was written ages back and, naturally enough, was covered by unit tests. So how could it be wrong?

Well, firstly, the unit tests checked past and current dates, but never a date in the future. Worse, one of the tests was checking the wrong thing. Ouch. Now I'm in a quandry. If I fix this method, my code will work, but how many other places will be broken. Thankfully, Eclipse has a References search. The answer came back that only my code was using this method (maybe other had found the bug and avoided it?). I fixed the unit tests, the code defective method and verified that the missing data did appear.

Unit tests are good. Just don't believe they are always right. Murphy's law applies, as always.

[Comments turned off and comment spam removed: 5/Sep/2005]
Category: Software, Programming, Java, Unit Tests

Magic Numbers

Our code guidelines state clearly that "magic number" are not permitted. We run Checkstyle to pick up style related violations. So, despite this, what do I find when I'm poking around fixing something unrelated?


private static final int PARAM_INDEX_THREE = 3;
private static final int PARAM_INDEX_FOUR = 4;
private static final int PARAM_INDEX_FIVE = 5;
private static final int PARAM_INDEX_SIX = 6;
private static final int PARAM_INDEX_SEVEN = 7;
private static final int PARAM_INDEX_EIGHT = 8;
private static final int PARAM_INDEX_NINE = 9;
private static final int PARAM_INDEX_TEN = 10;

The values above were used to index the result of this:

String [] fields = record.split ("\\|");

Ignore the fact that a pipe separated string might not be the ideal way to send this data. That's another topic altogether. Ignore that fact that elements 0, 1 and 2 were indexed with hard coded numbers. I still don't fathom what was the point of going to this much effort and still not giving the constants meaningul names. And yes, the code used these to index the array and set the values to attributes of the class.

Freedom Languages

A thought provoking article over at Code Craft. Kevin compares languages that try to remove constructs that are deemed confusing or easy to misuse with languages that try to remove constructs that reduce programmer freedom. According to Kevin, C++, Java, C are examples of safety languages; his freedom languages include Python and Ruby.

For what it's worth, I find programming in a safety language (C++) as satisfying as writing in a freedom language (Ruby). In a sense, I feel both do their job well. Somehow, I can never get real satifaction from writing Java (OK, it pays the bills, but I mean that pleasure of writing something beautiful; something that has merit beyond its practical purpose). The C++ language has beauty in its austerity. Ruby has a different charm, but ultimately the attraction of both is their power to let me express the problem without hiding it in the messy housekeeping details. My first reaction when I discovered C++ (back in the old cfront days) was that it improved the signal to noise ratio (that's my time in HW slipping through the cracks). And there's my chief complaint about Java; I spend too much time (and too many lines of code) writing things that are better left unseen. In trying to make the laguage safe, it has made it more difficult to write and maintain.

If Java's greatest strength is that it is easy enough that anyy programmer can write it; that is also its greatest weakness. Today we have far too many people who can write Java (and are Java Certified Engineers) but don't know how to program.

Category: Software, Programming, Java, Ruby

Lucky Typo

Our project uses Cruise Control for continuous integration, and of late, the builds have been taking way to long to complete. The main culprit are the Struts Test Case tests. One single test was taking over 12 seconds to run, and we have many of them. Something was wrong, but no one could figure out what it was. It was the usual problem; we were so busy, there was no time to look at it. And, of course, the unreasonably long build times were increasing our work load, making it less likely that anyone had the time to find the roor cause.

A couple of days back, while struggling to release a build, I finally decided that something serious had to be done. I had once before noted that the Ant junittask now (that is from 1.6.2) has a parameter called forkmode that can be set to run all unit tests in a single JVM. We had briefly tried that before; noted that it didn't work with our build script and given up. The problem was that the reorted message wasn't very helpful; it reports ClassNotFoundError on a properties file.

That properties file turns out to contain the list of tests to run. It did exist, though I had to create a specially hacked Ant build to discover that (the standard Ant build deletes the temporary file to fast to exmine it). Maybe the file was corrupt? I re-constructed the command line that Ant issues and discovered, in dismay, that the line length exceed the Windows 2000 limit 2048 characters. Our classpath was grotesquely long; with junk and duplicates. I managed to prune it down, only to find out that some items were actually missing. Strangely, the pre-forkmode builds did work, but how, I'm not sure.

And finally, a much hacked classpath contained everything needed, but nothing more. The tests ran blisteringly fast; about 50ms per test -- admittedly in a totally different environment from our actual build. I just needed to plug it back into the build scripts. My classpath was hard-coded; the build script has a nice set of property definitions that define library folders and the like. It took a while, but I was confident that this was going to work. It was worth the effort

I ran the tests and stared in dismay at the timings; better, for sure, but still about 5 secondsper test. What happened. I switched back to my hardcoded classpath; very fast, even in the build environment. Now I am really confused. Ant resolves the definitions up front, so there couldn't possibly be a per-test overhead. Yet that is what appeared to be happening. As I went home, my mind was struggling to make sense of the situation.

That evening, I tried to rerun the situation on my laptop. While setting up the classpath, I noticed that the log4j jar file was in the classpath, but its path was mis-typed. We use lag4j, but indirectly via the apache commons logging library. The commons logging library cleverly uses log4j, if present, but falls back to java logging if it doesn't find log4j. Not strictly needed, in other words. And then I test with and without log4j and reach the moment of truth. By turing off logging (tidily now, using the log4j.properties file) my time per test went down by a factor of 10. Not bad. Tweaking the build to use forkmode could reduce it by almost the same amount again. Not quite down the my standalone and highly artificial environment, but not bad.

A final analsis brought all the pieces together. StrutsTestCase mocks up a servlet container in which it runs our Action classes. It is faithful enough to the real servlet container that it uses the struts-config.xml file to map a path into an Action. Each test re-initialises the mock container; and in doing so re-reads the .xml file. Our log4j.properties file is kept in the same directory (WEB-INF) as the struts-config.xml file, so was visible to the the test setup. Apache have been liberal with debug logging in the libraries -- specifically the commons-digester library, which is used to parse the struts-config.xml file. Our default logging level is still DEBUG (we're still a few months away from going live), so a lot of output. All this chewed up an enormous amount of CPU cycles and disk space. Odd that we didn't notice it, but then our build process generates a monsterous amount of output, so it just got lost the volume.

We may not use the forkmode tweak for the build machine as there are all sorts of complications; it requires Ant 1.6.2 or newer, which again requires Xalan 2, which has compatibility problems with the jdk 1.4.1 we are stuck with. I won't get off track about java wierdness again. Right now everyone on the project is ecstatic that builds are fast again. Sometimes a typo can be the source of good fortune.
Category: Software, Programming, Java

Java is wierd

It all began when I wrote a simple class with overloaded methods:

   public abstract Object apply (Object obj);

   public Object apply (Collection collection)
   {
      for (Iterator i = collection.iterator (); i.hasNext(); )
      {
         apply (i.next());
      }
      return this;
   }

Note that the first version is abstract (as was the class, of course). The second method, which takes a Collection and applies the first method on each item. The intent is that you extend the class and provide a implemetation of the abstract method. Invoking the apply (Collection) method will invoke your apply (Object) over the collection. Seemed simple enough to me. My unit tests green barred, so I was ready to plug the code into the project.

I get a compilation error. Huh? I run my unit tests again. Green bar. Now I'm confused. I isolate the offending code into a sandbox environment. and repeated the above. Unit tests green bar; code fails to compile. Now wait a minute. To run a unit test, the code has to compile, correct? Stop and think.

So, here's the situation. When I wrote the code in Eclipse, it gave no errors. Ecplise compiles in the background; so my code does compile. When I build the project I run an Ant build script (started from Eclipse, usually). That build script reports the following:

[javac] C:\John\Closures\src\functor\ClosureTest.java:138: reference to apply is ambiguous, both method apply java.util.Collection) in functor.Closure and method apply(java.lang.Object) in match
[javac] new BufferedClosure ()

So Eclipse thinks my code is good; javac disagrees. That's just dandy, right?

As the practical person I am, I did the quick fix; rename apply (Collection) to applyAll (Collection). Project build works and we are back on track. I have no peace, however, until I understand what is going on.

The answer turns up after some googling. It's a bug; Bug Id 4761586. Furthermore, it's fixed in java 1.5, and interstingly enough, the fix was quietly back ported to 1.4.2. There is a discussion of the issues in the JDJ article, Overloaded Method Matching. (Published Oct 2003 -- how did I miss it?).

The current project is locked down to 1.4.1 — where it is definitely not fixed. Another thing is that Bug Id 4761586 (and it's resolution) refers to "the class in which an overloaded method is declared (the 'declaring class')". Now my two methods were declared in the same class; even if one was first defined in the extending class. As far as I can figure it out, that's a compiler bug, even according to wording of the old spec. which, as the above mentioned article declares, "was simply a mistake in the original specification" with "no logical explanation".

Java is wierd.

Category: Software, Programming, Java

Approaching desktop usability

I read this tongue-in-cheek article over at News Forge. Nicely written from the perspective of a (Gnu/)Linux user evaluating Windows XP. The title say it all; Windows rapidly approaching desktop usability. Miller makes an issue out of a problem with an LCD monitor during installation. More seriously, he complains "Windows XP networking: Not for amateurs". I know his pain; I have struggled long with my wife's laptop (running XP Home) to get it to connect to our wireless network. Each time I declare victory, each time my wife reports back after a few days that she doesn't have Internet. connection. Most recently, I found that a mis-typed ssid was being remembered, when the correctly typed ssid was not. Only brute force deletion (editing the registry) fixed that one. Curiously, my laptop which has inbuilt 802.11g has little trouble connecting; and that despite there being no Linux driver; I use the windows driver together with ndiswrapper.
And the old trusty old W2K Server box that we connect to; that runs fine -- as long as I don't use the the console (and definately not Windows Explorer, which freezes at the drop of a hat). But all is not rosy in Linux either. Recently, the sound system started to report CPU overload. The DVD player Xine used to work fine, but now consumes 100% CPU. MPlayer has no such problem, but it suddenly refused to start because the there was no sound system. Printing -- a major headache on Windows XP -- works just fine using CUPS, apart from one irritating little detail; every time our wireless printer cycles power, When it changes IP address, I have to feed it manually into the CUPS setup. But at least that is a deterministic process compared to coaxing the XP laptop to connect to our printer server. That is pure black magic.
Swapping between W2K (at work) and Linux lets me see the good sides (and the not so good sides) of both. On Linux there is the Gimp; an amazing tool that I use to process RAW images from my digital camera. On Windows there is the awesome Rational Rose. Amazing functionality, disasterously poor UI. An increasing number of aplications are available on both platforms; Eclipse (though the Linux version has some small cosmetic defects); Firefox/Thunderbird; even the Gimp (though it doesn't work for me as there are DLL conflicts).
To be honest, I cannot say that either is approaching "desktop usability". I spend far too much time futzing with non-productive things. Where is the Desktop OS that "just works"?

Category: Desktop

Debugging Javascript

Debugging Javascript has its own set of challenges; often aggrevated by a lengthy build/deploy cycle time. With the increased usage of asynchronous server calls (now with the fancy buzz word, Ajax), life has become a few notches more difficult. We still have the tried and tested alert pop-ups, And there are Javascript debuggers where one can step through the code line by line.
Alas, with a cascade of asynchrous actions, you are quickly lost. Timing and sequence matters, and your debugging tools freeze the action and most likely disturb the very events you are trying to understand. In situations like this, it is helpful to write to a log that you can inspect when things have calmed down. Some time back, I recall seeing a little utility that allowed you to write to a scrolling window on the page. I no longer have a refernce to that article, so I spent last Saturday creating my own little utility.
The concept is simple; a simple function ( called debug() ) writes to the window. If the window doesn't exist, it gets created on-the-fly. Each call to debug adds a new paragraph to the window -- I used an iframe to save me some hassle with selects shining through on IE.

   var box = document.createElement ('iframe');

That gets appended to the body element. Inside (this.iframe.contentWindow;) I create a container <div>. I add the text, wrapped in a paragraph element.

   var p = this.doc.createElement ('p');
   p.appendChild (this.doc.createTextNode (text));
   this.container.insertBefore (p, top);

All formatting is done by setting styles directly on the javascript objects; no need for an external stylesheet. To add a little finesse, I resize the window (up to a certain maximum height. After that, the iframe body takes car of the scrolling for me. I highlight the most recent paragraph and all debug text are prefixed with time [hh:mm:ss]. As the final touch, I set the opacity to 0,75 so the underlying shines through. That's it. Life just became a little simpler.
Category: Programming,Javascript