7 November 2008

Forking parallel subants from Ant

Ever wanted to do something in parallel in your Ant build file? Well there is the Parallel-Task, which should do the trick, shouldn't it? Unfortunately the Ant Manual says that

The primary use case for <parallel> is to run external programs such as an application server, and the JUnit or TestNG test suites at the same time.

And that's what all the examples are about - to start Tomcat in a separate thread. In my case I needed to fork several <subant>-calls, i.e. to fork several Java virtual machines with the same properties as the original one and wait till all of them were finished.

To start an independent Ant run, which means to fork a sibling Java process with the same Ant properties, you have to use the <java> task, which allows forking with the fork attribute:
<parallel>
<java classname="org.apache.tools.ant.Main"
fork="true" clonevm="true" dir="${basedir}"
resultproperty="sub_res_1" output="sub_1.log">
<arg value="-buildfile" />
<arg value="${basedir}\sub_1.xml" />
<arg value="target" />
<!-- add jvmarg sysproperty as needed -->
</java>
<java classname="org.apache.tools.ant.Main"
fork="true" clonevm="true" dir="${basedir}"
resultproperty="sub_res_2" output="sub_2.log">
<arg value="-buildfile" />
<arg value="${basedir}\sub_2.xml" />
</java>
<java classname="org.apache.tools.ant.Main"
fork="true" clonevm="true" dir="${basedir}"
resultproperty="sub_res_3" output="sub_3.log">
<arg value="-buildfile" />
<arg value="${basedir}\sub_3.xml" />
</java>
</parallel>
The output of these parallel Ant calls is collected and printed to the screen:
<concat>
<fileset dir="." id="spawn.logs">
<include name="sub_1.log" />
<include name="sub_2.log" />
<include name="sub_3.log" />
</fileset>
</concat>
<delete>
<fileset refid="spawn.logs" />
</delete>
Finally we have to determine if one of the <subant>s failed:
<condition property="spawn.error">
<isfailure code="${sub_res_1}" />
</condition>
<condition property="spawn.error">
<isfailure code="${sub_res_2}" />
</condition>
<condition property="spawn.error">
<isfailure code="${sub_res_3}" />
</condition>
<fail if="spawn.error" message="sub failed" />
That's it. Note that Ant versions below 1.7 do not support the <isfailure/> condition and you have to use
<not>
<equals arg1="0" arg2="${sub_res_1}" />
</not>
(Download Ant scripts.)

24 October 2008

Technical Progress and Joel Test

The Joel Test: 12 simple yes/no questions verbalised by Joel Spolsky in 2000 to determine the pain of developing software in some company. Since he wrote them, I have been using them to check companies if to work for a them or not.

In 2003 and 2004, I "checked" several companies during interviews or asked friends about their jobs. Back then I found that marketing and web design agencies scored lowest (1 out of 12), followed by little software shops and in-house development of small companies (2-3, sometimes up to 4). More seriously working software companies usually reached 6. That was it. Only one out of 20 companies scored 9, still not ideal. (Especially noteworthy was a mobile phone provider with more than 1000 employees having only a score of 3.)

In 2008 while looking for a new job, the companies I met were scoring about 9 or 10. I do not have much data to state any facts, but this feels much better than 4 years ago. The question is if there is such a tendency, a technical progress of the software engineering craft, that could be measured by Joel's test? Or is it just my experience telling me where to look for proper work now?

Update 28 February 2009

All Lies

After accepting an offer of one of these companies I found another reason: They were lying! They said they are able to build the product in one step. Well, that's somehow true but only after changing some hundred configurations in different files. And yes, it's also true that they have daily builds, continuous builds even. But nobody is watching the build status until a new release has to be created. That's hardly what I expect from a daily build. So their real score is 6 at most.

16 August 2008

JRuby Performance fails in Batch

You JRuby advocates will say it's obvious: I did not use Java 6, did not compile to byte code etc. You JRuby objectors will say it's obvious: JRuby is so sloooow. Probably the truth is somewhere in between and depends on the circumstances. So here is my little story about trying out JRuby in the enterprise...

We have some kind of ETL Loader that collects data into objects before inserting into the warehouse. Depending on the state of every particular instance some value is precalculated and stored to speed up the queries later. (DWH-data is usually not normalised.) Until now these calculations have been spread across the whole import and transform process, which was a real pain whenever something had to be changed. Recently we had to add some new calculation rules and we decided to make it proper: 1) Put all calculation rules in one place and 2) put them all in some kind of configuration file.

As the checks of the state vary considerably, we need a "powerful" configuration language. How about a dynamic language running on top of the JVM? (Which looks like the mainstream method to solve many problems nowadays ;-) In an XML file (boring mainstream again) we have a list of simple JRuby expressions.
<ACTION_REQUESTOR_EVAL>
<requestor active="true" type="SUPPRESSED">
<code>$action.suppressed</code>
</requestor>
<requestor active="true" type="SPECIAL_CASE">
<code>['GPDA', 'LIA'].include? $action.name</code>
</requestor>
<requestor active="true" type="OVERLOAD">
<code>$action.overload</code>
</requestor>
<!-- all others are normal -->
<requestor active="true" type="NORMAL">
<code>true</code>
</requestor>
</ACTION_REQUESTOR_EVAL>
A centralised component evaluates them in a Chain of Responsibility style.
BSFManager manager = pContext.getManager();
int line = 1;
manager.declareBean("action", pActionData, ActionData.class);
for (ActionRequestorEvaluation eval : getRequestors()) {
boolean isOfType = ((Boolean) manager.eval("ruby",
"action eval command " + eval.getType(), line++,
1, eval.getCode())).booleanValue();
if (isOfType) {
pActionData.setRequestorType(eval.getType());
break;
}
}
manager.undeclareBean("action");
After adding some unit tests we deployed the loader to the test system. Unfortunately this solution ran 3 times slower than before, taking 25 instead of 8 minutes for 100.000 entries, adding approx. 10 milliseconds per entry. Considering that in most cases 10 expressions are evaluated, that's only 1 millisecond per JRuby call, so it's not that bad. (The numbers were got using Java 5 with BSF 2.4.0 to access JRuby 1.1.3.)

The conclusion is that the straight forward use of JRuby in batch processing (i.e. when called millions of times) does not work. So we had to undo the changes and put the rules hard-coded into the code :-( Unfortunately we had only time to try the "plain" approach. Obviously there is room for improvement, I would like to hear about.