11 December 2008

New in Java 6

Java 6 New Features coverIt has been two years since Java 6 was released, so it's really high time to get to know it better.

Book about new features
Earlier this year I read Java 6 New Features by Budi Kurniawan to get an idea of the new features. I liked the book a lot because it is really unique. All the other books you find for Java 6 are general Java books that more or less cover the new features of Mustang. I do not want to read about the methods of java.lang.Object or the Collections Framework again. Budi Kurniawan does an excellent job focusing only on the new stuff. If you want to know what is new in release 6, go for this book!

List of all new classes in Mustang
Then I updated my list of all classes included in the runtime of Java, similar to what I did two years ago. I used Sun's Java 1.6.0_11 and created the updated list of all runtime environment classes in Java 1.6. Let's see what I found: Java 6 added only 74 new types in the java. name-space,
java.awt.Desktop
java.awt.font.LayoutPath
java.awt.geom.Path2D
java.awt.GridBagLayoutInfo
java.awt.LinearGradientPaint
java.awt.MultipleGradientPaint
java.awt.RadialGradientPaint
java.awt.SplashScreen
java.awt.SystemTray
java.awt.TrayIcon
java.beans.ConstructorProperties
java.io.Console
java.io.IOError
java.lang.management.LockInfo
java.lang.management.MonitorInfo
java.net.CookieManager
java.net.CookiePolicy
java.net.CookieStore
java.net.HttpCookie
java.net.IDN
java.net.InterfaceAddress
java.security.PolicySpi
java.security.URIParameter
java.sql.ClientInfoStatus
java.sql.NClob
java.sql.RowId
java.sql.RowIdLifetime
java.sql.SQLClientInfoException
java.sql.SQLDataException
java.sql.SQLFeatureNotSupportedException
java.sql.SQLIntegrityConstraintViolationException
java.sql.SQLInvalidAuthorizationSpecException
java.sql.SQLNonTransientConnectionException
java.sql.SQLNonTransientException
java.sql.SQLRecoverableException
java.sql.SQLSyntaxErrorException
java.sql.SQLTimeoutException
java.sql.SQLTransactionRollbackException
java.sql.SQLTransientConnectionException
java.sql.SQLTransientException
java.sql.SQLXML
java.sql.Wrapper
java.text.Normalizer
java.text.spi.BreakIteratorProvider
java.text.spi.CollatorProvider
java.text.spi.DateFormatProvider
java.text.spi.DateFormatSymbolsProvider
java.text.spi.DecimalFormatSymbolsProvider
java.text.spi.NumberFormatProvider
java.util.ArrayDeque
java.util.concurrent.BlockingDeque
java.util.concurrent.ConcurrentNavigableMap
java.util.concurrent.ConcurrentSkipListMap
java.util.concurrent.ConcurrentSkipListSet
java.util.concurrent.LinkedBlockingDeque
java.util.concurrent.locks.AbstractOwnableSynchronizer
java.util.concurrent.locks.AbstractQueuedLongSynchronizer
java.util.concurrent.RunnableFuture
java.util.concurrent.RunnableScheduledFuture
java.util.Deque
java.util.NavigableMap
java.util.NavigableSet
java.util.ServiceConfigurationError
java.util.ServiceLoader
java.util.spi.CurrencyNameProvider
java.util.spi.LocaleNameProvider
java.util.spi.LocaleServiceProvider
java.util.spi.TimeZoneNameProvider
java.util.zip.DeflaterInputStream
java.util.zip.InflaterOutputStream
java.util.zip.ZipError
and around 400 in javax. packages,
javax.activation
javax.annotation
javax.jws
javax.lang.model
javax.script
javax.smartcardio
javax.tools
javax.xml.bind
javax.xml.crypto
javax.xml.soap
javax.xml.stream
javax.xml.transform.stax
javax.xml.ws
mainly the Java Activation Framework, some extended annotations and the model of the Java language. Further there are the new Scripting (JSR 223) and Compiler (JSR 199) APIs. More than half of the new classes are XML related, including XML Binding/JAXB 2.0 (JSR 222), XML signatures (JSR 105), Streaming API 4 XML/StaX (JSR 175) and XML based Web Services/JAX-WS 2.0 (JSR 224).

The need for a better tool
Till now I used some ugly shell scripts to extract the needed information about the classes in rt.jar. With the growth of the JRE the repeated call of javap -public %class% for each class took more and more time. I needed something else to determine the accessibility of each class. I remembered that there was a field that determined various attributes of a class. I wanted to load the class file's bytes, jump to the offset of this field and read the package/public flag. The Java class file format is simple, but unfortunately not that simple. In order to read the access_flags, one needs to understand the whole constant pool because the access flag is stored afterwards and the constant pool is of variable length. So I started disassembling the class files. I do not know why, but I happened to write the new script in Ruby:
# Unpack HI-LO byte orders contained in the String.
class String
  # Return the _index_ 'th and the next element as unsigned word.
  def u2(index)
    self[index..index+1].unpack('n')[0]
  end
end

class JavaClassHeader

  # Parse the binary _data_ from the class file.
  def initialize(data)

    # u4 magic;
    # u2 minor_version;
    # u2 major_version;

    # u2 constant_pool_count;
    # cp_info constant_pool[constant_pool_count-1];

    constant_pool_count = data.u2(8)
    pos = 10
    cnt = 1
    while cnt <= constant_pool_count - 1
      case data.u1(pos) # cp_info_tag
        when 7
          pos += 3
        when 9
          pos += 5
        when 10
          pos += 5
        when 11
          pos += 5
        when 8
          pos += 3
        when 3
          pos += 5
        when 4
          pos += 5
        when 5
          pos += 9
          cnt += 1
        when 6
          pos += 9
          cnt += 1
        when 12
          pos += 5
        when 1
          length = data.u2(pos+1)
          pos += 3 + length
        else
          raise "const ##{cnt} unknown tag #{data[pos]}"
      end
      cnt += 1
    end

    # u2 access_flags;
    @access_flags = data.u2(pos)
  end
After (sort of) parsing the class file, the check if a class is public or only package accessible was straight forward to implement.
  ACC_PUBLIC = 0x0001

  # Return +true+ if the class is public.
  def accessible?
   (@access_flags & ACC_PUBLIC) != 0
  end
This improved performance a lot and I am looking forward to the release of Java 7 to use my new ClassList script again.

(Download the list of all classes in Mustang.)

Update 14 Mar 2009

javaclass-rb Project Initiated

The ClassList script was fun and made me want to play more with class files. Based on the source code above, I created javaclass-rb, a parser and disassembler for Java class files, similar to the javap command. The initial version reads the class file version and the package/public flag of a class as well as the constant pool. Expect updates soon!

7 November 2008

Forking parallel subants from Ant

Ever wanted to do something in parallel in your Ant build file? Well there is the Parallel-Task, which should do the trick, shouldn't it? Unfortunately the Ant Manual says that

The primary use case for <parallel> is to run external programs such as an application server, and the JUnit or TestNG test suites at the same time.

And that's what all the examples are about - to start Tomcat in a separate thread. In my case I needed to fork several <subant>-calls, i.e. to fork several Java virtual machines with the same properties as the original one and wait till all of them were finished.

To start an independent Ant run, which means to fork a sibling Java process with the same Ant properties, you have to use the <java> task, which allows forking with the fork attribute:
<parallel>
<java classname="org.apache.tools.ant.Main"
fork="true" clonevm="true" dir="${basedir}"
resultproperty="sub_res_1" output="sub_1.log">
<arg value="-buildfile" />
<arg value="${basedir}\sub_1.xml" />
<arg value="target" />
<!-- add jvmarg sysproperty as needed -->
</java>
<java classname="org.apache.tools.ant.Main"
fork="true" clonevm="true" dir="${basedir}"
resultproperty="sub_res_2" output="sub_2.log">
<arg value="-buildfile" />
<arg value="${basedir}\sub_2.xml" />
</java>
<java classname="org.apache.tools.ant.Main"
fork="true" clonevm="true" dir="${basedir}"
resultproperty="sub_res_3" output="sub_3.log">
<arg value="-buildfile" />
<arg value="${basedir}\sub_3.xml" />
</java>
</parallel>
The output of these parallel Ant calls is collected and printed to the screen:
<concat>
<fileset dir="." id="spawn.logs">
<include name="sub_1.log" />
<include name="sub_2.log" />
<include name="sub_3.log" />
</fileset>
</concat>
<delete>
<fileset refid="spawn.logs" />
</delete>
Finally we have to determine if one of the <subant>s failed:
<condition property="spawn.error">
<isfailure code="${sub_res_1}" />
</condition>
<condition property="spawn.error">
<isfailure code="${sub_res_2}" />
</condition>
<condition property="spawn.error">
<isfailure code="${sub_res_3}" />
</condition>
<fail if="spawn.error" message="sub failed" />
That's it. Note that Ant versions below 1.7 do not support the <isfailure/> condition and you have to use
<not>
<equals arg1="0" arg2="${sub_res_1}" />
</not>
(Download Ant scripts.)

24 October 2008

Technical Progress and Joel Test

The Joel Test: 12 simple yes/no questions verbalised by Joel Spolsky in 2000 to determine the pain of developing software in some company. Since he wrote them, I have been using them to check companies if to work for a them or not.

In 2003 and 2004, I "checked" several companies during interviews or asked friends about their jobs. Back then I found that marketing and web design agencies scored lowest (1 out of 12), followed by little software shops and in-house development of small companies (2-3, sometimes up to 4). More seriously working software companies usually reached 6. That was it. Only one out of 20 companies scored 9, still not ideal. (Especially noteworthy was a mobile phone provider with more than 1000 employees having only a score of 3.)

In 2008 while looking for a new job, the companies I met were scoring about 9 or 10. I do not have much data to state any facts, but this feels much better than 4 years ago. The question is if there is such a tendency, a technical progress of the software engineering craft, that could be measured by Joel's test? Or is it just my experience telling me where to look for proper work now?

Update 28 February 2009

All Lies

After accepting an offer of one of these companies I found another reason: They were lying! They said they are able to build the product in one step. Well, that's somehow true but only after changing some hundred configurations in different files. And yes, it's also true that they have daily builds, continuous builds even. But nobody is watching the build status until a new release has to be created. That's hardly what I expect from a daily build. So their real score is 6 at most.

16 August 2008

JRuby Performance fails in Batch

You JRuby advocates will say it's obvious: I did not use Java 6, did not compile to byte code etc. You JRuby objectors will say it's obvious: JRuby is so sloooow. Probably the truth is somewhere in between and depends on the circumstances. So here is my little story about trying out JRuby in the enterprise...

We have some kind of ETL Loader that collects data into objects before inserting into the warehouse. Depending on the state of every particular instance some value is precalculated and stored to speed up the queries later. (DWH-data is usually not normalised.) Until now these calculations have been spread across the whole import and transform process, which was a real pain whenever something had to be changed. Recently we had to add some new calculation rules and we decided to make it proper: 1) Put all calculation rules in one place and 2) put them all in some kind of configuration file.

As the checks of the state vary considerably, we need a "powerful" configuration language. How about a dynamic language running on top of the JVM? (Which looks like the mainstream method to solve many problems nowadays ;-) In an XML file (boring mainstream again) we have a list of simple JRuby expressions.
<ACTION_REQUESTOR_EVAL>
<requestor active="true" type="SUPPRESSED">
<code>$action.suppressed</code>
</requestor>
<requestor active="true" type="SPECIAL_CASE">
<code>['GPDA', 'LIA'].include? $action.name</code>
</requestor>
<requestor active="true" type="OVERLOAD">
<code>$action.overload</code>
</requestor>
<!-- all others are normal -->
<requestor active="true" type="NORMAL">
<code>true</code>
</requestor>
</ACTION_REQUESTOR_EVAL>
A centralised component evaluates them in a Chain of Responsibility style.
BSFManager manager = pContext.getManager();
int line = 1;
manager.declareBean("action", pActionData, ActionData.class);
for (ActionRequestorEvaluation eval : getRequestors()) {
boolean isOfType = ((Boolean) manager.eval("ruby",
"action eval command " + eval.getType(), line++,
1, eval.getCode())).booleanValue();
if (isOfType) {
pActionData.setRequestorType(eval.getType());
break;
}
}
manager.undeclareBean("action");
After adding some unit tests we deployed the loader to the test system. Unfortunately this solution ran 3 times slower than before, taking 25 instead of 8 minutes for 100.000 entries, adding approx. 10 milliseconds per entry. Considering that in most cases 10 expressions are evaluated, that's only 1 millisecond per JRuby call, so it's not that bad. (The numbers were got using Java 5 with BSF 2.4.0 to access JRuby 1.1.3.)

The conclusion is that the straight forward use of JRuby in batch processing (i.e. when called millions of times) does not work. So we had to undo the changes and put the rules hard-coded into the code :-( Unfortunately we had only time to try the "plain" approach. Obviously there is room for improvement, I would like to hear about.

27 July 2008

Viewing Dependencies with Eclipse

Dependencies between classes are somehow double faced. Classes need other classes to do their work because code has to be broken down into small chunks to be easier to understand. On the other hand, growing quadratic with the number of classes (n nodes can define up to n*(n-1) directed edges in a graph), they can be a major source of code complexity: Tight coupling, Feature Envy and cyclic dependencies are only some of the problems caused by relations between classes "going wild".

I try to avoid (too many) usages of other classes in my source. But it's not that easy. Consider a class in some legacy code base that shows signs of Feature Envy for classes in another package. Should I move it there? To be sure I need to know all classes my class is dependent upon? I mean all classes, so checking imports does not help. Even if I forbid wildcard imports (what I do), that are still not all classes. Classes in the same package are rarely shown by tools. Even my favourite Code Analysis Plugin (CAP) for Eclipse does not show them.

So here is my little, crappy, amateur Java Dependency View Plugin for Eclipse. Unpack it and put the jar into Eclipse's plugins directory. Restart Eclipse and open the Dependency View.open Dependency View in Eclipse with Open View
(The small blue icon is my old coding-logo, a bit skewed.) In the view you'll see all kinds of outgoing references: inner classes, classes in the same package, other classes, generic types and sometimes - a bug in the view. Dependency View in Eclipse showing all outgoing references of java.util.ArrayList
The plugin was developed with Eclipse 3.2 and tested in 3.3. Source code is included in the jar.

A nice feature, that I was not able to implement would have been reverse search, i.e. a list of all classes depending on the current selected one. If you happen to know how to to that, i.e. how to search for all classes that refer the current class, drop me a note.

23 July 2008

DOS Commandline Tools

A Bit of History
I always liked coding, right from my very first line of Basic back in 1986. Later I did some 65xx-Assembler work and in 1991 I finally went for Turbo Pascal. During the nineties I created a lot of stuff for MS-DOS based systems either during university projects or just for fun. Being a lazy programmer, I produced a lot of tools to perform repetitive tasks more easily. DOS and Windows never shipped with many utilities, unlike other operation systems. (Today there are Windows ports of more powerful tools available.)

Recently, when reorganising all my historic code I recovered these command-liners. Well, good old Turbo Pascal is definitely out-dated and most executables wouldn't even run any more. The reason is that the CRT unit as supplied by Borland has a few defects causing RTE200 on fast computers or at least inconsistent delay problems. Fortunately there are CRT Replacement Units for Turbo & Borland Pascal Compilers which fix all these problems. So I spent some time recompiling all my 280k lines of Pascal code. Now they work on newer computers as well. The tools are divided into several groups: Data-File-Hacking, File-Utilities, Image-, Sound/Music- and Text-Manipulation.

Rusty Ring Most of the tools provided here were created in 1994 to 1998. They were developed for 16 bit MS-DOS based systems and do not support long file names or recently introduced file formats. Most likely some of them are superseded. Use them at your own risk! Almost all tools show their usage when called with parameter "-?" or "/?". Unfortunately the help messages are in German... Some of them are highly experimental. and some are probably totally useless. I say it again, use them at your own risk!

The Data-File-Hacking package contains programs for disassembling, extracting data from files and memory, extracting files from special data files of some games, hex- and eor/xor-representation of binary data and extensive search and compare procedures. (Download hacking tools).
  • ????HACK contains code to "hack" i.e. unpack or uncompress data files of certain games.
  • BIN2HEX converts binary data into hex table text file.
  • BIT2BYTE expands every bit of binary data to a full byte.
  • BUB2ASM converts Bubble-Asm format into TASM-Asm (prefix hex numbers with 0).
  • BYTE2BIT compress every byte with value != 0 to a set bit, else not set bit.
  • CHANGBYT changes a single byte in a file.
  • CUTBIN removes the first bytes of a binary file.
  • CUTBYTES removes bytes of a binary file starting at a certain offset.
  • CUTNOP removes all lines from a text file that start with 90<space> (NOP command).
  • CV2ASMDB converts CodeView output into ASM "DB" format.
  • CV2INLIN converts CodeView output into Turbo Pascal inline assembler format.
  • EOV exclusive-or viewer to find byte sequences of 'protected' files.
  • EXTRACT extracts parts of files from larger files.
  • HEX converts hex, decimal, octal and binary numbers into each other.
  • HEX2BIN converts a sequence of hex number into a binary file.
  • HEXTSR (TSR) shows hex conversion window on key stroke ALT-S.
  • MEMLOAD loads binary data into memory without executing it.
  • MEMSAVE saves memory contents to a file.
  • MEMSVCNV saves contents of conventional memory to a file.
  • MEMSVXMS saves contents of free XMS memory to a file.
  • MEMTSR (TSR) saves memory contents to a file on key stroke ALT-S.
  • RECH calculates arithmetic expressions.
  • SAVESCRN saves current contents of video text memory to a file.
  • SCRNTSR (TSR) saves current contents of video text memory to a file on key stroke ALT-S.
  • VOCEXTR finds and extracts Creative Labs Voice (VOC), Wave (WAV) and XMidi (XMI) from larger files.
  • WADIT edits and displays contents of ID software's WAD files (DOOM data files).
  • WAVEXTR finds and extracts RIFF Wave and IFF files from larger files.
  • XADD adds/xors all bytes of a file to a checksum.
  • XCHGBYTE replaces or swaps specific values in binary files.
  • XCOMP extended version of DOS COMP.
  • XSEARCH searches several files simultaneously.
  • XSEQUENZ searches results of XCOMP runs for ascending/descending sequences.
  • XWHAT prints the value of a certain position in files.
  • XWHAT2 prints all the value of a certain position in files.

File-Manipulation features tools for comparing, renaming, setting time and date, wildcard operations and hardware detection. (Download file tools.)
  • BRAKE! slows down CPU using busy waits in interrupt 8 (INT08).
  • CHKFILES reads files from disk to check their integrity.
  • DETECT ultimate hardware detection and benchmark tool.
  • DFOR executes a DOS command for all file names in a list.
  • DUPLICAT calls COPY to duplicate directory trees, copies only new files.
  • FILLDISK calculates somehow optimal distribution of files when saved to floppy disks.
  • GETTIME measures the time another DOS command needs to execute.
  • ISWIN checks for currently running Windows 3.1 or 95.
  • LENCOMP compares files similar to COMP but based on length, ignoring path and name.
  • NOTHING Do Nothing Loop.
  • RENALL renames a wildcard file set according to a name list.
  • SCOMP compares files similar to COMP and deletes identical ones.
  • SFOR executes a DOS command for a wildcard file set.
  • TDIR lists files similar to DIR with their date time to be used for TTDAT.
  • TIMETOUC (or short TT) changes the date and time of a file.
  • TO searches for a directory and changes current working directory to there.
  • TTDAT executes TIMETOUC for a list file produced with TDIR.

Field Working Tools
The Image-Manipulation collection provides help for viewing header information of images, viewing special picture formats, viewing special animation formats, palette manipulation, Windows BMP generation and font designing and using. (Download image tools.) Some of the tools need a VESA compatible driver to display images in SVGA graphics mode.
  • ANIM creates an rotating animation from a Bitmap (BMP).
  • ANM plays 320x200x256 sized ANM animations (with LPF ANIM magic bytes).
  • APAL2BIN converts ASCII RGB-palette into binary.
  • BGI2DOOM converts Turbo Pascal's BGI images into Doom 1.2 textures.
  • BMP2ASC converts 640x400x256 sized BMP into an ASCII image.
  • BMP2BGI converts uncompressed BMP into Turbo Pascal's BGI format.
  • BMP2DRW converts 80x50x16 sized BMP into my own DRW format for display in text mode.
  • BMP2PAL saves colour palette of a BMP.
  • BMP2PLAN converts 640x480x16 sized BMP into 4 colour planes.
  • BMP2RAW converts 320x200x256 sized BMP into raw-format.
  • BMP2TXT converts BMP into ASCII image depending on brightness values.
  • BPAL2ASC converts binary RGB-palette (768 bytes) into decimal text format as needed for RAW2GIF.
  • COLTABL displays table of DOS colours.
  • DRAWPIC editor for drawing 80x25 DRW image files.
  • DRAWTXT editor for drawing ASCII images.
  • FNTMAGIC editor for drawing 16x8/8x8 fonts.
  • FONTHACK displays binary data as 16x8/8x8 fonts.
  • FONTVIEW displays an F16 font.
  • GIF2BGI converts a GIF into one or more BGI images.
  • GIF2SPR converts 160x160 sized GIF into 64 BGI images useable as sprites in games.
  • GIFHEAD displays header information of GIF images.
  • HLS2RGB converts ASCII HLS-palette into ASCII RGB-palette.
  • KPACK compresses in my own R9 format, similar to RLE8 with slightly better compression.
  • LINESET set number of lines text mode (25, 28, 33 or 50).
  • PACK compresses files with RLE algorithm.
  • PACKFL4 compress 4-bit animations with distinct frames, my own FL4 format.
  • PAL2BMPP converts binary palette into BMP palette with empty padding.
  • PLAY plays 320x200x256 sized Autodesk FLI and 640x480x256 sized FLC.
  • PLAYFL4 plays 640x480x16 sized FL4 animation.
  • PLAYFLI plays 320x200 sized Autodesk FLI and FLC animation.
  • RAW2BMP converts RAW images into BMP.
  • RGB2HLS converts ASCII RGB-palette into ASCII HLS-palette.
  • SETBORD sets border colour in console.
  • SETFONT sets an F16 font in console.
  • SHOWBMP displays unpacked 1024x768x256 sized BMP.
  • SHOWC64 displays Commodore 64 Koala Painter and other image formats.
  • SHOWGIF loads a few GIFs into XMS memory, displays and cycles them (mini slide show).
  • SHOWIT displays GIFs and slide shows of GIFs using external VPIC.EXE.
  • SHOWPAL displays colours of a palette.
  • SHOWPIC displays arbitrary data as colour values of an image.
  • SHOWRAW displays arbitrary data as 16 or 256 colour values of an image up to 1024x768x256.
  • SHOWSVGA (or short EV) displays arbitrary data as colour values of an image up to 1280x1024x256.
  • SHOWVGA displays arbitrary data as VGA image (320x200x256), part of EOV.
  • TXT2BMP converts text into 80 times lines sized BMP.
  • TXT2RAW converts text into 80 times lines sized raw image data.
  • UNPCKFL4 unpacks and displays 4-bit animations (FL4 files).
  • UNRLE8 unpacks RLE8 compressed files.
  • XCOLOR changes specific colour values of a BMP in image data, not in palette.
  • XTRC_BMP extracts sub-image out of an uncompressed BMP into a new BMP.

The Sound/Music-Manipulation tools support conversion of special sound formats into common formats and between different types of formats, playing of VOCs, WAVs and XMidis via Soundblaster and PC-Speaker and getting header information of audio files. (Download sound tools.) To use your audio hardware you need a DOS styled driver emulator like VDMSound.
  • AMIG2SND converts Amiga signed raw sound data (SND) into PC format.
  • DWD2SND converts DiamondWare digitised data into SND format.
  • MAC2SND converts Apple Mac sound data (SND) into SND format.
  • MANYINST lists instruments of an XMidi found in the *.ad file.
  • MIDHEAD displays header information and event list of Midi files (MID).
  • PAWE plays XMidi (XMI) using AWE32 hardware.
  • PCM2VOC converts Space Hulk PCM into VOC.
  • PCM2WAV converts AquaNox's PCM into WAV.
  • PCMF plays Creative Music File (CMF), needs SBFM to be started before.
  • PCMF17 plays CMF.
  • PMID plays XMidi using Midi hardware.
  • PMPU plays XMidi using MT32MPU hardware.
  • PSOUND plays sound files like VOC and WAV using PC speaker hardware.
  • PVOC plays VOC.
  • PWAV plays WAV.
  • PXMI plays XMidi using Soundblaster hardware.
  • RAW16TO8 converts raw 16 bit samples into 8 bit by ignoring every second byte.
  • RAW2SGN converts signed into unsigned samples by adding 080h.
  • RAW2SND converts raw samples into SND format.
  • SAM2WAV converts Modplayer Sample data (SAM) into WAV.
  • SND2WAV converts SND into WAV.
  • SYN2WAV split SYN data files into separate WAVs.
  • VAC2WAV converts VAC, RAC and MAC (ST-TNG CD, 11025Hz) into WAV.
  • VCMF plays all CMFs in the current directory, needs SBFM to be started before.
  • VCMF17 plays all CMFs in the current directory.
  • VMF2WAV converts Creatix sound 4800Hz (VMF) into WAV.
  • VMPU plays all XMidis in the current directory using MT32MPU hardware.
  • VOCHEAD displays header information of VOC files.
  • VVOC plays all VOCs in the current directory.
  • VWAV plays all WAVs in the current directory.
  • VXMI plays all XMidis in the current directory.
  • WAV16TO8 converts 16 bit WAVs into 8 bit.
  • WAV2SAM converts WAV into Modplayer Sample data (SAM).
  • WAV2SYN combines several WAVs into a SYN.
  • WAV2VMF converts WAV into Creatix VMF.
  • WAV2VOC converts WAV into VOC.
  • WAVHEAD displays header information of WAV files.
  • WAVMIX mixes WAVs using different algorithms.
  • WAVMORPH morphs a WAV into another (very experimental).
  • WAVST2MO converts 8 bit stereo WAV into mono format.
  • XMIHEAD displays header information of XMidi files.
  • XMITRACK extracts single tracks from a multi-track XMidi.

Abandoned to nature
Text-Manipulation (ASCII) offers methods for converting, replacing, sorting and different kinds of removing of pieces of text in text files. (Download text tools.)
  • ASCTABL displays table of ASCII characters.
  • ASCTABL2 displays table of ASCII characters together with their decimal values.
  • BIN2ASC removes all bytes 0-31 from a file to edit it as text.
  • C64TOASC converts Commodore 64 text into ASCII format.
  • COPYLINE copies all lines of a text file which contain a search term.
  • CUT10 removes the first 10 columns in each line (for DIS86).
  • CUTLAST removes the last given characters in each line.
  • CUTLEAD removes the first given characters in each line.
  • CUTLINES removes every some lines some more lines, e.g. remove two lines every ten lines.
  • CUTPOS removes all text after a given column position.
  • CUTSPACE removes trailing spaces.
  • D2E replaces all 'D' with 'E', used to convert FORTRAN output into gnuplot data.
  • DELNEWS removes all lines of a text file which start with search term.
  • DELSPACE removes all double blanks.
  • DIR2HTML creates an HTML index file containing the current folder entries.
  • DIR2TEX creates a TeX index file containing the current folder entries.
  • DOC2TXT converts WinWord umlauts back into DOS characters.
  • DOWN converts all letters into lowercase.
  • EXP2DEZ converts scientific numbers (e.g. 1.3E-1) into decimal numbers (e.g. 0.13).
  • FIT2PIX reduces data points to one value per pixel, used for gnuplot and TeX.
  • FORM2TXT converts HTML form data into plain text.
  • GERADEN solves a linear equation.
  • GNUSUB subtracts values from the second column, used for translation of gnuplot data.
  • INSP2TEX converts INSPEC search results to LaTeX bibitems.
  • LINECNT counts the lines of a text file.
  • LINESORT sorts like SORT but blocks of lines instead of single lines.
  • LINESUB replaces whole lines of a text file, configured by line number.
  • MERGLINE merges two text files line wise (today's editors know this as 'column mode').
  • MOVE_TEX moves TeX elements inside EmTeX pictures.
  • NOSPACE removes all single blanks.
  • ROUND rounds all decimal numbers.
  • SEQ2TXT converts Commodore 64 sequential file data (SEQ) into PC char-set.
  • SKALIER scales numbers by a factor, used for gnuplot data.
  • SLIM_TEX removes all unnecessary horizontal \put(lineto), used for gnuplot and TeX.
  • STRIP_% removes all comments (%) from TeX source.
  • STRIP_C removes all comments from FORTRAN 77 source.
  • TASTCODE prints the key code of the pressed key.
  • TEST7BIT checks if a text only contains ASCII characters > 127
  • TEXTDEL removes every some lines some more lines, e.g. remove two lines every ten lines.
  • TEXTINS inserts a text file into another at a certain position.
  • TEXTSUB replaces characters in a text file.
  • TURNBACK save a file in reverse order, i.e. last byte first.
  • TURNLINE turn all lines backwards, i.e. from right to left.
  • UMBRUCH joins all lines and adds new line breaks at given position.
  • UP converts all letters into uppercase.
  • UP11 converts all letters and umlauts into uppercase.

8 July 2008

Eclipse Europa GlassFish Problems

OK, it's my fault. I do not RTFM, but usually try out the stuff immediately without much reading.

Recently I wanted to check out the GlassFish application server (version 2) together with Eclipse (version 3.3 or Europa) and encountered some problems. Here is my minimalist conclusion:
  • Download the latest version of GlassFish, in my case it was 2ur2-b04.
  • Read the documentation, there is more to do than just unpack the archive and start it.
  • Do it! Go there! Read it!
  • java -Xmx256m -jar glassfish-installer-v2ur2-b04-windows.jar is obvious...
  • Now comes an important part: Probably you do not want GlassFish to live in the /glassfish-folder, I would rather have it in something like /glassfish-2ur2-b04 for this version. So
  • Copy and/or rename the newly created and filled directory glassfish to your liking. This has to be done before you continue! You can't do that later.
  • Go into this folder and execute lib\ant\bin\ant -f setup.xml as shown in the GlassFish documentation. This should finish the installation.
For the integration of GlassFish with Eclipse, there is an excellent description on the GlassFish site. Unfortunately, there are still some holes in it (and of course I did not read it in the first place, remember, I do not RTFM :-)
  • To create a new server in Eclipse JEE perspective, use the "New Server" wizard in the "Server" tab. But where is GlassFish?
Eclipse Europa - New Server - where is GlassFish
  • So click on the "Download additional server..." link
  • When working behind a proxy, this may take long and not list GlassFish in the end. The reason is that an https-Proxy has to be configured in Eclipse.
Eclipse Europa - Network Settings SSL Proxy
  • Now you are able to create the new server. If the wizard tells you that it can't find some appserv-rt.jar, then you forgot to call ant -f setup.xml during installation of GlassFish (as listed above).
Eclipse Europa - GlassFish V2 - where is appserv-rt.jar
  • This is because GlassFish comes with it's jars in the pack200 format, which do not end with .jar but with .jar.pack.gz and are unpacked during installation. (Manually unpacking unpack200 lib\appserv-rt.jar.pack.gz lib\appserv-rt.jar does not help here. You need to call the set-up...)
  • Start the server. If you get error messages that the system can't find some path and the start-up fails with a timeout after a minute then you did not move/rename the /glassfish-folder to the desired destination before calling the setup.
  • This is because GlassFish has his scripts like asadmin.bat contain absolute paths written during setup, e.g.
    setlocal
    call "C:\download\glassfish\config\asenv.bat"
    set Path=%AS_INSTALL%\bin;%AS_ICU_LIB%;%PATH%
  • Enjoy the finally starting GlassFish, serving it's index page at port 8080:
GlassFish V2 or SunAppServer V9
Now is it GlassFish 2.2 or Sun's Application Server 9.1? ;-)

25 April 2008

backwater, in-house development

Indien BackwatersMaybe I am too strict, or my expectations are far too high for the average company (which is likely the case). However, while staying with Herold I reached the conclusion that "backwater, in-house development" does not provide footing for IT innovation. Obviously this is true but the ironic part is, that management pretends that it's there. Listen to this:

Two years ago they came up with the idea of science Fridays. Every Friday the development department would be locked, so no one would disturb us and all developers would try out new stuff or study books. Well, it never happened. Last year the idea was still around, we would just start after this particular next release. And there is always some pressing next release to work on. The WTF of this paragraph is that two months ago I was asked by the boss of my boss if I used the science Friday time for some project. I had to laugh out loud.

Another idea was born at the beginning of last year: every month a member of the team would prepare some topic, e.g. new features in .NET 3, and present an overview. There was not a single one of these presentations, the monthly meetings were always full with lists of features to be put into the next release. The proposed topics, already outdated, are still in my boss' drawer, waiting for their time.

How about professional training? "Sorry, there is no budget for that. Sales revenue just increased by 5 percent last year. The company is in critical condition. We all have to work harder."

So I experiment with new technologies at home and try to bring in something new from time to time. But I was told that we do not need it, because we are no IT company and there is no point in spending time to stay up to date because it is not our main business. And anyway I am not supposed to spend "so" much work time on education and research. But I have to, because as Heinz Kabutz once said, we're in an industry with a knowledge half-life of at most 18 months. Keep in mind that half of what you knew 18 months ago is worthless today, so you need to keep learning new things.

Do you work under similar conditions? Tell me how you overcome them to stay a top notch developer.

23 April 2008

Redland RDF library and Ruby

Ever wanted to use the Redland RDF library with Ruby? In case you do not know Redland, it's a library to manipulate RDF triples and perform SPARQL queries as needed for semantic web applications. Redland is written in C++ and provides bindings to other languages like Java or Ruby.

Although current version is 1.0.7, windows binaries are only available for 1.0.3. Ok, it's my fault, I could always compile the stuff myself, but as a true Java person, I try to stay away from such obscenities. (This is probably another argument for Joel that Java coders can't be real programmers).

So I took version 1.0.3 and hoped that everything would be fine. When testing some of its command line tools, it turned out that still some libraries were missing: msvcp71.dll and msvcr71.dll. This should have been a hint to me, obviously these files belong to Microsoft's Visual C++ 7.1, but I did not bother and was happy to find them online in some dll archive.

Bound But the Ruby binding still needed another library (read a dll) to run, which was nowhere to find. So I was forced to fool around with Devccp (Mingw) myself to compile it. Unfortunately Ruby's extconf.rb works only with MSVC, so the auto-generated make file was of little use. After a painful day I finally managed to compile the given redland_wrap.c into a nice little redland.dll.

So far so good. Redland's Ruby API looked easy and some unit tests showed the way. But why did some tests work and some failed with a segmentation fault. In fact, why did most Redland functions crash for URIs longer than 20 characters? So after 2 days wasted, this looked like obviously incomplete compilation or linkage problems.

You might think that the Mingw compiled code might not be fully compatible with Ruby's core libraries, which are supposed to be MSVC6 stuff on Windows systems. Well, there are some issues about this topic, but here it was not the case. Much later I was able to access an old MSVC6 installation and compile the Redland Ruby binding again. This time the Ruby's extconf.rb worked nicely and make produced a redland.so. Still the functions crashed the same.

In the end it turned out, that the Redland 1.0.3 binaries for Windows where compiled using Visual C++ .NET 2003 (aka VC 7.1) , which needs the dotNET 1.1 framework runtime, which I choose not to install. Couldn't the system tell me that some runtime dependency is missing instead of producing segmentation faults all the time?

Downloads
In case you face similar problems, here are the results of my "hard" work:

19 April 2008

Birds in the Bush

An old man's wisdom:
If you're not hitting the bush soon enough,
one day the birds will come out of the bush by themselves.
Applying Murphy's Law we can easily see that the birds will choose the worst moment thinkable to come out of the bush.

(Thank you Richard for this little gem. I know you are not that old ;-)

10 April 2008

My Rants

After years of being a maintenance developer I am pissed off. During maintenance the software systems are always messed up. Obviously the release date of the next batch or upgrade is set by the business analysts long before the exact extend of work is known. There is never time to fix anything. In my early days I would have argued with managers to add extra time for fixing the most ugly things during the next release, but finally I resigned from wasting my energies.

anger The Angry Developer
So I became an angry developer. Always in pain, I am using biting comments to point at serious problems in systems, processes and whole organisation. I will rant from time to time about IT work and software development related stuff.

For any legal departments reading this: The events and companies I am posting about are completely fictional and any resemblance to any real people or companies that may or may not exist is purely coincidental.

19 January 2008

Daily Build Articles

In 2007 I started writing articles about Daily Builds for Java applications, which I called the 'Code Cop' Series. Unfortunately I just managed to finish two articles so far, shown below. I have a lot of material for further articles about adding automated testing and enforcing architecture to our daily build, just have to squeeze in the time to do it ;-)

The Daily GrindPart 1 - Daily Build with Anthill OS
This article describes my experiences when introducing a daily build in 2004 when I used the Anthill tool. The first steps were to create JavaDoc pages daily and to compile the Java sources. It turned out that the initial set-up of these build routines did not cost much and were supported by the team. Obviously this is only the start of better quality code. Read more in the full article Täglicher Build mit Anthill OS published in JavaSPEKTRUM, 5/2007.

ReferencesOther build tools are Anthill Pro, Continuum, Cruise Control, Maven, Luntbuild (incomplete list).

Part 2 - Daily Code Analysis with PMD
This article introduces static code analysis with PMD. The existing daily build was extended easily. A daily report of the code quality metrics awakened the management and was used as a base to check for a small set of errors. The most serious of them were fixed and part of the coding conventions have been checked automatically since then. Read more in the full article Tägliche Code-Analyse mit PMD published in JavaSPEKTRUM, 1/2008.

ReferencesTurn on the magicOther code analysis tools are Checkstyle, Enerjy CQ2, Findbugs, JavaNCSS, JLint, Jtest, Optimal Advisor, Sotograph (incomplete list).

(Download source code of Ant-PMD integration and BuildStatistics.java.)

FAQ
Q: Why did you favour Anthill OS instead of all other build tools listed? A: There was no evaluation or decision process. Anthill OS was just the first tool I got my hands on. Everything worked fine, so I did not look further. For a new project I would use Cruise Control because it is actively maintained and has a strong community. Paul Duvall has written a nice comparison of open source CI servers. If your build process is "heavy", you might want to have a look at commercial build servers. Some of them offer build farms. e.g. Anthill Pro, Parabuild or Team City.

Q: You divided all rules of PMD into four groups: error, warning, information and formatting. How can I get this categorisation? A: PMD comes with build in severity. Each rule definition contains an <priority> element. There are also some commercial tools that use PMD under the hood and have their own severity levels. Some of them even have references for each rule, why it's bad.

Q: You use a program called BuildRuleDoc to document used rules. Is it free or home-grown? A: I wrote it myself, but you can use it if you want to. The BuildRuleDoc.zip contains the code, the template to create the rule document, an Ant fragment and some test scripts. You will have to adapt the scripts in order to run them. Finally you need the XML rule set file of active PMD rules to generate the report.

(List of all my publications with abstracts.)