Integrating Io Addons

This is the third time I write about making existing Io extensions work under Windows. While I still struggle with Socket - a crucial dependency - I install and compile random addons to see what could go wrong. Almost every addon I touch has a different issue that I need to resolve for my Io Windows build dating back to 5.11.2013. Till now I have covered the following situations:

• Clone current addons right into the addons folder, see part 1, Kano.
• Rename addon folders to start with an uppercase letter.
• Create the required protos file with a list of all exported prototypes and depends file with a list of required prototypes, which is usually empty.
• Create the missing Windows starter bat for tools like Kano.
• If the name of the addon is missing in the package.json, Docio will be unable to process the package. I will cover Docio next time.
• Compile the native code using MingW GCC, see part 2, Rainbow.
• Fix linker issues regarding one versus two underscores in name decoration.
• Copy native headers to _build/​headers, which I will need today.
• Resolve initialisation recursion issue when loading addons.

I walk through the installation of addons with different needs and increasing difficulty using concrete examples. As many addons are wrappers around C libraries, which "may not be installed on your system" - I will work on integrating these libraries today.

Addons with (Pre-Compiled) Dependencies: ReadLine
The Io ReadLine addon is a binding to GNU readline. I have no use for it but it is small and a good example. I start as usual with cloning, creating the init file and required files:

> cd "%IO_HOME%\lib\io\addons"
> git clone git@github.com:IoLanguage/ReadLine.git
> io generate.io . ReadLine
> cd ReadLine
> touch depends
> echo ReadLine > protos
> cd source
> gcc -fPIC -D _NO_OLDNAMES -c IoReadLine.c IoReadLineInit.c
IoReadLine.c: fatal error: readline/readline.h: No such file or directory

This is expected as ReadLine binds to libreadline. For more complex cases it helps to look at the Eerie package manifest package.json:

{
  "name": "ReadLine",
  ...
  "dependencies": {
    "libs":     ["readline"],
    "headers":  ["readline/readline.h", "readline/history.h"],
    "protos":   [],
    "packages": []
  }
}

I need libreadline and two of its headers. The easiest way to build a Linux library for Windows is not build it. There is GnuWin32 @ SourceForge. It was last updated 2010, which is around the same time that active Io development stopped. Lovely there is readline-5.0-1-bin.zip which is all I need. Now I can continue building the addon.

> gcc -liovmall -lbasekit -lreadline5 ^
      -shared -Wl,--out-implib,libIoReadLine.dll.a ^
      IoReadLine.o IoReadLineInit.o -o libIoReadLine.dll
IoReadLine.o: undefined reference to `IoState_registerProtoWithFunc_'

Sigh, this was supposed to be the easiest case. I have seen this method call before. Older addons from the 2009 source release used this function, while addons bundled with my 2013 version use IoState_registerProtoWithId_. The required change in source/​IoReadLine.c, line 47 is

  using_history();

-  IoState_registerProtoWithFunc_((IoState *)state, self, protoId);
+  IoState_registerProtoWithId_(state, self, protoId);

  IoObject_addMethodTable_(self, methodTable);

Now compilation and linking succeeds.

> gcc -fPIC -D _NO_OLDNAMES -c IoReadLine.c IoReadLineInit.c
> gcc -liovmall -lbasekit -lreadline5 ^
      -shared -Wl,--out-implib,libIoReadLine.dll.a ^
      IoReadLine.o IoReadLineInit.o -o libIoReadLine.dll
> del *.o
> move /Y lib*.* ..\_build\dll
> copy /Y *.h ..\_build\headers
> cd ..\..
> io -e "ReadLine readLine() println"

The last step is to copy readline5.dll into the %IO_HOME%\​bin folder, where Io's DLLs are located. This makes running Io independent from my build environment. Here is my Io ReadLine with the necessary fixes. A nice surprise after installing ReadLine is that the Io CLI remembers my input.

Addons with Dependencies: Markdown
Io's Markdown addon is next in line. It is a Markdown parser for Io, based on discount. Its package.json states that it needs libmarkdown and a header mkdio.h. I need to compile the library first. The addon comes with the library's source code, and even a compiled version for Windows x64. It has a build.io, which will rewrite Eerie's AddonBuilder to represent a receipt for your package. Eerie should be able to build it. What does it say?

AddonBuilder clone do(

  srcDir := Directory with(Directory currentWorkingDirectory .. "/source/discount")

  compileDiscountIfNeeded := method(
    if((platform == "windows") or(platform == "mingw"),
      appendLibSearchPath(Path with(Directory currentWorkingDirectory, "deps/w64/lib") asIoPath)
      appendHeaderSearchPath(Path with(Directory currentWorkingDirectory, "/deps/w64/include") asIoPath)
    ,
      prefix := Directory currentWorkingDirectory .. "/_build"
      Eerie sh("cd #{srcDir path} && " ..
               "CC='cc -fPIC' ./configure.sh --prefix=#{prefix} && " ..
               "make && " ..
               "make install" interpolate)
      appendHeaderSearchPath(Path with(Directory currentWorkingDirectory, "_build/include") asIoPath)
      appendLibSearchPath(Path with(Directory currentWorkingDirectory, "_build/lib") asIoPath)
    )
  )

  compileDiscountIfNeeded

  dependsOnLib("markdown")
  dependsOnHeader("mkdio.h")
)

I have little experience with building C, still the bold line above gives me enough information to build discount. Tweaking its makefile and librarian.sh a bit does the trick of creating DLLs. It would have been nice to use Eerie instead... I follow the usual steps as for ReadLine above to build the addon and in the end io -e "Markdown toHtml(\"# A1\") println" displays <h1>A1</h1>.

Addons with (Conflicting) Dependencies: UUID
There waits a different problem when installing UUID, Io's wrapper around libuuid. To get started there is a MinGW compatible libuuid, thank you Alessandro Pilotti. The typical commands sh.exe ./configure, make && make install work. Between configure and make I had to #define HAVE_USLEEP 1 in the configured config.h. Having built libuuid, I can work on Io's UUID. The problem is that both libuuid (in uuid.h) and MinGW (in basetyps.h) define the type uuid_t differently. I dislike messing with MinGW include files.

Maybe I can avoid including basetyps.h from the code I want to compile? Io's source contains more than 40 headers, one for each Io object. Even the smallest addon needs to include IoObject.h and IoState.h, which provides access to the whole Io virtual machine - the largest headers in the source - which include other headers on the way. I start copying required includes into a stand-alone header. It is a boring and repetitive work, and brute force wins the day. I end up copying 500 lines of structs, method declarations, and macros from various headers. Using that isolated header instead of the provided ones, addon compilation and linking works as expected.

Addons with Io and Third Party Dependencies: Regex
My whole exploration of Io addons started with the Regex package. I really wanted to use Regular Expressions. I am a big fan of Regex, and Mastering Regular Expressions is one of my favourite books since 2006. I still consider it one of the most exciting technical books of all times. The Regex addon supports Perl regular expressions using the PCRE library. There is a suitable GnuWin32 PCRE build. Maybe I am lucky today.

This addon is special, it depends on a third party library libpcre3, and it depends on the native code of another Io addon, Range. It is shown inside build.io:

AddonBuilder clone do(
  dependsOnLib("pcre")
  dependsOnHeader("pcre.h")

  dependsOnBinding("Range")

  debs atPut("pcre", "libpcre3-dev")
  ebuilds atPut("pcre", "pcre")
  pkgs atPut("pcre", "pcre")
  rpms atPut("pcre", "pcre-devel")
)

I will have to consider this when building it.

> cd "%IO_HOME%\lib\io\addons"
> git clone git@github.com:IoLanguage/Regex.git
> io generate.io . Regex
> cd Regex/source
> gcc -fPIC -D _NO_OLDNAMES -I ..\..\Range\_build\headers ^
      -c IoRegex.c IoRegexInit.c IoRegexMatch.c IoRegexMatches.c Regex.c
> gcc -liovmall -lbasekit -lpcre3 -L ..\..\Range\_build\dll -lIoRange ^
      -shared -Wl,--out-implib,libIoRegex.dll.a ^
      IoRegex.o IoRegexInit.o IoRegexMatch.o IoRegexMatches.o Regex.o -o libIoRegex.dll
> del *.o
> move /Y lib*.* ..\_build\dll
> copy /Y *.h ..\_build\headers
> cd ../..
> io -e "Regex; \"11aabb\" matchesOfRegex(\"aa*\") asStrings println"
list(aa)

If there is a linker error,

Creating library file: libIoRegex.dll.a
IoRegexMatches.o: undefined reference to `_imp__IoRange_new'
IoRegexMatches.o: undefined reference to `_imp__IoRange_setRange'

it is the same name decoration problem as before. The solution is to move out libIoRange.dll.a from Range\​_build\​dll temporarily.

If io -e Regex fails during runtime with Exception: Error loading object 'lib\​io\​addons\​Regex\​_build\​dll\​libIoRegex.dll': 'Did not find module' or similar, then Range functionality was not loaded in advance. Make sure that depends contains Range and that it does not end with a newline at the end.

The next episode (part 4) will deal with patching some broken addons like Docio.