Patching Io Addons

This project is getting out of hand. I just wanted to use Regular Expressions in Io. This is the forth part of me trying different extensions of Io. First I installed addons without native code, then I compiled native code of addons and in the third part I:

• Checked package.json and build.io for hints about needed third party libraries and headers.
• Found ported libraries for Windows in GnuWin32, e.g. ReadLine.
• Compiled Io addons with dependencies.
• Fixed the undefined reference to 'IoState_registerProtoWithFunc_' error, which occurs using addons created for older versions of Io.
• Worked around conflicting headers included by IoObject.h or IoState.h.
• Finally compiled with dependencies on the native code of another addon.

On the way I added minor fixes to several addons, see my (forked) Io repositories and today I cover addons which needed more extensive modifications.

JSON Parsing
Io has "half" JSON support, most objects provide an asJson() method to represent the contained data in JSON. Some addons (like Docio) require Sequence.parseJson() but omit a specific dependency for that and my Io (iobin-​win32-​current.zip) misses this method. These addons are for a newer version of Io, which has not been released (yet?). Writing a JSON parser is a nice exercise and there is already a JSON parser for Io. It is neither an addon nor an Eerie package. Preparing some arbitrary Io code as addon means moving the existing files into the proper folder structure, and adding the files needed to run as addon (i.e. proto and depends files). I even added a test. My Io JSON addon can be cloned directly into the addons folder %IO_HOME%\​lib\​io\​addons.

Missing Io Core Functions
I started changing addon source code to include the JSON parser addon, at the same time I wanted to keep my changes as little as possible. The Io Guide said that if you have a .iorc file in your home folder, it will be eval'ed before the interactive prompt starts.. In my %HOMEDRIVE%%HOMEPATH%\.iorc I added all definitions of missing Io core functions:

false isFalse := method( true )
true isFalse := method( false )

// load parseJson
Sequence hasSlot("parseJson") ifFalse(
    Json
)

false asJson := "false"
true asJson := "true"
nil asJson := "null"

This made me ask myself, what else was added to Io since my Windows binary was built in 2013? I cloned Io and compared the proper tag with master. There were many changes and I filtered out formatting and comments. The final result was a short list of additions and modifications like

• Addon and AddonLoader extensions to use Eerie.
• Core tildeExpandsTo() using UserProfile instead of HOME on Windows.
• Sequence power() and powerEquals().
• TestRunner run() returning the number of errors.

Docio
With JSON support and Markdown working, I can run Docio. Docio is the documentation generator for Eerie packages. It extracts tagged comments from C and Io code and generates Markdown and HTML documentation like this. To make Docio work, I had to work around several issues:

1. Docio's repository name is lowercase. To install it as addon the folder name has to be uppercase, as for Kano:

   > cd "%IO_HOME%\lib\io\addons"
   > git clone git@github.com:IoLanguage/docio.git Docio

2. Docio does not depend on native code, and it has Markdown as its sole dependency.

   > echo Markdown > Docio\depends
   > echo Docio > Docio\protos

3. Docio loads Eerie eagerly in the first line of Docio.io. It uses Eerie to query its template path. I lack Eerie and the eager loading fails, so I remove that. With Eerie out of the picture, I always have to provide the full path to the template (%IO_HOME%\lib\io\addons\Docio\template). Now I can initialise Docio with io -e "Docio println". Success.
4. Docio has a starter script in bin/docio which needs a Windows version, i.e. a bin/docio.bat which calls the original starter script,

   io "%IO_HOME%\lib\io\addons\Docio\bin\docio" %*

5. With the starter script Docio is available as a command line tool. One thing to know is that its help is wrong. It says

   docio package=/path/to/package [template=/path/to/template]

   while it really needs two dashes for each option,

   docio --package=/path/to/package [--template=/path/to/template]

   Both paths need to be absolute, otherwise documentation files appear in strange places.
6. There is a syntax error in DocsExtractor.io line 65 and following.
7. Copying of binary resources, i.e. fonts, corrupts the files. You will have to copy them manually. This is because Io's low level C code does fails to set the binary flag when opening files. On Windows the C library functions consider text files to be finished on the first byte 0x1A. I am unable to fix that right now.

After fixing all that, Docio provides help, even inside the Io REPL, which comes handy:

Io> Docio printDocFor("ReadLine")
Binding to GNU readline.

Docio will generate the documentation for any package on the fly which has a package.json with a name field. Bundled addons miss that file, and I create empty ones containing each addon's name for addons where I want to generate documentation. Here is my own Docio with all the fixes.

Socket
Finally I am going for the Socket addon. I was scared of Socket, some sources stated that Io socket support has always been tricky. Socket depends on libevent 2.0.x, an event notification library. Surprisingly autogen.sh, configure and make install compile and link libevent without any problems. Now I have some libevent_*.dll files. This is way too easy.

But of course Socket's C code does not compile. The header sys/​queue.h is not available on Windows. Fortunately libevent comes with its own compat/​sys/​queue.h. Some header files, e.g. IoEvConnection.h, IoEvHttpServer.h and several others need the conditional include:

#include "Socket.h"
#if !defined(_WIN32) || defined(__CYGWIN__)
#include <sys/queue.h>
#else
#include <compat/sys/queue.h>
#endif
#include <event.h>

Further UnixPath.c fails compilation, and I drop it from the list of files to compile. Even if it would compile, it would not work as indicated by the error returned from IoUnixPath for defined(_WIN32) || defined(__CYGWIN__): Sorry, no Unix Domain sockets on Windows MSCRT.

Linking the compiled files produces an undefined reference to 'GetNetworkParams@8', which is part of the Windows GetNetworkParams interface in iphlpapi.dll, as listed on build.io together with ws2_32. It seems that after my exploration of building addons with native dependencies like ReadLine or Markdown, compiling Socket as difficult than that. Here are the important pieces of the steps:

> ...
> cd source
> gcc -fPIC -D _NO_OLDNAMES -D IO_WINSOCK_COMPAT -c ...
> gcc -liovmall -lbasekit -levent -liphlpapi -lws2_32 ^
      -shared -Wl,--out-implib,libIoSocket.dll.a ^
      ... -o libIoSocket.dll
> ...

My fork of Socket contains the required fixes for Windows.

Using the Socket addon on current Ubuntu
In my product development Coderetreat template, I have GitHub actions to verify the sample code. Based on Sandro's installation instructions I first install the required dependencies:

sudo apt-get install libpcre3-dev libevent-dev

(This might be unecessary as GitHub's Ubuntu image has both of them installed.) The current branch of libevent is 2.1.x and the none of the older 2.0.x versions do compile due to wrong dependencies. At the same time, the Linux version of Io contains Socket but needs libevent 2.0.5 specifically. I have no idea if that is the way to fix these kind of issues, but it works, so I link the installed version (2.1.7) as the required (2.0.5) one.

ls -la /usr/lib/x86_64-linux-gnu/libevent*.so.*
sudo ln -s /usr/lib/x86_64-linux-gnu/libevent-2.1.so.7.0.1 /usr/lib/x86_64-linux-gnu/libevent-2.0.so.5

(I leave the ls is the action to show the current version when the GitHub runner changes and there is a newer version of libevent.) After that I download the "latest" version of Io and install it:

wget -q http://iobin.suspended-chord.info/linux/iobin-linux-x64-deb-current.zip
unzip -qq iobin-linux-x64-deb-current.zip
sudo dpkg -i IoLanguage-2013.11.04-Linux-x64.deb
sudo ldconfig
io --version

The Linux version of Io comes with almost all addons pre-compiled, and there is no need for any compilation. Success. Because this version is without Eerie, custom addons have to be installed manually into the System installPrefix folder, e.g. adding Docio

git clone https://github.com/codecop/Docio.git
io -e "System installPrefix println"
sudo mv Docio /usr/local/lib/io/addons/
io -e "Docio println"

Now the runner is ready to execute some tests which is usually done with io ./tests/correctness/run.io. The full GitHub action for Io is here.

Random Facts about Addons
During my exploration I learned more things about Io's addon structure:

• The protos file does not need to contain the name of the "main" prototype, which is also the name of the addon, and often it does not. I put the name of all (exported) prototypes there to simplify my scripts. Then package.json, the Eerie manifest, does contain all prototypes.
• In the beginning I though depends was some kind of manifest, listing dependencies. But it is only needed for dependencies of native code, so AddonLoader loads dependent addons before they are used in native code. For non native addons, Io will load whatever is needed when parsing any unknown type name. Till now the only addon I have seen which needs that feature is Regex.
• When an addon is loaded all its files are evaluated. Only then is it registered as active. This can lead to addon initialisation loops. The initialisation order seems to be by file name. Some addons with complex inter dependencies - like Socket - prefix Io files with A_0, A_1 and so on to ensure ordered initialisation. (This is a bit annoying for tooling as the prototype name usually equals the file name in Io.)

Summary: My Feelings Towards Io
While Io is dead since more than ten years, working with it feels bleeding edge, even more it is outside the frontier. You are at your own, there is no help. Nobody is coming to save you. The latest article I found was written in 2019. There are less than ten (!) active blog posts: Blame it on Io (2006), Io language on Windows (2014), Io Basics (2015) and Io Programming Language (2019) - to list the best ones. There are a handful of Stack Overflow questions and a few repositories on GitHub - which are sometimes incompatible with the "latest" Io. ChatGPT understands the language but fantasises the libraries, so no help from AI neither. I am used to modern languages with a rich ecosystem, e.g. Java, C#, Python and this is an unfamiliar feeling. At the same time it is a refreshing puzzle. Maybe I will come back for a vacation in uncharted territory.