Unit Testing Scheme

For more than four years this article has been waiting to be written. Now a rainy summer day can do wonders. Back in 2018, when I fell into Scheme, I needed a unit testing framework. I was digging into Scheme with coding exercises and wanted the quick feedback of my TDD cycle so see if I was on the right track. I did not know anything about modules, dependency management or how to get custom code in Scheme and searched the web for a small test framework. As minimalism is in the spirit of Scheme, I initially went with the Scheme Unit outlined on Cunningham's Wiki,

;
; Unit test framework for scheme
; see http://c2.com/cgi/wiki?SchemeUnit
;
(define (report-error msg)
    (display "ERROR: ")
    (display msg)
    (newline))
    
(define (assert msg b)
    (if (not b)
        (report-error msg)))

Evolution
Now that was minimalist indeed. But as good as it was, it lacked important features: I wanted more assertions, at least for all basic data types, and needed to see some colours. What is the point of a green bar, if it is not shown in green. I had created my own xUnit libraries before, one for good old Turbo Pascal and one for assembly (written in assembly - usually you would use C). While both were useful, I did not get famous for them, it seems that am focusing on "niche products" in the testing framework world. Creating my own xUnit was a kata in itself and always helped me while learning a new programming language.

I started adding more assertions already in 2015 and test cases in 2017. I kept extending it whenever I needed something new, always copying the latest file to the next code kata. In 2018 I decided to put my "SchemeUnit" up on GitHub under the name of assert-scm. Now my tests, for example the tests for the Parrot kata, look like

(include "assert.scm")
(include "parrot.scm")

(test-case "it gets the speed of an european parrot"
    (assert= 12.0
             (parrot-speed 'european-parrot 0 0.0 #f)))

(test-case "it gets the speed of an african parrot with one coconut"
    (assert= 3.0
             (parrot-speed 'african-parrot 1 0.0 #f)))

Features of xUnit
Which features are expected from a true xUnit framework? From my knowledge of JUnit, I derive the core elements of xUnit:

• Assertions: First we need assertions. These are typically called assertSomething. Assertions are necessary to verify the actual result versus the expected one. If these values are not equal, the assertion should fail in some way. There should be assertions for equality of basic data types. In my Scheme xUnit there are (assert-true actual) and assert-false, assert= for integer numbers and symbols (and everything you can compare with = in Scheme), assert-char= and assert-string= for these primitives and assert-inexact= for floating point numbers which allows a delta for rounding errors. There are assert-null, assert-not-null, and more. As lists are the basic, all encompassing data structure in Lisp, and therefore Scheme, any testing framework for these languages needs support for comparing lists for equality: assert-list= and assert-list-deep= for flat and deep list comparison.
  
  
• Failure Messages: Assertions need to fail with descriptive messages. For example, if two values expected and actual are not equal, I would like to see "expected: <expected value> but was: <actual value>". I hate testing frameworks which just stop with "Assertion failed." Creating good messages gets more interesting when comparing lists as they can be of different length and nested. After assertions, this is the second important thing to have.
  
  
• Test Cases: In xUnit, test cases are often a bit weird, as they are classes containing multiple methods. Each of these methods is an individual test case because during test execution the class is instantiated for each method. In some frameworks test methods are named testSomething(), or annotations or other markers are used. In frameworks without classes, e.g. Jest or Pytest, each test function is a test case. A test case has a name, often the name of the method, some arrange code, some logic and one or more assertions. Test cases should be run independently of each other and report success or failure individually.

  (test-case "(test-case) allows several assertions"
      (assert-true #t)
      (assert-true #t))

  will print (test-case) allows several assertions .. OK.
  
  
• Ignoring Test Cases: Sometimes I want to ignore a certain test case. Most frameworks offer ways to do that, e.g. @Ignore, @Disabled, @mark.skip or using other markers. I like the Mocha way of replacing it('') with xit('') and went for a different function ignored-test-case:

  (ignored-test-case "(ignored-test-case) is ignored, else it would fail"
      (assert-true #f))

• Test Suites: Test suites are used to group test cases. Naturally these are Java classes, Python modules or Jest/Mocha describe blocks containing test methods. In Scheme 5 that would be files. Files can include other files which allows me to build arbitrary test suites. I rarely use test suites in any language, as I am running all tests most of the time.
  
  
• Fixtures: Fixtures contain the necessary creation and release of resources needed for the test and make sure these are released even if the test failed. Older test frameworks allow setup and teardown methods or @Before/@After markers. Other approaches include injecting necessary dependencies, as for example JUnit 5 and Pytest do. Till now I did not need fixtures in my exercises. In small test sets, I am fine when tests stop at the first failing test.
  
  
• Asserting on Exceptions: Few testing frameworks offer assertions for exceptions. For example in Java, before JUnit 5's assertThrows, there were 5+ ways to test that a method threw an exception. Maybe this is a special case, something that is rarely used. As I was building my assert-scm Scheme xUnit from scratch, I wanted to be sure the assertions work. How would I test for a failing assertion? I had to dig deeper into Scheme. Standard R5RS Scheme has no function to catch exceptions and different implementations handle this differently. Gambit Scheme, the Scheme I started, offers some proprietary extension for exceptions, whereas Chicken Scheme, another common Scheme, has some support for handling exceptions - called conditions. At least Chicken version 4 does, but it is outdated now. Portability is an issue between different Scheme implementations. It seems Scheme R6RS has standard support for exceptions, but its structure is way more complicated. I would like to avoid this for fun exercises.

If you are interested in features of xUnit and how to use them I recommend you work through my Unit Testing-Koans exercises.

Prime Factors
Exploring a new language is incomplete for me, until there is a nice TDD, step by step implementation of the Prime Factors kata. It is my goto exercise and I am collecting my Prime Factors solutions - which usually look the same. Here are the tests:

(include "prime-factors.scm")
(include "../assert-r5rs.scm")

(test-case "one"
    (assert-null (prime-factors 1)))

(test-case "two"
    (assert-number-list= (list 2) (prime-factors 2)))

; ...

(test-case "nine"
    (assert-number-list= (list 3 3) (prime-factors 9)))

(test-case "max"
    (assert-number-list= (list 2147483647) (prime-factors 2147483647)))

which execute in assert-scm's GitHub build,


and verify the code

(define (prime-factors n)
    (test-candidate n 2))

(define (test-candidate n candidate)
    (cond ((= n 1) (list))
          ((too-large? n candidate) (prime n))
          ((divides? n candidate) (keep-candidate n candidate))
          (else (next-candidate n candidate))))

(define (too-large? n candidate)
    (> candidate (sqrt n)))

(define (prime n)
    (list n))

(define (divides? n candidate)
    (= (modulo n candidate) 0))

(define (keep-candidate n candidate)
    (append (prime candidate)
            (test-candidate (/ n candidate) candidate)))

(define (next-candidate n candidate)
    (test-candidate n (+ candidate 1)))

This is neat, isn't it? If you fancy playing with Scheme and do not want to miss unit tests, check out assert-scm, a minimalist unit test framework for Scheme R5RS.

How to Unit-Test Assembly

After my last trip into Assembly I got the feedback that I should have used TDD, especially as it was supposed to be a code kata. (Thank you Emmanuel Gaillot for reminding me of my duties.) And yes, I should have. But as I said, I could not find any real unit testing support, at least not without resorting to C or C++ unit testing frameworks. On the other hand, creating an xUnit implementation is a great exercise to get to know a language. So I decided to create my own, minimal unit testing framework for Windows IA-32 Assembly using NASM. (The following code uses stdcall call convention because it is used in the Microsoft Win32 API anyway.)

Failure
So what are the most required features for unit testing support? I believe the most important feature is to mark a test as failed. I also want to stop executing this test in case of failure, like JUnit does. This is the fail() method,

%macro fail 0
        log msg_failed, msg_failed_len

        leave
        ret 0 ; test method never has any arguments
%endmacro

msg_failed:     db 'FAILED'
msg_failed_len  equ $ - msg_failed

In fact it is a NASM macro which gets copied into each invocation. For simplicity, it does not support a message argument right now, hence zero macro parameters. It uses the log macro, which prints the given string to Standard Out, just like _printGrid of my Assembly Minesweeper. Then it resets the stack frame (leave) and returns from the current function. I expect all test methods to have no arguments, and fail's code gets copied into each test method, this skips further execution of the test method and returns immediately to the calling code, i.e. the test runner. Adding the count of failed tests in fail is straight forward but I will leave that for later.

Assertion
Next I need assertions to check my expectations, at least an assert_equals,

%macro assert_equals 2
        cmp     %1, %2
        je      .%1_%2_end
        fail
.%1_%2_end:
        call    _show_progress
%endmacro

This has to be a macro as well so fail works correctly as explained above. But as the macro gets expanded into the calling code, the local label .%1_%2_end has to be unique. While different assertions are possible, a specific assertion like assert_equals eax, 2 can only be used once per test method. This is a weird constraint but I do not mind as my tests tend to have a single assertion per test method anyway. (Later I changed the macro to use a macro local label %%_end which removed this problem.)

The function _show_progress just prints a single dot to show the progress during test execution. It needs to preserve all registers because it is called in the middle of my test code.

_show_progress:
        push    eax
        push    ebx
        push    ecx
        push    edx

        push    dot_len
        mov     eax, dot
        push    eax
        call    _print

        pop     edx
        pop     ecx
        pop     ebx
        pop     eax

        ret

dot:    db      '.'
dot_len equ     1

Test Cases
For my test cases I want descriptive names. Using long, regular labels for function names and the assertion macros I write my first test

_should_add_one_and_one_to_be_two:
        create_local_variables 0

        mov     eax, 1
        add     eax, 1
        assert_equals eax, 2

        leave
        ret

The test succeeds and prints a single dot to Standard Out. The create_local_variables macro creates the stack frame and local variables if needed. If a test fails, e.g.

_should_add_one_and_two_to_be_three:
        create_local_variables 0

        mov     eax, 1
        add     eax, 2
        assert_equals eax, 2 ; (6)

        ; not reached
        hlt

        leave
        ret

it prints FAILED and stops execution in line 6.

Test Runner
Finally I want to run all my tests one by one. In this simple case I just call the test methods from the main entry point, followed by printing DONE at the end.

global  _main

_main:

        ; welcome message
        log     msg_hello, msg_hello_end - msg_hello

        call    _should_add_one_and_one_to_be_two
        call    _should_add_one_and_two_to_be_three
        ; (10)

        ; completion message
        log     msg_done, msg_done_end - msg_done

        jmp     _exit

msg_hello:      db 'HELLO asmUnit'
msg_hello_end:

msg_done:       db 'DONE'
msg_done_end:

This gets the job done but I am not happy with it. I need to add new test method invocations by hand in line 10. By seeing the new test fail I cannot forget to add the call of the new test method, but automatic test case discovery would be more convenient.

Before and After
What else does a unit testing framework need? For complex cases, before and after test execution hooks might be necessary. A before macro defines the proper label and the begin_test macro just calls it.

%macro before 0-1 0
        %ifdef ctx_before
                %error "before used more than once"
        %endif

        %define ctx_before, 1
_before_hook:
        create_local_variables %1
%endmacro

%macro begin_test 0-1 0
        create_local_variables %1
        %ifdef ctx_before
                call _before_hook
        %endif
%endmacro

The after and end_test macros work accordingly, just for the test clean-up. The actual test code stays the same, the macro calls just changed.

_should_add_one_and_one_to_be_two:
        begin_test

        mov     eax, 1
        add     eax, 1
        assert_equals eax, 2

        end_test

While this works well, I am unhappy with all the (NASM specific) macros in the code. There remains little real Assembly, but I guess that is the way to go. There are also plenty of missing features but this minimal first version gets me started. I will see how it works out in my next Assembly kata. (The complete source of asmUnit is here.)

Turbo Pascal Prime Factors Kata

Recently I started performing the Prime Factors Kata. After playing with Java and Ruby I had the weird idea of performing it in every programming language I ever knew. Well, maybe not every - I don't plan to use 6502 or 80x86 assembler, that wouldn't be fun. But I already did it in BASIC. Going forward in time Turbo Pascal would be next. So it's time for another retro post ;-)

Where Is My TPUnit?
I couldn't find any unit testing framework for Turbo Pascal. The closest thing I could find was FPCUnit packaged with Free Pascal. Unfortunately it's not compatible with good old TP. So I had to roll my own. I started with some minimalist infrastructure.

TYPE TestCase = OBJECT
    PROCEDURE AssertEquals(msg:String; expect, act:Longint);
    PROCEDURE AssertNil(msg:String; act:Pointer);
    { other asserts ... }
    PROCEDURE Fail(msg:String);
    { TestCase }
    PROCEDURE SetUp; VIRTUAL;
    PROCEDURE TearDown; VIRTUAL;
  END;

PROCEDURE TestCase.AssertEquals(msg:String; expect, act:Longint);
VAR ex, ac:String;
BEGIN
  IF expect <> act THEN
  BEGIN
    Str(expect, ex);
    Str(act, ac);
    Fail(Concat(msg,' expected ',ex,' but was ',ac));
  END;
END;

...

PROCEDURE TestCase.Fail(msg:String);
BEGIN
  TearDown;
  Writeln(' - FAILED');
  Writeln(msg);
  Halt(1);
END;

PROCEDURE TestCase.SetUp;
BEGIN
END;

PROCEDURE TestCase.TearDown;
BEGIN
END;

Subclasses may overwrite SetUp or TearDown, add test methods and call Asserts. To keep it simple the first failed assertion stops program execution. What's missing is some kind of procedure RunTest that would wrap a particular test method inside calls to SetUp and TearDown. Hmm - function variables might be handy here. In case you are not familiar with them, here is an example:

{$F+} {needs far calls for function variables}
TYPE FuncVar = PROCEDURE;

PROCEDURE FancyMethod(method:FuncVar);
BEGIN
  method; { invokes the method }
END;

PROCEDURE SomeMethod;
...

VAR v:FuncVar;
BEGIN
  v := SomeMethod;
  FancyMethod(v);
END.

Unfortunately Turbo Pascal does not allow class methods (e.g. TestCase.AssertEquals) to be used as function variables. (At least I couldn't figure.) Obviously self is an implicit parameter of all such methods. Well not obviously, but analysing the generated machine code helps ;-)

SomeMethod; { -> 0E E8 D1 FB }
push CS { because of $F+ }
call fbe1 { address of SomeMethod in CS }

cls.ClassProc; { -> BF 70 00 1E 57 0E E8 ED FB }
mov DI, #0070 { address of object cls in DS }
push DS
push DI { first parameter is self }
push CS
call fbed { address of ClassProc in CS }

Using this knowledge TestCase.RunTest is implemented a bit dirty using an untyped Pointer argument:

PROCEDURE CallClassPtr(pt:Pointer; VAR cls:TestCase);
VAR s,o:Word;
BEGIN
  s := Seg(cls);
  o := Ofs(cls);
  ASM
    mov DI, [o]
    mov AX, [s]
    push AX
    push DI
    call [pt.dword]
  END;
END;

PROCEDURE TestCase.RunTest(name:String; testMethod:Pointer);
BEGIN
  Write('TEST ', name);
  SetUp;
  CallClassPtr(testMethod, self);
  TearDown;
END;

The Prime Factors Kata
Having a simple TPUnit in place, it's time for the kata itself. The seven test methods of PrimeFactorsTest,

PROCEDURE PrimeFactorsTest.Run;
BEGIN
  RunTest('TestOne', @PrimeFactorsTest.TestOne );
  RunTest('TestTwo', @PrimeFactorsTest.TestTwo );
  RunTest('TestThree', @PrimeFactorsTest.TestThree );
  RunTest('TestFour', @PrimeFactorsTest.TestFour );
  RunTest('TestSix', @PrimeFactorsTest.TestSix );
  RunTest('TestEight', @PrimeFactorsTest.TestEight );
  RunTest('TestNine', @PrimeFactorsTest.TestNine );
END;

yield

FUNCTION TPrimeFactors.generate(i:Longint):ArrayListPtr;
VAR factors:ArrayListPtr;
    candidate:Longint;
BEGIN
  factors := new(ArrayListPtr, Init);

  FOR candidate := 2 TO i DO
  BEGIN
    WHILE i MOD candidate = 0 DO
    BEGIN
      factors^.Add(candidate);
      i := i DIV candidate;
    END;
  END;

  generate := factors;
END;

ArrayListPtr is a pointer to a variable sized, user defined list backed by an array of Longints, similar to Java's ArrayList<Integer>. I can't deny I'm a Java guy. Everything I code looks like Java :-) (Probably I would have used a linked list back then instead of a complex object. Something like

TYPE PrimeFactorPtr = ^PrimeFactor;
  PrimeFactor = RECORD
    value:LongInt;
    next:PrimeFactorPtr;
  END;

Still the procedure body looks the same and the kata does not change much.)

(Download full source)