Unit Testing Scheme

For more than four years this article has been waiting to be written. Now a rainy summer day can do wonders. Back in 2018, when I fell into Scheme, I needed a unit testing framework. I was digging into Scheme with coding exercises and wanted the quick feedback of my TDD cycle so see if I was on the right track. I did not know anything about modules, dependency management or how to get custom code in Scheme and searched the web for a small test framework. As minimalism is in the spirit of Scheme, I initially went with the Scheme Unit outlined on Cunningham's Wiki,

;
; Unit test framework for scheme
; see http://c2.com/cgi/wiki?SchemeUnit
;
(define (report-error msg)
    (display "ERROR: ")
    (display msg)
    (newline))
    
(define (assert msg b)
    (if (not b)
        (report-error msg)))

Evolution
Now that was minimalist indeed. But as good as it was, it lacked important features: I wanted more assertions, at least for all basic data types, and needed to see some colours. What is the point of a green bar, if it is not shown in green. I had created my own xUnit libraries before, one for good old Turbo Pascal and one for assembly (written in assembly - usually you would use C). While both were useful, I did not get famous for them, it seems that am focusing on "niche products" in the testing framework world. Creating my own xUnit was a kata in itself and always helped me while learning a new programming language.

I started adding more assertions already in 2015 and test cases in 2017. I kept extending it whenever I needed something new, always copying the latest file to the next code kata. In 2018 I decided to put my "SchemeUnit" up on GitHub under the name of assert-scm. Now my tests, for example the tests for the Parrot kata, look like

(include "assert.scm")
(include "parrot.scm")

(test-case "it gets the speed of an european parrot"
    (assert= 12.0
             (parrot-speed 'european-parrot 0 0.0 #f)))

(test-case "it gets the speed of an african parrot with one coconut"
    (assert= 3.0
             (parrot-speed 'african-parrot 1 0.0 #f)))

Features of xUnit
Which features are expected from a true xUnit framework? From my knowledge of JUnit, I derive the core elements of xUnit:

• Assertions: First we need assertions. These are typically called assertSomething. Assertions are necessary to verify the actual result versus the expected one. If these values are not equal, the assertion should fail in some way. There should be assertions for equality of basic data types. In my Scheme xUnit there are (assert-true actual) and assert-false, assert= for integer numbers and symbols (and everything you can compare with = in Scheme), assert-char= and assert-string= for these primitives and assert-inexact= for floating point numbers which allows a delta for rounding errors. There are assert-null, assert-not-null, and more. As lists are the basic, all encompassing data structure in Lisp, and therefore Scheme, any testing framework for these languages needs support for comparing lists for equality: assert-list= and assert-list-deep= for flat and deep list comparison.
  
  
• Failure Messages: Assertions need to fail with descriptive messages. For example, if two values expected and actual are not equal, I would like to see "expected: <expected value> but was: <actual value>". I hate testing frameworks which just stop with "Assertion failed." Creating good messages gets more interesting when comparing lists as they can be of different length and nested. After assertions, this is the second important thing to have.
  
  
• Test Cases: In xUnit, test cases are often a bit weird, as they are classes containing multiple methods. Each of these methods is an individual test case because during test execution the class is instantiated for each method. In some frameworks test methods are named testSomething(), or annotations or other markers are used. In frameworks without classes, e.g. Jest or Pytest, each test function is a test case. A test case has a name, often the name of the method, some arrange code, some logic and one or more assertions. Test cases should be run independently of each other and report success or failure individually.

  (test-case "(test-case) allows several assertions"
      (assert-true #t)
      (assert-true #t))

  will print (test-case) allows several assertions .. OK.
  
  
• Ignoring Test Cases: Sometimes I want to ignore a certain test case. Most frameworks offer ways to do that, e.g. @Ignore, @Disabled, @mark.skip or using other markers. I like the Mocha way of replacing it('') with xit('') and went for a different function ignored-test-case:

  (ignored-test-case "(ignored-test-case) is ignored, else it would fail"
      (assert-true #f))

• Test Suites: Test suites are used to group test cases. Naturally these are Java classes, Python modules or Jest/Mocha describe blocks containing test methods. In Scheme 5 that would be files. Files can include other files which allows me to build arbitrary test suites. I rarely use test suites in any language, as I am running all tests most of the time.
  
  
• Fixtures: Fixtures contain the necessary creation and release of resources needed for the test and make sure these are released even if the test failed. Older test frameworks allow setup and teardown methods or @Before/@After markers. Other approaches include injecting necessary dependencies, as for example JUnit 5 and Pytest do. Till now I did not need fixtures in my exercises. In small test sets, I am fine when tests stop at the first failing test.
  
  
• Asserting on Exceptions: Few testing frameworks offer assertions for exceptions. For example in Java, before JUnit 5's assertThrows, there were 5+ ways to test that a method threw an exception. Maybe this is a special case, something that is rarely used. As I was building my assert-scm Scheme xUnit from scratch, I wanted to be sure the assertions work. How would I test for a failing assertion? I had to dig deeper into Scheme. Standard R5RS Scheme has no function to catch exceptions and different implementations handle this differently. Gambit Scheme, the Scheme I started, offers some proprietary extension for exceptions, whereas Chicken Scheme, another common Scheme, has some support for handling exceptions - called conditions. At least Chicken version 4 does, but it is outdated now. Portability is an issue between different Scheme implementations. It seems Scheme R6RS has standard support for exceptions, but its structure is way more complicated. I would like to avoid this for fun exercises.

If you are interested in features of xUnit and how to use them I recommend you work through my Unit Testing-Koans exercises.

Prime Factors
Exploring a new language is incomplete for me, until there is a nice TDD, step by step implementation of the Prime Factors kata. It is my goto exercise and I am collecting my Prime Factors solutions - which usually look the same. Here are the tests:

(include "prime-factors.scm")
(include "../assert-r5rs.scm")

(test-case "one"
    (assert-null (prime-factors 1)))

(test-case "two"
    (assert-number-list= (list 2) (prime-factors 2)))

; ...

(test-case "nine"
    (assert-number-list= (list 3 3) (prime-factors 9)))

(test-case "max"
    (assert-number-list= (list 2147483647) (prime-factors 2147483647)))

which execute in assert-scm's GitHub build,


and verify the code

(define (prime-factors n)
    (test-candidate n 2))

(define (test-candidate n candidate)
    (cond ((= n 1) (list))
          ((too-large? n candidate) (prime n))
          ((divides? n candidate) (keep-candidate n candidate))
          (else (next-candidate n candidate))))

(define (too-large? n candidate)
    (> candidate (sqrt n)))

(define (prime n)
    (list n))

(define (divides? n candidate)
    (= (modulo n candidate) 0))

(define (keep-candidate n candidate)
    (append (prime candidate)
            (test-candidate (/ n candidate) candidate)))

(define (next-candidate n candidate)
    (test-candidate n (+ candidate 1)))

This is neat, isn't it? If you fancy playing with Scheme and do not want to miss unit tests, check out assert-scm, a minimalist unit test framework for Scheme R5RS.