How to Unit-Test Assembly

After my last trip into Assembly I got the feedback that I should have used TDD, especially as it was supposed to be a code kata. (Thank you Emmanuel Gaillot for reminding me of my duties.) And yes, I should have. But as I said, I could not find any real unit testing support, at least not without resorting to C or C++ unit testing frameworks. On the other hand, creating an xUnit implementation is a great exercise to get to know a language. So I decided to create my own, minimal unit testing framework for Windows IA-32 Assembly using NASM. (The following code uses stdcall call convention because it is used in the Microsoft Win32 API anyway.)

Failure
So what are the most required features for unit testing support? I believe the most important feature is to mark a test as failed. I also want to stop executing this test in case of failure, like JUnit does. This is the fail() method,

%macro fail 0
        log msg_failed, msg_failed_len

        leave
        ret 0 ; test method never has any arguments
%endmacro

msg_failed:     db 'FAILED'
msg_failed_len  equ $ - msg_failed

In fact it is a NASM macro which gets copied into each invocation. For simplicity, it does not support a message argument right now, hence zero macro parameters. It uses the log macro, which prints the given string to Standard Out, just like _printGrid of my Assembly Minesweeper. Then it resets the stack frame (leave) and returns from the current function. I expect all test methods to have no arguments, and fail's code gets copied into each test method, this skips further execution of the test method and returns immediately to the calling code, i.e. the test runner. Adding the count of failed tests in fail is straight forward but I will leave that for later.

Assertion
Next I need assertions to check my expectations, at least an assert_equals,

%macro assert_equals 2
        cmp     %1, %2
        je      .%1_%2_end
        fail
.%1_%2_end:
        call    _show_progress
%endmacro

This has to be a macro as well so fail works correctly as explained above. But as the macro gets expanded into the calling code, the local label .%1_%2_end has to be unique. While different assertions are possible, a specific assertion like assert_equals eax, 2 can only be used once per test method. This is a weird constraint but I do not mind as my tests tend to have a single assertion per test method anyway. (Later I changed the macro to use a macro local label %%_end which removed this problem.)

The function _show_progress just prints a single dot to show the progress during test execution. It needs to preserve all registers because it is called in the middle of my test code.

_show_progress:
        push    eax
        push    ebx
        push    ecx
        push    edx

        push    dot_len
        mov     eax, dot
        push    eax
        call    _print

        pop     edx
        pop     ecx
        pop     ebx
        pop     eax

        ret

dot:    db      '.'
dot_len equ     1

Test Cases
For my test cases I want descriptive names. Using long, regular labels for function names and the assertion macros I write my first test

_should_add_one_and_one_to_be_two:
        create_local_variables 0

        mov     eax, 1
        add     eax, 1
        assert_equals eax, 2

        leave
        ret

The test succeeds and prints a single dot to Standard Out. The create_local_variables macro creates the stack frame and local variables if needed. If a test fails, e.g.

_should_add_one_and_two_to_be_three:
        create_local_variables 0

        mov     eax, 1
        add     eax, 2
        assert_equals eax, 2 ; (6)

        ; not reached
        hlt

        leave
        ret

it prints FAILED and stops execution in line 6.

Test Runner
Finally I want to run all my tests one by one. In this simple case I just call the test methods from the main entry point, followed by printing DONE at the end.

global  _main

_main:

        ; welcome message
        log     msg_hello, msg_hello_end - msg_hello

        call    _should_add_one_and_one_to_be_two
        call    _should_add_one_and_two_to_be_three
        ; (10)

        ; completion message
        log     msg_done, msg_done_end - msg_done

        jmp     _exit

msg_hello:      db 'HELLO asmUnit'
msg_hello_end:

msg_done:       db 'DONE'
msg_done_end:

This gets the job done but I am not happy with it. I need to add new test method invocations by hand in line 10. By seeing the new test fail I cannot forget to add the call of the new test method, but automatic test case discovery would be more convenient.

Before and After
What else does a unit testing framework need? For complex cases, before and after test execution hooks might be necessary. A before macro defines the proper label and the begin_test macro just calls it.

%macro before 0-1 0
        %ifdef ctx_before
                %error "before used more than once"
        %endif

        %define ctx_before, 1
_before_hook:
        create_local_variables %1
%endmacro

%macro begin_test 0-1 0
        create_local_variables %1
        %ifdef ctx_before
                call _before_hook
        %endif
%endmacro

The after and end_test macros work accordingly, just for the test clean-up. The actual test code stays the same, the macro calls just changed.

_should_add_one_and_one_to_be_two:
        begin_test

        mov     eax, 1
        add     eax, 1
        assert_equals eax, 2

        end_test

While this works well, I am unhappy with all the (NASM specific) macros in the code. There remains little real Assembly, but I guess that is the way to go. There are also plenty of missing features but this minimal first version gets me started. I will see how it works out in my next Assembly kata. (The complete source of asmUnit is here.)