Empowering Testers

Developers need to be able to test
I strongly believe that we (software developers) have to be able to test our work. How else can we make sure that "it works". At the minimum we write unit tests for the code we just created. The whole idea of Test Driven Development is based on finding, prioritising and running unit (and sometimes acceptance) tests against our code. Even people that do not like TDD, put a strong focus on automated tests. Paul Gerrard recently put it nicely as "if they can't test they aren't developers". Unfortunately not all developers are able to do so. I often meet young developers who have no experience in deriving test cases from requirements or finding more than the most obvious test scenarios at all. They write tests because they want to make sure their program work, but their tests are not focused, testing too much or nothing at all. More skill and experience is needed.

Need testers be able to code?
When developers need to be able to test, do testers need to be able to code? This is an old debate. As hard-core coder I believe that testers need to be able to code. Period. For example at one my of previous employers, the testers were asked to automate their tests using Rational Function Tester. They had to write Java code to drive the test engine. And the code they wrote was abysmal. I do not blame them, nobody had taught them how to do it. But the code they wrote was important for the project's success, so it better had some quality at least. What was needed were Test Automation Engineers. As the test automation code is much simpler than the regular code, there is no need for explicit conditionals or loops in test cases, I concede that these testers (the engineers) do not have to know as much about coding as developers do. They do not have to know Design Patterns or SOLID principles, but they need at least to know naming, duplication and to a certain extend, abstraction.

Manual testing is immoral
Uncle Bob has explained years ago why manual testing is a bad thing. Probably he meant that manual checking is immoral, because the testing community likes to distinguish between aspects of the testing process that machines can do (checking) versus those that only skilled humans can do (testing). As there are more than 130 types of software testing, many of them can only be done by a skilled human, e.g. exploratory testing. I do not know how many testers do these kind of interesting things, but most of what I see is either (immoral) manual checking or test automation engineering, which is kind of coding.

Austrian Testing Board
Earlier this year I was invited to the Austrian Testing Board to give a presentation about Deliberate Practice. While Deliberate Practice is popular with developers, e.g. Coding Dojos or Code Retreats, it can be applied to all kind of learning. For example there are Testing Dojos and Testautomation (Code) Retreats. My presentation was received well and I enjoyed a heated discussions afterwards because - of course - I had provoked the "testers" a bit ;-) One of the attendees, Gerd Weishaar, was surprised to hear about my approach: Doing TDD, automating all tests (because "immorality") and running only a few integrated tests (because Test Pyramid and Scam). I also told him about my hate for UI test automation tools because of their brittleness. As the VP Product Management at Tricentis, a company focusing on software (end-to-end) test automation, he had a very different view than mine: Almost no developers writing unit tests, groups of testers who work offshore, doing manual testing. We immediately became friends. ;-)

Empowering Testers
Gerd's vision was to enable the testers who are not able to code, empowering them to do their job - to get something tested - without any technical knowledge. Despite our contradicting views I got curious: Would it be possible to test for example a web application without any technical knowledge? How much technical background is necessary to understand the application at all? How much automation can be achieved without being able to write a script? And most interestingly, how much support can be put into a testing tool, just by making it usable for the testers. How far can we go?

Tricentis Tosca Testsuite
The tool Gerd was talking about was Tosca, a functional end-to-end testing tool. I had heard about it but never used it myself. Gerd encouraged me to have a look at it and provided me with online material, but I failed to allocate time for that. In the end he invited me to one of their monthly instructor-led training, free of charge. Although I had never planned to dedicate three full days (!) to checking out Tosca, I accepted his invitation, because I find well prepared instructor-led training an efficient way to get started with a new topic. The training was tough, three full days working the tool on various exercises. The trainer and the other participants knew a lot about testing and we had great discussions. (If you need to work with Tosca on your job I strongly encourage you to take their instructor-led training. Afterwards you will be productive using the Tosca Testsuite.)

Small Disclaimer
I attended the three day training free of charge. Gerd asked for feedback and a blog post about it in return. This is this blog post. I liked the trainer and in the end I liked Tosca, so I am positively biassed. Still all the open questions from the beginning of the article remain.

Training
I had two goals for the training: First I wanted to learn more about the mind-set of "classic testers" and second I wanted to check Gerd's idea of empowering non-technical/non-coder testers. Tosca is very powerful. It includes tools for project planning, risk management, test case design as well as manual and automated testing. It took me three days to get an idea what the tool is capable of. I do not know how much more time one would need without the development background that I have. I am used to such kind of tools, after all Eclipse or IDEA are not easy to use either. So training is necessary to use Tosca. Maybe three days are enough to explain it to someone who has no idea about coding, maybe five days are needed. Anything less than ten years will do. As the tool only offers a fixed limit of combinations, it is possible to get familiar with it fast.

Context Sensitivity
Tosca is a professional tool. The user interface is well done and it supports power users by providing tab ordering and consistent keyboard short-cuts. Unfortunately some of its many options and functions are really hidden deep in dialogues or require you to type in certain text combinations. This is not usable in the general sense. I guess developers are used to writing text, after all that is the code we write day after day. Still we expect context sensitive help and code completion, something that non-coders need even more. So I guess there needs some work to be done in a few places, but that can be fixed easily, and Gerd told me they are planning to improve this.

Tosca Test Case Design
The coolest things of Tosca is its Test Case Design functionality. It enables the formulation of abstract test cases, working with happy path, boundary values and negative cases, to create a conceptual model of all tests necessary to cover the given functionality. While this modelling does not involve any kind of coding related skills, it definitely requires skilled test professionals with analytical thinking and a high level of abstraction. As I consider these skills important coding skills, the test case design requires at least someone "like a coder".

Tosca Test Automation
While it is possible to make test case organisation or execution easy to use, the real problem is always the automation. I cannot imagine a non-coder being able to create reasonable test case automation. To hide complexity, Tosca's automated test cases are constructed from building blocks which hide the real automation code, e.g. pressing a button on a web page. It might be possible to create an automated test case just by arranging and configuring the existing automation steps in the right order, but usually that is not the case, because new automation steps have to be created on the fly. Also the test automation allows for conditionals, repetitions (loops) and templates (macros), which are typical coding constructs. If someone has to work with assignments, conditionals and repetitions, she is already a programmer.

How far can we go?
The underlying question is if it is possible to lower the requirements for technical work by providing smarter user interfaces that support the way we work? I hope so. Enabling people to get some tests automated without knowledge in coding is a good example for this. Even if we can make the tools just a bit more usable our users will feel the difference. Tosca contains some interesting approaches regarding this. Reducing complexity lets users work on a higher level of abstraction. Unfortunately abstractions are sometimes leaky and suddenly we are back in the low level world and have to type "magic" text into plain text fields. For now there are still areas in Tosca where you need to be a coder and you have to stick to software development principles to get good results. But I like Gerd's vision and I am curious how far they will go to empower non technical testers.

Introducing Brutal Coding Constraints

Last year I teamed up with Martin Klose to run a workshop at the Agile Testing Days. We knew that some really experienced developers would be there and aimed for an expert level workshop. We wanted a really difficult session, something that was hard, maybe even impossible to do. So we came up with the idea of Brutal Coding Constraints.

A constraint is an artificial challenge during an exercise designed to help participants think about writing code differently than they would otherwise. Some constraints are an exaggeration of fundamental rules of object oriented design and are applicable during your day to day work. The more extreme ones might still help you understand the underlying concepts of object orientation.

Brutal Coding Constraints
The Brutal Constraints constraint is a composite constraint like Object Calisthenics, a combination of several constraints, some of them already difficult enough on their own. One particular combination that we like is

1. No Conditionals, i.e. no if, unless or ?: operator. while can be used as conditional too and is not allowed as well.
   
2. No Loops, i.e. no for, while, do or repeat until or whatever repetition constructs your language offers. Together No Conditionals and No Loops are sometimes called Cyclomatic Complexity One.
   
3. TTDaiymi (TDD as if you Meant it), a very strict interpretation of the practice of TDD. This is optional, a "bonus" constraint for experienced developers. If you never heard about it, just ignore this one.
   
4. No Naked Primitives, i.e. wrapping all "primitive" values, e.g. booleans, numbers or strings. Also general purpose containers like List, Map or Set are considered primitive. In extension all generic types of your language are primitive because they are not from your domain. A generic date (e.g. java.util.Date) is not from your domain even if you use dates, because it either does not define all methods you need or it defines other methods you do not need.
   
5. No void, i.e. all functions must return something, methods with no return value are forbidden.
   
6. Immutable, i.e. all data and data-structures must be immutable.
   

All six constraints are regular Code Retreat Activities so I will skip their further discussion.

Violating the Rules
When practising these Brutal Constraints with some kata, Martin and I were not able to find an implementation that would satisfy all constraints right away. We usually allowed violations in the beginning and refactored towards the constraints after the green phase. Sometimes we would leave a violation in for a few red-green-refactor cycles. It helped us to go through the list in each refactoring step to make sure we did not forget any constraint. It happened that we had put a condition somewhere in the code and forget about it - we are just that used to using conditionals and loop constructs. Because participants often ask for it, here is a list when to be strict about the rules. It is allowed to violate constraints:

• temporary until you fix them during the next refactoring step;
• temporary until you fix them after triangulating a solution, probably during a larger refactoring step after several cycles, e.g. after the third test;
• if an used framework requires it, e.g. using a Runnable needs a void run() method;
• if the testing frameworks requires it, e.g. JUnit test methods are void methods and @Parameterized tests need List<Object[]> which is a primitive container.

In general it seems easier to refactor later but that defeats the purpose of the whole exercise. We ordered the six constraints by some kind of priority or difficulty. If you have to violate a constraint, try to follow the ones higher up in the list at least.

Difficulty
As I said before, Martin and I aimed for a really difficult exercise and I think we offered the only expert level session at ATD2014. And yes, this exercise was hard. The difficulty of each constraint was multiplied by their combination. For example I wrote about combining TDDaiymi, No Naked Primitives and No Conditionals last year. The attendees of the workshop agreed, "it was really difficult" up to "WTF" ;-). Brutal Coding Constraints are definitely too difficult for programming beginners, who even struggle with the concept of immutability.

An exercise like the Brutal Constraints can get frustrating easily. We told the participants that we made the session impossible on purpose, so they would not feel bad when getting stuck. When creating the workshop we got stuck ourselves several times, so we knew what to expect. During the session we paid close attention to the participants' mood and were prepared to offer hints on how to proceed without violating constraints. All participants worked hard and enjoyed the exercise.

Why practise like that?
When we prepared for the workshop we experimented with these constraints several times both in Java and JavaScript. It was difficult and interesting at the same time. I discussed some of our findings already. Also the ATD2014 participants liked the exercise. In the feedback round several people said that the constraints forced them to "think outside of the box" and that they liked the opportunity to "deviate from usual way" how they create software.

What about Functional Programming?
The Brutal Constraints focus on Object Oriented Programming. No Naked Primitives is the main driving force to create more types. On the other hand, constraints 1, 2, 5 and 6 might not challenge in Functional Programming. Instead of explicit conditionals some languages provide an Option or Maybe type and filter operations remove unwanted elements from containers. Most loops are unnecessary because containers provide map, foreach or similar operations. Also a recursive function call is not a loop. Pure functions and immutability are base concepts of functional programming anyway. I would like to see a solution following the above constraints in Clojure or Haskel. I am unsure how No Naked Primitives translates into Functional Programming.

The Assignment
In general the actual assignment, i.e. the problem that participants are asked to solve, does not matter for a kata but we wanted a problem that did not support the constraints. (Evil grin ;-) Such a problem would have a linear or higher dimensional structure with a need for looping (which is not allowed) and business rules, which are conditionals (which are not allowed either). We started with the classic Game of Life but it took us too long to reach a point where the constraints forced us to think harder. So we switched to a smaller problem, finding the winner of a game of Tic-tac-toe, which worked well for us.

Hints for Facilitators
If you plan to host your own Brutal Constraints exercise, your first priority is to help the participants to meet the constraints. It is easy to miss an if or a void method. We recommend printing the list of constraints and TDDaiymi rules as handout for each pair in the workshop. Second you need to pay attention to people's mood, as I described above.

For a short workshop it would help to force participants into situations conflicting with the Brutal Constraints as early as possible. One way to do this is to start with a prepared code base that already contains the first loop or conditional which needs to be removed. But then the list of supported programming languages for the workshop is small, ruling out less popular ones. Another way is to ask participants to follow a list of predefined test cases. While this allows any language, it impedes the creativity of solution finding. We still have to find a good list of test cases though.

Moar Brutality
But why should we stop here? We can make the exercise even more difficult. A suitable constraint to add is No Duplication, i.e. being very aggressive about duplication in the code. Unfortunately detecting duplication is less straight forward than following constraints 1, 2, 4, 5 and 6 which just deny certain reserved keywords or library classes. Another option is to add Baby Steps to force smaller working steps and Baby Steps has been combined with TDDaiymi already. When practising the constraints we committed every five to six minutes without problems.

Credits
Thanks to Martin Klose for creating the Brutal Coding Constraints with me. Pair facilitation is just so much more rewarding than solo work.

How to Unit-Test Assembly

After my last trip into Assembly I got the feedback that I should have used TDD, especially as it was supposed to be a code kata. (Thank you Emmanuel Gaillot for reminding me of my duties.) And yes, I should have. But as I said, I could not find any real unit testing support, at least not without resorting to C or C++ unit testing frameworks. On the other hand, creating an xUnit implementation is a great exercise to get to know a language. So I decided to create my own, minimal unit testing framework for Windows IA-32 Assembly using NASM. (The following code uses stdcall call convention because it is used in the Microsoft Win32 API anyway.)

Failure
So what are the most required features for unit testing support? I believe the most important feature is to mark a test as failed. I also want to stop executing this test in case of failure, like JUnit does. This is the fail() method,

%macro fail 0
        log msg_failed, msg_failed_len

        leave
        ret 0 ; test method never has any arguments
%endmacro

msg_failed:     db 'FAILED'
msg_failed_len  equ $ - msg_failed

In fact it is a NASM macro which gets copied into each invocation. For simplicity, it does not support a message argument right now, hence zero macro parameters. It uses the log macro, which prints the given string to Standard Out, just like _printGrid of my Assembly Minesweeper. Then it resets the stack frame (leave) and returns from the current function. I expect all test methods to have no arguments, and fail's code gets copied into each test method, this skips further execution of the test method and returns immediately to the calling code, i.e. the test runner. Adding the count of failed tests in fail is straight forward but I will leave that for later.

Assertion
Next I need assertions to check my expectations, at least an assert_equals,

%macro assert_equals 2
        cmp     %1, %2
        je      .%1_%2_end
        fail
.%1_%2_end:
        call    _show_progress
%endmacro

This has to be a macro as well so fail works correctly as explained above. But as the macro gets expanded into the calling code, the local label .%1_%2_end has to be unique. While different assertions are possible, a specific assertion like assert_equals eax, 2 can only be used once per test method. This is a weird constraint but I do not mind as my tests tend to have a single assertion per test method anyway. (Later I changed the macro to use a macro local label %%_end which removed this problem.)

The function _show_progress just prints a single dot to show the progress during test execution. It needs to preserve all registers because it is called in the middle of my test code.

_show_progress:
        push    eax
        push    ebx
        push    ecx
        push    edx

        push    dot_len
        mov     eax, dot
        push    eax
        call    _print

        pop     edx
        pop     ecx
        pop     ebx
        pop     eax

        ret

dot:    db      '.'
dot_len equ     1

Test Cases
For my test cases I want descriptive names. Using long, regular labels for function names and the assertion macros I write my first test

_should_add_one_and_one_to_be_two:
        create_local_variables 0

        mov     eax, 1
        add     eax, 1
        assert_equals eax, 2

        leave
        ret

The test succeeds and prints a single dot to Standard Out. The create_local_variables macro creates the stack frame and local variables if needed. If a test fails, e.g.

_should_add_one_and_two_to_be_three:
        create_local_variables 0

        mov     eax, 1
        add     eax, 2
        assert_equals eax, 2 ; (6)

        ; not reached
        hlt

        leave
        ret

it prints FAILED and stops execution in line 6.

Test Runner
Finally I want to run all my tests one by one. In this simple case I just call the test methods from the main entry point, followed by printing DONE at the end.

global  _main

_main:

        ; welcome message
        log     msg_hello, msg_hello_end - msg_hello

        call    _should_add_one_and_one_to_be_two
        call    _should_add_one_and_two_to_be_three
        ; (10)

        ; completion message
        log     msg_done, msg_done_end - msg_done

        jmp     _exit

msg_hello:      db 'HELLO asmUnit'
msg_hello_end:

msg_done:       db 'DONE'
msg_done_end:

This gets the job done but I am not happy with it. I need to add new test method invocations by hand in line 10. By seeing the new test fail I cannot forget to add the call of the new test method, but automatic test case discovery would be more convenient.

Before and After
What else does a unit testing framework need? For complex cases, before and after test execution hooks might be necessary. A before macro defines the proper label and the begin_test macro just calls it.

%macro before 0-1 0
        %ifdef ctx_before
                %error "before used more than once"
        %endif

        %define ctx_before, 1
_before_hook:
        create_local_variables %1
%endmacro

%macro begin_test 0-1 0
        create_local_variables %1
        %ifdef ctx_before
                call _before_hook
        %endif
%endmacro

The after and end_test macros work accordingly, just for the test clean-up. The actual test code stays the same, the macro calls just changed.

_should_add_one_and_one_to_be_two:
        begin_test

        mov     eax, 1
        add     eax, 1
        assert_equals eax, 2

        end_test

While this works well, I am unhappy with all the (NASM specific) macros in the code. There remains little real Assembly, but I guess that is the way to go. There are also plenty of missing features but this minimal first version gets me started. I will see how it works out in my next Assembly kata. (The complete source of asmUnit is here.)