10 January 2019

Interview Hanno Embregts

I am happy to start the year with another guest interview on ethics in Software Development: Hanno Embregts is a developer, teacher and frequent speaker from the Netherlands.

Hanno, how did you become a software developer?
Just before I graduated high school I was fairly certain I would end up in Economics. During college I took a few Computer Science classes on top of my Economics curriculum and I quickly discovered I liked programming a lot more than analysing micro-economic models. The concept of creating something new instead of just rehashing other people's ideas really appealed to me. The rest is, as they say, history.

I have been a software developer for over 11 years now, and I still love it immensely. I have worked on for about 10-15 clients, including insurance companies, banks, health care and public transport. I have spoken at 12 international conferences, while actually enjoying it. I used to be terrified of public speaking when I was younger, but it seems I gradually got used to it. Doing something you are quite comfortable with in front of an audience (in my case: playing music and singing) really helps you to get used to 100 pairs of eyes staring at you.

I saw you discussing ethics in software development on Twitter. Why does that matter to you?
Much of it has to do with my Christian faith. As a Christian I try to follow the example that Jesus Christ set when he lived on the earth. He devoted his time mostly to doing good for the people around him. And he specifically noticed the people that were less important in society. So I try to do the same, both in my private and my work life.

Women Baker In BadakhshanWhich topic are you most concerned about?
I am very concerned about the gap between the rich and the poor people. The rich are getting richer, and the poor are getting poorer. And although I can consider myself fortunate that I was born in a rich country with a lot of possibilities for education and career, I realise I could have been a lot less lucky. And if that would have been the case, I would want the rich people to help me out.

I try to share my assets with those who are less fortunate. I am quite enthusiastic about Kiva, a lending platform which supplies loans with low or no interest to boost local economies. Give a man a fish, and you feed him for a day. Teach a man to fish, and you feed him for a lifetime.

What do you consider the biggest challenge for humanity?
I think our biggest challenge is to love each other. Not just those who are special to us, but more importantly the other people: Your neighbours, your colleagues, even your enemies. If everyone would do this, there would be no need for wars and conflicts. Food, money and other assets would be distributed more fairly.

When I talk to people, many express concern about meat mass production or pollution. What could we do to engage in the topics?
Caring about the planet is important, because it shows you care about future generations. No-one can solve this problem on their own, but I try to contribute through a few small things in my life. I eat less meat than I did before, and I go to work by bike whenever I can. I moved my financial life (bank accounts, credit card etc.) to ASN Bank, which invests in socially responsible and sustainable projects only. And every month I donate money to Trees For All, an charity organisation that plants trees all around the world. As I said it does not solve the problem instantly, but it makes me feel good and gives me hope for our future.

I am looking for more impact on important topics while doing my regular work. Do you think that is possible in general?
I need to keep working in software development, because I love doing it and I cannot imagine earning my money in any different field. And there are not many jobs for software developers that directly change the world in a significant way. But that does not mean I cannot make an impact. As is pointed out in Episode 7 of the excellent podcast Soft Skills Engineering, somebody has to write the mainframe app that the bank uses so that charities can use this app to change the world in a significant way. And an insurance company has to insure the charity that produces clean water in underdeveloped countries. It may be directly or indirectly, but we are bound to make an impact in the world.

We take many decisions during projects. Which guidance do we have to navigate these decisions?
While I am developing software I try to think of myself as the end user. Would I feel comfortable with my decisions if I had to use this software on a daily basis? Would I find it acceptable to permit this app to access my contact list? Would I be happy when a website after creating an account immediately registers me for an email newsletter that I have not explicitly signed up for? Asking myself these questions nudges me in the right direction more often than not.

How do you think about selecting industry, customer or project based on your values?
I think if you have the chance to choose a project that clearly improves the world in a profound way, then obviously you should go for it. I have worked on a few medical software projects and the fact that my code helped medical staff to cure people motivated me immensely. At the moment I am working on public transport software, where the goal in itself is entirely different. I find that my motivation focuses on different things now, for example on teaching some junior developers the tricks of the trade.

FMSC Distribution Partner KenyaLet's be more specific: Would you work for an animal factory? Would you work for a company producing equipment for an animal factory? Would you work for a sweat shop exploiting kids in Asia?
I would not work for any of them. As I wrote earlier I am a big believer in sharing things equally over all people, so any company that violates human rights, exploits kids or does not pay fair wages is a no-go for me - both as a consumer and a software developer for that matter. This in itself is a challenge to say the least, because how do you find out which industries are OK according to this definition? Sure, there are a lot of certification labels that can help you decide (such as UTZ, FSC, Fairtrade), but every once in a while I read stories about these labels that really disappoint me. So I try to do some research on these industries to get to an informed decision.

That being said, there is also a 'grey area' of companies that I might work for, but only if there is no better alternative. For example, I am a big believer in electric- and hydrogen-powered cars. But I might consider working for Volkswagen or Mercedes anyway. The clothing industry is another good example. There are brands that pay their people fairly and there are those that exploit their workers. But there are also a lot of brands 'in between'. I might consider working for them. Who knows, I could perhaps help them to become a fair-paying brand.

Did you ever reject a client or project based on your values?
I have not, actually. But as I wrote before, in theory there are a lot of projects that I would reject based on my values. I think it has not happened until now, because our industry is booming and there are job openings everywhere. So because of that I have so far been able to choose projects that match my values fairly easily.

On the other hand, what would be projects that you would love to work on?
As I mentioned I work in public transport for a while now, and I like the fact that the train company in my country uses renewable electricity only to power their trains. I might work in the finance industry again and if that happens, I would be really interested to work for a socially responsible bank. Also working for an NGO would be interesting I guess. Still, I like what I do now and from teaching junior colleagues and sharing knowledge at various international conferences I derive a lot of meaning. Perhaps they will use the things I have learnt them in the future when they work at a socially responsible bank or an NGO. It may be directly or indirectly, but we are bound to make an impact in the world.

Thank you Hanno.

20 December 2018

Scheme Programming Language

With the beginning of this year I "fell" into Scheme. I blame SoCraTes BE 2015 for that. (SoCraTes is a group of international unconferences focusing on Software Craft and Testing. They are about sustainable creation of useful software in a responsible way and usually run as self-organised Open Space.) During that SoCraTes I happened to enter a session where a small group worked on some Scheme code. The Scheme programming language is one of the two main dialects of Lisp and I recognise Lisp when I see it. The session was so much fun that it continued for the whole of the second day. We used available slots in the schedule of the open space, and in cases there were none, we continued coding Scheme in the hallway, the kitchen and other places. (Now such things only happen on SoCraTes and I encourage you to attend one if possible.)

SICP Exhibit, MIT Museum Includes a battered copy of SICPThis SoCraTes experience broke the ice with Scheme, i.e. it gave me enough exposure to a new language and enabled me to continue on my own. For the next two years I kept playing with Scheme, experimenting with code katas, e.g. the Bank-OCR Kata, porting well known exercises like Parrot, Gilded Rose and Game Of Life and I even solved some Project Euler problems using Scheme.

Earlier this year I started reading SICP. SICP stands for Structure and Interpretation of Computer Programs, a computer science text originally used in introductory courses taught at the MIT in the eighties. It had been on my reading list since 2010, when Uncle Bob recommended it in one of his Clean Coder videos. SICP had been recommended again and again and I was very happy to find the time and energy to read it.

SICP is one of the greatest books - if not the greatest book - I have ever read. It is fast paced and written in the style of all scientific books - too dense and hard to follow. Reading it is hard work. I used to like such books when I was studying applied mathematics many years ago - actually I was eating such books for breakfast ;-) The high density gives me new information fast. As I like to read books from cover to cover, I do not mind that the information is not accessible directly. Yes it is challenging to read. And it is very insightful. I appreciate most the skill of the authors to define abstractions. I am stunned again and again how they name their composed methods.

The Scheme Programming Language
SICP is strongly related to Scheme as it is one of the bibles of the Lisp/Scheme world. SICP uses Scheme for its code samples and exercises as Gerald Jay Sussman, one author of SICP, is also one of the creators of Scheme. Scheme is a version of Lisp and as such a divine language per se. It is a great language and unlike Lisp it follows a minimalist design philosophy. I really like its simplicity: There are no reserved words, not much syntax, just atoms, pairs and lists.
(define atom?
  (lambda (x)
    (and (not (pair? x))
         (not (null? x)))))
Atoms are numbers, strings, booleans, characters and symbols. Symbols i.e. names usually represent other atoms or functions. Maybe you do not like all the extra parenthesis, but it is compact and uniform. Because of the uniformity of the S-expression, it is easy to create parsers, embedded languages and even full featured Schemes, e.g. in Python, Haskell or any language. (To be fair, the last link contains implementations of Lisp not Scheme in 73 languages.)

As I said before Scheme is based on Lisp. The Lisp programming language was created 60 years ago by John McCarthy. And Lisp is very special, it seems to transcend the utilitarian criteria used to judge other languages. In 1975 Scheme was created by Gerald Jay Sussman and Guy Lewis Steele. Watch this great talk by Guy Steele about issues associated with designing a programming language.

Ancient HistoryStandards
One thing which confused me a lot when I started playing with Scheme were its versions. The Scheme language is standardised by IEEE and the de facto standard is called the Revised n Report on the Algorithmic Language Scheme (RnRS). The most widely implemented standard is R5RS from 1998. I use that mainly because the first Scheme implementation I downloaded was an R5RS compliant one. R5RS files use the .scm filename extension. The newer R6RS standard, ratified in 2007, is controversial because it departs from the minimalist philosophy. It introduces modularity which is breaking everything. Being used to the strong compatibility of Java or at least the major compatibility of Python I did not expect versions to be incompatible at all. R6RS files use the .ss, .sls and .sps filename extensions. The current standard is R7RS from 2013, a smaller version, defining a subset of the large R6RS version retaining the minimalism or earlier versions.

(Almost) Complete List of Special Forms and Functions
I did not read any tutorials or books on the Scheme language - I was exploring the language on my own. Still I wanted to know which library functions I could use. Similarly I used to list all classes available in each Java release. I searched for an exhaustive list of Scheme functions. There are many Schemes and I did not want to depend on any implementation specific extensions. In the end I decided to scrape the standards.

Scheme makes a difference between forms and functions. Function calls evaluate all arguments before control is passed to the function body. If some expressions should not be evaluated - as in if or switch (which is cond in Scheme) - a special form is needed. Special forms evaluate their arguments lazily. For more information see Why is cond a special form in Scheme.

R5RS Forms and Functions
I scraped the list of special forms and built-in functions from the R5RS HTML documents. The list is incomplete since I had to rely on formatting of the document. It misses define and the abbreviations like ' and @, but looks pretty good (to me) otherwise. Browsing the 193 forms and functions gives an idea of built in data types, i.e.
complex (number)
exact (number)
inexact (number)
rational (number)
as well as possible conversions between them available in any R5RS compliant Scheme.
The Leeds LibraryR6RS Forms and Functions
As mentioned earlier, R6RS is much larger than R5RS. It defines 630 forms and functions, most of them in the (new) libraries. The standard separates built-in forms and functions from the ones defined in libraries. (This is still very small, considering that the Java 8 core library contains 6000 public classes.) My list of forms and functions contains all, built-in and library alike. It looks complete, but I am not sure, I did not work with R6RS. From a quick glance R6RS adds bitwise and floating point operations as well as different types of byte vectors, enum-set and hashtable. When looking at my list now, I see that I should have scraped the names of the library modules as well. (I added that to my task list for 2019 ;-)

Scheme Requests for Implementation
Next to the RnRS standards, there are the SRFIs, a collection of concrete proposals and reference implementations. People implementing Scheme chose to implement SRFIs or not. Most Schemes support some of then, see Arthur A. Gleckler's report of SRFI support by Scheme implementations in 2018. Some Schemes have package managers which allow to download and instal packages. Usually some of those are SRFI implementations. I guess I also need to scrape these.

Scheme (Lisp) is so much fun, especially when you do not have to deliver anything. Most people I meet got in touch with Lisp during university but never followed up. Some are even afraid of it. Since 2016 I run Scheme coding sessions at every unconference or SoCraTes event I attend. I invite people to mob with me and I do all the typing. We always have a good time, and - after some warm up with Scheme - people really like it. They all enjoy the opportunity to dive into Lisp again. Will you?

11 November 2018

Widespread Architectural Changes using IDEs

This is the second part of my list of tools to automate Widespread Architectural Changes. Now Widespread Architectural Change is a category of architectural refactoring which do not change the overall structure but need to be applied in many places consistently across a whole code base. For example, consider changing or unifying coding conventions or fixing violations. The main challenge of these changes is the high number of occurrences.

The goal of this article is to introduce you to ways to automate such changes. I have been using some of these techniques because as Code Cop I value code consistency, still this is a raw list. I have only used a few of the mentioned tools and need to explore many in more detail.

In the first part I listed some basic options, e.g. how to support manual changes with fast navigation, search and replace across the whole file system, use scripted search and replace using regular expressions, Macros and finally JetBrains' Structural Search and Replace. This time I focus on options available for modern IDEs. Tools like Eclipse or IntelliJ use an internal representation of the code, the Abstract Syntax Tree (AST). Structural Search and Replace works the AST and therefore belongs into this group as well.

Rescripter (Eclipse)
Modern IDEs support many Refactoring steps - why not use the existing tools to make our job easier? One of such tools is Rescripter, built for making large-scale changes that you can describe easily but are laborious to do by hand. Rescripter is an Eclipse plugin which runs JavaScript in the context of the Eclipse JDT. David Green added some helper objects to make searching, parsing and modifying Java code easier and wrapped a thin JavaScript layer around it. Rescripter scripts are working the AST.

Here are two examples from Rescripter's documentation: Finding a method named getName inside the class Person:
var type = Find.typeByName("Person");
var method = Find.methodByName(type, "getName").getElementName();
and adding a method getJobTitle to that type:
var edit = new SourceChange(type.getCompilationUnit());
edit.addEdit(ChangeType.addMethod(type, "\n\tpublic String getJobTitle() {\n" +
                                        "\t\treturn this.jobTitle;\n" +
Rescripter is an old project and has not been updated since 2011. While it has some demo code and documentation, it is a bit raw. I guess it is not compatible with newer versions of Eclipse. Still - like Structural Search and Replace - it is very powerful. If you have a large code base and a repetitive change that can be expressed in terms of the AST, the high effort to get into the tool and create a script will pay off. (A useful plugin to help you working the AST is the AST View. The AST View is part of the JDT but not installed out of the box. It is visualising the AST of a Java file open in the editor.)

JackpotJackpot (NetBeans)
Another, very interesting tool in this area is the Jackpot DSL for NetBeans used in the IDE actions Inspect and Transform aka Inspect and Refactor. Jackpot is a NetBeans IDE module for modelling, querying and transforming Java source files. In the Jackpot context, a program transformation is a script or Java class which queries sources in Java projects for patterns, changes them, and then writes these changes back to the original source files. [...] Jackpot transformations tend to be applied globally, such as at a project level or across several projects. Win! Unfortunately there is very little information about it. It took me hours just to find a little bit of information about it:

Dustin Marx describes how to create NetBeans 7.1 custom hints. He agrees that the most difficult aspect of using NetBeans's custom hints is finding documentation on how to use them. The best sources currently available appear to be the NetBeans 7.1 Release Notes, several Wielenga posts (Custom Declarative Hints in NetBeans IDE 7.1, Oh No Vector!, Oh No @Override!), and Jan Lahoda's jackpot30 Rules Language (covers the rules language syntax used by the custom inspections/hints). The Refactoring with Inspect and Transform in the NetBeans IDE Java Editor tutorial also includes a section on managing custom hints. All listed documentation is from 2011/2012. The most recent one I found is a short Jackpot demo from JavaOne Brazil 2016.

NetBeans hints are warnings that have a quick fix. For example the hint "Can Use Diamond" finds places where the diamond operator of Java 7 can be used instead of explicit type parameters. When the offered action is taken, the code is migrated. In the Inspect and Transform dialogue, the inspections can be managed. Custom hints are stored in .hint files. For example, Dustin Marx' hint to remove extraneous calls to System.currentTimeMillis() from new java.util.Date constructor is written as
<!description="Unnecessary use of System.currentTimeMillis on new Date">

new java.util.Date(java.lang.System.currentTimeMillis())
=> new java.util.Date()
The Java Declarative Hints Format allows matching on variables, modifiers and statements. Fixes can be applied conditionally too.

I do not know if Jackpot hints can be applied across a whole code base at once. As they are inspections, I expect them to be displayed for the whole project - making them fast navigation markers at least. Anyway this is very exciting. It is so exciting that some people wanted to port it to Eclipse (but they never did).

Using Specific Refactoring Plugins
There are a few specific plugins that combine various Refactoring. Eclipse's Source Clean Up Action deserves a honorary mention: It fixes basic warnings on a whole code base, but is very limited. An interesting plugin for Eclipse is Autorefactor which aims to fix language/API usage to deliver smaller, more maintainable and more expressive code bases. Spartan Refactoring is another Eclipse plugin that performs automatic refactoring of Java source code, making it shorter, more idiomatic and more readable.

All these plugins change a predefined, limited set of code patterns, mainly focusing on technical debt. Maybe someone implemented a refactoring plugin for part of the widespread change you need to perform. Search the market places first.

Using Refactoring APIs of IDEs
A possible way is to write your own refactoring plugin. Maybe start with code of a refactoring plugin listed above. Both Eclipse and IntelliJ IDEA offer APIs to manipulating Java code, i.e. the JDT and PSI APIs. I have not done that because it seems too much effort for one time migrations and widespread changes. And reusing the available refactoring tools might raise some problems like waiting for user input which is problematic.

Using (APIs of) Refactoring Browsers
The Refactoring Browser was the first tool that automated Refactoring for Smalltalk. It set the standard for all modern Refactoring tools. Today we have Refactoring Browsers for many languages, e.g. CScout - the C Refactoring Browser, Ruby Refactoring Browser for Emacs or PHP Refactoring Browser which is controlled via the command-line and has plugins for Vim and Emacs. Stand-alone Refactoring tools should be easier to use and script than full blown IDEs, especially if they were designed for different plugins. The idea would be to create code which controls Refactoring Browsers to apply certain changes - again and again.

RopePython Refactoring Libraries
While searching for Refactoring Browsers I just learnt that there are at least three stand-alone Python Refactoring libraries: Rope comes with Vim and Emacs plugins and is also used in VS Code. The second is Bicycle Repair Man. Further there is Bowler which supposedly is better suited for use from the command-line and encourages scripting. All these libraries are rich in features and used by Vim users. (Yeah, Vim is still going strong.)

Rope (Python)
For example, Rope can be used as a library, which allows custom refactoring steps. Even more, it offers something similar to Jackpot's hints, Restructurings. For example, we split a method f(self, p1, p2) of a class mod.A into f1(self, p1) and f2(self, p2). The following Restructuring updates all call sites:
pattern: ${inst}.f(${p1}, ${p2})

 inst: type=mod.A
The code to perform the Restructuring using Rope as a library is
from rope.base.project import Project
from rope.refactor import restructure

project = Project('.')

pattern = '${inst}.f(${p1}, ${p2})'
goal = '...'
args = '...'

restructuring = restructure.Restructure(project, pattern, goal, args)

I never used that but it looks cool. The promise of scripted refactoring makes me excited. Another item to add to my to-research list. ;-) And there are more tools on my list, which will have to wait for part 3.