8 September 2018

IDE Support for Scripts

Last month I wrote about dealing with modified files during build in Jenkins. The solution uses Groovy scripts and the Jenkins Groovy plugin which allows execution of scripts in Jenkins' system context. These System Scripts have access to Jenkins' internal model like build status, changed files and so on. The model includes several classes which need to be navigated.

SupportIn many situations little additional scripts make our lives as developers easier: For example the Groovy System Script to customise Jenkins when building a NATURAL code base mentioned above, or a Groovy Script Transformation to perform repeated code changes in a large Java project using WalkMod, or a custom build during a Maven build, using any language supported by the Apache Bean Scripting Framework, e.g. Groovy, Ruby, Python or others. Now if the scripts are getting complicated IDE support would be nice.

Getting support for the language in the IDE
Depending on the language, most IDEs need installation of a plugin or extension. I will briefly describe the steps to set up Groovy support into Eclipse because it is more elaborate and the organisation I initially wrote this documentation used Eclipse. Getting plugins for IntelliJ IDEA is straight forward.
  • First obtain the version of Eclipse you are using. Customised distributions like the Spring Tool Suite, IBM/Rational products or NaturalONE follow different version schemes than Eclipse. Navigate to the Help menu, item About, button Installation Details, tab Features, line Eclipse Platform to see the real version. For example, NaturalONE 8.3.2 is based on Eclipse 4.4.
  • The Groovy plugin for Eclipse is GrEclipse. For each major version of Eclipse there is a matching version of GrEclipse. The section Releases lists the Update Site for each release. Find the suitable version. (If you use a newer Eclipse, maybe you have to use a snapshot version instead of a release version.) Copy the URL of the Update Site.
  • The section How to Install explains in detail how to continue and install GrEclipse.
After that Eclipse can talk Groovy.

Enabling support for the language in the project
Next we configure the project for the additional language. In Eclipse this is often possible through the (right click) context menu of the project, menu item Configure. I usually do not bother and edit Eclipse's .project file directly to add the needed natures and build commands. Adding Groovy to a non-Java project needs adding to .project,
<projectDescription>
  <!-- existing configuration -->
  <buildSpec>
    <!-- existing configuration -->
    <buildCommand>
      <name>org.eclipse.jdt.core.javabuilder</name>
      <arguments>
      </arguments>
    </buildCommand>
  </buildSpec>
  <natures>
    <!-- existing configuration -->
    <nature>org.eclipse.jdt.groovy.core.groovyNature</nature>
    <nature>org.eclipse.jdt.core.javanature</nature>
  </natures>
</projectDescription>
and .classpath
<classpath>
  <!-- existing configuration -->
  <classpathentry exported="true" kind="con" path="GROOVY_SUPPORT"/>
  <classpathentry exported="true" kind="con" path="GROOVY_DSL_SUPPORT"/>
</classpath>
Similar changes are needed for Ruby (using org.eclipse.​dltk.core.​scriptbuilder and org.eclipse.​dltk.ruby.​core.nature) and Python (org.python.​pydev.PyDevBuilder and org.python.​pydev.pythonNature). If I am not sure, I create a new project for the target language and merge the .projects manually. Sometimes other files like .classpath, .buildpath or .pydevproject have to be copied, too.

Enabling support for libraries
Adding a nature to .project gives general support like syntax highlighting and refactoring for the respective language but there is no completion of library classes. Especially when using new libraries, like Jenkins Core, using code completion helps exploring the new API because you get a list of all possible methods to call. The trick is to add the needed dependency to the project in a way that it is not accessible from or packaged together with the production code.

An Example: Adding Jenkins System Script support to a NaturalONE project
To add dependencies to the NaturalONE project earlier, I converted the Java/Groovy project to Maven (by setting another nature and builder) and added the needed classes as fake dependencies to the pom.xml.
<dependencies>
  <!-- other dependencies -->
  <dependency>
    <groupId>org.jenkins-ci.main</groupId>
    <artifactId>jenkins-core</artifactId>
    <version>2.58</version>
    <scope>test</scope> <!-- (1) -->
  </dependency>
  <dependency>
    <groupId>org.jenkins-ci.plugins</groupId>
    <artifactId>subversion</artifactId>
    <version>2.7.2</version>
    <scope>test</scope>
  </dependency>
</dependencies>
Groovy has optional typing and after adding the types to the definitions and arguments, Eclipse will pick up the type and we get code completion on available fields and methods:
import hudson.model.Build
def Build build = Thread.currentThread()?.executable
build._ // <-- code completion, yeah!
This made developing and understanding the script to find modified files much easier. The Jenkins dependencies in the pom.xml are never used, as the System Script runs inside Jenkins where the classes are provided. The dependencies are declared to "trick" Eclipse into analysing them and providing type information. The scope of the dependencies is set to test (see line (1) in code snippet above) so the dependencies are not packaged and cannot be called from the production code. This repository contains the NATURAL project together with all the Eclipse configuration files.

Another Example: Adding WalkMod Transformation support to a Maven project
Another situation where I wish for script support in Eclipse is when writing WalkMod Script Transformations. WalkMod is an open source Java tool that can be used to apply code conventions automatically. Read this tutorial by Raquel Pau to see how WalkMod works. WalkMod allows for Groovy scripts to define code transformations which manipulate the AST of Java classes. Navigating the AST is difficult in the beginning.

After adding the Groovy nature as shown in the previous example, the relevant dependencies to get code completion for the AST are
<dependencies>
  <dependency>
    <!-- API -->
    <groupId>org.walkmod</groupId>
    <artifactId>walkmod-core</artifactId>
    <version>3.0.4</version>
    <scope>test</scope>
  </dependency>
  <dependency>
    <!-- AST -->
    <groupId>org.walkmod</groupId>
    <artifactId>javalang</artifactId>
    <version>4.8.8</version>
    <scope>test</scope>
  </dependency>
  <dependency>
    <groupId>com.squareup</groupId>
    <artifactId>javapoet</artifactId>
    <version>1.10.0</version>
    <scope>test</scope>
  </dependency>
</dependencies>
In case of Walkmod transformations, the interesting class is (org.walkmod.​javalang.ast.​CompilationUnit) and it is already in scope as variable node. To add type information we need to alias it:
import org.walkmod.javalang.ast.CompilationUnit
def CompilationUnit cu = node
cu._ // <-- code completion on available fields and methods
Conclusion
Supporting scripts in Groovy, Ruby or Python make our lives easier. With some additional extensions or plugins and some tweaks in the project configuration our IDEs will support the scripting language. Why should we accept code editing like in the early 1990s?

2 September 2018

Work Harder

With this article I want to prompt you to work hard(er). What does working hard mean? When I looked for different translations of German Strengt Euch an, I found several ones, which are all suitable: Work hard. Keep it tight. Push yourselves. Put some heart into it. Put some muscle in it. Put your backs to it. Make your best effort. Just play it like you fucking mean it. Obviously hard work needs your strength. It also needs your heart - motivation and dedication. It might test your limits, both physical and psychical ones.

Domesday BooksMy favourite example of hard work was writing books in the Dark Ages. Monks were copying books by hand, adding drawings and decorations as they went. An extreme example is the Codex Gigas, which was handwritten by a single, anonymous monk. Because the scribe was a monk he may only have been able to work for about three hours a day, and this means that the manuscript including decoration probably took at least 20 years to finish, and could even have taken 30. Now that is a whole lifetime for collecting and handwriting a single book. Imagine the dedication.

Maybe less extreme is the life of master craftspeople. Several years ago I wrote about a master craftsman hand-crafting rakes. He could retire any time, still chose to improve his methods and work hard all day.

Benefits of hard work
Most of you will agree, that hard work pays off. Here is a little TED talk by Richard St. John, who explains why it is so: Hard work is the real secret to success. I vividly remember a professor of Mathematical Analysis during my studies: At the beginning of a lecture he wrote a theorem on the blackboard, followed by an easy proof. The proof looked good and we students understood what was going on. Then he said "Was man leicht kriegt ist nichts wert" ("What you get easily is worth nothing"), pointed out a mistake in the proof and erased it. He used the remaining of the hour to sketch a valid proof which was complicated and much harder to follow.

PainHard work has more value
Theodore Roosevelt even said that "Nothing in the world is worth having or worth doing unless it means effort, pain, difficulty..." Maybe he was exaggerating, but a little pain never hurts. (Pun intended ;-) So is the motto in weightlifting training: No Pain, No Gain. Psychologist Katarina Veselko mentioned the topic in one of her talks. During the following conversation she explained: It not so much that we do not value things that are easy to get, it is more about the fact that when we need to invest a lot of energy, time, money, ... into achieving something, the result will be more valuable for us because we have invested a lot into it. It is some kind of Cognitive dissonance. She also believes that most things worth having are not easy to get; most things that are valuable to us are not achieved in a way that is always fun, fast and easy, but also includes obstacles and challenges.

Instant Gratification
The term Instant Gratification is often used to label the satisfactions gained by more impulsive behaviours: choosing now over tomorrow. And today it is harder to delay gratification than it used to be. We have been trained by technology and social media to expect results fast, without much effort. For example, there are more than 2 million hits on Google how to earn 5k extra, many of then offering "work from home, 30 minutes daily" up to "basically doing nothing". The same is true for losing weight without any diet and so on. If you want to read more on Instant Gratification and its problems, I recommend Courtney Ackerman's Definition with Examples.

Conclusion
It seems that working hard has become unpopular. Why work towards a long term goal when many things are handed over on a plate, or downloaded at the click of a button, or ours in twenty-four hours for just £9.99 extra? I do not think that life works like that. Why do I come up with this in my blog? I am writing this because I think we Craftspeople need to work harder. I am not talking about working more hours. In fact working overtime is against the idea of Craftsmanship because we need our free time to reflect, exchange and learn.

I am talking about pushing ourselves more: Following up on that refactoring you did not finish last week. Refactoring mercilessly, always pushing to keep the whole code base clean and consistent, even during architectural changes. I am talking about not accepting lesser standards because of low level, different or even obsolete technologies or environments. I am talking about overcoming pointless bureaucracy or managers who have no idea about code quality.

Defiance Cafe
I know it is hard. Years of arguments with colleagues, fighting superiors, struggling to grow and working broken processes have taken their toll. Sometimes I feel old. But I have to defy them. Let us continue pushing, trying to work harder and make an effort!

29 August 2018

Widespread Architectural Change Part 1

Last month I introduced Four Categories of Architectural Refactoring:
  • Substitute Architectural Decision
  • Refactor Architectural Structure
  • Widespread Architectural Change
  • Other Architectural Changes
Today I want to start discussing options to perform Widespread Architectural Changes. This category contains all kind of small changes repeated throughout the whole code base. The number of these changes is high, being hundreds or sometimes even thousands of occurrences that have to be changed in a similar way. Examples of such modifications are:
  • Migrating or upgrading (similar) APIs, frameworks or libraries
  • Changing or unifying coding conventions
  • Fixing compiler warnings and removing technical debt
  • Consistently applying or changing aspects like logging or security
  • Applying internationalization
  • Migrate between languages, e.g. SQL vs. HQL
Ways to do Widespread Architectural Changes (C) by SoftDevGang 2016All of them do not change the overall structure, they just work on the implementation. Often substituting architectural decisions or altering architectural structure contain similar changes.

Options
I have been using some techniques since 2004 because as Code Cop I value code consistency. Two years ago I had the opportunity to discuss the topic with Alexandru Bolboaca and other experienced developers during a small unconference and we came up with even more options, as shown in the picture on the right. The goal of this and the next article is to introduce you to these options, the power of each one and the cost of using it. A word of warning: This is a raw list. I have used some but not all of them and I have yet to explore many options in more detail.

Supporting Manual Changes With Fast Navigation
The main challenge of widespread changes is the high number of occurrences. If the change itself is small, finding all occurrences and navigating to them is a major effort. Support for fast navigation would make things easier. Often it is possible to create a list of all lines that need to be changed. In the past have created custom rules of static analysis tools like PMD to find all places I needed to change. Such a list can be used to open the file and jump to the proper line, one source file after another.

As example, here is a Ruby script that converts pmd.xml which is contains the PMD violations from the Apache Maven PMD Plugin to Java stack traces suitable for Eclipse.
#! ruby
require 'rexml/document'

xml_doc = REXML::Document.new(File.new('./target/site/pmd.xml'))
xml_doc.elements.each('pmd/file/violation') do |violation|
  if violation.attributes['class']
    class_name = violation.attributes['package'] + '.' +
                 violation.attributes['class']
    short_name = violation.attributes['class'].sub(/\$.*$/, '')
    line_number = violation.attributes['beginline']
    rule = violation.attributes['rule']

    puts "#{class_name}.m(#{short_name}.java:#{line_number}) #{rule}"
  end
end
The script uses an XML parser to extract class names and line numbers from the violation report. After pasting the output into Eclipse's Stacktrace Console, you can click each line one by one and Eclipse opens each file in the editor with the cursor in the proper line. This is pretty neat. Another way is to script Vim to open each file and navigate to the proper line using vi +<line number> <file name>. Many editors support similar navigation with the pattern <file name>:<line number>.

Enabling fast navigation is a big help. The power this option is high (that is 4 out of 5 on my personal, totally subjective scale) and the effort to find the needed lines and script them might be medium.

Search and Replace (Across File System)
The most straight forward way to change similar code is by Search and Replace. Many tools offer to search and replace across the whole project, allowing to change many files at once. Even converting basic scenarios, which sometimes cover up to 80%, helps a lot. Unfortunately basic search is very limited. I rate its power low and the effort to use it is also low.

Scripted Search and Replace using Regular Expressions
Now Regular Expressions are much more powerful than basic search. Many editors allow Regular Expressions in Search and Replace. I recommend creating a little script, e.g. using AWK, Ruby or Python and a lot of changes can be automated. While the script adds extra work to traverse all the source directories and files, the extra flexibility is needed for conditional logic, e.g. adding an import to a new class if it was not imported before. Also in a script Regular Expressions can be nested, i.e. analysing the match of an expression further in a second step. This helps to keep the expressions simple.

For example I used a script to migrate Java's clone() methods from version 1.4 to 5. Java 5 offers covariant return types which can be used for clone(), removing the cast from client code. The following Ruby snippet is called with the name of the class and the full Java source as string:
shortName = shortClassNameFromClassName(className)
if source =~ / Object clone\(\)/
  # use covariant return type for method signature
  source = $` + " #{shortName} clone()" + $'
end
if source =~ /return super\.clone\(\);/
  # add cast to make code compile again
  source = $` + "return (#{shortName}) super.clone();" + $'
end
In the code base where I applied this widespread change, it fixed 90% of all occurrences as clone methods did not do anything else. It also created some broken code which I reverted. I always review automated changes, even large numbers, as jumping from diff to diff is pretty fast with modern tooling. Scripts using Regular Expressions helped me a lot in the past and I rate their power to high. Creating them is some effort, e.g. a medium amount of work.

Macros and Scripts
Many tools like Vim, Emacs, Visual Studio, Notepad++ and IntelliJ IDEA (by use of a plugin) allow creation or recording of Keyboard and or mouse macros or Application scripts. With them, frequently used or repetitive sequences of keystrokes and mouse movements can be automated, and that is exactly what we want to do.

Alex has used Vim Scripts to automate repetitive coding tasks. I have not used it and relating to his experience. Here is a Vim script function which extracts a variable, taken from Gary Bernhardt's dotfiles:
function! ExtractVariable()
  let name = input("Variable name: ")
  if name == ''
    return
  endif

  " Enter visual mode
  normal! gv
  " Replace selected text with the variable name
  exec "normal c" . name
  " Define the variable on the line above
  exec "normal! O" . name . " = "
  " Paste the original selected text to be the variable value
  normal! $p
endfunction
I guess large macros will not be very readable and hard to change but they will be able to do everything what Vim can do, which is everything. ;-) They are powerful and easy to create - as soon as you know Vim script of course.

Structural Search and Replace
IntelliJ IDEA offers Structural Search and Replace which performs search and replace across the whole project, taking advantage of IntelliJ IDEA's awareness of the syntax and code structure of the supported languages.. In other words it is Search and Replace on the Abstract Syntax Tree (AST). Additionally it is possible to apply semantic conditions to the search, for example locate the symbols that are read or written to. I have not used it. People who use it tell me that it is useful and easy to use.

Online help says that you can apply constraints described as Groovy scripts and make use of IntelliJ IDEA PSI (Program Structure Interface) API for the used programming language. As PSI is the IntelliJ version of the AST, this approach is very powerful but you need to work the PSI/AST which is (by its nature) complicated. This requires a higher effort.

To be continued
And there are many more options to be explored, e.g. scripting code changes inside IDEs or using Refactoring APIs as well as advanced tools outside of IDEs. I will continue my list in the next part. Stay tuned.

15 August 2018

Creating your own NATstyle rules

Last month I showed how to use NATstyle from the command line. NATstyle is the utility to define and check the coding standard of your NATURAL program. Today I want to explain how to customize and create your own rules beyond what is explained in the manual. I used NaturalONE 8.3.5.0.242 CE (November 2016). Likely there are more options and rules available in newer versions of NaturalONE and NATstyle.

Basic Configuration
NATstyle comes packaged inside NaturalONE, the Eclipse-based IDE for NATURAL. As expected NATstyle can be configured in Eclipse preferences. The configuration is saved as NATstyle.xml which is used when you run NATstyle from the right click popup menu. We will need to modify NATstyle.xml later, so let's have a look at it:
<?xml version="1.0" encoding="utf-8"?>
<naturalStyleCheck version="1.0"
                   xmlns="http://softwareag.com/natstyle/rules"
                   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                   xsi:schemaLocation="http://softwareag.com/natstyle/rules checks.xsd">
  <checks type="source">
    <check class="CheckLineLength" name="Line length" severity="warning">
      <property name="max" value="72" />
      <property name="exclude" value="D3" />
    </check>
    <!-- more checks of type source -->
  </checks>
  <!-- more checks of other types -->
</naturalStyleCheck>
(The default configuration file is here together with its XML schema.)

Existing Rules
The existing rules are described in NaturalONE help topic Overview of NATstyle Rules, Error Messages and Solutions. Version 8.3 has 42 rules. These are only a few compared to PMD or SonarQube, which has more than 1000 rules available for Java. Here are some examples what NATstyle can do:
  • Source Checks: e.g. limit line length, find tab characters, find empty lines, limit the number of source lines and check a regular expressions for single source lines or whole source file.
  • Source Header Checks: e.g. force header or check file naming convention.
  • Parser Checks: e.g. find unused local variables, warn if local variable shadows view, find TODO comments, calculate Cyclomatic and NPath complexity, force NATdoc (documentation) tags and check function, subroutine and class names against regular expressions.
  • Error (Message File) Checks: e.g. check error messages file name.
  • Resource (File) Checks: e.g. check resource file name.
  • Library (Folder) Checks: e.g. library folder conventions, find special folders, force group folders and warn on missing NATdoc library documentation.
Same rule multiple times configured differently
Some rules like Source/Regular expression for single source lines only allow a single regular expression to be configured. Using alternation, e.g. a|b|c, in the expression is a way to overcome that, but the expression gets complicated quickly. Another way is to duplicate the <check> element in the NATstyle.xml configuration. Assume we do not only forbid PRINT statements, we also do not allow reduction to zero. (These rules do not make any sense, they are just here to explain the idea.) The relevant part of NATstyle.xml looks like
<checks type="source">
  <check class="CheckRegExLine"
         name="Regular expression for single source lines" ... >
    <property name="regex" value="PRINT '.*" />
  </check>
  <check class="CheckRegExLine"
         name="Regular expression for single source lines" ... >
    <property name="regex" value="REDUCE .* TO 0" />
  </check>
</checks>
While it is impossible to configure these rules in the NaturalONE preferences, it might be possible to run NATstyle with these modified settings. I did not verify that. I execute NATstyle from the command line passing in the configuration file name using the -c flag. (See the full configuration and script to to run the rules from the command line.)

rock piles of different sizeDefining your own rules
There is no documented way to create new rules for NATstyle. All rules' classes are defined inside the NATstyle plugin. The configuration XML contains a class attribute, which is a short name, e.g. CheckRegExLine. Its implementation is located in the package com.​softwareag.​naturalone.​natural.​natstyle.​check.​src.​source where source is the group of the rules defined in the type attribute of the <checks> element. I experimented a lot and did not find a way to load rules from other packages than com.​softwareag.​naturalone.​natural.​natstyle. All rules must be defined inside this name space, which is possible.

Source Rules
While I cannot see the actual code of NATstyle rules, Java classes expose their public methods and parent class. I did see the names of the rule classes in the configuration and guessed and experimented with the API a lot. My experience with other static analysis tools, e.g. PMD and Pylint and the good method names of NATstyle code helped me doing so. A basic Source rule looks like that:
package com.softwareag.naturalone.natural.natstyle.check.src.source; // 1.

import com.softwareag.naturalone.natural.natstyle.NATstyleCheckerSourceImpl;
// other imports ...

public class FindFooSourceRule
  extends NATstyleCheckerSourceImpl { // 2.

  private Matcher name;

  @Override
  public void initParameterList() {
    name = Pattern.compile("FOO").matcher(""); // 3.
  }

  @Override
  public String run() { // 4.
    StringBuffer xmlOutput = new StringBuffer();

    String[] lines = this.getSourcelines(); // 5.
    for (int line = 0; line < lines.length; i++) {
      name.reset(lines[line]);
      if (name.find()) {
        setError(xmlOutput, line, "Message"); // 6.
      }
    }

    return xmlOutput.toString(); // 7.
  }
}
The marked lines are important:
  1. Because it is a Source rule, it must be in exactly this package - see the paragraph above.
  2. Source rules extend NATstyleCheckerSourceImpl which provides the lines of the NATURAL source file - see line 6. It has more methods, which have reasonable names, use the code completion.
  3. You initialise parameters in initParameterList. I did not figure out how to make the rules configurable from the XML configuration, which will probably happen in here, too.
  4. The run method is executed for each NATURAL file.
  5. NATstyleCheckerSourceImpl provides the lines of the file in getSourcelines. You can iterate the lines and check them.
  6. If there is a problem, call setError. Now setError is a bit weird, because it writes an XML element for the violation report XML (e.g. NATstyleResult.xml) into a StringBuffer.
  7. In the end the return the XML String of all found violations.
Finally the rule is configured with
<checks type="source">
  <check class="FindFooSourceRule"
         name="Find FOO"
         severity="warning" />
</checks>
(In the example repository, there is a working Source rule FindInv02.java together with its configuration customSource.xml.)

Parser Rules
Now it is getting more interesting. There are 18 rules of this type, which is a good start, but we need moar! Parser rules look similar to Source rules:
package com.softwareag.naturalone.natural.natstyle.check.src.parser; // 1.

import com.softwareag.naturalone.natural.natstyle.NATstyleCheckerParserImpl;
// other imports ...

public class SomeParserRule
  extends NATstyleCheckerParserImpl { // 2.

  @Override
  public void initParameterList() {
  }

  @Override
  public String run() {
    StringBuffer xmlOutput = new StringBuffer();

    // create visitor
    getNaturalParser().getNaturalASTRoot().accept(visitor); // 3.
    // collect errors from visitor into xmlOutput

    return xmlOutput.toString();
  }
}
where
  1. Like Source rules, Parser rules must be defined under the package ...natstyle.​check.​src.​parser.
  2. Parser rules extend NATstyleCheckerParserImpl.
  3. The NATURAL parser traverses the AST of the NATURAL code. Similar to other tools, NATstyle uses a visitor, the INaturalASTVisitor. The visitor is called for each node in the AST tree. This is similar to PMD.
Using the Parser
Tree, MuthillThe visitor must implement INaturalASTVisitor in package com.​softwareag.​naturalone.​natural.​parser.​ast.​internal. This interface defines 48 visit methods for the different sub types of INaturalASTNode, e.g. array indices, comments, operands, system function references like LOOP or TRIM, and so on. Still there are never enough node types as the AST does not convey much information about the code, most statements end up as INaturalASTTokenNode. For example the NATURAL lines
* print with leading blanks
PRINT 3X 'Hello'
which are a line comment and a print statement, result in the AST snippet
+ TOKEN: * print with leading blanks
+ TOKEN: PRINT
+ TOKEN: 3X
+ OPERAND
  + SIMPLE_CONSTANT_REFERENCE
    + TOKEN: 'Hello'
Now PRINT is a statement and could be recognised as one and 'Hello' is a string. This makes defining custom rules possible but pretty hard. To help me understand the AST I created a visitor which dumps the tree as XML file, similar to PMD's designer: DumpAstAsXml.java.

Conclusion
With this information you should be able to get started defining your own NATstyle rules. There is always so much more we could and should check automatically.

13 August 2018

This is the last time

It is August, it is hot and it is time to do some alternate work:

Bare Brickwork
Quick Fix
Looking at the picture made me think. Every industry seems to have its quick fixes: Engineers use duct tape and WD-40. In software development, we take short cuts by violating existing structure, e.g. adding another conditional resulting in convoluted logic. And in the construction industry it clearly is the usage of polyurethane or spray foam. As every tool it has its use, and like in the picture it is especially useful for temporary work. Unfortunately most professional construction workers I commissioned wanted to use it for everything - like duct tape: A hole in the wall? Why use brick and mortar when it is faster to put some foam into it. A gap in the boards? Why care to work accurately when it is easier to close the gap with foam afterwards. Not enough steam brake to cover the ceiling? Let's fill the remaining area with foam. You get the idea. While some people like the speed such quick fixes provide, e.g. Joel Spolsky praises the Duct Tape Programmer, I do not agree. Anyway, this project must come to an end and I sincerely hope it is the last time. ;-)

31 July 2018

Strong Opinions

As Code Cop I meet a lot of developers and I have to tell some of them that their code is crap and that they need to drive down their technical debt. Others need to test more, maybe start doing TDD. There are a few who would not listen. Some of them are even proud of stubbornly not listening, they say that they have strong opinions about the topic. They do not seem interested in in another view on their code or design? Why is that? I have some ideas.

GorillaDunning Kruger Effect
The Dunning Kruger Effect is a cognitive bias in which people of low ability have illusory superiority and mistakenly assess their cognitive ability as greater than it is. In other words, beginners believe themselves senior and think that they know better. There is no need to listen to an outsider.

I know this effect myself: When I learnt my first programming language(s), as soon as I could write a few lines of code, I felt invincible. (I thought that) I could do everything. I ruled. I knew how to do it and I knew I was right and that there was no other way to do it properly. Today, after writing code for 20 years in more than 25 languages, I do not feel like that any more. Sometimes I would like to go back to my state of mind of the late nineties and I try to look at new languages like a child - with a beginner's mind - but I know too much details. I know there will be nasty details, even for that newest and hottest languages of today.

Expert Beginner
An expert beginner has quickly reached (what looks like) expert status. He or she voluntarily ceases to improve because of a belief that expert status has been reached and thus further improvement is not possible. (I recommend reading the whole series of Erik Dietrich, How Developers Stop Learning: Rise of the Expert Beginner and further posts of his series.) Expert beginners make up defending arguments because they are the experts.

I have met some of these. I vividly remember two senior developers who had been with one of my clients for more than 16 years. They were opposing me on everything I said. Sure they had a superior knowledge of the application they had built and maintained for so many years, and they knew the domain they were working in pretty well. I am sure they were adding value to the product. They both were strong influencers. As soon as they would speak their mind, all other team members would just agree. But, but, but... Sigh. They were still using Ant, did not know Maven, did not write unit tests, did not care for clean code and so on - they made me very unhappy.

Quadrants of Knowledge
The quadrants of knowledge seem connected to my previous points. The four quadrants are
  1. Known Knowns: What you know that you know
  2. Unknown Knowns: What you do not know that you know
  3. Known Unknowns: What you know that you do not know
  4. Unknown Unknowns: What you do not know that you do not know
When we advance our career in software delivery, we learn: We collect concrete knowledge (quadrant 1), we gain experience which allows us to have gut feelings about some things (maybe quadrant 2) and we hear about things that we have no idea about (quadrant 3). There are gazillion things in software that I have no idea of: ANTLR, APL, AWS, Category Theory, Distributed Computing, Elm, Gradle, Guava, Haskell, HBase, J2EE, Kubernetes, Mongo DB, Networking, Node.JS, Prolog, Security - these are just the first few that come to my mind. It is easier to list things which I do not know in areas that I do know. I have been Java developer for 15 years and there are so many things in the Java ecosystem which I have no clue about. On the other hand there are not many things I can say about C#, because I only know it little.

So the more we learn, the more we know that we do not know, increasing our Known Unknowns (quadrant 3). The philosopher Confucius (551–479 BC) even said, Real knowledge is to know the extent of one's ignorance. Like the Dunning Kruger Effect, beginners have no idea about the vast size of Known Unknowns quadrant and do not expect surprises or potential improvements.

Strong Opinions, Weakly Held
The ideas of the Dunning Kruger Effect, Expert Beginners and Known Unknowns offer some explanation why some junior developers are not open for discussions. I have also met senior and expert level developers who are the same. When attending Software Crafting unconferences, e.g. SoCraTes, attendees are eager to learn and open for discussion. After all, that is the idea of unconferences, right? Still, many of them have strong opinions about almost everything, e.g. Spring Boot is cool. No, Spring Boot is hell. ;-) Different from beginners, all of them know the problems of strong opinions and claim that they only hold onto them weakly.

Deal with itWeakly Held, what is that supposed to mean? Isaac Morehouse explains in These Four Words Will Help You ‘Hold Strong Opinions Weakly': You act as if they [your opinions] are true unless and until it is proven they are not. Maybe people like strong opinions because they give them the power of definiteness. Choices are much easier and life is more predictable with definiteness and absolutes. Holding opinions weakly also gives the power of openness. This sounds hard: Basing one's actions on some definite "truths" and re-evaluating these truths whenever they are challenged. Seems like a difficult balancing act, if not a contradiction. For sure that state of mind is not easy to get into. I still need to see someone with strong opinion - weakly held or not - to change his or her mind,

Rant
When I started this article, it was supposed to be a rant. As usual, writing down my thoughts helped me to structure the material in a better way. (I consider writing an act of learning.) So where is the rant? Here it is:

Person with strong opinion == arsehole who does not want to listen.

That is a bit harsh, I agree. Obviously beginners need to listen more and due to Known Unknowns I expect experienced and expert developers to be open to discussion on any topic at any time. The strong opinion itself is not the problem, but the resulting behaviour, especially if strong opinion is used as warning or justification of some sort. I witness strong opinion mainly as excuse to not discuss or challenge existing views.

What about me?
Of course I had strong opinions during my professional life. I did not earn the nick name of Code Cop for nothing. Someone even called me Code Nazi. And my opinions were definitely not weakly held. I like absolutes. (Now I see - as I just learned above - the power of definiteness.) I guess I still have some strong opinions, and they are not just opinions, they are dogma: a system of principles proclaimed as unquestionable.

I have changed and it seems that the Code Cop is getting soft. ;-) The extended experience and discussions have weakened my definiteness. There is always some special case, the almighty it depends. And since I am working with people, and try to help them, I am more compassionate. I understand their problems, the difficulty to get time for quality and the pain of legacy code. Sometimes I catch myself not recommending what I believe the technically best course of action because it will be a lot of unpleasant, hard work for the team. No, that is too soft. If they made a mess, they have to fix it. Time to fall back on dogma. (Sound of firing cannons in the background ;-) All the time these balancing acts...

25 July 2018

Only modified files in Jenkins

For a custom Jenkins build I needed to know all changed files since the last green build. I searched a lot and found a solution as combination of several StackOverflow answers. It took me some experimenting to get it working: I installed the Groovy plugin, configured the Groovy language and created a script which executed as system Groovy script. Here is the complete step by step guide for Jenkins 2.63, SVN and Groovy 2.4.11.

Jenkins only provides the current revision in the environment variable $SVN_REVISION. Of course Jenkins knows the information about changed files of each build as it is shown in the build status page. I guess a plugin would be able to access the model, but that is too much work. Fortunately the Jenkins Groovy plugin allows scripts to run under the system context having access to hudson.model.Build and other classes.

Groovy!Groovy Programming Language
The Groovy programming language is a dynamic language which runs on the JVM. It integrates smoothly with any Java program and is the first choice for scripting Java applications. While not strictly necessary I recommend downloading the SDK's zip and unpacking it on the host where you run Jenkins, usually into the folder where you keep your development tools. For testing and debugging I also install it on my local workstation (in the same location).

Groovy in Jenkins
Next comes the Jenkins Groovy plugin. Open Jenkins in the browser and navigate the menus:
  • Manage Jenkins
  • Manage Plugins
  • select tab Available
  • filter "groovy"
  • select Groovy
  • Install
(You will need Jenkins admin rights to do so.) Then tell Jenkins which Groovy to use and where to find it. To configure the Groovy language go to
  • Manage Jenkins
  • Global Tool Configuration
  • go to section Groovy
  • Add Groovy: Give it a name and set GROOVY_HOME to the folder you unpacked it, e.g. /tools/groovy-2.4.11.
  • deselect Install automatically
  • Save
Now Jenkins supports Groovy scripts.

Run a Groovy script in the build
Now let's use a Groovy script in the project. On the project page,
  • Configure
  • go to section Build
  • Add build step
  • select Execute system Groovy script
  • paste Groovy code into the script console
  • Save
Now when you trigger the build, the script will be executed.

Debugging the Script
Of course it does not work. How can I debug this? Can I print something to the console? Groovy's println "Hello" does not show up in the build log. Searching again, finally the gist by lyuboraykov shows how to print to the console in system scripts: Jenkins provides the build console as out variable,
out = getBinding().getVariables()['out']
which can be used like out.println "Hello". Much better, now I can debug. Let's wrap the out.println in a def log(msg) method for later.

The MisfitsGetting the changed files of the current build
StackOverflow answer by ChrLipp shows how to get the changed files of the current build:
def changedFilesIn(Build build) {
  build.getChangeSet().
    getItems().
    collect { logEntry -> logEntry.paths }.
    flatten().
    collect { path -> path.path }
}
This gets the change set hudson.scm.ChangeLogSet<LogEntry> from the build, gets the SubversionChangeLogSet.LogEntrys from it and collects all the paths in these entries - this is the list of all file paths of all changed items in all commits (LogEntrys). I guess when another SCM provider is used, another type of ChangeLogSet.LogEntry will be returned, but I did not test that. To better understand what is going on, I added explicit types in the final Groovy script, which will only work for Subversion projects.

Getting all builds since the last successful one
I want all changed files from all builds since the last green one because they might not have been processed in previous, failed builds. Again StackOverflow, answer by CaptRespect comes to the rescue:
def changedFileSinceLastSuccessfull(Build build) {
  if (build == null || build.result == Result.SUCCESS) {
    []
  } else {
    changedFilesIn(build) +
      changedFileSinceLastSuccessfull(build.getPreviousBuild())
  }
}
In case there is no previous build or it was successful the recursion stops, otherwise we collect changed files of this build and recurse into the past.

All Together
Let's put it all together,
def changedFiles() {
  def Build build = Thread.currentThread()?.executable
  changedFileSinceLastSuccessfull(build).
    unique().
    sort()
}
After collecting all duplicates are removed, as I do not care if a file was changed once or more times, and the list is sorted. In the end the list of changed files is saved as text changed_files.log into the workspace. (The full script is here.)

Leave space to VIPs and journalistsWhile developing the script, the Jenkins script console was very handy. As soon as the script worked, I created a file jenkins_list_changed_files.groovy, put that under version control and changed the build definition step to use the script's file name. Next time the build ran, the script file would be executed, or at least so I thought.

Script Approvals
Unfortunately system Groovy script files do not work as expected because Jenkins runs them in a sandbox. Scripts need certain approvals, see StackOverflow answer by Maarten Kieft. To approve a script's access to sensitive fields or methods navigate to
  • Manage Jenkins
  • In-process Script Approval (This is the one but last item in the list.)
  • Approve
The sandbox is very restrictive, the full jenkins_list_changed_files needs a lot of approvals:
field hudson.model.Executor executable
method groovy.lang.Binding getVariables
method hudson.model.AbstractBuild getChangeSet
method hudson.model.AbstractBuild getWorkspace
method hudson.model.Run getNumber
method hudson.model.Run getPreviousBuild
method hudson.model.Run getResult
method hudson.scm.SubversionChangeLogSet$LogEntry getPaths
method java.io.PrintStream println java.lang.String
new java.io.File java.lang.String
staticMethod java.lang.Thread currentThread
staticMethod org.codehaus.groovy.runtime.DefaultGroovyMethods flatten java.util.List
staticMethod org.codehaus.groovy.runtime.DefaultGroovyMethods println java.lang.Object java.lang.Object
staticMethod org.codehaus.groovy.runtime.DefaultGroovyMethods sort java.util.Collection
staticMethod org.codehaus.groovy.runtime.DefaultGroovyMethods withWriter java.io.File groovy.lang.Closure
Creating a new java.io.File might be a security risk, but even println is not allowed. Adding all these approvals is a boring process. The build breaks on each missing one until everything is well. As soon as you have all the approvals, you can copy Jenkins' scriptApproval.xml found in JENKINS_HOME (e.g. ~/.jenkins) and store it for later installations. Here is mine.

Conclusion
Jenkins' Groovy integration is very powerful. System scripts have access to Jenkins' internal model which allows them to query information about build, status, changed files etc. On the other hand, development and debugging is cumbersome and time consuming. IDE support helps a lot. Fortunately StackOverflow knows all the answers! ;-)

23 July 2018

Three Rules of Coaching Engagement

Last year software crafter and Kotlin enthusiast Oleksii Fedorov asked me about technical coaching. He was going to work with a small team and focus on technical enablement. He would do so by pair and mob programming with them for some time and later support them remotely if needed. During that time there would be no pressure or deadline to deliver new features so they could focus on improving.

Oleksii is a like-minded peer. I met him on several occasions in the past. He is fun to work with and considers teaching an integral part of pair programming. I did not expect any problems. He kept pushing me and made me think about my principles. In the end I came up with three points I like to make clear right from the beginning, i.e. set the ground rules of the engagement.

Drill Sergeant Giving Orders (maybe not coaching ;-)1. Manage expectations
Managing expectations is crucial. I learnt that the hard way during my Pair Programming Tour. Whenever I run a session, e.g. pair or mob programming and often also prepared workshops, I start with a discussion to collect expectations. For longer contracts I recommend to start with a dedicated meeting to do so. Usually expectations are fuzzy, like "I want to learn something", so keep pushing with questions like "How will you know it was worth it? How will you measure that you learned something?" and so on. In the end I want the team to come up with ways to measure their learning and define their own success metrics.

2. Learning is their responsibility
Oleksii planned for two weeks. In two weeks people will not change (much). Such a coaching engagement does not last long enough for much impact, rather it is a start of a (maybe even life long) journey of learning and improving. This is true from day one and during the initial meeting I make it clear that I am not a teacher. I will not push knowledge onto them to be forgotten the next day. To get something from working together, people have to participate, to engage, to invest time and effort into the topic. They will only get so much from it what they have put into it themselves. Learning is their responsibility.

3. Treat them as strangers
After removing the responsibility to teach anything - because learning is their responsibility - I feel less stressed. As coach I am still responsible for creating a safe space, providing content, managing exercises and a lot of other things. If someone does not want to participate or feels like doing something else, I have to be OK with that. Even being paid for "teaching" should not change that. My advice here is to treat the team members as "strangers", like we treat new visitors of a Coding Dojo. As a facilitator I try to help, but I will not oppose or coerce.

Feedback
Oleksii started the job soon afterwards. I was sure everything would work out fine. When we met again he told me that my three rules had helped him. 1. He collected expectations in the beginning and managed them while moving forward. 2. People understood that they had to invest and some participants even showed up for sessions during their free time. 3. People where entering and leaving sessions all the time, still he stayed relaxed.

13 July 2018

Categories of Architectural Refactoring

At the end of one of my refactoring workshops, we started discussing how to refactor
larger structures, sometimes even changing the architecture of a system. During that discussion we came up with four different categories of larger refactoring which use different techniques and serve different goals:
  • Replace whole parts
  • Change the structure of components
  • Similar change all over the place
  • Other changes
Seoul SkyscrapersWe struggled to find good names, so let me explain the different things a bit. When I talk about architecture, I mean Software Architecture of a single application, i.e. 1. the different software components and 2. their (inter-) dependencies making up the application. Components are sub-systems and usually contain one or more namespaces or packages. While we can discuss forever what exactly software architecture is and is not, I like the definition of IEEE which in addition to elements (1.) and relationships (2.) adds principles of its design and evolution. I describe these principles as 3. design guidelines which include smaller recommended usage patterns and coding conventions.

Martin Fowler and Neal Ford say that architecture is the stuff that's hard to change later and replacing parts of the architecture involves a lot of work: Imagine (the highly theoretical example of) changing the programming language a system is written in or (a more common example of) moving from a monolith to a Microservice based architecture. These two examples already fall into different categories.

Replacing whole parts of the architecture
Massive changes like changing the UI technology or targeting another platform are possible but a lot of code has to be rewritten. Substitute Algorithm is a classic Fowler refactoring, and so is Substitute Architectural Decision. (This is my third try to name these category of changes.) Architectural Decisions are made to support non-functional requirements like performance, security, connectivity etc and when revisiting them later - with more domain knowledge - better alternatives might come up. Real examples of changing Architectural Decisions include:
  • Replacing one relational database system with another
  • Replacing your relational database with a NoSQL one
  • Replacing UI technologies, e.g. moving from JSF to Vaadin
  • Changing API technologies, e.g. moving from SOAP to REST
  • Switching from a fat client to a client-server architecture
Depending on used frameworks and abstractions, some of these changes will be easy, others will not. Even when the necessary change is isolated, it likely touches more than one architectural component. For example moving to another data store might need changing all data access objects in all components which access the data store. The traditional approach is to change everything at once ("Big Leap") but it is possible to refactor the architecture in a safe way, using small steps. In Responsive Design (QCon 2009) Kent Beck describes several strategies to perform such changes:
  • Leap: Make the new design and use it immediately. (Only if the change is not too big.)
  • Parallel Change: Make the new design and run both new and old simultaneously.
  • Stepping Stone: Create a platform to bring the desired new design within reach.
  • Simplification: Only implement part of the design now and add complexity bit-by-bit later.
  • Hack it: Just do it and pay price later. (This is maybe not a real strategy ;-)
Changing the structure of architectural components
The next category is similar to Substitute Architectural Decision but is focused on the architectural building blocks - the components. Some examples are:
  • Introduce or extract new component, module or package
  • Remove or invert dependencies between components, e.g. remove cyclic dependencies
  • Introduce layering, add a new layer, clean up layering violations
  • Weaken potential trouble spots in the architecture, e.g. Butterfly, Breakable, Hub and Tangle
foundationThis category of changes is the "classic" Architectural Refactoring because that is how people call it. For example - in accordance to Martin Fowler's definition - Dave Adsit defines Architectural Refactoring as intentionally changing the structure of a system without altering its features. Similarly Sean Barow explains Architectural Refactoring as the deliberate process to remove architectural smells while not changing the scope or functionality of the code. So it is always structural and deals with components (1.), their relationships (2.) and their inner structure. Let's call it Refactor Architectural Structure to distinguish it from other architectural refactoring.

Sean Barow uses the term Code Refactoring for the refactoring defined by Fowler. Code refactoring focuses on software entities like classes, methods, fields and variables. Architectural refactoring involves code refactoring, which leads to confusion between the two. Working with dependencies needs extracting or moving classes, methods or fields. We can work each class or dependency at once, which supports an incremental approach.

Performing similar changes throughout the whole project
This is another category and it feels like Shotgun Surgery: A usually small change has to be repeated all over the place. (While Shotgun Surgery is a code smell for feature logic, it is unavoidable if not required for consistent structure and usage defined by the architecture.) For example let's assume we want to replace EasyMock with Mockito. Both libraries serve the same purpose and are used in a similar way, but their APIs are different. (Facing such a problem indicates that we are coupled to much to a used library - maybe we should have wrapped it, maybe not. But that discussion is outside the scope of this article.) Converting a single usage of EasyMock to Mockito is straight forward: We change the method invocations and execute the calls earlier or later in the test. The only problem is that we have hundreds of such cases. More examples of similar "Shotgun" changes are:
  • Upgrading a major version of a library with breaking API changes
  • Migrating between similar frameworks or libraries
  • Changing or unifying coding conventions and fixing violations
  • Consistently applying or changing aspects like logging or security
  • Applying internationalization
These changes do not change the overall structure, they just work on the implementation. They are similar to Refactor Architectural Structure, just deal with design guidelines, coding conventions and smaller usage patterns - item 3 from the definition of Software Architecture in the beginning. They are code refactoring applied in many places consistently across a whole code base.

These category is important, because some steps of both Substitute Decision and Refactor Structure need similar changes. The main challenge of these changes is the high number of occurrences. Changing conventions and aspects is a piece by piece work and can be done manually. Nevertheless some automation tool or script would help a lot. Even converting only basic scenarios, which often cover up to 80%, is a big win. Unfortunately some changes, e.g. migrating libraries, need to be done all at once. In such cases a Stepping Stone approach has worked well for me in the past: First wrapping the library piece by piece - incrementally - and then changing the wrapper and all invocations at once. Creating the wrapper is a lot of work. Again, automated migration removes the need for a wrapper. There are some options available, which I will cover in a future article.

I have not found a good name for this category. I used to call it Large Scale Refactoring, but I am not happy with that name. It is a similar or identical code refactoring applied to many places widespread throughout the code. It is a Widespread Architectural Change.

Other Architectural Changes
During our initial discussion, we also considered other refactoring. To be on the safe side, always allow a category of unknown things ;-). At the moment I have no real examples for them. These are smaller changes neither related to architectural decisions or structure nor are they widespread. I believe we find no examples because these are plain code changes and we do not consider them architectural even if they are related to the architecture of the applications in a way.

FavelaSummary
Next to the code we also can and should refactor the architecture of our applications. Architectural changes are expensive, still sooner or later things need to change if the application is in service just long enough. There are at least four categories of architectural refactoring:
  • Substitute Architectural Decision
  • Refactor Architectural Structure
  • Widespread Architectural Change
  • Other Architectural Changes
Next time I will dive into tools to automate Widespread Architectural Changes.

3 July 2018

Using NATstyle from the commandline

Last year a new client contracted me to help them make their development process, their development environment and tooling "state of the art" (aka up do date). From the coding perspective that included - among other things - using a decent IDE, writing unit tests, having some Continuous Delivery pipeline in place and using static code analysis. I thought that it should not be too difficult, right? It was, but that is not the goal of this article. Today I want to describe the static analysis features and how to use it.

Earth Sapphire Progress Picture 30 july-2011NATURAL
The code base was NATURAL. NATURAL is an application development and deployment environment using a proprietary language maintained by Software AG. It was created in 1979. André describes it as Cobol's ugly, brain-damaged, BABBLING IN ALL-CAPS - but regrettably still healthy and strong - cousin.. ;-) Well, it is a procedural language structured using modules, subroutines and functions. It feels as old as it is. For example, module names are limited to eight (8!) characters and there is no way to structure the modules further. An enterprise application might contain 5.000 to 10.000 modules. To me NATURAL feels much like early Pascal: units (modules), subroutines and functions together with the eight characters DOS name limit. I really liked Pascal back then and I do not feel that bad about NATURAL. Probably also because I do not have to use it on a daily basis.

NaturalONE and NATstyle
NaturalONE is the Eclipse-based tool for NATURAL by Software AG. It looks pretty good, similar to what you would expect from Eclipse. It contains the usual features like Outline, Search, Debug and something called NATstyle. Now NATstyle - which probably on purpose sounds similar to the well known Checkstyle - is an utility to define and check the coding standard in your programs. It is used inside NaturalONE, see a NATstyle Demo by Software AG. (For more information refer to the Eclipse/NaturalONE help topic Checking Natural Code with NATstyle.)

Invoking NATstyle from outside NaturalONE
I like static code analysis and consider it a mandatory component of every delivery pipeline. As I said above, I wanted to have some static code analysis and was wondering if I would be able to run NATstyle in the pipeline? I did not find any information on the topic and was forced to experiment. I used NaturalONE 8.3.5.0.242 CE (November 2016) and figured out several ways, which were hardly documented, and might be subject to change. Likely there are other options available in newer versions of NaturalONE and NATstyle.

Invoking NATstyle from command line
NATstyle is just an Eclipse plugin inside NaturalONE. In my edition it is the folder C:\Natural-CE\​Designer\​eclipse\​plugins\​com.​softwareag.​naturalone.​natural.​natstyle_8.3.5.0000-0242. With the proper class path it is possible to invoke NATstyle's main method, which prints help on its usage:
Usage: com.softwareag.naturalone.natural.natstyle.NATstyle [-options]

where options include:
  -projectpath <directory> Specify where to find the Natural project.
  -rootfolder              Library root folder support enabled.
  -c <file>                Specify the configuration file.
  -o <file>                Specify the output file.
  -sourcefiles <srcList>   Specify list of source files to be loaded.
                           separated with ;
  -libraries <libList>     Specify list of libraries to be loaded.
                           separated with ;
  -exclude <libList>       Specify a list of libraries to exclude.
                           separated with ;
  -p <directory>           Specify where to find additional packages.
  -help                    Display command line options and exit.

Make sure the following classes can be found in your class path:

com.softwareag.naturalone.natural.auxiliary
com.softwareag.naturalone.natural.common
com.softwareag.naturalone.natural.parser
Nice, that looks like the developers from Software AG prepared NATstyle to be called stand-alone in my pipeline. +1. All the necessary classes can be found in the following plugins of Eclipse/NaturalONE:
  • Folder com.softwareag.naturalone.natural.natstyle_8.3.5.0000-0242
  • Jar com.softwareag.naturalone.natural.auxiliary_*.jar
  • Folder com.softwareag.naturalone.natural.common_8.3.5.0000-0242
  • Jar com.softwareag.naturalone.natural.parser_*.jar
  • Jar org.eclipse.equinox.common_*.jar
These are all bundles defined by Eclipse/OSGi. The list of dependencies of these bundles also contains the following Jars
  • org.eclipse.swt.win32.*.jar
  • org.eclipse.ui.console_*.jar
which (in my experience) are not needed to use NATstyle. NATstyle does not use any other OSGi features and we can set the needed class path by hand. Here is an example how to do that in the Windows shell when NaturalONE is installed:
set N1=C:\Natural-CE\Designer\eclipse\plugins
set CLASSPATH=^
%N1%\com.softwareag.naturalone.natural.natstyle_8.3.5.0000-0242;^
%N1%\com.softwareag.naturalone.natural.auxiliary_8.3.5.0000-0242.jar;^
%N1%\com.softwareag.naturalone.natural.common_8.3.5.0000-0242;^
%N1%\com.softwareag.naturalone.natural.parser_8.3.5.0000-0242.jar;^
%N1%\org.eclipse.equinox.common_3.6.200.v20130402-1505.jar

java com.softwareag.naturalone.natural.natstyle.NATstyle %*
See run_natstyle for the script I used to run NATstyle directly. Copying the needed folders and Jars to another machine works well. I recommend to create a Jar from the folders before copying them around. create_jar how to do that. Make sure not to include META-INF\*.RSA or META-INF\*.SF into the new Jars because the checksums do not match. copy_jars does the actual copying into the lib folder.

Invoking NATstyle from Ant
NaturalONE supports Ant for some automation tasks. There is no specific NATstyle Ant task, but calling it from Ant is straight forward. If the path <path id="natstyle.classpath"> is set up to contain all the Folders and Jars, NATstyle is executed via Ant's java task:
<java taskname="natstyle"
      classname="com.softwareag.naturalone.natural.natstyle.NATstyle"
      dir="${basedir}" fork="true" maxmemory="128m"
      failonerror="true"
      classpathref="natstyle.classpath">
  <arg value="-projectpath" />
  <arg value="../NatUnit/NatUnit_L4N" />
  ...
</java>
A new JVM needs to be forked because NATstyle calls System.exit() at the end. All parameters are the same as for the command-line invocation and are passed in via the <arg /> element. See ant_NatStyle.xml for the complete Ant script.

Reporting
NATstyle writes an XML report of all files if worked and found rule violations. The structure is
<NATstyle>
  <file type='...' location='...'>
    <error severity='...' message='...' />
    ...
  </file>
  ...
</NATstyle>
Inside the folder of NATstyle (i.e. com.softwareag.naturalone.natural.natstyle_8.3.5.0000-0242) there is a subfolder NATstyle. It contains sample files. Using the provided NATstyleSimple.xsl the report XML (e.g. NATstyleResult.xml) can be transformed into a human-readable format (e.g. NATstyleSimple.html). When using NATstyle stand-alone, you might want to copy this folder as well. XSLT transformation task is part of Ant and converts the XML to HTML.
<xslt basedir="${basedir}"
      destdir="${basedir}/report"
      style="${basedir}/NATstyle/NATstyleSimple.xsl">
  <include name="NATstyleResult_*.xml" />
</xslt>
See ant_NatStyleResultToHtml.xml for the complete Ant build script to convert the NATstyle result XML into a readable HTML report. Using your own stylesheet (.xsl) allows you to customize the reports.

Next time I will show you how to write your own NATstyle rules.

29 June 2018

xUnit Testing Koans Reloaded

Koans are sequences of little exercises to help you "on your way to enlightenment". Testing Koans are similar exercises to help you learn unit testing. Usually the code is in place, but the assertions are missing. xUnit Koans are Testing Koans for SUnit inspired testing frameworks, like JUnit, minitest, NUnit, PHPUnit, xUnit.net and more. They are introductory exercises which cover the most basic usage of assertions and life cycle methods.

JUnit5 Koans RetrospectiveJUnit 4 and PHPUnit
Initially I created a few JUnit Testing Koans in 2012 as part of a JUnit training while I was with IBM. I wanted the training to be hands-on and interactive. (These koans are in the junit-koans repository.) In 2015 I picked up the idea and ported the code to PHP to run a PHPUnit workshop for my local community. (These koans are in the phpunit-koans repository.)

JUnit 5
In 2017, with the new version of JUnit, Görge Albrecht, the Code Mentor, helped me add koans for new features of JUnit 5. We used the koans as hands-on exercises which were a part of our JUnit 5 introductory workshop, delivered at Topconf Linz 2017 and other conferences. The exercises worked well and gave an overview of the (old and) new API and showed how to use it. People liked the workshop and had fun, see the points made during the retrospective of one of the workshops on the right. (The code is in the junit5-koans repository.)

Python's unittest
It seems that the koans keep coming back to me ;-) - and I keep coming back to them, always adding more exercises and languages. This year I ported them to Python, which took me around five hours. I ran them as a short, Pragmatic Introduction to Python Unit Testing at PyDays 2018. I liked PyDays for its enthusiastic crowd and consider the workshop successful because in the final feedback round several participants reported happily that they had "written their first test" and that "it's not that hard!". Even experienced participants reported some learnings, like "I never used skip" or "I did not know how to check the message in asserting exceptions". (The latest and greatest koans are in the python-unittest-koans repository.)

Teaching xUnit
Koans, i.e. learning progress verification by assertions, work well in both static and dynamic languages. There is no difference in workshops using Java and PHP or Python. Testing Koans are a suitable exercise to help people with their first steps in unit testing. In short two hours, people who have never seen a test will learn the concepts and flow of unit testing. I usually start with a few slides, about 20 minutes of theory about structure of xUnit tests before I ask people to work with the exercises.

JUnit5 handouts by GörgeLike with JUnit 5, Testing Koans also work well to introduce people to new testing framework APIs and their usage. People familiar with JUnit 4 will need no more than a migration guide, see Görge's handouts on the right, to get their first impression of the new features. Some more advanced aspects will require them to research the documentation, but that is part of the exercise.

People appreciate the "focused, step by step nature" of koans. The "exercise is clear" and "invites to explore xUnit". And most importantly "it was fun". In all the feedback I have collected for my xUnit Testing Koans, fun is dominant. And fun helps learning. (All quotes were taken as is from actual feedback I got.)

Moar Testing Koans!
Most xUnit frameworks are consistent with xUnit, and are similar in their usage. It is easy to port the koans to other languages, at least for basic examples. The same is true for newer versions. I should upgrade some koans to accommodate newer features of the used frameworks, on the other hand the basic features are still the same. I would love to see more ports and also Koans for different testing styles, e.g. RSpec or Jasmine Testing Koans.

Current List of Koans

25 February 2018

Complete Cofoja Setup Example

Design by Contract
Have you heard of Design by Contract (short DbC)? If not, here is a good introduction from Eiffel. (In short, Design by Contract is one of the major mechanisms to ensure the reliability of object-oriented software. It focuses on the communication between components and requires the interactions to be defined precisely. These specifications are called contracts and they contain Preconditions, Postconditions and Invariants. Unlike using assertions to ensure these conditions, DbC considers the contracts important parts of the design process which should be written first. It is a systematic approach to building bug-free object-oriented systems and helps in testing and debugging.)

Cofoja (Contracts for Java)
Cofoja is a Design by Contract library for Java. It uses annotation processing and byte code instrumentation to provide run-time checking. It supports a contract model similar to that of Eiffel, with added support for a few Java-specific things, such as exceptions. In Cofoja, contracts are written as Java code within quoted strings, embedded in annotations. Here is some sample code (derived from lost icontract library): A basic stack with methods to push, pop and to see the top element.
import java.util.LinkedList;
import com.google.java.contract.Ensures;
import com.google.java.contract.Invariant;
import com.google.java.contract.Requires;

@Invariant({ "elements != null",
             "isEmpty() || top() != null" }) // (1)
public class CofojaStack<T> {

  private final LinkedList<T> elements = new LinkedList<T>();

  @Requires("o != null") // (2)
  @Ensures({ "!isEmpty()", "top() == o" }) // (3)
  public void push(T o) {
    elements.add(o);
  }

  @Requires("!isEmpty()")
  @Ensures({ "result == old(top())", "result != null" })
  public T pop() {
    final T popped = top();
    elements.removeLast();
    return popped;
  }

  @Requires("!isEmpty()")
  @Ensures("result != null")
  public T top() {
    return elements.getLast();
  }

  public boolean isEmpty() {
    return elements.isEmpty();
  }

}
The annotations describe method preconditions (2), postconditions (3) and class invariants (1). Cofoja uses a Java 6 annotation processor to create .contract class files for the contracts. As soon as Cofoja's Jar is on the classpath the annotation processor is picked up by the service provider. There is no special work necessary.
javac -cp lib/cofoja.asm-1.3-20160207.jar -d classes src/*.java
To verify that the contracts are executed, here is some code which breaks the precondition of our stack:
import org.junit.Test;
import com.google.java.contract.PreconditionError;

public class CofojaStackTest {

  @Test(expected = PreconditionError.class)
  public void emptyStackFailsPreconditionOnPop() {
    CofojaStack<String> stack = new CofojaStack<String>();
    stack.pop(); // (4)
  }

}
We expect line (4) to throw Cofoja's PreconditionError instead of NoSuchElementException. Just running the code is not enough, Cofoja uses a Java instrumentation agent to weave in the contracts at runtime.
java -javaagent:lib/cofoja.asm-1.3-20160207.jar -cp classes ...
Cofoja is an interesting library and I wanted to use it to tighten my precondition checks. Unfortunately I had a lot of problems with the setup. Also I had never used annotation processors before. I compiled all my research into a complete setup example.

Maven
Someone already created an example setup for Maven. Here are the necessary pom.xml snippets to compile and run CofojaStackTest from above.
<dependencies>
  <dependency> <!-- (5) -->
    <groupId>org.huoc</groupId>
    <artifactId>cofoja</artifactId>
    <version>1.3.1</version>
  </dependency>
  ...
</dependencies>

<build>
  <plugins>
    <plugin> <!-- (6) -->
      <artifactId>maven-surefire-plugin</artifactId>
      <version>2.20</version>
      <configuration>
        <argLine>-ea</argLine>
        <argLine>-javaagent:${org.huoc:cofoja:jar}</argLine>
      </configuration>
    </plugin>
    <plugin> <!-- (7) -->
      <artifactId>maven-dependency-plugin</artifactId>
      <version>2.9</version>
      <executions>
        <execution>
          <id>define-dependencies-as-properties</id>
          <goals>
            <goal>properties</goal>
          </goals>
        </execution>
      </executions>
    </plugin>
  </plugins>
</build>
Obviously we need to declare the dependency (5). All examples I found register the contracts' annotation processor with the maven-compiler-plugin, but that is not necessary if we are using the Maven defaults for source and output directories. To run the tests through the agent we need to enable the agent in the maven-surefire-plugin (6) like we did for plain execution with java. The Jar location is ${org.huoc:cofoja:jar}. To enable its resolution we need to run maven-dependency-plugin's properties goal (7). Cofoja is build using Java 6 and this setup works for Maven 2 and Maven 3.

Gradle
Similar to Maven, but usually shorter, we need to define the dependency to Cofoja (5) and specify the Java agent in the JVM argument during test execution (6). I did not find a standard way to resolve a dependency to its Jar file and several solutions are possible. The cleanest and shortest seems to be from Timur on StackOverflow, defining a dedicated configuration for Cofoja (7), which avoids duplicating the dependency in (5) and which we can use to access its files in (6).
configurations { // (7)
  cofoja
}

dependencies { // (5)
  cofoja group: 'org.huoc', name: 'cofoja', version: '1.3.1'
  compile configurations.cofoja.dependencies
  ...
}

test { // (6)
  jvmArgs '-ea', '-javaagent:' + configurations.cofoja.files[0]
}
Eclipse
Even when importing the Maven project into Eclipse, the annotation processor is not configured and we need to register it manually. Here is the Eclipse help how to do that. Fortunately there are several step by step guides how to set up Cofoja in Eclipse. In the project configuration, enable Annotation Processing under the Java Compiler settings.
Eclipse Project Annotation Processing
Although Eclipse claims that source and classpath are passed to the processor, we need to configure source path, classpath and output directory.
com.google.java.contract.classoutput=%PROJECT.DIR%/target/classes
com.google.java.contract.classpath=%PROJECT.DIR%/lib/cofoja.asm-1.3-20160207.jar
com.google.java.contract.sourcepath=%PROJECT.DIR%/src/main/java
(These values are stored in .settings/org.eclipse.jdt.apt.core.prefs.) For Maven projects we can use the %M2_REPO% variable instead of %PROJECT.DIR%.
com.google.java.contract.classpath=%M2_REPO%/org/huoc/cofoja/1.3.1/cofoja-1.3.1.jar
Add the Cofoja Jar to the Factory Path as well.
Eclipse_ Project Factory Path
Now Eclipse is able to compile our stack. To run the test we need to enable the agent.
Eclipse Run Configuration JUnit
IntelliJ IDEA
StackOverflow has the answer how to configure Cofoja in IntelliJ. Enable annotation processing in Settings > Build > Compiler > Annotation Processors.
IDEA Settings Annotation Processors
Again we need to pass the arguments to the annotation processor.
com.google.java.contract.classoutput=$PROJECT_DIR$/target/classes
com.google.java.contract.classpath=$M2_REPO$/org/huoc/cofoja/1.3.1/cofoja-1.3.1.jar
com.google.java.contract.sourcepath=$PROJECT_DIR$/src/main/java
(These values are stored in .idea/compiler.xml.) For test runs we enable the agent.
IDEA Run Configuration JUnit
That's it. See the complete example's source including all configuration files for Maven, Gradle, Eclipse and IntelliJ IDEA.