8 September 2018

IDE Support for Scripts

Last month I wrote about dealing with modified files during build in Jenkins. The solution uses Groovy scripts and the Jenkins Groovy plugin which allows execution of scripts in Jenkins' system context. These System Scripts have access to Jenkins' internal model like build status, changed files and so on. The model includes several classes which need to be navigated.

SupportIn many situations little additional scripts make our lives as developers easier: For example the Groovy System Script to customise Jenkins when building a NATURAL code base mentioned above, or a Groovy Script Transformation to perform repeated code changes in a large Java project using WalkMod, or a custom build during a Maven build, using any language supported by the Apache Bean Scripting Framework, e.g. Groovy, Ruby, Python or others. Now if the scripts are getting complicated IDE support would be nice.

Getting support for the language in the IDE
Depending on the language, most IDEs need installation of a plugin or extension. I will briefly describe the steps to set up Groovy support into Eclipse because it is more elaborate and the organisation I initially wrote this documentation used Eclipse. Getting plugins for IntelliJ IDEA is straight forward.
  • First obtain the version of Eclipse you are using. Customised distributions like the Spring Tool Suite, IBM/Rational products or NaturalONE follow different version schemes than Eclipse. Navigate to the Help menu, item About, button Installation Details, tab Features, line Eclipse Platform to see the real version. For example, NaturalONE 8.3.2 is based on Eclipse 4.4.
  • The Groovy plugin for Eclipse is GrEclipse. For each major version of Eclipse there is a matching version of GrEclipse. The section Releases lists the Update Site for each release. Find the suitable version. (If you use a newer Eclipse, maybe you have to use a snapshot version instead of a release version.) Copy the URL of the Update Site.
  • The section How to Install explains in detail how to continue and install GrEclipse.
After that Eclipse can talk Groovy.

Enabling support for the language in the project
Next we configure the project for the additional language. In Eclipse this is often possible through the (right click) context menu of the project, menu item Configure. I usually do not bother and edit Eclipse's .project file directly to add the needed natures and build commands. Adding Groovy to a non-Java project needs adding to .project,
  <!-- existing configuration -->
    <!-- existing configuration -->
    <!-- existing configuration -->
and .classpath
  <!-- existing configuration -->
  <classpathentry exported="true" kind="con" path="GROOVY_SUPPORT"/>
  <classpathentry exported="true" kind="con" path="GROOVY_DSL_SUPPORT"/>
Similar changes are needed for Ruby (using org.eclipse.​dltk.core.​scriptbuilder and org.eclipse.​dltk.ruby.​core.nature) and Python (org.python.​pydev.PyDevBuilder and org.python.​pydev.pythonNature). If I am not sure, I create a new project for the target language and merge the .projects manually. Sometimes other files like .classpath, .buildpath or .pydevproject have to be copied, too.

Enabling support for libraries
Adding a nature to .project gives general support like syntax highlighting and refactoring for the respective language but there is no completion of library classes. Especially when using new libraries, like Jenkins Core, using code completion helps exploring the new API because you get a list of all possible methods to call. The trick is to add the needed dependency to the project in a way that it is not accessible from or packaged together with the production code.

An Example: Adding Jenkins System Script support to a NaturalONE project
To add dependencies to the NaturalONE project earlier, I converted the Java/Groovy project to Maven (by setting another nature and builder) and added the needed classes as fake dependencies to the pom.xml.
  <!-- other dependencies -->
    <scope>test</scope> <!-- (1) -->
Groovy has optional typing and after adding the types to the definitions and arguments, Eclipse will pick up the type and we get code completion on available fields and methods:
import hudson.model.Build
def Build build = Thread.currentThread()?.executable
build._ // <-- code completion, yeah!
This made developing and understanding the script to find modified files much easier. The Jenkins dependencies in the pom.xml are never used, as the System Script runs inside Jenkins where the classes are provided. The dependencies are declared to "trick" Eclipse into analysing them and providing type information. The scope of the dependencies is set to test (see line (1) in code snippet above) so the dependencies are not packaged and cannot be called from the production code. This repository contains the NATURAL project together with all the Eclipse configuration files.

Another Example: Adding WalkMod Transformation support to a Maven project
Another situation where I wish for script support in Eclipse is when writing WalkMod Script Transformations. WalkMod is an open source Java tool that can be used to apply code conventions automatically. Read this tutorial by Raquel Pau to see how WalkMod works. WalkMod allows for Groovy scripts to define code transformations which manipulate the AST of Java classes. Navigating the AST is difficult in the beginning.

After adding the Groovy nature as shown in the previous example, the relevant dependencies to get code completion for the AST are
    <!-- API -->
    <!-- AST -->
In case of Walkmod transformations, the interesting class is (org.walkmod.​javalang.ast.​CompilationUnit) and it is already in scope as variable node. To add type information we need to alias it:
import org.walkmod.javalang.ast.CompilationUnit
def CompilationUnit cu = node
cu._ // <-- code completion on available fields and methods
Supporting scripts in Groovy, Ruby or Python make our lives easier. With some additional extensions or plugins and some tweaks in the project configuration our IDEs will support the scripting language. Why should we accept code editing like in the early 1990s?

2 September 2018

Work Harder

With this article I want to prompt you to work hard(er). What does working hard mean? When I looked for different translations of German Strengt Euch an, I found several ones, which are all suitable: Work hard. Keep it tight. Push yourselves. Put some heart into it. Put some muscle in it. Put your backs to it. Make your best effort. Just play it like you fucking mean it. Obviously hard work needs your strength. It also needs your heart - motivation and dedication. It might test your limits, both physical and psychical ones.

Domesday BooksMy favourite example of hard work was writing books in the Dark Ages. Monks were copying books by hand, adding drawings and decorations as they went. An extreme example is the Codex Gigas, which was handwritten by a single, anonymous monk. Because the scribe was a monk he may only have been able to work for about three hours a day, and this means that the manuscript including decoration probably took at least 20 years to finish, and could even have taken 30. Now that is a whole lifetime for collecting and handwriting a single book. Imagine the dedication.

Maybe less extreme is the life of master craftspeople. Several years ago I wrote about a master craftsman hand-crafting rakes. He could retire any time, still chose to improve his methods and work hard all day.

Benefits of hard work
Most of you will agree, that hard work pays off. Here is a little TED talk by Richard St. John, who explains why it is so: Hard work is the real secret to success. I vividly remember a professor of Mathematical Analysis during my studies: At the beginning of a lecture he wrote a theorem on the blackboard, followed by an easy proof. The proof looked good and we students understood what was going on. Then he said "Was man leicht kriegt ist nichts wert" ("What you get easily is worth nothing"), pointed out a mistake in the proof and erased it. He used the remaining of the hour to sketch a valid proof which was complicated and much harder to follow.

PainHard work has more value
Theodore Roosevelt even said that "Nothing in the world is worth having or worth doing unless it means effort, pain, difficulty..." Maybe he was exaggerating, but a little pain never hurts. (Pun intended ;-) So is the motto in weightlifting training: No Pain, No Gain. Psychologist Katarina Veselko mentioned the topic in one of her talks. During the following conversation she explained: It not so much that we do not value things that are easy to get, it is more about the fact that when we need to invest a lot of energy, time, money, ... into achieving something, the result will be more valuable for us because we have invested a lot into it. It is some kind of Cognitive dissonance. She also believes that most things worth having are not easy to get; most things that are valuable to us are not achieved in a way that is always fun, fast and easy, but also includes obstacles and challenges.

Instant Gratification
The term Instant Gratification is often used to label the satisfactions gained by more impulsive behaviours: choosing now over tomorrow. And today it is harder to delay gratification than it used to be. We have been trained by technology and social media to expect results fast, without much effort. For example, there are more than 2 million hits on Google how to earn 5k extra, many of then offering "work from home, 30 minutes daily" up to "basically doing nothing". The same is true for losing weight without any diet and so on. If you want to read more on Instant Gratification and its problems, I recommend Courtney Ackerman's Definition with Examples.

It seems that working hard has become unpopular. Why work towards a long term goal when many things are handed over on a plate, or downloaded at the click of a button, or ours in twenty-four hours for just £9.99 extra? I do not think that life works like that. Why do I come up with this in my blog? I am writing this because I think we Craftspeople need to work harder. I am not talking about working more hours. In fact working overtime is against the idea of Craftsmanship because we need our free time to reflect, exchange and learn.

I am talking about pushing ourselves more: Following up on that refactoring you did not finish last week. Refactoring mercilessly, always pushing to keep the whole code base clean and consistent, even during architectural changes. I am talking about not accepting lesser standards because of low level, different or even obsolete technologies or environments. I am talking about overcoming pointless bureaucracy or managers who have no idea about code quality.

Defiance Cafe
I know it is hard. Years of arguments with colleagues, fighting superiors, struggling to grow and working broken processes have taken their toll. Sometimes I feel old. But I have to defy them. Let us continue pushing, trying to work harder and make an effort!

29 August 2018

Widespread Architectural Change Part 1

Last month I introduced Four Categories of Architectural Refactoring:
  • Substitute Architectural Decision
  • Refactor Architectural Structure
  • Widespread Architectural Change
  • Other Architectural Changes
Today I want to start discussing options to perform Widespread Architectural Changes. This category contains all kind of small changes repeated throughout the whole code base. The number of these changes is high, being hundreds or sometimes even thousands of occurrences that have to be changed in a similar way. Examples of such modifications are:
  • Migrating or upgrading (similar) APIs, frameworks or libraries
  • Changing or unifying coding conventions
  • Fixing compiler warnings and removing technical debt
  • Consistently applying or changing aspects like logging or security
  • Applying internationalization
  • Migrate between languages, e.g. SQL vs. HQL
Ways to do Widespread Architectural Changes (C) by SoftDevGang 2016All of them do not change the overall structure, they just work on the implementation. Often substituting architectural decisions or altering architectural structure contain similar changes.

I have been using some techniques since 2004 because as Code Cop I value code consistency. Two years ago I had the opportunity to discuss the topic with Alexandru Bolboaca and other experienced developers during a small unconference and we came up with even more options, as shown in the picture on the right. The goal of this and the next article is to introduce you to these options, the power of each one and the cost of using it. A word of warning: This is a raw list. I have used some but not all of them and I have yet to explore many options in more detail.

Supporting Manual Changes With Fast Navigation
The main challenge of widespread changes is the high number of occurrences. If the change itself is small, finding all occurrences and navigating to them is a major effort. Support for fast navigation would make things easier. Often it is possible to create a list of all lines that need to be changed. In the past have created custom rules of static analysis tools like PMD to find all places I needed to change. Such a list can be used to open the file and jump to the proper line, one source file after another.

As example, here is a Ruby script that converts pmd.xml which is contains the PMD violations from the Apache Maven PMD Plugin to Java stack traces suitable for Eclipse.
#! ruby
require 'rexml/document'

xml_doc = REXML::Document.new(File.new('./target/site/pmd.xml'))
xml_doc.elements.each('pmd/file/violation') do |violation|
  if violation.attributes['class']
    class_name = violation.attributes['package'] + '.' +
    short_name = violation.attributes['class'].sub(/\$.*$/, '')
    line_number = violation.attributes['beginline']
    rule = violation.attributes['rule']

    puts "#{class_name}.m(#{short_name}.java:#{line_number}) #{rule}"
The script uses an XML parser to extract class names and line numbers from the violation report. After pasting the output into Eclipse's Stacktrace Console, you can click each line one by one and Eclipse opens each file in the editor with the cursor in the proper line. This is pretty neat. Another way is to script Vim to open each file and navigate to the proper line using vi +<line number> <file name>. Many editors support similar navigation with the pattern <file name>:<line number>.

Enabling fast navigation is a big help. The power this option is high (that is 4 out of 5 on my personal, totally subjective scale) and the effort to find the needed lines and script them might be medium.

Search and Replace (Across File System)
The most straight forward way to change similar code is by Search and Replace. Many tools offer to search and replace across the whole project, allowing to change many files at once. Even converting basic scenarios, which sometimes cover up to 80%, helps a lot. Unfortunately basic search is very limited. I rate its power low and the effort to use it is also low.

Scripted Search and Replace using Regular Expressions
Now Regular Expressions are much more powerful than basic search. Many editors allow Regular Expressions in Search and Replace. I recommend creating a little script, e.g. using AWK, Ruby or Python and a lot of changes can be automated. While the script adds extra work to traverse all the source directories and files, the extra flexibility is needed for conditional logic, e.g. adding an import to a new class if it was not imported before. Also in a script Regular Expressions can be nested, i.e. analysing the match of an expression further in a second step. This helps to keep the expressions simple.

For example I used a script to migrate Java's clone() methods from version 1.4 to 5. Java 5 offers covariant return types which can be used for clone(), removing the cast from client code. The following Ruby snippet is called with the name of the class and the full Java source as string:
shortName = shortClassNameFromClassName(className)
if source =~ / Object clone\(\)/
  # use covariant return type for method signature
  source = $` + " #{shortName} clone()" + $'
if source =~ /return super\.clone\(\);/
  # add cast to make code compile again
  source = $` + "return (#{shortName}) super.clone();" + $'
In the code base where I applied this widespread change, it fixed 90% of all occurrences as clone methods did not do anything else. It also created some broken code which I reverted. I always review automated changes, even large numbers, as jumping from diff to diff is pretty fast with modern tooling. Scripts using Regular Expressions helped me a lot in the past and I rate their power to high. Creating them is some effort, e.g. a medium amount of work.

Macros and Scripts
Many tools like Vim, Emacs, Visual Studio, Notepad++ and IntelliJ IDEA (by use of a plugin) allow creation or recording of Keyboard and or mouse macros or Application scripts. With them, frequently used or repetitive sequences of keystrokes and mouse movements can be automated, and that is exactly what we want to do.

Alex has used Vim Scripts to automate repetitive coding tasks. I have not used it and relating to his experience. Here is a Vim script function which extracts a variable, taken from Gary Bernhardt's dotfiles:
function! ExtractVariable()
  let name = input("Variable name: ")
  if name == ''

  " Enter visual mode
  normal! gv
  " Replace selected text with the variable name
  exec "normal c" . name
  " Define the variable on the line above
  exec "normal! O" . name . " = "
  " Paste the original selected text to be the variable value
  normal! $p
I guess large macros will not be very readable and hard to change but they will be able to do everything what Vim can do, which is everything. ;-) They are powerful and easy to create - as soon as you know Vim script of course.

Structural Search and Replace
IntelliJ IDEA offers Structural Search and Replace which performs search and replace across the whole project, taking advantage of IntelliJ IDEA's awareness of the syntax and code structure of the supported languages.. In other words it is Search and Replace on the Abstract Syntax Tree (AST). Additionally it is possible to apply semantic conditions to the search, for example locate the symbols that are read or written to. I have not used it. People who use it tell me that it is useful and easy to use.

Online help says that you can apply constraints described as Groovy scripts and make use of IntelliJ IDEA PSI (Program Structure Interface) API for the used programming language. As PSI is the IntelliJ version of the AST, this approach is very powerful but you need to work the PSI/AST which is (by its nature) complicated. This requires a higher effort.

To be continued
And there are many more options to be explored, e.g. scripting code changes inside IDEs or using Refactoring APIs as well as advanced tools outside of IDEs. I will continue my list in the next part. Stay tuned.