Widespread Architectural Changes using IDEs

This is the second part of my list of tools to automate Widespread Architectural Changes. Now Widespread Architectural Change is a category of architectural refactoring which do not change the overall structure but need to be applied in many places consistently across a whole code base. For example, consider changing or unifying coding conventions or fixing violations. The main challenge of these changes is the high number of occurrences.

The goal of this article is to introduce you to ways to automate such changes. I have been using some of these techniques because as Code Cop I value code consistency, still this is a raw list. I have only used a few of the mentioned tools and need to explore many in more detail.

In the first part I listed some basic options, e.g. how to support manual changes with fast navigation, search and replace across the whole file system, use scripted search and replace using regular expressions, Macros and finally JetBrains' Structural Search and Replace. This time I focus on options available for modern IDEs. Tools like Eclipse or IntelliJ use an internal representation of the code, the Abstract Syntax Tree (AST). Structural Search and Replace works the AST and therefore belongs into this group as well.

Rescripter (Eclipse)
Modern IDEs support many Refactoring steps - why not use the existing tools to make our job easier? One of such tools is Rescripter, built for making large-scale changes that you can describe easily but are laborious to do by hand. Rescripter is an Eclipse plugin which runs JavaScript in the context of the Eclipse JDT. David Green added some helper objects to make searching, parsing and modifying Java code easier and wrapped a thin JavaScript layer around it. Rescripter scripts are working the AST.

Here are two examples from Rescripter's documentation: Finding a method named getName inside the class Person:

var type = Find.typeByName("Person");
var method = Find.methodByName(type, "getName").getElementName();

and adding a method getJobTitle to that type:

var edit = new SourceChange(type.getCompilationUnit());
edit.addEdit(ChangeType.addMethod(type, "\n\tpublic String getJobTitle() {\n" +
                                        "\t\treturn this.jobTitle;\n" +
                                        "\t}"));
edit.apply();

Rescripter is an old project and has not been updated since 2011. While it has some demo code and documentation, it is a bit raw. I guess it is not compatible with newer versions of Eclipse. Still - like Structural Search and Replace - it is very powerful. If you have a large code base and a repetitive change that can be expressed in terms of the AST, the high effort to get into the tool and create a script will pay off. (A useful plugin to help you working the AST is the AST View. The AST View is part of the JDT but not installed out of the box. It is visualising the AST of a Java file open in the editor.)

Jackpot (NetBeans)
Another, very interesting tool in this area is the Jackpot DSL for NetBeans used in the IDE actions Inspect and Transform aka Inspect and Refactor. Jackpot is a NetBeans IDE module for modelling, querying and transforming Java source files. In the Jackpot context, a program transformation is a script or Java class which queries sources in Java projects for patterns, changes them, and then writes these changes back to the original source files. [...] Jackpot transformations tend to be applied globally, such as at a project level or across several projects. Win! Unfortunately there is very little information about it. It took me hours just to find a little bit of information about it:

Dustin Marx describes how to create NetBeans 7.1 custom hints. He agrees that the most difficult aspect of using NetBeans's custom hints is finding documentation on how to use them. The best sources currently available appear to be the NetBeans 7.1 Release Notes, several Wielenga posts (Custom Declarative Hints in NetBeans IDE 7.1, Oh No Vector!, Oh No @Override!), and Jan Lahoda's jackpot30 Rules Language (covers the rules language syntax used by the custom inspections/hints). The Refactoring with Inspect and Transform in the NetBeans IDE Java Editor tutorial also includes a section on managing custom hints. All listed documentation is from 2011/2012. The most recent one I found is a short Jackpot demo from JavaOne Brazil 2016.

NetBeans hints are warnings that have a quick fix. For example the hint "Can Use Diamond" finds places where the diamond operator of Java 7 can be used instead of explicit type parameters. When the offered action is taken, the code is migrated. In the Inspect and Transform dialogue, the inspections can be managed. Custom hints are stored in .hint files. For example, Dustin Marx' hint to remove extraneous calls to System.currentTimeMillis() from new java.util.Date constructor is written as

<!description="Unnecessary use of System.currentTimeMillis on new Date">

new java.util.Date(java.lang.System.currentTimeMillis())
=> new java.util.Date()
;;

The Java Declarative Hints Format allows matching on variables, modifiers and statements. Fixes can be applied conditionally too.

I do not know if Jackpot hints can be applied across a whole code base at once. As they are inspections, I expect them to be displayed for the whole project - making them fast navigation markers at least. Anyway this is very exciting. It is so exciting that some people wanted to port it to Eclipse (but they never did).

Using Specific Refactoring Plugins
There are a few specific plugins that combine various Refactoring. Eclipse's Source Clean Up Action deserves a honorary mention: It fixes basic warnings on a whole code base, but is very limited. An interesting plugin for Eclipse is Autorefactor which aims to fix language/API usage to deliver smaller, more maintainable and more expressive code bases. Spartan Refactoring is another Eclipse plugin that performs automatic refactoring of Java source code, making it shorter, more idiomatic and more readable.

All these plugins change a predefined, limited set of code patterns, mainly focusing on technical debt. Maybe someone implemented a refactoring plugin for part of the widespread change you need to perform. Search the market places first.

Using Refactoring APIs of IDEs
A possible way is to write your own refactoring plugin. Maybe start with code of a refactoring plugin listed above. Both Eclipse and IntelliJ IDEA offer APIs to manipulating Java code, i.e. the JDT and PSI APIs. I have not done that because it seems too much effort for one time migrations and widespread changes. And reusing the available refactoring tools might raise some problems like waiting for user input which is problematic.

Using (APIs of) Refactoring Browsers
The Refactoring Browser was the first tool that automated Refactoring for Smalltalk. It set the standard for all modern Refactoring tools. Today we have Refactoring Browsers for many languages, e.g. CScout - the C Refactoring Browser, Ruby Refactoring Browser for Emacs or PHP Refactoring Browser which is controlled via the command-line and has plugins for Vim and Emacs. Stand-alone Refactoring tools should be easier to use and script than full blown IDEs, especially if they were designed for different plugins. The idea would be to create code which controls Refactoring Browsers to apply certain changes - again and again.

Python Refactoring Libraries
While searching for Refactoring Browsers I just learnt that there are at least three stand-alone Python Refactoring libraries: Rope comes with Vim and Emacs plugins and is also used in VS Code. The second is Bicycle Repair Man. Further there is Bowler which supposedly is better suited for use from the command-line and encourages scripting. All these libraries are rich in features and used by Vim users. (Yeah, Vim is still going strong.)

Rope (Python)
For example, Rope can be used as a library, which allows custom refactoring steps. Even more, it offers something similar to Jackpot's hints, Restructurings. For example, we split a method f(self, p1, p2) of a class mod.A into f1(self, p1) and f2(self, p2). The following Restructuring updates all call sites:

pattern: ${inst}.f(${p1}, ${p2})
goal:
 ${inst}.f1(${p1})
 ${inst}.f2(${p2})

args:
 inst: type=mod.A

The code to perform the Restructuring using Rope as a library is

from rope.base.project import Project
from rope.refactor import restructure

project = Project('.')

pattern = '${inst}.f(${p1}, ${p2})'
goal = '...'
args = '...'

restructuring = restructure.Restructure(project, pattern, goal, args)

project.do(restructuring.get_changes())

I never used that but it looks cool. The promise of scripted refactoring makes me excited. Another item to add to my to-research list. ;-) And there are more tools on my list, which will have to wait for part 3.

What is Ethical Coding and how to get involved

Three years ago I started thinking about work with meaning. Since then I had great discussions with peer Software Professionals about our social responsibility and ethics in software development. Today I am happy to present a guest post written by Kayleigh Alexandra, a content writer for Micro Startups, a site dedicated to spreading the word about startups. I like seeing a focus on ethical approaches in startups, i.e. Social entrepreneurship. Kayleigh Alexandra shares her ideas about ethical coding in this introductory article. For her latest startup news and inspiring stories, visit the Micro Startups blog or follow along on Twitter.

Technology provides amazing possibilities, but its power also makes it a threat. Almost everything we do relies on complex interactions of computer systems, with those systems running on code from countless distinct sources - you cannot trust one chunk of code to meet the same standard as another.

Unknown quality is not the only concern we can have about code, however. A function devised to achieve one thing can be extended and repurposed to do numerous other things - does not the developer bear some responsibility in how it is used? Those who share that sentiment argue for ethical coding, targeting a future of much broader social awareness. But what exactly does ethical coding involve, what are some relevant cases to consider, and how can you get involved in it? Let's take a look.

What does ethical coding involve?
Ethical coding involves acknowledging and acting upon the social responsibilities of a developer - factoring it alongside corporate responsibility - seeking to adhere to a set of meaningful values. It has becoming more prominent in recent years due to rising social awareness in general.

Because of the layer of abstraction between low-level source code and the consequences of the projects people work on, it is easy for everyone in a development company to absolve themselves of any culpability. Ethical coding requires the executive level to take ownership of the entire company output, committing to holding every last employee to a revised standard.

A developer with an ethical approach will think carefully about what her software will be used (or has been used) to achieve, and adjust course accordingly. They will also have strong thoughts about the general responsibilities of the software development industry, making them eager to advocate for higher standards and more transparency.

Since ethical stances vary wildly, though, there can be no precise definition of ethical coding - the best we can do is to use a broad ethical framework that loosely mirrors the societal standard for other fields such as medicine: minimising harm, being truthful, and making life better. See the Programming Ethical Guidelines by ACM and IEEE.

The rise of ethical software development
As noted, social awareness of the many issues that face the world continues to rise, driven by the 24/7 connectivity of the Internet and the move into adulthood of an ethically-conscious generation. For better or worse, social media users now act as moral guardians of sorts. Because technology in general plays a huge role in industry and the opaque nature of many coding projects is at odds with a desire for transparency, the tech world has attracted a lot of flak. This has led to various projects that have sought to demonstrate that technology can be a force for good, such as the following:

• The Hummingbird Clock. In 2016, as part of the Liverpool Biennial, this timepiece was installed as a commentary on government corruption and to serve as a tool for the public. Set up as three binoculars looking upon the Town Hall, it actually records the hum of the electrical grid and streams it online - since the government has long used that hum for surveillance, this was to make it available to anyone. As a result, anyone in legal difficulty can now use the hum to verify the time and date of a particular recording.
  
  
• The Empathy Deck. Social media can be brutal at times, deeply unpleasant and unempathetic, because being online and having some degree of anonymity drives people to embrace their darkest impulses. The Empathy Deck account was set up to respond to tweets with thought-provoking cards assembled from pieces of an artist's body of work, commenting on the perils of automation while providing some relevant musings and snippets of poetry to brighten someone's day.
  
  
• Cody Rocky. To many, the world of technology still feels quite dry and alienating, which can limit the types of people who choose to pursue it as a career. Cody Rocky is a programmable robot designed to appeal to children, lending a degree of accessibility to the field and making learning both productive and entertaining. It also crosses slightly into the nascent field of digital companionship.

In light of some high-profile fiascos making people much more concerned about the dangers of technology (such as the Cambridge Analytica scandal, or the death caused by a self-driven car), the pressure on developers to implement ethics policies has increased significantly. There is also the thorny issue of so-called hacktivism - is hacking done for a good cause any less objectionable? Who gets to decide where to draw the line?

Though industry-wide ethical frameworks have been suggested before, there is so much more impetus on companies to be proactive now. In the coming years, I think it is fairly likely that it will become standard for software developers - well, any companies in the digital industry - to lay out and commit to their ethical codes.

How to get involved in ethical coding
If you are a developer and you want to get involved in ethical coding, then you can. It is actually fairly simple, with the following two major steps:

• Code ethically. This is straightforward enough. Whenever you create applications, be keenly aware of any and all ethical issues that may arise as a result. For example credit everyone whose code you adapt, be efficient to avoid needlessly taxing a system, maintain professional integrity, and refrain from working on any project that you expect to be used irresponsibly or immorally. You cannot know every potential issue, but you can do your best.
  
  
• Contribute to ethical projects. These can be your own, or existing projects that need help. This is not about your code, but about the goals of the project: for instance, you could volunteer some time for a healthcare-related project, or help out a charity with its operations. It does not cost much to start your own business these days, so it is easy for someone with good intentions to launch a startup, but it is much harder to make it a contender. Your support could be the key to a business getting big enough to do some real good.

Beyond that, the specifics are entirely up to you. It is really a matter of getting involved with online communities, finding people who share your values, and slowly discovering how you can best use your time to engage in ethical coding.

Ethical coding is only going to grow as a concern while massively-influential fields such AI or the IoT get larger. If you develop software and are strongly ethical, give a lot of thought to how you can better reconcile your beliefs with your work. There is no reason why you cannot have professional and personal contentment at the same time.