13 July 2018

Categories of Architectural Refactoring

At the end of one of my refactoring workshops, we started discussing how to refactor
larger structures, sometimes even changing the architecture of a system. During that discussion we came up with four different categories of larger refactoring which use different techniques and serve different goals:
  • Replace whole parts
  • Change the structure of components
  • Similar change all over the place
  • Other changes
Seoul SkyscrapersWe struggled to find good names, so let me explain the different things a bit. When I talk about architecture, I mean Software Architecture of a single application, i.e. 1. the different software components and 2. their (inter-) dependencies making up the application. Components are sub-systems and usually contain one or more namespaces or packages. While we can discuss forever what exactly software architecture is and is not, I like the definition of IEEE which in addition to elements (1.) and relationships (2.) adds principles of its design and evolution. I describe these principles as 3. design guidelines which include smaller recommended usage patterns and coding conventions.

Martin Fowler and Neal Ford say that architecture is the stuff that's hard to change later and replacing parts of the architecture involves a lot of work: Imagine (the highly theoretical example of) changing the programming language a system is written in or (a more common example of) moving from a monolith to a Microservice based architecture. These two examples already fall into different categories.

Replacing whole parts of the architecture
Massive changes like changing the UI technology or targeting another platform are possible but a lot of code has to be rewritten. Substitute Algorithm is a classic Fowler refactoring, and so is Substitute Architectural Decision. (This is my third try to name these category of changes.) Architectural Decisions are made to support non-functional requirements like performance, security, connectivity etc and when revisiting them later - with more domain knowledge - better alternatives might come up. Real examples of changing Architectural Decisions include:
  • Replacing one relational database system with another
  • Replacing your relational database with a NoSQL one
  • Replacing UI technologies, e.g. moving from JSF to Vaadin
  • Changing API technologies, e.g. moving from SOAP to REST
  • Switching from a fat client to a client-server architecture
Depending on used frameworks and abstractions, some of these changes will be easy, others will not. Even when the necessary change is isolated, it likely touches more than one architectural component. For example moving to another data store might need changing all data access objects in all components which access the data store. The traditional approach is to change everything at once ("Big Leap") but it is possible to refactor the architecture in a safe way, using small steps. In Responsive Design (QCon 2009) Kent Beck describes several strategies to perform such changes:
  • Leap: Make the new design and use it immediately. (Only if the change is not too big.)
  • Parallel Change: Make the new design and run both new and old simultaneously.
  • Stepping Stone: Create a platform to bring the desired new design within reach.
  • Simplification: Only implement part of the design now and add complexity bit-by-bit later.
  • Hack it: Just do it and pay price later. (This is maybe not a real strategy ;-)
Changing the structure of architectural components
The next category is similar to Substitute Architectural Decision but is focused on the architectural building blocks - the components. Some examples are:
  • Introduce or extract new component, module or package
  • Remove or invert dependencies between components, e.g. remove cyclic dependencies
  • Introduce layering, add a new layer, clean up layering violations
  • Weaken potential trouble spots in the architecture, e.g. Butterfly, Breakable, Hub and Tangle
foundationThis category of changes is the "classic" Architectural Refactoring because that is how people call it. For example - in accordance to Martin Fowler's definition - Dave Adsit defines Architectural Refactoring as intentionally changing the structure of a system without altering its features. Similarly Sean Barow explains Architectural Refactoring as the deliberate process to remove architectural smells while not changing the scope or functionality of the code. So it is always structural and deals with components (1.), their relationships (2.) and their inner structure. Let's call it Refactor Architectural Structure to distinguish it from other architectural refactoring.

Sean Barow uses the term Code Refactoring for the refactoring defined by Fowler. Code refactoring focuses on software entities like classes, methods, fields and variables. Architectural refactoring involves code refactoring, which leads to confusion between the two. Working with dependencies needs extracting or moving classes, methods or fields. We can work each class or dependency at once, which supports an incremental approach.

Performing similar changes throughout the whole project
This is another category and it feels like Shotgun Surgery: A usually small change has to be repeated all over the place. (While Shotgun Surgery is a code smell for feature logic, it is unavoidable if not required for consistent structure and usage defined by the architecture.) For example let's assume we want to replace EasyMock with Mockito. Both libraries serve the same purpose and are used in a similar way, but their APIs are different. (Facing such a problem indicates that we are coupled to much to a used library - maybe we should have wrapped it, maybe not. But that discussion is outside the scope of this article.) Converting a single usage of EasyMock to Mockito is straight forward: We change the method invocations and execute the calls earlier or later in the test. The only problem is that we have hundreds of such cases. More examples of similar "Shotgun" changes are:
  • Upgrading a major version of a library with breaking API changes
  • Migrating between similar frameworks or libraries
  • Changing or unifying coding conventions and fixing violations
  • Consistently applying or changing aspects like logging or security
  • Applying internationalization
These changes do not change the overall structure, they just work on the implementation. They are similar to Refactor Architectural Structure, just deal with design guidelines, coding conventions and smaller usage patterns - item 3 from the definition of Software Architecture in the beginning. They are code refactoring applied in many places consistently across a whole code base.

These category is important, because some steps of both Substitute Decision and Refactor Structure need similar changes. The main challenge of these changes is the high number of occurrences. Changing conventions and aspects is a piece by piece work and can be done manually. Nevertheless some automation tool or script would help a lot. Even converting only basic scenarios, which often cover up to 80%, is a big win. Unfortunately some changes, e.g. migrating libraries, need to be done all at once. In such cases a Stepping Stone approach has worked well for me in the past: First wrapping the library piece by piece - incrementally - and then changing the wrapper and all invocations at once. Creating the wrapper is a lot of work. Again, automated migration removes the need for a wrapper. There are some options available, which I will cover in a future article.

I have not found a good name for this category. I used to call it Large Scale Refactoring, but I am not happy with that name. It is a similar or identical code refactoring applied to many places widespread throughout the code. It is a Widespread Architectural Change.

Other Architectural Changes
During our initial discussion, we also considered other refactoring. To be on the safe side, always allow a category of unknown things ;-). At the moment I have no real examples for them. These are smaller changes neither related to architectural decisions or structure nor are they widespread. I believe we find no examples because these are plain code changes and we do not consider them architectural even if they are related to the architecture of the applications in a way.

FavelaSummary
Next to the code we also can and should refactor the architecture of our applications. Architectural changes are expensive, still sooner or later things need to change if the application is in service just long enough. There are at least four categories of architectural refactoring:
  • Substitute Architectural Decision
  • Refactor Architectural Structure
  • Widespread Architectural Change
  • Other Architectural Changes
Next time I will dive into tools to automate Widespread Architectural Changes.

3 July 2018

Using NATstyle from the commandline

Last year a new client contracted me to help them make their development process, their development environment and tooling "state of the art" (aka up do date). From the coding perspective that included - among other things - using a decent IDE, writing unit tests, having some Continuous Delivery pipeline in place and using static code analysis. I thought that it should not be too difficult, right? It was, but that is not the goal of this article. Today I want to describe the static analysis features and how to use it.

Earth Sapphire Progress Picture 30 july-2011NATURAL
The code base was NATURAL. NATURAL is an application development and deployment environment using a proprietary language maintained by Software AG. It was created in 1979. André describes it as Cobol's ugly, brain-damaged, BABBLING IN ALL-CAPS - but regrettably still healthy and strong - cousin.. ;-) Well, it is a procedural language structured using modules, subroutines and functions. It feels as old as it is. For example, module names are limited to eight (8!) characters and there is no way to structure the modules further. An enterprise application might contain 5.000 to 10.000 modules. To me NATURAL feels much like early Pascal: units (modules), subroutines and functions together with the eight characters DOS name limit. I really liked Pascal back then and I do not feel that bad about NATURAL. Probably also because I do not have to use it on a daily basis.

NaturalONE and NATstyle
NaturalONE is the Eclipse-based tool for NATURAL by Software AG. It looks pretty good, similar to what you would expect from Eclipse. It contains the usual features like Outline, Search, Debug and something called NATstyle. Now NATstyle - which probably on purpose sounds similar to the well known Checkstyle - is an utility to define and check the coding standard in your programs. It is used inside NaturalONE, see a NATstyle Demo by Software AG. (For more information refer to the Eclipse/NaturalONE help topic Checking Natural Code with NATstyle.)

Invoking NATstyle from outside NaturalONE
I like static code analysis and consider it a mandatory component of every delivery pipeline. As I said above, I wanted to have some static code analysis and was wondering if I would be able to run NATstyle in the pipeline? I did not find any information on the topic and was forced to experiment. I used NaturalONE 8.3.5.0.242 CE (November 2016) and figured out several ways, which were hardly documented, and might be subject to change. Likely there are other options available in newer versions of NaturalONE and NATstyle.

Invoking NATstyle from command line
NATstyle is just an Eclipse plugin inside NaturalONE. In my edition it is the folder C:\Natural-CE\ Designer\ eclipse\ plugins\ com.softwareag.naturalone.natural.natstyle_8.3.5.0000-0242. With the proper class path it is possible to invoke NATstyle's main method, which prints help on its usage:
Usage: com.softwareag.naturalone.natural.natstyle.NATstyle [-options]

where options include:
  -projectpath <directory> Specify where to find the Natural project.
  -rootfolder              Library root folder support enabled.
  -c <file>                Specify the configuration file.
  -o <file>                Specify the output file.
  -sourcefiles <srcList>   Specify list of source files to be loaded.
                           separated with ;
  -libraries <libList>     Specify list of libraries to be loaded.
                           separated with ;
  -exclude <libList>       Specify a list of libraries to exclude.
                           separated with ;
  -p <directory>           Specify where to find additional packages.
  -help                    Display command line options and exit.

Make sure the following classes can be found in your class path:

com.softwareag.naturalone.natural.auxiliary
com.softwareag.naturalone.natural.common
com.softwareag.naturalone.natural.parser
Nice, that looks like the developers from Software AG prepared NATstyle to be called stand-alone in my pipeline. +1. All the necessary classes can be found in the following plugins of Eclipse/NaturalONE:
  • Folder com.softwareag.naturalone.natural.natstyle_8.3.5.0000-0242
  • Jar com.softwareag.naturalone.natural.auxiliary_*.jar
  • Folder com.softwareag.naturalone.natural.common_8.3.5.0000-0242
  • Jar com.softwareag.naturalone.natural.parser_*.jar
  • Jar org.eclipse.equinox.common_*.jar
These are all bundles defined by Eclipse/OSGi. The list of dependencies of these bundles also contains the following Jars
  • org.eclipse.swt.win32.*.jar
  • org.eclipse.ui.console_*.jar
which (in my experience) are not needed to use NATstyle. NATstyle does not use any other OSGi features and we can set the needed class path by hand. Here is an example how to do that in the Windows shell when NaturalONE is installed:
set N1=C:\Natural-CE\Designer\eclipse\plugins
set CLASSPATH=^
%N1%\com.softwareag.naturalone.natural.natstyle_8.3.5.0000-0242;^
%N1%\com.softwareag.naturalone.natural.auxiliary_8.3.5.0000-0242.jar;^
%N1%\com.softwareag.naturalone.natural.common_8.3.5.0000-0242;^
%N1%\com.softwareag.naturalone.natural.parser_8.3.5.0000-0242.jar;^
%N1%\org.eclipse.equinox.common_3.6.200.v20130402-1505.jar

java com.softwareag.naturalone.natural.natstyle.NATstyle %*
See run_natstyle for the script I used to run NATstyle directly. Copying the needed folders and Jars to another machine works well. I recommend to create a Jar from the folders before copying them around. create_jar how to do that. Make sure not to include META-INF\*.RSA or META-INF\*.SF into the new Jars because the checksums do not match. copy_jars does the actual copying into the lib folder.

Invoking NATstyle from Ant
NaturalONE supports Ant for some automation tasks. There is no specific NATstyle Ant task, but calling it from Ant is straight forward. If the path <path id="natstyle.classpath"> is set up to contain all the Folders and Jars, NATstyle is executed via Ant's java task:
<java taskname="natstyle"
      classname="com.softwareag.naturalone.natural.natstyle.NATstyle"
      dir="${basedir}" fork="true" maxmemory="128m"
      failonerror="true"
      classpathref="natstyle.classpath">
  <arg value="-projectpath" />
  <arg value="../NatUnit/NatUnit_L4N" />
  ...
</java>
A new JVM needs to be forked because NATstyle calls System.exit() at the end. All parameters are the same as for the command-line invocation and are passed in via the <arg /> element. See ant_NatStyle.xml for the complete Ant script.

Reporting
NATstyle writes an XML report of all files if worked and found rule violations. The structure is
<NATstyle>
  <file type='...' location='...'>
    <error severity='...' message='...' />
    ...
  </file>
  ...
</NATstyle>
Inside the folder of NATstyle (i.e. com.softwareag.naturalone.natural.natstyle_8.3.5.0000-0242) there is a subfolder NATstyle. It contains sample files. Using the provided NATstyleSimple.xsl the report XML (e.g. NATstyleResult.xml) can be transformed into a human-readable format (e.g. NATstyleSimple.html). When using NATstyle stand-alone, you might want to copy this folder as well. XSLT transformation task is part of Ant and converts the XML to HTML.
<xslt basedir="${basedir}"
      destdir="${basedir}/report"
      style="${basedir}/NATstyle/NATstyleSimple.xsl">
  <include name="NATstyleResult_*.xml" />
</xslt>
See ant_NatStyleResultToHtml.xml for the complete Ant build script to convert the NATstyle result XML into a readable HTML report. Using your own stylesheet (.xsl) allows you to customize the reports.