31 October 2018

More on Architectural Refactoring

Last month I showed my categorisation of Architectural Refactoring to some friends in the Software Crafting community. Explaining my ideas and getting feedback helped me to sharpen the definitions further. In my original article I defined four groups of Architectural Refactoring. Here is what we discussed about each one of them:

1. Substitute Architectural Decision
In this category I collected Refactoring techniques which replace whole parts of an architecture, e.g.
  • Replacing one relational database system with another
  • Replacing the relational database with a NoSQL one
  • Replacing UI technologies, e.g. moving from JSF to Vaadin
Adrian Bolboaca further split these categories into groups and used the following three names:

Kinds of Architectural Refactoring (C) SoftDevGang 20181.1. Tool Change
A Tool Change affects the implementation of a "tool", e.g. the database system. The first change in the previous list - replacing one relational database system with another - is a Tool Change. Upgrading libraries or migrating between similar frameworks is a Tool Change too. Initially I had listed them as Widespread Architectural Change. While upgrading or migrating libraries might require similar changes throughout the whole project, they are motivated by the Architectural Decision to use a certain version of a library.

1.2. Technology Change
A Technology Change modifies a used technology. Items two and three of the previous list are Technology Changes. In my experience, most Architectural Refactoring is of this type.

1.3. (Other) NFR Caused Change
Changing a tool or technology is never done for its own sake. We might upgrade a framework for the improved performance of its new version, or switch to another data store because it is more reliable or its license is more suited for the business. NFRs (Non Functional Requirements) are requirements that specify how a system is supposed to be in contrast to functional requirements which define what a system is supposed to do. NFRs usually contain long lists of attributes describing a system like performance, reliability and also security, traceability and globalisation. Examples of such changes are changing aspects like logging or security or applying internationalization. Again I had initially listed them as Widespread Architectural Change.

2. Refactor Architectural Structure
The second category deals with architectural building blocks like components, layers or packages. There are two causes for changing these:

RSAR Package Tangle found at an unnamed client2.1. Cleaning up Legacy Code
When we start a new project, its architecture is clear. When the system becomes more complex, effects known as Software Architecture Erosion or Architectural Drifts happen. These effects can lead to systems where the realisation of the system diverges from the intended architecture with resulting negative impacts on quality attributes associated with the intended architecture. Uncle Bob uses other words: The architecture starts to rot like a piece of bad meat.

For example look at the package tangle on the right: Several teams were working on a shared code base under an aggressive schedule and produced an architecture where each package was depending on all others, resulting in the legendary Big Ball of Mud architecture. To clean up this mess components with clear interfaces had to be introduced.

2.2. Evolving an Architecture
Today we rather embrace evolving requirements than stick to a big up-front plan. Techniques like Evolutionary Design accommodate the need of changing the design while systems are growing. And also their architectures need to evolve, e.g. we need to extract a new component or aspect which became duplicated during addition of structure to support new features. With the rising popularity of Microservices and their focus on incremental change in the architecture, Evolutionary Architecture has become a reality.

3. Widespread Architectural Change
I am mostly interested in these changes because I want to automate them. I described some basic tools and some more powerful ones. Most examples of changes I used were motivated in the change of Architectural Decisions. Even coding conventions can be seen as part of the NFR Maintainability. There is definitely an overlap, as some architectural Refactoring and also a few structural ones, will be widespread. Adi drew me a Venn diagram to explain that overlap:

Overlap of Categories of Architectural Refactoring, by Adrian Bolboaca
4. Other Architectural Changes
I am still uncertain if there are other kinds of architectural changes. I miss examples. If you have some, please come forward!

19 October 2018

NATstyle Custom Reports

This summer I wrote about NATstyle, a utility to define and check the coding standard in your NATURAL program. Now NATURAL is not a language you will encounter frequently (hopefully not ;-) but it is an excellent show case for neither being intimidated by strange environments nor accepting idiosyncratic vendor views. First I showed how NATstyle can be used outside of the NATURAL IDE - a prerequisite for setting up Continuous Integration, then I experimented with ways to define my own custom analysis rules. Now is the time to talk about reporting.

Custom Stylesheets
NATstyle creates an XML report. Its structure is
  <file type='...' location='...'>
    <error severity='...' message='...' />
NATstyle comes with a stylesheet NATstyleSimple.xsl. The XSLT transformation is not part of the NATstyle execution. Using custom stylesheets allows you to convert the XML into arbitrary reports.

Suitable Target Structure
When working on weird environments, I try to use standard tools as much as possible. Emulating common output formats is the way to go. For example in the case of test results, the JUnit XML format is used in similar scenarios, e.g. Erlang, Cobol and others.

What would be a good structure for reporting static code analysis findings? PMD is a common tool in the Java space and it creates XML reports which are understood by tools like Jenkins, making integration much easier. In the report, violations are grouped by files:
<pmd version="..." timestamp="...">
  <file name="SampleClass.java">
    <violation beginline="2" endline="16"
               begincolumn="8" endcolumn="1"
               rule="TooManyFields" ruleset="Code Size"
               externalInfoUrl="..." priority="3">
      Too many fields
XML Parser
Changing the NATstyle result to PMD's format is possible using XSLT alone, but I favour XML parsers for more complex transformations. I established the following process to convert the results: As I used Ant to execute NATstyle on Jenkins, I created an Ant task NatStyleResultToPmdTask.java. (Ant declaration for that task is in ant/ant_NatStyleResultToPmd.xml.) The Ant task sets the input and output XML file names and invokes the NatStyleToPmd.java conversion which uses a SAX parser to parse the NATstyle result and feeds the violations into net.​sourceforge.​pmd.​Report#​addRuleViolation. Then core PMD classes create the final XML report. (The project containing the Ant task and complete, working setup is here.)

In legacy environments it is often possible to "bridge" to standard tools and make use of the rich tooling we have. We can apply modern practises like automated code reviews or Continuous Integration. Another example is bringing TDD and automated testing to the mainframe platform. It is 2018, we have all these great tools, let's make use of them!

8 October 2018

Interview Johannes Link

I am happy to announce my next guest in the series of interviews on ethics in Software Development and meaningful work: Johannes Link is a well known XP practitioner, trainer and frequent speaker from Germany.

Johannes, please introduce yourself.
I have been developing software in a professional context for more than 20 years and that is still one of my passions. My understanding of how to do programming was largely shaped by Extreme Programming. Especially its focus on quality and close collaboration are an intrinsic part of how I look on everything software-related.

ethicsI know that you are concerned with ethics in Software Development. Why do you care?
I think that everyone's personal views on ethics, morale, purpose of life, "good and bad" should also be reflected in their professional activities. Thus it is obvious to me that I cannot support activities in my job that I would reject when discussing politics with my friends. It came rather as a surprise to me when I learned that other people intentionally differentiate between their personal opinions and their professional activities.

What other topics are you concerned about?
Besides my intention to not harm society I would love to do something meaningful or even "to give back" to the world. I guess it came with age and with having children: Just implementing the next e-commerce shop or the next workflow-system for travel expenses won't excite me any more - regardless of the technology in use. That is why I am always looking for "purpose" in the projects I work on; it might be a personal purpose - e.g. building a new tool to make my life as a developer easier - or a more general purpose like teaching kids the fundamentals of programming.

There is also a "meta"-topic: Raising the awareness in our industry that what we do and how we do it shapes the future life of all. That is why we have to start thinking about responsibility in all our projects, and that contains aspects like security, usability, accessibility and risks.

Outside the topics discussed so far, what do you consider the biggest challenges (for humanity) of our times?
Preserving a world that is worth living in. It is not only the climate that is at stake but also democracy and other traits of humaneness that came with the Age of Enlightenment.

When I talk to people, many express concern about meat mass production or pollution, but almost nobody really acts on it. What could we do to engage in these topics?
I am probably as lazy as most when it comes to getting into action. I donate money and I sign petitions, but does that really help? Maybe it is even corrupting the purpose because it makes us feel better and takes some of the uneasiness away that could otherwise push us to real action.

That said, I do not think we can (and should) all fight the same battles. Choose one or a few topics in which your interest or your anger is the biggest and tackle those. As for me the rise of nationalism and xenophobia inside many countries that seemed to be very stable democracies until recently is finally driving me to speak up. Just watching is no longer enough.

Whistle-BlowerAs a software developer, what options do we have to act and do "the right things"?
You always have the possibility to question certain aspects of your work. For example, when you are supposed to collect vast amounts of data that could potentially be used to harm people, you can raise the related ethical and legal issues with your employer. In the end you must be willing to suffer the consequences and not all of us have the luxury (the savings) to do that.

What looks like technical decisions to us might have a real consequence for others: Choose a certain browser as target platform and you exclude some users from using the product. Switch from on-premise deployment to a cloud solution and you make some operation staff redundant. That is why I think that we cannot hide behind the "It is just technology" argument.

Whatever we do we should always ask ourselves:
  • Is there a conceivable way that the system to which I contribute will intentionally or unintentionally harm others? If so, how can it be changed to (at least) reduce the probability?
  • Am I competent enough to take the technical decisions that I take? This is especially relevant when it comes to security because even as senior developer it is hard to keep up-to-date in that field.
  • Is the quality we build high enough to not put the system's purpose at risk? This decision is so context-specific that we often get it wrong when switching domains, e.g. from web store to medical device.
I run into quality conflicts with my customers from time to time. Sometimes, as a last resort, I might refuse to go with a Product Owner's prioritisation because I consider fixing a certain technical debt or design flaw crucial for either the project's sanity or the team's capability to change the system in the future. Saying that I will not continue with the current prioritization can get me the accusation of blackmailing whereas I consider it to be an additional piece of information for the business side. And yes, in those cases I will stand with my word if we won't be able to find a common resolution; so far that happened only once.

How do you think about selecting industry, customer and project based on your values?
Well, we have to earn our daily pasta. When arguing these topics with others, I had to concede that the decision if a certain industry, company or project is in alignment with my ethical views is sometimes very tricky or even undecidable. It can be difficult to see if my contract is contributing to a good or a bad purpose. Sometimes it is obvious. In those cases you have the choice to quit a contract and to tell others why you did it.

There are a few industries that are no-go areas for me. In most other cases it is the concrete project that makes me decide. It is easier for me as an independent developer than it is for an employee, though.

Since the amount of information you get in advance of joining a project is somewhat limited I tend to find out more details through my network or go with my gut feeling. It has not happened yet that I wanted to leave a project for ethical reasons afterwards.

Boeing B-52 dropping bombsLet's be more specific: Would you work for an animal factory? Would you work for a company producing equipment for an animal factory?
Animal Factory: No. Company producing equipment for an animal factory: I might not even know. If I knew, it would probably depend on the type of equipment since a lot of equipment is very unspecific, e.g. an electric engine can be used for all kinds of things.

Do you have problems with any industries?
There is currently only one area that I would reject on principle: Military and weapons. I consider intelligence organisations to be part of military so add that to the list. And there are a bunch of companies whose offers I would reject without thinking twice.

I struggle with the distinction of "working for" and "buying from". Buying also means supporting, but I am certainly less consequential there.

Did you ever reject a project, that would bring you money based on your values?
Yes. I was recently contacted by an agent offering me a very interesting position both location-wise and considering the technologies in use. When I was told the name of the customer however, I had to reject, since the company's business model is known for harming people and mostly serving the rich. When I told the agent that I won't take the position "out of principal considerations" she was astonished and I explained my motives. She seemed to be really surprised that anyone would do that.

On the other hand, what would be industries, customers and projects that you would love to work on?
The medical field has a lot of potential to improve the lives of many. But the current equilibrium of forces usually takes care that the established industry players win and the patients lose. So that is tricky and I have been working in medicine-related projects a few times. I would love to do paid work for NGOs that work along my personal political views. I have not seen a single project in that area, though.

Thank you Johannes for sharing your views on this important topic.

2 October 2018

Developer Melange Episode 3

Yesterday the third episode of the Developer Melange podcast was published. Developer Melange is a monthly podcast which brings you regular discussions about software engineering topics. All of them, in one way or another, related to building great software products. It is recorded by my friends and fellow software crafters Paul Rohorzka, David Leitner and Christian Haas and they release around an hour of high quality discussion every month.

Hello MicI am supporting them since the very beginning. In fact I was playing with the idea of recording a monthly podcast already for some time. After my summer project I finally found the time to visit them during recording: We discussed the essence of Behaviour Driven Development, starting with Dan North's introductory article from 2006. It seems that we could not agree what the essence or core of BDD might be. In the second part we tried to answer the question if it is better to be a specialist or generalist in the field of software delivery. Of course the time was too short for such deep topics. ;-)

Get it here
Listen to this episode here. All previous (and future) episodes can be found on the Developer Melange home page and Developer Melange SoundCloud page. If you need an RSS feed for an old-fashioned podcast app, use Get RSS feeds from SoundCloud URLs, which is this RSS.

The team and I are curious what you think about Developer Melange. You can reach us on Twitter or leave comments on SoundCloud. Tell us what you think! Any suggestions, e.g. topics you would like to hear about, and also criticism are highly welcomed. Stay tuned!