3 December 2017

PMD Check and Report in same build

lane one, lane twoI am working together with senior developer and (coding) architect Elisabeth Bl├╝melhuber to set up a full featured continuous delivery process for the team. The team's projects use Java and are built with Maven.

Using PMD for Static Code Analysis
After using Jenkins for some time to run the tests, package and deploy the products, it was time to make it even more useful: Add static code analysis. As a first step Elisabeth added a PMD report of a small set of important rules to the Maven parent of all projects. PMD creates a pmd.xml in the target folder which is picked up by Jenkins' PMD Plugin. Jenkins displays the found violations and tracks changes over time, showing a basic trend graph. (While SonarQube would be more powerful, we decided to stay with Jenkins because the team was already "listening" to it.)

Breaking the Build on Critical Violations
I like breaking the build on critical violations to ensure the developers' attention. It is vital, though, to achieve the acceptance of the team members when changing their development process. We thus started with a custom, minimal set of rules (in src/config/pmd_mandatory.xml) that would break the build. The smaller the initial rule set is the better. In the beginning of adding static code analysis to the build process, it is not about the code but getting the team aboard - we can always add more rules later. The first rule set might even contain a single rule, e.g. EmptyCatchBlock. Empty Catch blocks are a well known problem when analysing defects and usually developers agree with the severity of having them in the code and accept breaking the build for that. On the other hand, breaking the build on minor or formatting issues is not recommended in the beginning.

Here is the snippet of our pom.xml that breaks the build:
<build>
  ...
  <plugins>
    <plugin>
      <groupId>org.apache.maven.plugins</groupId>
      <artifactId>maven-pmd-plugin</artifactId>
      <configuration>
        <failOnViolation>true</failOnViolation>
        <printFailingErrors>true</printFailingErrors>
        <rulesets>
          <ruleset>.../pmd_mandatory.xml</ruleset>
        </rulesets>
        ... other settings
      </configuration>
      <executions>
        <execution>
          <id>pmd-break</id>
          <phase>prepare-package</phase>
          <goals>
            <goal>check</goal>
          </goals>
        </execution>
      </executions>
    </plugin>
  </plugins>
</build>
This is more or less taken directly from the PMD Plugin documentation. After running the tests, PMD checks the code.

Keeping a Report of Major Violations
We wanted to keep the report Elisabeth had established previously. We tried to add another <execution> element for that. As executions can have their own <configuration> we thought that this would work, but it did not. PMD just ignored the second configuration. (Maybe this is a general Maven issue. For example the Maven Failsafe Plugin is a copy of the Surefire plugin to allow both plugins to have different configurations.)

The PMD plugin offers a report for the Maven site which is configured independently. As a workaround for the above problem, we used the site report to check the rules listed in src/config/pmd_report.xml. The PMD report invocation created the needed target/pmd.xml as well as a readable target/site/pmd.html.
<reporting>
  <plugins>
    ... other plugins
    <plugin>
      <groupId>org.apache.maven.plugins</groupId>
      <artifactId>maven-pmd-plugin</artifactId>
      <configuration>
        <rulesets>
          <ruleset>.../pmd_report.xml</ruleset>
        </rulesets>
        ... other settings
      </configuration>
    </plugin>
  </plugins>
</reporting>
Skipping Maven Standard Reports
Unfortunately mvn site also created other reports which we did not need and which slowed down the build. Maven standard reports can be selected using the Maven Project Info Reports Plugin. It is possible to set its <reportSet> empty, not creating any reports:
<reporting>
  <plugins>
    ... other plugins
    <plugin>
      <groupId>org.apache.maven.plugins</groupId>
      <artifactId>maven-project-info-reports-plugin</artifactId>
      <version>2.9</version>
      <reportSets>
        <reportSet>
          <reports>
            <!-- empty - no reports -->
          </reports>
        </reportSet>
      </reportSets>
    </plugin>
  </plugins>
</reporting>
Now it did not create the standard reports. It only generated target/site/project-reports.html with a link to the pmd.html and no other HTML reports. Win.

Skipping CPD Report
By default, the PMD plugin invokes PMD and CPD. CPD is checking for duplicate code - and is very useful - but we did not want to use it right now. As I said before, we wanted to start small. All plugins have goals which are explained in the documentation. Obviously the Maven report invokes PMD plugin's goals pmd:pmd and pmd:cpd. How do we tell a report which goals to invoke? That was the hardest problem to solve because we could not find any documentation on that. It turned out that each reporting plugin can be configured with <reportSets> similar to the Maven Project Info Reports Plugin:
<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-pmd-plugin</artifactId>
  <configuration>
    ... same as above
  </configuration>
  <reportSets>
    <reportSet>
      <reports>
        <report>pmd</report>
      </reports>
    </reportSet>
  </reportSets>
</plugin>
Putting Everything Together
We execute the build with
mvn clean verify site
If there is a violation of the mandatory rules, the build breaks and Maven stops. Otherwise site generates the PMD report. If there are no violations at all, Maven does not create a pmd.html. There is always a pmd.xml, so Jenkins is always happy.

(The complete project, ready to clone, is here.)

29 November 2017

Interview Rea

Palestinians Collect Belongings from Gaza RuinsThe last Coderetreat was one of the rare Coderetreats that I was participating instead of organizing. I used the opportunity to pair with strangers and Rea was one of my partners. We came to talk about the sponsor of the Coderetreat and if she would work for a company offering products in the defence sector. So I asked her to answer my usual questions. These were her answers:

Rea, why did you choose to become a software developer?
I consider myself a beginner in the field of software development. I chose to switch to software development because in this world where computers play an increasingly important role I don't want to be a bystander, but be able to help shape it. I believe that in the near future software development should be as basic a skill as writing and arithmetic.

You said that you would not work for a company in the defence industry. Why is that so?
Call it defence industry or call it arms industry - its goal is to build something that harms or kills other people. Calling it defence industry rather than attack industry is just a euphemism. In the end it doesn't matter who started it. No defence industry is meant do be just passive-defensive. It always includes active attack as well. I don't want to have a part in this. I don't see why some humans should be worth living, while others can be harmed, maimed, crippled or killed at the whim of somebody in an office somewhere far from the actual action. Even thinking that there is a state funded - and therefore tax payer funded - industry that has the goal to kill makes me sick.

What other topics are you concerned about and what do you do about them?
I am concerned about many topics: Microbial resistance, climate change, loss of privacy, growing fear that manifests itself in hatred against others. I am not a politician and don't want to be one. What I do is to think hard about what I, as a single human being, can do, and then take little steps, one at a time. Even if they are not the most comfortable steps. Doing something, as little as it may be, is more valuable than to despair and do nothing at all. "Many a mickle makes a muckle" - I am convinced that each and every little step counts.

To be more specific, I do my little share against climate change, try to reduce waste and emissions, and have a small footprint. Although I have given up being a strict vegetarian (which I was for about 20 years), I eat meat less than once a month. I don't own a car and try to avoid using one as often as possible. I don't think it is necessary to travel extensively, although I really would love to see the world. I just think the harm it does is worse than the benefits I get. I buy locally and seasonally whenever possible. For me, this has precedence over organic food.

I have recently read that the volume of insect biomass has declined by almost 80% in the last 40 years in areas where it was measured, and other areas are likely to have seen a similar decline. This frightens me. Especially as the reasons can only be speculated about, and there is nothing I can do.

I speak up against racism, homophobia, sexism, etc. when I encounter it. I embrace diversity and try to raise awareness in others. I try not to judge what I don't know. Often, its not a matter of 'good' versus 'bad', but just of 'different'.

As for loss of privacy, again, I try to raise awareness. When people tell me they don't mind being under surveillance, I tell them this: You have nothing to hide when you go to the bathroom. But I bet you would rather not have me watch you there. I don't own a smart phone and still resist the urge to buy one. I try to reduce being tracked while surfing the Internet. I don't want a smart TV or any other smart device in my apartment. I still have an email address with Google, this is something I want to change soon. I do quite well without most of social media. And I wonder again and again how much longer I can keep up this retro lifestyle.

Methicillin-Resistant Staphylococcus aureus BacteriaWhat do you consider the biggest challenges for humanity at the moment?
I think bacteria resistance to antibiotics is one of the biggest dangers we are facing. This, combined with today's mobility, is all you need for a ghastly Hollywood movie scenario. I hope reality won't be as bad.

What could we do to engage in topics like meat mass production or pollution? For example, did you take part in public protests, donate money to NGOs or sign petitions?
As to what we can do - see my answer above. Signing petitions with Avaaz is something I did for a while. I realised that doing this makes me lazy in other ways. Signing gives me this feeling of accomplishment, but there is more I can do, in my own vicinity. The best way to fight meat mass production is not to consume this meat. For some people this is a bigger departion from their habits than for others, granted. But if you skip meat once a week, this is better than nothing. Or skip every other day. Or have meat once a week only. Just reduce your consumption. I think the difference in production cost between vegetables and meat should be reflected in the price. Not only the monetary price, but also the price as concerns the ecological footprint.

I have a split relationship with NGOs. I used to support Greenpeace, for example, but don't do that any more. One of the reasons is that I strongly disagree with their campaign against GMOs. Let's not get too deep into this here. But I do support NGOs as such, I think it is important that there are organisations that are independent of governments and other big influencers.

Do you think it is possible to work on "the right things"?
This is not my own idea, and I don't remember where I heard it first, but it had a great effect on me: Thinking about how much good I can do with my own work and how much good can be done with the money I earn, I have to realise that my money can do more than I can. While I would prefer to do both - work a job that matters and give money to others to do good stuff, I am not currently in the position to do so. I find it hard enough to avoid doing things I am not comfortable doing, working for industries I would rather not support. But what I can do is live a frugal life and give away the money I don't need to help others do better things. Again, not because I don't want to get my hands dirty or anything, but because other people or organisations are better suited to doing this stuff.

I wish I could do both: work on the things I believe to be the right ones, and support others to do the same at the same time. But then again, what I believe to be the right things might turn out to be completely wrong. I think it is important to keep an open mind on what the right things are, and change directions in the face of new evidence.

If it happened, please tell the story of your decisions regarding your work because of your values and social responsibility.
I don't remember any such decisions in my short life as a developer. I imagine that they would include security or privacy issues, or issues of inclusion/exclusion of certain groups. Before becoming a developer, I worked at a staffing firm for a very short time. After about a month I figured out how they treated their staff and quit immediately. I did not want to be associated in any way with such business practices. In hindsight I regret not having done more for the staff.

How do you think about selecting industry, customer and project based on your values?
An important choice for me was to work with a company that embraces Open Source Software instead of working for a closed source company. However, the software we produce is not Open Source, which I deeply regret. Choosing the customers I work with directly is unfortunately out of reach at the moment. This is something I aspire to.

Do you have problems with any industries? Why? What about Porn industry or weapon manufacturing?
I don't think porn is bad per se, as long as it is consenting adults working in it. (Of course, more could be said about the gender issues in most porn, but this is not the place and time for that.) As for the industries I don't want to work for: I don't have a list ready. But it is very obvious for me that the weapons and defence industry definitely have place high on that list. I would put animal factories and suppliers on that list too. As a teen I had to work as a farm hand for a few weeks, and was assigned to a small scale chicken farm. Every morning I had to walk through the big hall with all the chicks and collect the ones that had choked over night. Not a good memory.

Cow PortraitObviously, companies that use child labour are out of question as well. I also see combustion engines as a problem for climate change - although there are bigger problems, and worse polluters. Many car companies are looking into ways of making individual transport more sustainable making it a less clear-cut question. On the other hand, I would not want to work for companies in the petrochemical sector. Banks and insurance companies probably get a space on that list as well. And the pseudo-pharma industry that produces and sells pseudo-medicine. I mean those who sell sugar beads or other quackery and convince people they are as good as - or even better than - proper medicine. Any company, basically, that is concerned more with benefits for the few, instead of adding value for the many. Which is, unfortunately, quite common practice.

Did you ever reject a customer based on your values?
As I mentioned in an earlier question, I quit a job I actually needed because of fraudulent behaviour on their side.

On the other hand, what would be projects that you would love to work on?
I would love to work for social and educational projects. I see education as the most important tool to build a better world. Education on all levels, and for all social groups, everywhere. I am convinced that education is the only way to solve the problems we have today. (This is one of the reasons why I don't want to give up my small teaching gig.)

Some branches of research are important too. No future development without research. Not related to any specific project or customer, I find the use, propagation, and development of Open Source Software to be of great importance. Apart from the often mentioned four freedoms, I see it as a means to more equality. I have used and propagated Open Source for a long time, and want to start contributing to it as my next goal.

Thank you Rea for taking the time to answer my questions. (I saw the time of your last email was 1:18 am.)
Thank you, Peter, for the opportunity to think about these topics in a structured way! If anyone has any suggestions, please let me know.

30 October 2017

Managing the Difficulty of Coding Exercises

There are different scenarios when we might want to change the difficulty of coding exercises. This depends on our skill and the topic we want to practise. If an exercise is too easy we get bored. There is still value in repeating the very same exercise, e.g. internalising certain patterns or improving keyboard navigation, but boredom does not help learning. Here is an unsorted list of options to increase (and decrease) the difficulty of coding exercises:

Most DifficultMaking it Harder: Constraints
A constraint, or activity, is an artificial challenge during an exercise. I have discussed some of them in the past. Some constraints like No If, Cyclomatic Complexity One or Only Void Methods are easy to follow but make it hard to write your usual code. To have more challenge chose constraints that work against the assignment, e.g. use an algorithmic challenge together with Only Void Methods. Algorithms are often functional in nature but void methods are no functions. Win!

To make things more interesting, constraints can be combined. For example, Object and Functional Calisthenics are constraints that combine several rules. When creating combined constraints, it is important to make sure the constraints work together. There is no point in forcing a functional style with No Void Methods and an object oriented style with Only Void Methods at the same time. When Martin Klose and I combined the Brutal Coding Constraints we spent around 20 hours experimenting and fine tuning them. By the way, these Brutal Coding Constraints are probably one of the most challenging.

When the list of constraints gets long, it is easy to make a mistake and forget to follow one or another. In these situations you need a reviewer, e.g. Coding Dojo facilitator, pair programming partner or static code analysis tool, who checks for violations of constraints.

Harder: Changing Requirements
Another way to spice up an exercise is to introduce requirement changes. This is particularly useful for groups, e.g. Coding Dojos, when participants do not know which requirement is going to change. For the usual Coderetreat exercise Game of Life, several interesting changes have been proposed, e.g. Hex Life, Vampire Cells and the toughest constraint (Wormholes) by Adrian Bolboaca.

I witnessed Martin Klose taking this to the next level: In his exercise Wind of Change, he puts on a tie (because he is the product owner now) and keeps changing the requirements every few minutes. This is a lot of fun and adds some time pressure as well.

Requirement changes is useful to verify a design, usually used in double sessions on software design during Coderetreats. When you are on your own, as soon as you finished the exercise, you think of changes to the requirements and how they would affect your current design.

Harder: Algorithmic Challenges
Algorithmic challenges vary from easy to impossible. Project Euler even has a difficulty rating on each exercise. Often algorithmic challenges are based on mathematics, which makes them not useful for people with less academic background. Also, as soon as you found a solution, the exercises get boring. Using additional constraints can make them fun again, but that would be different exercises then.

I have seen senior developers being more interested in algorithms than XP practises like Pair Programming or TDD. Algorithms are a perfect way to "lure" them into attending Coding Dojos. After a few dojos, people understand the value of practise and will agree to do basic katas with focus on TDD.

If you need a challenge, go for an algorithmic kata and chose a difficult exercise like Potter, Searching or one from Project Euler above number 20.

Harder: Try to be Faster
I do not like to apply time pressure during exercises, because people get sloppy when under pressure. On the other hand, this is what needs to be trained to not get sloppy. Houssam Fakih explains this with a short video (where three people throw basket balls. One is a beginner and fails from time to time, one is experienced and wins repeatedly and one, a "master", is doing the same, but much faster.) Houssam's suggestion is to do the same, but try to be faster. I did that once because I wanted to squeeze an one hour life refactoring demo into a 45 minutes presentation slot. It was hard work, exactly what I wanted.

BalanceWarning
When using constraints and other techniques I describe above, it is easy to go over the top. The exercises become too difficult and working on it is frustrating and eventually we stop doing it. While this might be OK for yourself, it must not happen when working with a group. Exercises like Brutal Coding Constraints are very difficult and not - I repeat - not suitable for a general audience. People tend to overestimate their skill and get frustrated easily.

When facilitating a Coding Dojo, I want to stay in control of the difficulty of the exercise for all participants. I aim for easier, simpler exercises and keep the difficult ones for myself. In rare cases, when I meet very skilled people, I assign them individual constraints, because I know them and I am confident that they will handle. I also make sure everyone understands that it is difficult what they want to do.

Making it Easier: Simpler Assignments
Start with a simple problem. There is always a smaller assignment, The smallest kata I know is FizzBuzz, it is just a single function. There is nothing wrong with FizzBuzz and its friends. I do it from time to time when I explore a new language or try different constraints (or when it is very late and I feel tired). Some function katas like Prime Factors are small too, but algorithmic in nature, so stay away from them. These katas are called FizzBuzz of Function Katas.

Easier: Use Well Known Problems
Solving a programming assignment includes many steps: e.g. understanding the problem, finding a solution, implementing the solution, testing it, etc. The assignment is easier if we get rid of some of these steps. If we use a well known problem, e.g. a ticket machine or a game everyone knows, we already know what is expected.

Easier: Clarify/Understand the Problem
Often the problem with an exercise is that people do not understand the problem. We are eager to get into the code, but we need to understand the assignment first: Take time to analyse the problem you want to solve. Google it. If it is a game like Tic-Tac-Toe, Minesweeper or Pac-Man, find an online version and play for a while. Draw some sketches or diagrams of what needs to be done. Create an list of initial acceptance criteria. To find them, you have to think about the problem. In Coding Dojos I ask people to spend the first ten minutes on creating a test list. This forces them into thinking about the problem.

If you practise with a partner, which I highly recommend, try Adi's pair programming game Solution Seeker. Solution Seeker makes you find at least three different solutions to your problem before you are allowed to implement one of them. This forces you to think hard about the problem and different options to solve it.

As a facilitator, make sure you fully understand the problem so you can answer any question about it. Give more explanations and discuss the problem from different angles. Provide posters or handouts of the problem for participants for later reference.

EasierEasier: Repeat the Same Exercise
Repeating the exactly same exercise is considered boring, but it helps. You will understand the assignment better after working on it once. After implementing it several times, maybe even in different programming languages, more and more aspects of the implementation are known and you can go deeper. (This why we run Game of Life in a Coderetreat six times. We do not want to fight with a new problem each session.) This is especially true for hard problems or if you are not satisfied with your process or final solution.

Easier: Focus on One Thing
After repeating the same exercise one or more times, the problem is sufficiently known and you can shift your focus to something else. Using a well known problem is also a way to focus on one thing, in this case you do not focus on solving the problem. There are exercises that isolate different aspects of development: For example, if I want to focus on finding test cases and designing unit tests, I go for the Gilded Rose. If I want to practice refactoring, I do Tennis or Yatzy. Both code bases contain ugly code which is more or less fully covered with tests, making it safe to refactor. There are exercises isolating other things, like incremental development, emergent design, SOLID principles, etc.

Koans belong into this category. Koans are series of little exercises, starting with basic things and building on each other to move to more advanced topics. They are useful to learn programming languages. They contain a list of failing test cases, where tiny pieces of code have to be filled in to make them pass. The idea is not only applicable to programming language constructs. For example I have created Unit Testing Koans to teach xUnit assertions and life cycle to junior developers.

All these exercises require prepared code. For example Gilded Rose is available in 26 languages, including lesser common ones like ABAP and PL/SQL. Trivia even contains COBOL and VB6 - which is very suitable for a legacy code exercise. Obviously prepared code limits the number of languages which can be used. If you want to practice in a new language like Elixir, Elm or Swift, you might need to port the code base first. Although, if the new language is trending, chances are high that someone already ported it.

Easier: Prepared (Helper) Code
Prepared code is useful in many situations, especially outside the core of the practise. Even code snippets or cheat sheets help. For example when I run the Data Munging exercise with focus on functional programming in Java, I show participants code snippets how to read the text file. File IO is not related to the exercise and I want them to spend time working with Lambda expressions and Stream.

Prepared code allows us to focus on one thing, but we need to understand the code first. Unfortunately this adds extra complexity. Unless you want to practice working with unreadable code, prepared code must be simple and super clean. Try to make it more expressive, maybe even verbose, than your usual code and use very descriptive names. Describe the code in the assignment. If there are more methods or classes, visualise their relations. For example in my Test Double exercise, I added a simple diagram of the prepared classes and their collaborators.

Easier: Guide Step by Step
Alex Bolboaca once told me that as facilitator of an exercise it is most important to manage participants' frustration. When I notice that most of the participants are unable to move forward, I take control of the group and guide them step by step. I am not giving them answers, but moderate the necessary process. Maybe we need to discuss the problem before hand on a white board. Or we discuss potential solutions up front. To get an initial test list, I keep asking how we will verify our product until we have a reasonable number of test cases. Sometimes I switch to Mob Programming where the whole group works on the assignment together and I am able to support them best (a.k.a. micro manage).

Conclusion
There are many options to make coding exercises easier or more difficult. I recommend starting easy. There is no point in hurting yourself or others. ;-)

Thanks to Kacper Kuczek and Damian Lukasik for discussing this with me.