Only modified files in Jenkins

For a custom Jenkins build I needed to know all changed files since the last green build. I searched a lot and found a solution as combination of several StackOverflow answers. It took me some experimenting to get it working: I installed the Groovy plugin, configured the Groovy language and created a script which executed as system Groovy script. Here is the complete step by step guide for Jenkins 2.63, SVN and Groovy 2.4.11.

Jenkins only provides the current revision in the environment variable $SVN_REVISION. Of course Jenkins knows the information about changed files of each build as it is shown in the build status page. I guess a plugin would be able to access the model, but that is too much work. Fortunately the Jenkins Groovy plugin allows scripts to run under the system context having access to hudson.model.Build and other classes.

Groovy Programming Language
The Groovy programming language is a dynamic language which runs on the JVM. It integrates smoothly with any Java program and is the first choice for scripting Java applications. While not strictly necessary I recommend downloading the SDK's zip and unpacking it on the host where you run Jenkins, usually into the folder where you keep your development tools. For testing and debugging I also install it on my local workstation (in the same location).

Groovy in Jenkins
Next comes the Jenkins Groovy plugin. Open Jenkins in the browser and navigate the menus:

• Manage Jenkins
• Manage Plugins
• select tab Available
• filter "groovy"
• select Groovy
• Install

(You will need Jenkins admin rights to do so.) Then tell Jenkins which Groovy to use and where to find it. To configure the Groovy language go to

• Manage Jenkins
• Global Tool Configuration
• go to section Groovy
• Add Groovy: Give it a name and set GROOVY_HOME to the folder you unpacked it, e.g. /tools/groovy-2.4.11.
• deselect Install automatically
• Save

Now Jenkins supports Groovy scripts.

Run a Groovy script in the build
Now let's use a Groovy script in the project. On the project page,

• Configure
• go to section Build
• Add build step
• select Execute system Groovy script
• paste Groovy code into the script console
• Save

Now when you trigger the build, the script will be executed.

Debugging the Script
Of course it does not work. How can I debug this? Can I print something to the console? Groovy's println "Hello" does not show up in the build log. Searching again, finally the gist by lyuboraykov shows how to print to the console in system scripts: Jenkins provides the build console as out variable,

out = getBinding().getVariables()['out']

which can be used like out.println "Hello". Much better, now I can debug. Let's wrap the out.println in a def log(msg) method for later.

Getting the changed files of the current build
StackOverflow answer by ChrLipp shows how to get the changed files of the current build:

def changedFilesIn(Build build) {
  build.getChangeSet().
    getItems().
    collect { logEntry -> logEntry.paths }.
    flatten().
    collect { path -> path.path }
}

This gets the change set hudson.scm.ChangeLogSet<LogEntry> from the build, gets the SubversionChangeLogSet.LogEntrys from it and collects all the paths in these entries - this is the list of all file paths of all changed items in all commits (LogEntrys). I guess when another SCM provider is used, another type of ChangeLogSet.LogEntry will be returned, but I did not test that. To better understand what is going on, I added explicit types in the final Groovy script, which will only work for Subversion projects.

Getting all builds since the last successful one
I want all changed files from all builds since the last green one because they might not have been processed in previous, failed builds. Again StackOverflow, answer by CaptRespect comes to the rescue:

def changedFileSinceLastSuccessfull(Build build) {
  if (build == null || build.result == Result.SUCCESS) {
    []
  } else {
    changedFilesIn(build) +
      changedFileSinceLastSuccessfull(build.getPreviousBuild())
  }
}

In case there is no previous build or it was successful the recursion stops, otherwise we collect changed files of this build and recurse into the past.

All Together
Let's put it all together,

def changedFiles() {
  def Build build = Thread.currentThread()?.executable
  changedFileSinceLastSuccessfull(build).
    unique().
    sort()
}

After collecting all duplicates are removed, as I do not care if a file was changed once or more times, and the list is sorted. In the end the list of changed files is saved as text changed_files.log into the workspace. (The complete jenkins_list_changed_files.groovy script is inside the zipped source.)

While developing the script, the Jenkins script console was very handy. As soon as the script worked, I created a the file jenkins_list_changed_files.groovy, put that under version control and changed the build definition step to use the script's file name. Next time the build ran, the script file would be executed, or at least so I thought.

Script Approvals
Unfortunately system Groovy script files do not work as expected because Jenkins runs them in a sandbox. Scripts need certain approvals, see StackOverflow answer by Maarten Kieft. To approve a script's access to sensitive fields or methods navigate to

• Manage Jenkins
• In-process Script Approval (This is the one but last item in the list.)
• Approve

The sandbox is very restrictive, the full jenkins_list_changed_files needs a lot of approvals:

field hudson.model.Executor executable
method groovy.lang.Binding getVariables
method hudson.model.AbstractBuild getChangeSet
method hudson.model.AbstractBuild getWorkspace
method hudson.model.Run getNumber
method hudson.model.Run getPreviousBuild
method hudson.model.Run getResult
method hudson.scm.SubversionChangeLogSet$LogEntry getPaths
method java.io.PrintStream println java.lang.String
new java.io.File java.lang.String
staticMethod java.lang.Thread currentThread
staticMethod org.codehaus.groovy.runtime.DefaultGroovyMethods flatten java.util.List
staticMethod org.codehaus.groovy.runtime.DefaultGroovyMethods println java.lang.Object java.lang.Object
staticMethod org.codehaus.groovy.runtime.DefaultGroovyMethods sort java.util.Collection
staticMethod org.codehaus.groovy.runtime.DefaultGroovyMethods withWriter java.io.File groovy.lang.Closure

Creating a new java.io.File might be a security risk, but even println is not allowed. Adding all these approvals is a boring process. The build breaks on each missing one until everything is well. As soon as you have all the approvals, you can copy Jenkins' scriptApproval.xml found in JENKINS_HOME (e.g. ~/.jenkins) and store it for later installations. The full scriptApproval.xml is inside the zipped source.

Conclusion
Jenkins' Groovy integration is very powerful. System scripts have access to Jenkins' internal model which allows them to query information about build, status, changed files etc. On the other hand, development and debugging is cumbersome and time consuming. IDE support helps a lot. Fortunately StackOverflow knows all the answers! ;-)