Don’t Do Eclipse Headless Builds Ever

October 29, 2012 SteveHawley

We standardized on using Eclipse for Java development for a few reasons, not the least of which was the having Team Explorer Anywhere integrated.  This was a key feature for us since we use TFS for our regular source control and would prefer to not have another infrastructure system, if possible.

Since our shop uses continuous integrated and have all our releases built by automation, we needed to build the Eclipse projects on our own system.  If you search for “eclipse headless build” you will find a number of hits on how to do this.  They work.  Sometimes.  Sort of.  We found a couple things – we can’t check in the .metadata folder since it stores absolute paths and one particular engineer’s .metadata folder won’t necessarily work on the build machine or any other engineer’s machine.  So we kept a private copy on the build server with the provision that if you added a new sub project, it would be necessary to update the .metadata folder on the build server (and on engineering machines).  We found that even with that, builds were flakey.  They worked most of the time, but not all the time.  By adding a call to refresh the projects before the headless build, we found that we could get a more reliable build.  Then we had a new project added that didn’t really change the build order on the engineering machines, but it was causing errors on the build server that didn’t halt the build.  We only found out about them by reading the logs and finding out that later in the build obfuscation was failing.  Interestingly enough, the java compiler was spouting errors, but was writing .class files that were technically correct but would never load or run (the compiler injected throws in all the constructors).

Long story short: Don’t Do Headless Builds with Eclipse.  They may work in simple projects, but will ultimately fail in the worst possible way: generating bad code and not failing the build.

So what do we do?  All our builds are run from CruiseControl which then fires up NAnt tasks.  When we added Java, we kept the same infrastructure but now fire off Ant tasks to do more Java specific things.  First, I broke down a project build into 5 tasks (there’s actually a 6th, but I haven’t integrated it in this build step fully yet):

  1. validate
  2. wipe previous output (not technically necessary, but I like to make sure the environment is clean)
  3. compile
  4. make jar
  5. obfuscate

With this, I added one file to the global workspace and one file to each project.  The global file is a csv that contains a project directory name on each line (with the notion that there may be other information later on each line), then I loop over it with a NAnt foreach (this implies that the file is in build order):

<foreach item="Line" in="ProjectList.csv" property="buildDirName">

Then I set up some properties including the .classpath  file, the src folder, the bin folder and so on.

Validation – I want to make sure that each project folder really is intended to be a project.  There should be a .classpath and a file I call ProjectManifest.xml.  It don’t actually contain anything yet, but in the future, there may be project specific properties set in this file so that there need not be as much logic and special cases in the actual build script.

Wipe Previous Output – this is easy, just a delete task on the bin folder.

Compile – this is somewhat complicated.  I wrote an embedded C# script in NAnt that parses the .classpath file and turns it into a string suitable for a command line option to the java compiler.  Here’s the code, which works well enough:

<script language="C#" prefix="pathmunge">
  <references>
    <include name="System.Xml.dll" />
  </references>
  <imports>
    <import namespace="System.IO" />
    <import namespace="System.Collections.Generic" />
    <import namespace="System.Xml" />
    <import namespace="System.Xml.Serialization" />
  </imports>
  <code>
    <![CDATA[
    [Function("to-class-path")]
    public static string ToClassPath(string projectDir, string dotClassPath)
    {
        List<ClassPathEntry> paths = GetPaths(dotClassPath);
        if (paths == null)
          throw new Exception("unable to find class path entries.");
        return ToClassPath(paths, projectDir);
    }
   
    private static List<ClassPathEntry> GetPaths(string dotClassPath)
    {
      using(TextReader reader = new StreamReader(dotClassPath)) {
        XmlSerializer serializer = new XmlSerializer(typeof(ClassPath));
        ClassPath cp = (ClassPath)serializer.Deserialize(reader);
        if (cp == null) return null;
        return cp.Paths;
      }
    }
   
    private static string ToClassPath(List<ClassPathEntry> list, string projectDir)
    {
      StringBuilder sb = new StringBuilder();
      foreach(ClassPathEntry entry in list)
      {
        string pathPart = null;
        switch (entry.Kind)
        {
          case "lib":
            pathPart = projectDir + entry.Path;
            break;
          case "src":
            if (entry.Path == "src") continue;
            else {
              pathPart = projectDir + entry.Path + @"\bin";
            }
            break;
          default:
            continue;
        }
        if (pathPart != null)
        {
          if (sb.Length > 0 && sb[sb.Length - 1] != ';') sb.Append(';');
          sb.Append(pathPart);
        }
      }
      return sb.ToString();
    }
   
    [Serializable()]
    public class ClassPathEntry {
      private string _kind;
      [System.Xml.Serialization.XmlAttribute("kind")]
      public string Kind { get { return _kind; } set { _kind=value; } }
      private string _path;
      [System.Xml.Serialization.XmlAttribute("path")]
      public string Path { get { return _path; } set { _path = value; } }
      private bool _combineaccessrules;
      [System.Xml.Serialization.XmlAttribute("combineaccessrules")]
      public bool CombineAccessRules { get { return _combineaccessrules; } set { _combineaccessrules = value; } }
    }
    [Serializable()]
    [System.Xml.Serialization.XmlRootAttribute("classpath", Namespace = "", IsNullable = false)]
    public class ClassPath
    {
      private List<ClassPathEntry> _paths = new List<ClassPathEntry>();
      [XmlElement("classpathentry")]
      public List<ClassPathEntry> Paths { get { return _paths; } set { _paths = value; } }
    }
    ]]>
  </code>
</script>

You can see that I leverage the Xml serializer to get the information that I need.  My first job after writing the compile-java task was to verify that a compiler error would fail the build appropriately.  It does.  Here is the compile-java task:

<!-- compile-java - compiles a single java project
  this needs to have Dir.CurrentJavaSrc set
-->
<target name="compile-java" >
  <echo message ="Compiling ${CurrentJavaProject}" />
  <if test="${not file::exists(File.DotClassPath)}" >
    <fail message="compile-java: bad project structure - looking for ${File.DotClassPath}"/>
  </if>
  <property name="JREMainJars" value="${environment::get-variable('JAVA_HOME')}\jre\lib\*" />
  <property name="JREExtJars" value="${environment::get-variable('JAVA_HOME')}\jre\lib\ext\*" />
  <property name="DerivedClassPath" value="${pathmunge::to-class-path(Dir.Build, File.DotClassPath)};${JREMainJars};${JREExtJars}" />
  <echo message="initial class path - ${DerivedClassPath}" />
  <property name="ant.commands" value="-Dcompile-classpath=${quote}${DerivedClassPath}${quote} -Dcompile-srcpath=${quote}${Dir.CurrentJavaSrc}${quote} -Dcompile-binpath=${quote}${Dir.CurrentJavaBin}${quote}" />
  <property name="ant.task" value="compile-project" />
  <call target ="ant" />
</target>

And the actual Ant task is straight forward:

<target name="compile-project">
  <echo message="compiling with srcpath ${compile-srcpath} binpath ${compile-binpath} classpath ${compile-classpath}" />
  <javac srcdir="${compile-srcpath}" destdir="${compile-binpath}" classpath="${compile-classpath}" />
</target>

The only other missing piece is how we invoke ant:

  <target name="ant">
  <exec program="ant.bat"
        basedir="${Dir.Ant}\bin"
        workingdir="${Dir.Build}"
        output="${Progress.Log.File}"
        append="true"
        timeout="7200000"
        verbose="true"
        failonerror="true"
        commandline="-buildfile ${quote}${Dir.Build}\JavaImageAnt.xml${quote} ${ant.commands} ${ant.task}" />
  <property name="ant.commands" value="" />
</target>

Oh and ${quote} is used to get an embedded double-quote into a string  <property name="quote" value='"' />.

Jar – jar is similar to compile.  We need the bin directory to pass in and then invoke an Ant task to build the jar file.  This is complicated only by the one project that has to have a few external jar’s combined into it, so one project gets handled with a special case Ant task.  Every project gets a jar even though unit test projects don’t strictly need them.

Obfuscate – the obfuscation task is nearly identical.  At present, all our jars get obfuscated with the same settings using Zelix Klassmaster.  We skip the obfuscate step on unit tests as we don’t need them and the dependencies would be unusual.

There are a number of ways that this type of build automation could be done – for example, keeping a completely separate ant file that handles the build entirely.  In an ideal world, the build would be done by the same system as an engineering workstation, but since that isn’t currently possible, we strike a balance to get the build as automatic as possible.  In theory, we should be able to analyze all the .classpath files and intuit the build order and this might be a good step for the future.

Previous Article
What is Partial Function Application?
What is Partial Function Application?

I’m going to talk about Partial Function Application in F# and show how...

Next Article
Atala-Luau Lunch
Atala-Luau Lunch

What is one to do when the boss is on vacation in Hawaii?  Possible...

Try any of our Imaging SDKs free for 30 days with Full Support

Download Now