Twitter github

Maven Tycho, Hudson (Jenkins) and Eclipse

I’ve helped setup quite a few Maven Tycho builds for people as of late, with more requests coming in. Personally, I’ve had quite a bit of experience working with Tycho-based builds as being one of the first early adopters with the EGit and JGit projects at Eclipse. At that time, Tycho was pretty rough around the edges but I’m pleased to report I believe now Tycho to be one of the best, if not the best option for building Eclipse/OSGi related artifacts. Sure, it’s not perfect but it’s pretty damn good now. Furthermore, there are even large projects like Mylyn at eclipse.org that have moved to Tycho to manage their whole build if you need further proof.

One of the common complaints from folks within the eclipse.org ecosystem is how difficult it is to setup build. In the past, we had the Athena Common Build project at eclipse.org with some success. Athena helped many projects get off the ground when it comes to building and publishing their artifacts. However, it still suffered the drawbacks of PDE Build based systems when it came to debugging, customizing and extending the build system. For example, if you wanted to integrate with Hudson, add FindBugs to your build or run SWTBot unit tests, you suffered quite a bit. Well, I’m happy to report that Maven Tycho helps quite a bit with these customization issues. To help alleviate some of the pain we’re experiencing with build at eclipse.org, I decided to put up sample Maven Tycho project for people to look at and refine. I’m calling the effort Minerva as I think it plays nicely with what was done in the Athena project (Minerva is the Roman equivalent of Athena and the word Maven is in Minerva!).

The goal of the Minerva project would be to get a well documented sample build for eclipse.org projects. I can even envision a future when a new eclipse.org project comes around and they get a build already generated for them and hooked up to hudson.eclipse.org! I think once you have something working and it’s well documented, it becomes pretty easy to expand as you add new features.

To get started, I’d take a look at what is on the wiki already. If you want to play with code, ensure that Maven 3 is installed and then do a quick clone.

git clone git://github.com/caniszczyk/minerva.git
mvn -Dskip-ui-tests=true clean install

Once you do that, you should notice a p2 repository (update site) generated in the update site project’s target/site folder. To sweeten the deal, if you have Hudson (Jenkins) installed anywhere, you can configure it quickly to build the code base.

Remember how long something like that took before? In reality, Tycho isn’t that complicated to setup.

In Maven, the parent pom.xml serves as the central point on adding things to the build. It’s also generally the most complicated piece of the build as it contains information that is shared by children pom.xml files. The first part of the parent pom.xml contains identifying information:

...
<groupId>org.aniszczyk.minerva</groupId>
<artifactId>minerva-parent</artifactId>
<version>1.0.0-SNAPSHOT</version>
<packaging>pom</packaging>

<name>Minvera Parent</name>
...

The second part contains profile information and where to get dependencies.

  <properties>
    <tycho-version>0.10.0</tycho-version>
    <platform-version-name>helios</platform-version-name>
    <eclipse-site>http://download.eclipse.org/releases/${platform-version-name}</eclipse-site>
    <wikitext-site>http://download.eclipse.org/tools/mylyn/update/weekly</wikitext-site>
    <swtbot-site>http://download.eclipse.org/technology/swtbot/${platform-version-name}/dev-build/update-site</swtbot-site>
  </properties>

  <profiles>
    <profile>
      <id>platform-helios</id>
      <activation>
        <property>
          <name>platform-version-name</name>
          <value>helios</value>
        </property>
      </activation>
      <properties>
        <eclipse-site>http://download.eclipse.org/releases/helios</eclipse-site>
        <platform-version>[3.6,3.7)</platform-version>
        <swtbot-site>http://download.eclipse.org/technology/swtbot/helios/dev-build/update-site</swtbot-site>
      </properties>
    </profile>
    <profile>
      <id>platform-indigo</id>
      <activation>
        <property>
          <name>platform-version-name</name>
          <value>indigo</value>
        </property>
      </activation>
      <properties>
        <eclipse-site>http://download.eclipse.org/releases/indigo</eclipse-site>
        <platform-version>[3.7,3.8)</platform-version>
        <swtbot-site>http://download.eclipse.org/technology/swtbot/indigo/dev-build/update-site</swtbot-site>
      </properties>
    </profile>
  </profiles>
...
  <repositories>
    <repository>
      <id>helios</id>
      <layout>p2</layout>
      <url>${eclipse-site}</url>
    </repository>
    <repository>
      <id>swtbot</id>
      <layout>p2</layout>
      <url>${swtbot-site}</url>
    </repository>
    <repository>
      <id>wikitext</id>
      <layout>p2</layout>
      <url>${wikitext-site}</url>
    </repository>
  </repositories>

What’s cool about setting up these repositories and profiles is that we can easily switch the build to run against Helios or Indigo repositories. If you want to run Indigo, simply set -Dplatform-version-name=indigo and you’re good to go. You could even setup multiple hudson jobs that build against various Eclipse releases if you want to ensure everything works properly.

The third part lists the modules (e.g., features, plug-ins) that are part of the build:

  <modules>
    <module>org.aniszczyk.minerva.core</module>
    <module>org.aniszczyk.minerva.ui</module>

    <module>org.aniszczyk.minerva-feature</module>
    <module>org.aniszczyk.minerva.source-feature</module>

    <module>org.aniszczyk.minerva-updatesite</module>

    <module>org.aniszczyk.minerva.tests.core</module>
    <module>org.aniszczyk.minerva.tests.ui</module>
   </modules>

Each of those modules has a simple pom.xml that tells Tycho what type of project it is (think feature or plug-in). You should be able to piece things together by looking at the documentation on the wikiLet me know if you have any questions and please feel free to add anything. My plan is eventually to get things up to shape so the code could live at eclipse.org and get things up to a point where a Maven archetype could be built and used to generate new projects.

Thoughts? Also, if you’re an eclipse.org project and interested in moving to Tycho, feel free to contact me and look at the example.

  • Jan (from Tycho) started something similar a couple days ago in preparation for our EclipseCon tutorial. Maybe we should just converge.

  • Steve

    I’m not getting very far, any idea why?

    [INFO] Scanning for projects…
    [INFO] ————————————————————————
    [ERROR] FATAL ERROR
    [INFO] ————————————————————————
    [INFO] Error building POM (may not be this project’s POM).

    Project ID: org.aniszczyk.minerva:minerva-parent

    Reason: Cannot find layout implementation corresponding to: ‘p2’ for remote repository with id: ‘helios’. for project org.aniszczyk.minerva:minerva-parent

  • Make sure you have the latest version of Maven 3. It looks like you’re using Maven 2.

  • Sure, I think we could collaborate. I see two main things lacking, a simple tycho example that Eclipse people can follow to see how they can migrate their builds. The other is an example that is geared towards the eclipse.org project space. For example, the best way to integrate with the signer (/usr/bin/sign) and how to publish your build within the eclipse.org infrastructure.

  • David Carver

    What we need to come up with is a standard Parent Pom, that eclipse projects can inherrit from. Something that already has setup FindBugs, CheckStyle, and CPD for running things. The parenet pom should be deployed to a standard Maven Repository either at eclipse.org, or hosted on SonaTypes OSS repository. This in itself would go a long way to easing adoption. As the rest of the setup is pretty straight forward after that.

  • Great! Combined with that bug : https://bugs.eclipse.org/bugs/show_bug.cgi?id=283745
    Maven will become very popular @eclipse!

  • +1. And also deployment configuration to publish Eclipse artifacts not only to p2, but also to Central staging/Maven central.

  • +1. And also deployment configuration to publish Eclipse artifacts not only to p2, but also to Central staging/Maven central.

  • Hi Chris,

    Do you think that a Tycho based build needs to be a multi-module build? My team and I are trying to create a build system for our Eclipse plugin/feature/product/update site components that is as modular as possible but are having a few issues.

    Some of the decisions we are trying to keep to (but may have to be sacrificed) are:

    – Be Maven un-aware at development time (within the IDE)
    – Be OSGi manifest driven
    – A module (OSGi bundle, Eclipse Feature, Eclipse Product) should be built once for a given state of it’s source code and configuration.
    – The build of one module should resolve its dependencies as binary dependencies from a repository, not from source.
    – Building an Eclipse Feature should be a packaging task without compilation as it’s constituent bundles (plugins) should already have been built.
    – Continuous Integration (CI) – commits to the managed resources of a module should trigger an automated build and test of that module.
    – A developer should only need to have the source of the modules he/she is working on in their IDE.

    Some of these objectives _may_ be an overreaction to an existing build process for another project which is an unwieldy PDEBuild that builds everything from source – but it just seems that there should be an easy way to build the minimum for a lean mean modular product…

    We are using Tycho (0.10.0) and Jenkins and have been able to build the bundles and deploy into both Artifactory and a local P2 repository but started to come a little unstuck when it came to automated testing as part of a bundle build (our initial plan was to use test fragments). The following questions I posted to the tycho users mailing list a couple of days ago (no bites as yet) – hope you don’t mind me reposting them here but I’d be very interested to hear your thoughts:

    1. If using a test fragment, how should CI jobs be set up to ensure that deployment of the host bundle is dependent on unit test success?

    2. Is it possible to create separate jobs in Jenkins for the host bundle and the test fragment and have the dependency between the jobs be derived automatically from the Fragment-Host entry in the fragment manifest?

    3. Would it be necessary to use the integration-test Maven goal to run unit tests in a test fragment?

    Many thanks,
    Dan

  • Hi Chris,

    I know I already mentioned it on twitter but, that might interest your readers as well : there’s a maven archetype available at https://issues.sonatype.org/browse/TYCHO-442

    regards,

    Fred Bricon

  • Claude

    I greatly appreciate this project. The example project builds smoothly on my machine. It would be great, if you could extend the example project to build a product using a product configuration.

    Regards,
    Claude

  • Thanks for the kudos Claude. Right now the example project is specifically targetted for eclipse.org projects and the associated infrastructure. I can’t think of one eclipse.org project that uses a product definition file so I didn’t include it in the example (I will if I can find one). For your benefit, you should checkout the Tycho reference card:

    https://docs.sonatype.org/display/TYCHO/Tycho+reference+card

  • – Be Maven un-aware at development time (within the IDE)

    This is not a problem… the current we way use Tycho within EGit and JGit is that we use PDE at development time to do everything. At build time with Hudson, we simply kick off Tycho.

    – Be OSGi manifest driven

    Yes, see the response above. This isn’t a problem.

    – A module (OSGi bundle, Eclipse Feature, Eclipse Product) should be built once for a given state of it’s source code and configuration.

    Yes.

    – The build of one module should resolve its dependencies as binary dependencies from a repository, not from source.

    Tycho handles this already, just list your repositories.

    – Building an Eclipse Feature should be a packaging task without compilation as it’s constituent bundles (plugins) should already have been built.

    Sure, it’s a packaging step. That’s what tycho does essentially.

    – Continuous Integration (CI) – commits to the managed resources of a module should trigger an automated build and test of that module.

    Hudson/Jenkins do this already as they can watch SCM commits. This is how most projects are configured at eclipse.org… to watch the SCM server for commits.

  • See http://maven.eclipse.org… it’s a work in progress.

  • Cristiano Gavião

    Hi Chris, 
    What OS are you using? Have you tried to build minerva on MacOs before?I’m getting errors trying to…

    [INFO] BUILD FAILURE

    [INFO] ————————————————————————

    [INFO] Total time: 7:26.101s

    [INFO] Finished at: Fri May 20 11:43:55 BRT 2011

    [INFO] Final Memory: 68M/123M

    [INFO] ————————————————————————

    [ERROR] Failed to execute goal org.eclipse.tycho:tycho-p2-repository-plugin:0.13.0-SNAPSHOT:assemble-repository (default-assemble-repository) on project org.aniszczyk.minerva-repository: Could not assemble p2 repository: Copying p2 repository content failed: “Problems resolving provisioning plan.”: [“Unable to satisfy dependency from org.aniszczyk.minerva.source.feature.group 1.0.0.201105201143 to org.aniszczyk.minerva.core.source [1.0.0.201105201143].”, “Unable to satisfy dependency from org.aniszczyk.minerva.source.feature.group 1.0.0.201105201143 to org.aniszczyk.minerva.ui.source [1.0.0.201105201143].”] -> [Help 1]

    I’m using 0.13.0-SNAPSHOT because 0.12.0 doesn’t work either…Thanks

  • This issue should be fixed with the latest commit on GitHub…

    https://github.com/caniszczyk/minerva/commit/50b83d94f92b0ecc57d6dd068732c6032e28c66b

  • Walter Boring

    I’m trying to figure this out and am having a problem with the mvn install build.  The ui tests are failing.

    I’m using maven 3.0.2 on linux ubuntu 11.04

    [INFO] [INFO] — tycho-packaging-plugin:0.12.0:package-plugin (default-package-plugin) @ org.aniszczyk.minerva.tests.ui —[INFO] Building jar: /home/waboring/devel/sample/minerva/org.aniszczyk.minerva.tests.ui/target/org.aniszczyk.minerva.tests.ui-1.0.0-SNAPSHOT.jar[INFO] [INFO] — tycho-p2-plugin:0.12.0:p2-metadata (default-p2-metadata) @ org.aniszczyk.minerva.tests.ui —[INFO] [INFO] — tycho-surefire-plugin:0.12.0:test (default-test) @ org.aniszczyk.minerva.tests.ui —[INFO] Adding repository file://home/waboring/devel/sample/minerva/org.aniszczyk.minerva.tests.ui/../org.aniszczyk.minerva-updatesite/target/site[WARNING] Failed to access p2 repository local-p2 (file://home/waboring/devel/sample/minerva/org.aniszczyk.minerva.tests.ui/../org.aniszczyk.minerva-updatesite/target/site), will try to use local cache. Reason: org.eclipse.equinox.p2.core.ProvisionException: URI has an authority component[INFO] Adding repository (cached) http://download.eclipse.org/releases/helios%5BINFO%5D Adding repository (cached) http://download.eclipse.org/technology/swtbot/helios/dev-build/update-site%5BINFO%5D Adding repository (cached) http://download.eclipse.org/tools/mylyn/update/weekly%5BINFO%5D Adding repository http://download.eclipse.org/tools/orbit/downloads/drops/S20110124210048/repository%5BWARNING%5D Failed to access p2 repository orbit (http://download.eclipse.org/tools/orbit/downloads/drops/S20110124210048/repository), will try to use local cache. Reason: org.eclipse.equinox.p2.core.ProvisionException: No repository found at http://download.eclipse.org/tools/orbit/downloads/drops/S20110124210048/repository.%5BINFO%5D Expected eclipse log file: /home/waboring/devel/sample/minerva/org.aniszczyk.minerva.tests.ui/target/work/data/.metadata/.log[INFO] Command line: /bin/sh -c cd /home/waboring/devel/sample/minerva/org.aniszczyk.minerva.tests.ui && /usr/lib/jvm/java-6-sun-1.6.0.24/jre/bin/java -Dosgi.noShutdown=false -Dosgi.os=linux -Dosgi.ws=gtk -Dosgi.arch=x86_64 -Xmx512m -XX:MaxPermSize=256m -jar /home/waboring/.m2/repository/p2/osgi/bundle/org.eclipse.equinox.launcher/1.1.1.R36x_v20101122_1400/org.eclipse.equinox.launcher-1.1.1.R36x_v20101122_1400.jar -data /home/waboring/devel/sample/minerva/org.aniszczyk.minerva.tests.ui/target/work/data -dev file:/home/waboring/devel/sample/minerva/org.aniszczyk.minerva.tests.ui/target/dev.properties -install /home/waboring/devel/sample/minerva/org.aniszczyk.minerva.tests.ui/target/work -configuration /home/waboring/devel/sample/minerva/org.aniszczyk.minerva.tests.ui/target/work/configuration -application org.eclipse.tycho.surefire.osgibooter.uitest -testproperties /home/waboring/devel/sample/minerva/org.aniszczyk.minerva.tests.ui/target/surefire.properties -testApplication org.eclipse.ui.ide.workbench -product org.eclipse.sdk.ide -nouithreadjava.lang.RuntimeException: Bundle org.aniszczyk.minerva.tests.ui is not found at org.eclipse.tycho.surefire.osgibooter.OsgiSurefireBooter.getBundleClassLoader(OsgiSurefireBooter.java:120) at org.eclipse.tycho.surefire.osgibooter.OsgiSurefireBooter.run(OsgiSurefireBooter.java:48) at org.eclipse.tycho.surefire.osgibooter.AbstractUITestApplication.runTests(AbstractUITestApplication.java:44) at org.eclipse.ui.internal.testing.WorkbenchTestable$1.run(WorkbenchTestable.java:71) at java.lang.Thread.run(Thread.java:662)

  • Thomas

    Hi Chris,

    i extracted your minerva project into separate cvs projects and added it to separate jenkins jobs (also the parent). Unfortunately jenkins/hudson can’t resolve the dependencies specified within the Manifest.mf (for example within the org.aniszczyk.minerva.ui.tests project). Am I doing anything wrong or does jenkins need another plugin… Can you help me?

    Best wishes, Thomas

  • Michael Pan Zhan

    I wish to see some example for building eclipse plug-in project. any suggetion?

  • Jandro

    Hi Chris! I’m a starter with the whole Maven, Tycho, Eclipse projects looking forward to install a CI Server such as Jenkins. I’ve tried to build Minerva tons of times but I keep getting the same error in the build and it is driving me crazy.

    I have added a pic to show the error. 

    Thank you so much for your time and congratulations! You did an awesome job!

  • Vikram

    Good job Chris!!…I picked a lot up referring your wiki pages while trying to build my tycho for eclipse plugins…I still have one scenario that I am unable to build using tycho. I have some third-party plugins/features on which my plugins/features are dependent (“Import-Package” in manifest). So I host these plugins on a locally created p2 repo .This works as I verified using the p2-director application with -list option that shows all my bundles from local p2 style repo. In my parent pom I make a reference to this p2-repo expecting tycho will pull these bundles for compiling/building my plugins. However, Tycho does not do as expected here….resulting in compile errors for my plugins. Any pointers how to get around this?