Continuous Integration for Addons and other openFrameworks

bakercp · November 1, 2016, 10:47pm

Hello all,

I’m in the process of adding adding continuous integration (CI) support for my addons to auto generate documentation, check for compilation errors, etc.

I’m roughly basing my solution on the CI support for the openFrameworks master repository, but the system is pretty complex (as one would expect for so many platforms).

I’d like to run have my addons’ “stable” branch build against the latest stable release of oF, and my develop branch build against the latest nightly build.

I suppose I could basically copy the complete openFrameworks CI system and add a single script to run my tests / and compile my examples, but that seems a bit excessive, but perhaps that’s necessary?

@arturo do you have any thoughts on a more streamlined way to set up a cross-platform CI system for addons?

Seems like a simple system for testing all desktop platforms would be pretty straight-forward, but it would be nice to test across all platforms, as we have that working with the main openFrameworks already.

Anyway, thoughts?

arturo · November 1, 2016, 11:33pm

i’ve been thinking about this for a while and it’s not so straight forward, not even for desktop platforms. the main problems are:

you need to download an compile all of OF to test each addon
most addons change less often than OF itself so they usually break because OF changes not because of changes on the addon but travis doesn’t allow to test a repository when another changes or with a certain frequency only when that repository has new commits or PRs
it’s pretty complex to setup and would require everyone to add the config to their addons

another possibility would be to have our own server constantly download every addon listed in ofxaddons, compile OF and then compile all addons examples one after another and publish any potential errors through a web page or even open an issue in the corresponding addon github page through the github api

arturo · December 11, 2016, 2:03pm

i’ve just pushed a travis config file to ofxAddonTemplate that should work with any addon, it needs to clone and compile all of OF but with the libraries gone from the repo cloning is pretty fast now. the config file tests the addon with both stable and release branches of OF but could be easily separated for people who keep 2 branches to track the corresponding in the OF repo.

to use it just copy .travis.yml from the template to any addon, create a travis account and enable the addon in your travis to be tested.

perhaps we can have a script that make a PR to addons repositories everytime there’s a new OF release so addons get tested automatically against the newest release

bakercp · January 5, 2017, 3:39am

Just one other note – while it would be nice to have a “hook” for repos to run builds when oF master / stable is updated, it might be useful to set up a “cron” job on addon repos …

https://github.com/travis-ci/beta-features/issues/1

It’s not a perfect system, but we’re getting there!

armadillu · February 1, 2017, 12:22am

Hey guys, I’m trying to decipher what to do for this to work for my ridiculous addon collection too; I got it working for some simple addons, but it seems to ignore each example’s config.make (which I need for addons that depend on other addons, or addons that require some macros defined).

Looking through the travis logs, I think I found the issue, but not really sure where to solve.

For example, I am trying to compile the example of this addon with a pre-processor macro defined in config.make; as seen here

But it seems the definition is ignored. Looking at the logs, I see this entry

cat: obj/linux64/Debug/.compiler_flags: No such file or directory

Complete Log Here

and looking through the makefile related scripts in OF, it seems the “.compiler_flags” are saved to disk into a file with that name at some point. Might me a simple case of the filed being saved in the wrong dir; but I really can’t keep up with the working dir changes all over the nested scripts… But of course, if the file is not there, the compiler flags are lost.

Also, I wonder if the OF scripts in “OF/scripts/ci/addons” should look at “addon_config.mk” and clone automatically any dependencies (although where to clone them from is a bit of a long shot, I could see some sort of scapper of addon_config.mk files to build an index addon urls by name); or maybe forcing the addon_config.mk to add a URL for each addon dependency… Or maybe based on “addons.make” for each example; there seem to be quite a few places where this can be defined and its a bit confusing to read which one does what (I know some are for PG, but maybe they could be consolidated)

I like what bakercp has done in his addon/scripts/ci/install.sh script install.sh and it should maybe be added in the ofxAddonTemplate bc the instructions in the .travis.yml file are quite vague travis.yml. Maybe there can be a install.sh that is a user-driven script, and some other plain text file like “addons.make” where you just input repo URLs for other addons?

Another thing that I am seeing is that the OSX ci tests seem to hang forever; but they don’t even start. Are you also seeing those?

Sorry for this messy post, and thanks for working on this guys!

bakercp · February 1, 2017, 3:00am

Regarding dependencies, I’m working on that and have it mostly working (for my own dependencies and core dependencies … after several days of work).

Check out my growing collection here:

https://travis-ci.org/bakercp/

One point of note, I made a few modifications to the core scripts which can be found in my openFrameworks CI-TESTS branch. Here’s an example:

github.com

bakercp/ofxGeo/blob/master/.travis.yml#L191


language: c++
compiler: gcc
sudo: true
matrix:
include:
# fully specify builds, include can't dynamically expand matrix entries
# relative order of sudo and env is important so that addons: is recognized


# Linux 64bit, OF master
- os: linux
  dist: trusty
  sudo: required
  env: TARGET="linux64" OF_BRANCH="master"
  addons:
    apt:
      sources:
        - ubuntu-toolchain-r-test
      packages:
        - gcc-4.9
        - g++-4.9
        - gdb


# OSX, OF master
- os: osx
  osx_image: xcode8
  compiler: clang
  env: TARGET="osx" OF_BRANCH="master"


# Exclude the default build that would otherwise be generated
# see https://github.com/travis-ci/travis-ci/issues/1228
exclude:
  - compiler: gcc
before_install:
- curl -SLsO https://raw.githubusercontent.com/bakercp/ofxAddonScripts/${OF_BRANCH}/scripts/ci/tools/install.sh
- source install.sh
install:
- do_install
script:
- do_script


git:
depth: 10

Note the dependencies install script (which defaults to the matching branch and my username for the moment)

github.com

bakercp/ofxGeo/blob/master/scripts/ci/install.sh

#!/usr/bin/env bash
set -e

OF_ROOT=$( cd "$(dirname "$0")/../../../.." ; pwd -P )

# Default addon github info.
GH_USERNAME='bakercp'
GH_BRANCH='master'
GH_DEPTH=1

# An array of required addons that will be gathered below.
REQUIRED_ADDONS=()

# Extract ADDON_DEPENDENCIES from addon_config.mk file.
if [ -f ${OF_ROOT}/addons/${OF_ADDON_NAME}/addon_config.mk ]; then
  while read line; do
    if [[ $line == ADDON_DEPENDENCIES* ]] ;
    then
      line=${line#*=}
      IFS=' ' read -ra ADDR <<< "$line"

This file has been truncated. show original

Here are travis results so far (check any of them for their .travis file and CI scripts) https://travis-ci.org/bakercp/

And here are the corresponding appveyor apps:

e.g.

https://ci.appveyor.com/project/bakercp/ofxGeo etc (not sure how to list all of them …).

There is only a minor change to the oF core here:

https://github.com/openframeworks/openFrameworks/compare/master...bakercp:CI-TESTS?expand=1

I’ll make a PR on that soon.

On the addon front, I haven’t made an update to the addon template, but check out any of my current addons and they all have identical CI scripts and .travis / .appveyor updates.

Things that we need I believe …

We need to be able to install dependencies automagically. I do this in my addon’s install.sh file currently, but it is limited to choosing them by name from my personal repositories.

Ideally, we would be able to label dependencies more in the addon_config.mk / addons.make file with greater precision – see this discussion:

https://github.com/openframeworks/projectGenerator/pull/90 (most recent comments).

CB

bakercp · February 1, 2017, 3:24am

Also, on other note – the reason for a lot of my updates were motivated by my desire to write a single bash script that would work on all Appveyor MSYS2+VS32/64. The same install.sh script installs the dependencies for all platforms on all systems currently.

In the core files that I modified, they were primarily changes to make the appveyor and travis config files as similar as possible because they were both installing in different places and doing things a bit differently. Currently config files and sequence are now almost identical, which should make it easier for new users to understand.