Automated Testing

I don’t know that much about automated testing for massive multiplatform c++ projects, but it is something that I think is worth having a discussion about. I know a few people have suggested having a CI server, something like http://www.jetbrains.com/teamcity/.

from the list:

When we have adopted the new git workflow, maybe we can automate the release process with the github hook system - every new commit in master should be a release now.
btw, someone proposed on the forum that he could set up a hudson server for automated building/testing: nothing came of it, but it may be a good time now to take this up again.
http://forum.openframeworks.cc/t/hudson-for-ci/5340/0
http://forum.openframeworks.cc/t/licensing-of-fmodexpossible-alternatives/5313/6
Yeah, ben, I’ll go over the github wiki page again. I just wanted to just avoid reiterating the article. I’ll include an executive summary. :wink:

yes, being able to see that everything is compiling, or what is not in all the platforms will be super useful, didn’t knew that there were multiplatform services for free. will take a look. Also if you want to begin setting it up that will be great

if looking at hudson, be aware that there was a split/fork in the community (because of oracle, basically), and there’s now hudson and jenkins; most of the community followed the original/main dev to jenkins. http://en.wikipedia.org/wiki/Jenkins-%28software%29

In thinking about this a bit more, we maybe jumping the gun quite a bit to start talking about CI servers…

The first step is actually to get a unit testing (or even better, ref testing) framework in place so that we can start writing the tests. I took a brief look around, and here are a few options for unit testing:

cppUnit
googletest
cpptest

As for ref testing, we may be able to get there even faster. Generally ref testing dumps out images based off of code and then compares them to a set of reference images. I imagine that this community has more than one person that can roll the code to make this work…

I think my last post got eaten by the spam filter, but it was basically saying that we need to start jamming on an actual test suite before we can jump forward into the integration server. I threw together a quick concept build that has googleTest and a really simple test to just see how it would work.

I don’t think this is necessarily the best way to handle things (it would be cool if the tests were actually in the core lib), but it is more than zero, and maybe a conversation starter.

https://github.com/jonbro/openFrameworks/tree/gUnitTests

yeah it’s strange. I even got an email notification for your post. It said:

In thinking about this a bit more, we maybe jumping the gun quite a bit to start talking about CI servers…

The first step is actually to get a unit testing (or even better, ref testing) framework in place so that we can start writing the tests. I took a brief look around, and here are a few options for unit testing:

cppUnit
googletest
cpptest

As for ref testing, we may be able to get there even faster. Generally ref testing dumps out images based off of code and then compares them to a set of reference images. I imagine that this community has more than one person that can roll the code to make this work…

could it be just for now something that compiles the OF library + all the examples and reports if something is not compiling?

that would be a pretty solid! especially if it was super easy to run.

Not to keep bringing up the processing-js project, but the way their ref tests run is pretty neat… first you put some working code in a box, then it spits out a huge chunk of rgb values, then all further tests check to see that the values being output by the code are the same:

  
  
  
var pixArray = [100, 200, 100, ...];  
  
// working code  
  
size(100, 100);  
background(100, 200, 100);  
  

it would be awesome to have something like that for OF eventually.

jonbro, how does that work exactly?

is it like the equivalent of having a screenshot of a known good result, then rendering from the current version, and doing a per-pixel comparison?

exactly!

Their ref tester also has a certain amount of variance that it allows for so that things like different graphics cards don’t get in the way of rendering issues. I imagine for us there will be some things that will be hard to test for, e.g. shaders, fbos… but for things like the camera, or the meshes, or the points, it might be useful to make sure that there are no major breakdowns.

how is this discussion going?

i have an idea to add unit testing on openframeworks.

https://github.com/diasbruno/ofCoreTest

just clone in myapps.

it uses cpptest. i think it’s simple.

TODO:

* check the license of cpptest.
* make it cross platform. it is xcode only, for now.
* remove dependency of cpptest/config/config.h. it doesn’t do much, but i think we can remove it. or…
* compile cpptest for each platform (lib/osx, lib/win, lib/linux,…).
* integrate with CI. (?)

hope it helps. :slight_smile: