managing multiple openframeworks copies, projects, backwards compatibility etc.

[memo]

UPDATE
This turned out to be quite a long post, so here’s a little summary:

Imagine you develop an (openframeworks) application which uses a lot of non-core addons: ofxMidi, ofxKinect, ofxTimeline, ofxMSAInteractiveObject, ofxSimpleGuiToo, ofxDelanauy, ofxOpenCL, ofxBulletPhysics, ofxSplineInterpolator, ofxFx, ofxArtnet, ofxPD, ofxARToolKit etc.

How do you make sure you can easily re-compile and run this application in 3 years time? Do you create a snapshot copy of openFrameworks and all relevant addons for that particular project?

What if you are have tens, if not hundreds of applications (some big, some small) accumulating over the years.

The general consensus reply is ‘this is what version controls systems are for’. Yes of course, but most VCS are too low-level to easily handle this particular style of dependency on multiple repo’s. How do you realistically manage tens - if not hundreds - of applications, each using a large number of different addons. Currently it seems by the replies that a lot of people do have multiple copies of openframeworks lying around, and duplicate addons per application, so clearly the VCS isn’t being used efficiently and higher level tools are needed.

P.S. This isn’t a dilemma unique to us, a lot of other developers have run into similar workflow issues and each have their own solutions.
E.g. Google’s solution for android development: http://source.android.com/source/version-control.html
and other tools to manage multiple repo’s and/or submodules simultaneously http://gitslave.sourceforge.net/ , https://github.com/patmaddox/giternal , https://github.com/apenwarr/git-subtree/ , https://github.com/evilchelu/braid/wiki
in fact https://www.google.co.uk/search?q=git+manage+multiple+repositories
none of which seem ideal for our purposes.

Original Post:
Hey all, I just tweeted in frustration about a dilemma I’ve been having for years. It’s getting a lot better, but still I don’t feel it’s fully resolved (judging by the amount of instant replies I’ve gotten back on the webz i think most people agree this needs to be discussed!).

The problem is that of how to manage your source code for apps in relation to different instances of openframeworks and addons. I.e. putting your source code inside one central openframeworks folder, vs outside, possibly with it’s own bespoke copy of openframeworks, possibly not.

I’ve raised this discussed a few times on the devlist, and now that we have the project generator things are MUCH MUCH better. But I still think there is room to improve (or at least I haven’t nailed the solution I’m after).

This post is more of an improvised brainfart as opposed to a specific question. We’ll see where it goes. I might just rant on about stuff. I might ask some constructive questions, I might even find some answers on this journey.

First let me start with the options I see and use:

1. old skool #1. All of your apps go inside one openframeworks folder. Inside of/apps you create a folder for each project (e.g. of/apps/myproject) or category (e.g. of/apps/demos, of/apps/experiments etc), and inside those are subfolders for each specific application.

1a. The most obvious solution is to have one openframeworks folder per OF version. E.g. OF005, OF00573 (remember that freaky one!?), OF006, OF007, OF0071, OF0072 etc. This maintains nice backwards compatibility, you can go back and compile old projects you wrote for OF005 if need be.

1b. the problem with the above, is it doesn’t factor in the possibility of working on the bleeding edge develop HEAD. As new updates come to OF (e.g. via github), you’re not benefitting from bug fixes. and often, you need to work on the develop head for latest features. In fact sometimes you need to work on branches ahead of develop HEAD! i.e. you modify the core, submit a pull request, and then it makes it into the core. if you are working off the latest on git, how do you manage that? So in addition to having a folder for each version of openframeworks, I also have a folder for openframeworks git. And that’s when it gets tricky.

2. Traditional way #2. The last scenario above starts to get tricky if you develop your app not for OF0072, or OF0073, but for something somewhere in between. So the safest bet, is for each project, create a whole copy of openframeworks elsewhere, and put your app there. This (almost) guarantees that you’ll be able compile and run in years to come (assuming you’re not depending on OS libraries and features that deprecate). But of course this takes up loads of space. Also if you are using custom addons, you’re likely to end up with hundreds of versions of the same addons. Often you might even make fixes and tweaks to a particular version of an addon for one project, but then other fixes and tweaks to another version of the same addon for another project. It’s relatively trivial to merge them all and update a master central repo, but of course in the real world that usually doesn’t get done straight away, and will eventually be forgotten, lost in time, like tears in rain.

3. New independent way. Thanks to the awesome Project generator, you can place your project anywhere, outside of any openFrameworks folder. This means that it’s not linked to any particular version (or instance) of OF. This has the most potential, but it’s not fully there yet. You can’t easily change instances i.e. make a project point to another instance of openframeworks (actually the PG has an ‘update project’ option, I don’t know if this is supposed to do that, but it doesn’t. more on that below).

In my *ideal* world what I’d like to be able to do is possibly this:

1. put my app code inside it’s own folder in a project folder. e.g. /myproject/src

2. point that to use my central openframeworks git repository so I can benefit from all current developments e.g. /dev/openframeworks/git

3. at the end of the project, automate the transfer of the current state of openframeworks and all relevant addons into the myproject folder. This seems like a bit of a diva thing to ask for. Why not copy it across in the finder. Well sometimes I have upto 20 addons in my projects. And I right now I have exactly 72 addons in my central openframeworks folder. To go through them all one by one comparing what I have in xcode vs finder seems very inefficient. It is also prone to human error. As a programmer, this kind of inefficiency, which is prone to human error aggravates my OCD Also since I’m developing in git version, I have all libs for all platforms, which pushes the size of OF to over 1GB i think. I really don’t want 1GB per project. So stripping out all those unnecessary libs would be great too.
   This ensures that I have a backup of the state which can make my app compile in years to come.

4. I would like a very simple way to change to which instance of openframeworks, my project points to. I think perhaps the ‘update’ option in the project generator is supposed to do this. But I’ve not managed to get it to work. (e.g. my application is inside the openframeworks folder, I move it outside, and ‘update’ it via the PG. When I open the project, it can’t find ofMain.h, project settings aren’t correct.

One big problem with this is addons. Any addons you might have in one openframeworks instance might be different to another instance. I’m thinking keeping addons inside each project folder is a safer option than keeping them with the openframeworks instance.

Also another problem with the above is sharing projects (i.e. collaborative development on github). If your project folder is outside of openframeworks folder, if you add it to the git repo, the same project/workspace file won’t work for anyone else - unless they have the exact same folder structure as you for where they keep their openframeworks. So you either have to keep your project inside openframeworks folder, or don’t add the project file to the repo. If you don’t add the project file to the repo, everytime someone adds a new file to the project (which is quite common during development other people get errors. So you can say, ‘if you see files which are new that aren’t in the project, just add them to the project’. But of course that doesn’t always work either. Sometimes you might create a cpp, develop it a bit, then decide you want to do it another way, with another set of classes. So you remove that from the project (i.e. exclude it from the target, or remove it altogether - but without deleting the file). then someone else sees the file and adds it to their project. And it doesn’t compile anymore. Of course good communication between developers solves all of these problems, and they are very manageable. But still, it is a system prone to human error. And again, any system which is prone to human error, which can be avoided through automation of some sort, aggravates my OCD

I guess this also really brings in the importance of semantic versioning (http://semver.org/ as also discussed many times). You could argue that as long as the major version doesn’t change, backwards compatibility doesn’t break, so you can be sure that you can keep upgrading openframeworks and all of your app code will work. All you need to do is just keep a single copy of each major version of openframeworks, and all of your old apps will compile and run. Of course that’s just theory. In reality, bugs will be introduced, so the app that you wrote for openframeworks 0.7.3 today, might not work exactly as it did in a years time on 0.7.7, because someone didn’t realize that changing one little implementation detail in one function would have an impact on the way another class functions. This is just a normal part of development.

I think I definitely asked some questions, I possibly found a glimmer of hope of an answer, but definitely not something I can grab onto. Definitely not something which I can call ‘rules - or strict guidelines - of how to structure the file system and folder structure of any new openframeworks application, whether it be a demo, test, or project’. Maybe the solution is right there under my nose but I can’t see it. Maybe we need to make some small changes to something (no idea what), and then we’ll all be much happier. Just wanted to start a discusion about this.

any thoughts, suggestions, comments, feedback much welcome!

/massive rant over

[bilderbuchi]

hi memo!

i won’t answer all of the points you touched, but I can at least contribute to some:
your options 1 and 2 (and a bit of your ideal world point 4):
I don’t get why you have to keep multiple separate instances of the OF folder, while also having an OF git repo? (i didn’t think people would actually consider doing this, honestly - that’s exactly what version control is for after all)

using just the OF git repo (with your projects inside at the appropriate places, as per your #1) would really reduce space and complexity, and solve some of your path problems (no instance pointers needed, no collaboration path problems, no multiple copies of OF littering your HDD) - use only the git repo, and either remember or record in a text file in your project’s directory which version/commit/tag of OF (and any relevant addons) you were using for making that project, and when at any point in the future you need to compile/work on that project again, you just check out that state of OF, compile the OF lib, and you’re good to go.
this is why we have tags, you can get any version just by saying git checkout 0071. or you make your own tags in the OF repo for marking the spot when your project was finished (tags don’t get pushed by default, so no fear of polluting origin/upstream). this can be automated of course, but obviously isn’t yet.
at the same time, you can work on your newest project on bleeding edge - just a git checkout develop away.

the same goes for any addons you use - if you cloned them from github, you have a repo already, so just record the commit they were at when you stopped working on your project, and you can easily go back to that exact state at any point later.

thoughts?

(edited in some clarifications)

[memo]

Hey Christoph, thanks for the input.

What you say is very wise and excites me. However for it to be realistic (for me) there’s a quite few things which I think need to be improved:

1. Addons. Yes some addons are from git, others aren’t. Even if they aren’t, of course I can create repos for them. But then saving the commit for every single addon, for every single application I create, just doesn’t seem feasible. How and where do I save this info?

2. Archive (And sharing). I forgot to mention this in my post above. I don’t want to keep every project i’ve ever written on my harddisk. Everything I did with OF005 is on an external harddrive (that folder was about 10GB, back then OF wasn’t a lib, so everything was always compiled for every project). But even most of my OF006 projects I want to archive onto external. So i need a simple way to manage this and save self-contained backups on an external. (self-contained: including the state of openframeworks and only addons I need for that project).

So in short, it sounds like git could be the answer to all of this:

1. one single openframeworks folder /openframeworks/git
2. all addons in there too /openframeworks/git/addons
3. all applications living outside, in their relevant project folders /myprojects/src

At the end of each project (or before updating openframeworks core or any addons), I somehow tag the state of the core and all addons. The question is:

• do I somehow store this information with the application? (i.e. commit number for each component: core+addons, saved in a text file, or a series of ‘git-state’ files - if such a thing exists). OR
• apply a tag to each component (core+addons) with the name of the application.

The former seems WAAAY more logical to me. I’ve written hundreds of apps for openframeworks (including all little demos, experiments, etc). I don’t want hundreds of tags on very single addon with every single application. Also, 3 months after thinking i’ve finished, i might revisit an application, develop it some more. So then do I have to re-tag the core and all of the addons? Ending with multiple tags per application?

So it seems the former is the way to go if possible. A system (i.e. a magic button) for saving the commit name for every single component (core+addons) along with the project. Then another system (i.e. another magic button) for re-applying that to all components. Then another system (i.e. a magic button), to extract and collect all of the relevant components and libraries (not all) to a separate folder - similar to ‘collect files’ in aftereffects or indesign - to consolidate only the files you need, so if you want to back up the project to an external hard drive, you can. Could work. In fact I love it. What do you think? How would that system work?

[bilderbuchi]

thank you! apparently I have more work to do as git leader if people let this get to themselves over years.
some more thoughts:

1. you are correct that this doesn’t scale very well. also, it has to be done manually for now. I’m confident that this can (and if the need arises, will) be automated in the future. it’s slow going, though - we have an addon template now (https://github.com/benben/ofxAddonTemplate) which we have to entice addon devs to use, then should come versioning schemes for addons (first step: switch OF to semver), and later dependency checks etc.

in any case, it would be a good idea to have some version and requirements info in a project README, so that may be a good place with not too much maintenance overhead (since you should have one anyway, especially if you expect to open the project years later).
alternatively (maybe better) I’d argue for some structured dependencies file (prolly xml) containing required addons and the commit point/tag they were used at. there’s nothing official in the works like that yet (only some discussions on the issue tracker), but xml or e.g. json, or even csv, could conceivably be automatically parsed and/or written into by a script. e.g., that script could parse your project’s addons.make (when the cross-platform make system which is currently WIP has landed), go to the addon’s location, and get the current commit via a git command, something like that.

ok I just read the rest of your post, what I said above applies to much of it.

I think the magic button system is something which could live in the project generator once it’s matured enough. for now, scripts should be feasible for that.
as a real-world example, at work i do scientific simulations, in matlab, and I got everything in git. it’s important to be able to precisely recall the state of the (ever-evolving :P) code at which results were produced. so my code calls git to confirm that there are no uncommitted changes (meaning the current commit precisely represents the current state), otherwise it aborts. then it puts the current commit sha (in short 6-char form) in a variable, which is added in the report generated, in any summary tables, and the result files which are stored on disk. thus, whenever I want to re-run simulations, I just have to check out the relevant git commit, and hit run.

that’s only 4 lines of code (without comments). matlab syntax, but I guess you get the idea

  
% If solution is to be saved, check if all code has been committed properly  
if (~strcmp(ecc.purpose,'donotsave') && system('git diff --quiet HEAD'))  
    error('Uncommitted changes in the repository, aborting! Commit all changes before repeating the calculation.');  
end  
% git diff --quiet HEAD exits with 1 if there were differences and 0 means no differences.  
% Untracked files are not considered.  
  
%   git commit identification  
%   relevant command: git rev-parse HEAD  
[~, commit]=system('git rev-parse HEAD');  
commit=commit(1:6);  

so yeah, that can be automated.

[cerupcat]

I’m also curious about this. How are people currently handling OF in one repo and their projects in another?

It seems that the following could make sense:

1. A repo for each project (or all projects)
2. OF as a submodule of that repo

Having a submodule would make sure that when a certain commit of a project is checked out, the right commit of OF would also be checked out since submodules commit version are committed as part of their parent.

The only issue I’m seeing is the folder structure of OF. If I put my projects in the ‘apps’ folder and create a repo there. I can’t create a submodule for OF because a submodule has to be a child directory of where the git file is.

Any thoughts?

[sphaero]

Hi all,

We’re using a setup in which our apps can easily switch OF versions and addons. Apps and their needed libs are in seperate repos.

We have library directory structure consisting of stable, testing and unstable. (Like Debian) Every branch has its own addons we need for our apps. We’re using svn externals to directly link to the github versions or if no repo is available we include the files directly in the repo. We haven’t switched to git yet, nor do I know we will in the short future.

dir structure:

  
libs/stable/openFrameworks   
libs/stable/ofAddons  
libs/testing/openFrameworks  
libs/testing/ofAddons  
libs/unstable/openFrameworks  
libs/unstable/ofAddons  

We put our apps in seperate svn repo’s. Using config.make we link to the version of OF we develop for.

  
OF_ROOT = ../../libs/testing/openFrameworks  
  

Now we’ve done a little trick for the addons. Since addons are found in the addons directory of OF we just tell the addons.make where to find the addon relative to the addons directory:

  
../../ofAddons/ofxSphereCam  
ofxXmlSettings  
../../ofAddons/ofxJSON  
../../ofAddons/ofxTween  
ofxAssimpModelLoader  
../../ofAddons/ofxArgs  
  

This works very nice for linux. For OSX you needed to include the addon to the project manually anyway. For Windows it has always been manual work. However project files for windows are easy to switch by doing a find/replace in a text editor. In this setup it’s important to clean the build after switching versions.

This setup has been very helpful to us. I’m curious of other approaches.

Rg,

Arnaud

[bilderbuchi]

@seth I have all my projects in the OF/apps/myApps/ folder of the OF repo, each in their own git repo. this works without issue since that folder is ignored by the OF repo. This works well for me, but I don’t have huge projects, so maybe there are scaling issues involved I don’t know about.

yeah I’m not sure your submodule approach is feasible due to the child folder issue you mention. also, even if it’d work, you’d have a copy of OF in every one of your project repos, so you might as well leave a plain separate OF copy in there, and we’re back to square one w.r.t duplicated OF copies all over the place.

arnaud: your approach is pretty elegant! I like the trick with addons.make, wasn’t aware that this would work (but why not, actually? :P)

[cerupcat]

@sphaero thanks for sharing. It’s an interesting approach.

@sphaero you’re right about the submodule issue. I forgot that it’d mean there’s a submodule per app repo.

The only reason I’m thinking submodules make some kind of sense is because it ensures that all the code (submodule included) are using the right version. If you have your projects in a separate repo, but that repo isn’t associated to any version of OF, there’s no guarantee that it’ll run if you update OF in the future. But, if it’s associated to a OF commit, then it’d always be able to compile if you checkout that version. Are there alternative ways of letting the ‘app repo’ know what version of OF to associate to?

Maybe it’s hacky, but could do a parent repo that only has submodules, OF and projects? So there’d be a ‘top level’ repo that has a submodule ‘OF’ and submodules for each project repo. You could still commit and push the submodule code when it needs updating, but then to keep if all in sync, you can push the ‘parent’ repo.

Does that sound too weird?

[jvcleave]

I do keep multiple instances of OF around. One reason is that it the only way that I know of that I can run the same project side by side (say exampleApp in both 0072 and 0073). For instance, if ofFloatImage isn’t drawing the same in 0073 but was fine in 0072 I can open them both up and step through to see what was changed.

For “real” projects I always try to

• work against the latest release
• include all of the addons into the project (even core addons)

This way anyone can open and compile the project with the requirements being the OF release version and platform I was using. It is also free of any type of versioning system as I personally use git but still work with SVN.

Concerning memo’s 1b/bleeding edge scenario - yeah - that is the toughest for me (“the bleeding edge demands blood”). These are the projects that I typically expect to be broken when opened later but are worth keeping a special version of OF around for them. I often tell myself that I will port these to an official OF release at some point but can’t rely on that.

Another benefit of copying ofxAddons into my project is that I have more freedom to make tweaks more quickly for that particular project. This is what I find I do the most often - make a particular ofxAddon member public, add a special function

A lot of this is also because I use 3 computers, tower at home, laptop when I travel and an iMac at work. I am also coming from using SVN for years (I often leave work on local branches on other machines as pushing isn’t as common with git).

Great discussion - thinking/writing it out and seeing how others operate is really helpful.

[bilderbuchi]

ok, I started making a system to aid you in doing this. basically it should implement Memo’s “magic buttons” - record, checkout, collect projects.
https://github.com/bilderbuchi/ofStateManager

It’s Python. Tested on Linux, should work on MacOS (only used python, git, some bash). It relies on addons.make and config.make files being present (this is where it gets the OF and addons locations from)

Metadata is stored in a metadata.json file in your project directory.
Please use with caution, it’s alpha software and not fully defensively coded yet. So far, I have implemented the record function, rest is to follow, but I thought I’d wait for some feedback first.

Usage: ./ofStateManager.py record -p . add -h for some help texts.

I think when the project manager has matured more, this could be integrated into it, in an advanced mode or so.

[kylemcdonald]

i just had a chance to read through this discussion now… after reading memo’s initial message, the response from bilderbuchi was exactly what i was going to say (“isn’t this what vcs is for?”)

personally i have a few versions of OF, but for two main reasons:

1. some versions were pre-git, and i haven’t migrated those apps
2. sometimes i distribute versions of OF with extra addons during workshops

everything else happens in my git copy of OF. when i need to check an old version i do a checkout. sometimes i roll addons back, but usually it’s faster to just fix the little bugs that creep in.

that said, having some clean tools for managing state like the ones you’re creating bilderbuchi – that could be a life changer

[bilderbuchi]

I just finished the basic implementation of ofStateManager: https://github.com/bilderbuchi/ofStateManager
It’s still rough around the edges, I still have to do more error/input checking, and tighten up the code structure and logging, but it can be used/tested. It should be fairly safe to use, too - it’s heavily leveraging git after all, and the script will bail out if anything is wrong. Use -v to get more detailed output, and read the README.md.

Some testing and feedback would be very much appreciated before I invest more effort in polishing/finishing this.
Is the CLI interface/argument set alright/understandable? Does it work for you? Does it work on MacOS (i can’t test this)? Any things I could do better? Does it even make sense to continue, or should we just take it as it is and port it into the project manager?

[memo]

awesome, will check it out thanks!

[underdoeg]

This looks awesome! Will try it with our current project…

[memo]

hey, The description of the 3 components of what it does (record, checkout and archive) sounds absolutely spot on. However I have problems on OSX, this is the error that I get.

Also, in this case i tested with an app in OF/apps/[xxx]/[yyy] (the normal path). But usually my apps are outside of the OF folder, will it also work then?

MSA-MBP15:Amoeba Dance memo$ python /Users/memo/Dropbox/lab/openFrameworks/ofStateManager/ofStateManager.py record
INFO:__main__:Start processing.
Traceback (most recent call last):
File “/Users/memo/Dropbox/lab/openFrameworks/ofStateManager/ofStateManager.py”, line 48, in
if args.func(args,metadata_filename) is not 0:
File “/Users/memo/Dropbox/lab/openFrameworks/ofStateManager/manager_functions.py”, line 77, in record
with open(‘addons.make’,‘r’) as addons_make:
IOError: [Errno 2] No such file or directory: ‘addons.make’

[underdoeg]

@memo

Didn’t try it yet. But from what I see in the code: I think you need the (originally) linux specific addons.make and config.make files. Config.make contains the path to the OF root.

https://github.com/openframeworks/openFrameworks/blob/develop/apps/devApps/projectGenerator/addons.make
https://github.com/openframeworks/openFrameworks/blob/develop/apps/devApps/projectGenerator/config.make

I guess the script will read the OF_ROOT value in config.make.

[bilderbuchi]

ah yes. it’s in the readme, but i probably should make it more clear that your project needs an addons.make file and a config.make file in your project directory.
these are used to find the location of OF and any addons - analysing project files of 3+ IDEs is out of scope for this, and i expect that with the current rewrite of the makefile system those will be more heavily utilised.
however, the addons.make just has a line with the name of every addon used (can even deal with addons outside of OF, like sphaero outlined above), and if you don’t have a config.make, it just has to contain the line
OF_ROOT = …/…/… (or whatever the path to OF from the project is)

I excluded the config.make and addons.make that I used for testing from the repo since those would be specific to my setup and are only used for testing anyway.

in your case, memo, you would supply the location of your project via the -p parameter. this works irrespective of location of your project w.r.t. OF, but the path to OF in config.make in the project has to be correct.

edit: ah, thanks underdoeg, didn’t see your post. you’re right.

[underdoeg]

no prob

Btw I like that you’re parsin the *.make files. It is by far the easiest system to store that kind of information.

[bilderbuchi]

yeah, from what I saw in the OF project generator, I didn’t want to parse any IDE files. also, the additional effort for maintaining those files (for people who don’t already have them) is pretty small, and a cross-platform make system is incoming, so I thought this way is most feasible and future-proof.

I just pushed an update for the readme and some sample files.

[memo]

ah ok. all makes sense. The Mac Project Generator doesn’t generate config.make. So I manually created one, gave it a shot, and the recording worked perfectly (I just checked the JSON file, I haven’t had a chance to try archive or checkout yet).

A few comments so far:

• I agree parsing 3+ IDE project file formats is out of scope for this script. The responsibilities it addresses are perfect.

• Mac PG doesn’t generate config.make, but I’m sure that isn’t a problem to add (or even creating the file manually is pretty straightforward).

• regarding addons.make, this puts a bit of responsibility on the app developer to make sure the addons.make is in sync with the project. This is completely acceptable, but it just needs a big red flag to be waved to warn people I’m pretty sure quite a lot of people (at least on mac) will have created a project with the PG - or duplicated another project - and then just drag/dropped addons directly into xcode, without using the PG - so addons.make very quickly becomes irrelevant. Also I have so many addons (https://www.dropbox.com/s/iwxkmddgqa5hwga/ofxAddons.png, that the PG can’t list them all, so I can’t use it anyway

• related to the above question, I keep all of my own addons in a separate folder (outside of addons). What is the syntax to add external addons into the addons.make file? Is the path relative to the addons folder?

• I really like the fact that the script tells me which git repos (if any) have uncommitted changes (because I sometimes have about 20 addons in a project!), and warns me if any of the addons don’t have git repos!