managing multiple openframeworks copies, projects, backwards compatibility etc.

I’m also curious about this. How are people currently handling OF in one repo and their projects in another?

It seems that the following could make sense:

  1. A repo for each project (or all projects)
  2. OF as a submodule of that repo

Having a submodule would make sure that when a certain commit of a project is checked out, the right commit of OF would also be checked out since submodules commit version are committed as part of their parent.

The only issue I’m seeing is the folder structure of OF. If I put my projects in the ‘apps’ folder and create a repo there. I can’t create a submodule for OF because a submodule has to be a child directory of where the git file is.

Any thoughts?

Hi all,

We’re using a setup in which our apps can easily switch OF versions and addons. Apps and their needed libs are in seperate repos.

We have library directory structure consisting of stable, testing and unstable. (Like Debian) Every branch has its own addons we need for our apps. We’re using svn externals to directly link to the github versions or if no repo is available we include the files directly in the repo. We haven’t switched to git yet, nor do I know we will in the short future.

dir structure:

  
libs/stable/openFrameworks   
libs/stable/ofAddons  
libs/testing/openFrameworks  
libs/testing/ofAddons  
libs/unstable/openFrameworks  
libs/unstable/ofAddons  

We put our apps in seperate svn repo’s. Using config.make we link to the version of OF we develop for.

  
OF_ROOT = ../../libs/testing/openFrameworks  
  

Now we’ve done a little trick for the addons. Since addons are found in the addons directory of OF we just tell the addons.make where to find the addon relative to the addons directory:

  
../../ofAddons/ofxSphereCam  
ofxXmlSettings  
../../ofAddons/ofxJSON  
../../ofAddons/ofxTween  
ofxAssimpModelLoader  
../../ofAddons/ofxArgs  
  

This works very nice for linux. For OSX you needed to include the addon to the project manually anyway. For Windows it has always been manual work. However project files for windows are easy to switch by doing a find/replace in a text editor. In this setup it’s important to clean the build after switching versions.

This setup has been very helpful to us. I’m curious of other approaches.

Rg,

Arnaud

@seth I have all my projects in the OF/apps/myApps/ folder of the OF repo, each in their own git repo. this works without issue since that folder is ignored by the OF repo. This works well for me, but I don’t have huge projects, so maybe there are scaling issues involved I don’t know about.

yeah I’m not sure your submodule approach is feasible due to the child folder issue you mention. also, even if it’d work, you’d have a copy of OF in every one of your project repos, so you might as well leave a plain separate OF copy in there, and we’re back to square one w.r.t duplicated OF copies all over the place.

arnaud: your approach is pretty elegant! I like the trick with addons.make, wasn’t aware that this would work (but why not, actually? :P)

@sphaero thanks for sharing. It’s an interesting approach.

@sphaero you’re right about the submodule issue. I forgot that it’d mean there’s a submodule per app repo.

The only reason I’m thinking submodules make some kind of sense is because it ensures that all the code (submodule included) are using the right version. If you have your projects in a separate repo, but that repo isn’t associated to any version of OF, there’s no guarantee that it’ll run if you update OF in the future. But, if it’s associated to a OF commit, then it’d always be able to compile if you checkout that version. Are there alternative ways of letting the ‘app repo’ know what version of OF to associate to?

Maybe it’s hacky, but could do a parent repo that only has submodules, OF and projects? So there’d be a ‘top level’ repo that has a submodule ‘OF’ and submodules for each project repo. You could still commit and push the submodule code when it needs updating, but then to keep if all in sync, you can push the ‘parent’ repo.

Does that sound too weird?

I do keep multiple instances of OF around. One reason is that it the only way that I know of that I can run the same project side by side (say exampleApp in both 0072 and 0073). For instance, if ofFloatImage isn’t drawing the same in 0073 but was fine in 0072 I can open them both up and step through to see what was changed.

For “real” projects I always try to

  • work against the latest release
  • include all of the addons into the project (even core addons)

This way anyone can open and compile the project with the requirements being the OF release version and platform I was using. It is also free of any type of versioning system as I personally use git but still work with SVN.

Concerning memo’s 1b/bleeding edge scenario - yeah - that is the toughest for me (“the bleeding edge demands blood”). These are the projects that I typically expect to be broken when opened later but are worth keeping a special version of OF around for them. I often tell myself that I will port these to an official OF release at some point but can’t rely on that.

Another benefit of copying ofxAddons into my project is that I have more freedom to make tweaks more quickly for that particular project. This is what I find I do the most often - make a particular ofxAddon member public, add a special function

A lot of this is also because I use 3 computers, tower at home, laptop when I travel and an iMac at work. I am also coming from using SVN for years (I often leave work on local branches on other machines as pushing isn’t as common with git).

Great discussion - thinking/writing it out and seeing how others operate is really helpful.

ok, I started making a system to aid you in doing this. basically it should implement Memo’s “magic buttons” - record, checkout, collect projects.
https://github.com/bilderbuchi/ofStateManager

It’s Python. Tested on Linux, should work on MacOS (only used python, git, some bash). It relies on addons.make and config.make files being present (this is where it gets the OF and addons locations from)

Metadata is stored in a metadata.json file in your project directory.
Please use with caution, it’s alpha software and not fully defensively coded yet. So far, I have implemented the record function, rest is to follow, but I thought I’d wait for some feedback first.

Usage: ./ofStateManager.py record -p . add -h for some help texts.

I think when the project manager has matured more, this could be integrated into it, in an advanced mode or so.

i just had a chance to read through this discussion now… after reading memo’s initial message, the response from bilderbuchi was exactly what i was going to say :slight_smile: (“isn’t this what vcs is for?”)

personally i have a few versions of OF, but for two main reasons:

  1. some versions were pre-git, and i haven’t migrated those apps
  2. sometimes i distribute versions of OF with extra addons during workshops

everything else happens in my git copy of OF. when i need to check an old version i do a checkout. sometimes i roll addons back, but usually it’s faster to just fix the little bugs that creep in.

that said, having some clean tools for managing state like the ones you’re creating bilderbuchi – that could be a life changer :slight_smile:

I just finished the basic implementation of ofStateManager: https://github.com/bilderbuchi/ofStateManager
It’s still rough around the edges, I still have to do more error/input checking, and tighten up the code structure and logging, but it can be used/tested. It should be fairly safe to use, too - it’s heavily leveraging git after all, and the script will bail out if anything is wrong. Use -v to get more detailed output, and read the README.md.

Some testing and feedback would be very much appreciated before I invest more effort in polishing/finishing this.
Is the CLI interface/argument set alright/understandable? Does it work for you? Does it work on MacOS (i can’t test this)? Any things I could do better? Does it even make sense to continue, or should we just take it as it is and port it into the project manager?

awesome, will check it out thanks!

This looks awesome! Will try it with our current project…

hey, The description of the 3 components of what it does (record, checkout and archive) sounds absolutely spot on. However I have problems on OSX, this is the error that I get.

Also, in this case i tested with an app in OF/apps/[xxx]/[yyy] (the normal path). But usually my apps are outside of the OF folder, will it also work then?

MSA-MBP15:Amoeba Dance memo$ python /Users/memo/Dropbox/lab/openFrameworks/ofStateManager/ofStateManager.py record
INFO:__main__:Start processing.
Traceback (most recent call last):
File “/Users/memo/Dropbox/lab/openFrameworks/ofStateManager/ofStateManager.py”, line 48, in
if args.func(args,metadata_filename) is not 0:
File “/Users/memo/Dropbox/lab/openFrameworks/ofStateManager/manager_functions.py”, line 77, in record
with open(‘addons.make’,‘r’) as addons_make:
IOError: [Errno 2] No such file or directory: ‘addons.make’

@memo

Didn’t try it yet. But from what I see in the code: I think you need the (originally) linux specific addons.make and config.make files. Config.make contains the path to the OF root.

https://github.com/openframeworks/openFrameworks/blob/develop/apps/devApps/projectGenerator/addons.make
https://github.com/openframeworks/openFrameworks/blob/develop/apps/devApps/projectGenerator/config.make

I guess the script will read the OF_ROOT value in config.make.

ah yes. it’s in the readme, but i probably should make it more clear that your project needs an addons.make file and a config.make file in your project directory.
these are used to find the location of OF and any addons - analysing project files of 3+ IDEs is out of scope for this, and i expect that with the current rewrite of the makefile system those will be more heavily utilised.
however, the addons.make just has a line with the name of every addon used (can even deal with addons outside of OF, like sphaero outlined above), and if you don’t have a config.make, it just has to contain the line
OF_ROOT = …/…/… (or whatever the path to OF from the project is)

I excluded the config.make and addons.make that I used for testing from the repo since those would be specific to my setup and are only used for testing anyway.

in your case, memo, you would supply the location of your project via the -p parameter. this works irrespective of location of your project w.r.t. OF, but the path to OF in config.make in the project has to be correct.

edit: ah, thanks underdoeg, didn’t see your post. you’re right. :slight_smile:

no prob :slight_smile:

Btw I like that you’re parsin the *.make files. It is by far the easiest system to store that kind of information.

yeah, from what I saw in the OF project generator, I didn’t want to parse any IDE files. also, the additional effort for maintaining those files (for people who don’t already have them) is pretty small, and a cross-platform make system is incoming, so I thought this way is most feasible and future-proof.

I just pushed an update for the readme and some sample files.

ah ok. all makes sense. The Mac Project Generator doesn’t generate config.make. So I manually created one, gave it a shot, and the recording worked perfectly (I just checked the JSON file, I haven’t had a chance to try archive or checkout yet).

A few comments so far:

  • I agree parsing 3+ IDE project file formats is out of scope for this script. The responsibilities it addresses are perfect.

  • Mac PG doesn’t generate config.make, but I’m sure that isn’t a problem to add (or even creating the file manually is pretty straightforward).

  • regarding addons.make, this puts a bit of responsibility on the app developer to make sure the addons.make is in sync with the project. This is completely acceptable, but it just needs a big red flag to be waved to warn people :slight_smile: I’m pretty sure quite a lot of people (at least on mac) will have created a project with the PG - or duplicated another project - and then just drag/dropped addons directly into xcode, without using the PG - so addons.make very quickly becomes irrelevant. Also I have so many addons (https://www.dropbox.com/s/iwxkmddgqa5hwga/ofxAddons.png, that the PG can’t list them all, so I can’t use it anyway :slight_smile:

  • related to the above question, I keep all of my own addons in a separate folder (outside of addons). What is the syntax to add external addons into the addons.make file? Is the path relative to the addons folder?

  • I really like the fact that the script tells me which git repos (if any) have uncommitted changes (because I sometimes have about 20 addons in a project!), and warns me if any of the addons don’t have git repos!

yes, I hope when the new makefile system lands we will all be able to use them ootb. in the meantime, manual creation will have to do (but it’s not complicated)

sounds good, I’ll put some bold text in the readme somewhere. :slight_smile:

yes, relative to OF/addons, just like sphaero explains in that post a bit above. similar for projects, they can be wherever, as long as the OF_ROOT path in config.make is correct (can also be relative)

yes, this is kind of a necessity - recording state via git SHA becomes meaningless if there are uncommitted changes (cause then that’s not the state).
currently I think there’s a weak point in that untracked changes (i.e. files not yet committed to the repo) don’t trigger the uncommitted changes abort, I have to tweak the git magic some more.

the warning about non-git repos is important too - state tracking is impossible short of calculating MD5 sums or whatnot, and so easy with git. tracking just doesn’t make too much sense if you don’t track everything. it’s not a show-stopper, though (e.g. for the archive and record command); only checkout gives you a WARNING log message at the end that the state might not be what you’d expect.

Just a heads-up, I have started to implement some unit/functional tests, so in a bit I’ll be able to automatically test and confirm most of the behaviour during further development.

This should raise confidence in the code. It should already be fine now, I just wanted to cover my ass with the alpha-quality warning (not that people use this on their irreplacable project and then blame me :-P)

If anybody finds bugs or wants features, the issue tracker is open. pull requests, too… :wink:
Also, usage experiences are welcome.

Hey bilderbuchi,

How different is the current Makefile setup from the one found in RaspberryPi version of OF or is it the same? If you know?

the new one is much more powerful, and cross-platform. addons get their own config file for adjusting e.g. linking stuff and addon dependencies. I’m not sure how much changes on the user-side, but I think not much.