Building, deployment

Hellas,

I have some concerns about the way openFrameworks is currently built and deployed on the Linux platform, which I would like to discuss. I noticed that prebuilt versions of all of the dependencies are included in the source tree. This general practice of deploying software is IMHO generally very error-prone and not reasonable for different reasons which I would like to elaborate.

I see the idea is for the project to be ‘self-contained’ in order to simplify deployment at first glance. For the Windows and Mac OS platforms, this might be a reasonably pragmatic way of deploying. This practice, however, fails to acknowledge the general heterogenity of Linux systems. I see that you support specific Ubuntu version by supplying binaries for it, however I work, among others, on Arch and Gentoo, which generally use more up-to-date and more rapidly changing versions of programs and libraries. I see (and have experienced with many projects) the following problems with the self-contained deployment approach:

  • You can not account for the vast amount of different standard library / compiler versions out there.
  • People might be interested in using, apart from the openFrameworks API, one of the libraries on which it depends directly, and have specific requirements on it regarding versions / features compiled in.
  • Libraries can have certain runtime behavior, such as using sockets or other IPC mechanisms, depending on global system configuration, writing to temporary files, etc. which creates conflicts if different versions of the same library are used in parallel on the same system.
  • For some libraries, continuous and quick updating via distribution means is critical. Supplying a custom-built openssl library which will not get updated in case of a critical vulnerability is just insane in my opinion.
  • Certain libraries will depend on having been built against other libraries on the system, e.g. if you supply glew and glu, you should probably also supply libGL, which you can’t since it will be different GPU vendor’s implementations.
  • Having 1.1Gb of binary files in a Git repository which needs to be cloned over and over is a waste of people’s time and bandwidth, and just plain nonsense. One thing that should be done in any case is to remove those binaries from version control and provide a separate zip-file or whatever with the third-party libs (as you probably will still want to do for Mac/Win).

Of course having to account for different versions of a library on which you depend, with possibly incompatible APIs, introduces some additional complexity and maintenance effort, no question about that. However it will allow users of all the different target systems to use the library without running into any of the abovementioned problems. I’m also a firm believer in that it will increase overall code quality of a project, since you have to think about, and account for, those differences and structure your code more clearly, abstracting those variations in a meaningful way. Thus, in my opinion, getting rid of all the third-party libs, at least on Linux, would be the Right Thing To Do.

An important aspect is also that, in order to lower the effort (of non-Ubuntu users) to get in touch with the software and increase its adoption, the ideal scenario is of course to have it in the main software repositories of the big Linux distributions. However as things stand right now, no distribution will ever think about including it if it will not integrate with the rest of the system. Something like supplying an own libssl version etc. will keep every distributor from including that software, and in my opinion rightfully so! I was actually (probably naively) expecting, since oF is a rather mature lib by now, that I will find it in the repos of my favorite distros. But as things stand, that is not going to happen.

I would be interested in your opinions on those matters! Since I seem to be the only one right now with such requirements / concerns, I would be gladly willing to do the work, at least porting to those platforms on which I can test everything, and adapt the build system accordingly. I might need some help getting a grip on the specific build mechanisms you use (it seems you are not using standard tools such as CMake/autotools/scons, which is another issue…), so a little help of experienced oF devs will be useful, but I’ll manage to work the stuff out either way.

Best regards

Hey there,

Unlike OSX, windows, etc at this point there are very few compiled linux dependencies included for linux. Most of them are included as source or used from the package manager on the target system (i.e. we don’t include openssl for linux it links against the system libs). What version of openFrameworks are you referring to – or what specific libs?

Regarding the obnoxiously huge git repo, it’s a known issue and we’re working toward a fix.

See https://github.com/openframeworks/openFrameworks/issues/3833 and others.

Also just did a quick survey on the linux nightly (~100MB). Binaries are included for libfmodex (dynamic libs due to licensing afaik), libglfw (due to the latest bug-free versions not being a part of package management systems), libkiss (for convenience, also included as source), poco (the version of poco in package management systems is woefully out of date), tess2 (not sure why this one is included as a binary – @arturo?), utf8cpp (included as source). Everything else is linked to system libs.

For larger packages like Poco / glfw (and smaller ones for that matter), custom static libs can be easily generated from source using the apothecary management system (included with the nightlies and in the main repo).

The goal in the near future is to remove all binaries from the git repo and generate all required static libs with apothecary. Developers can do this themselves if they work off the master branch or they will be automatically generated and included in the platform downloads for convenience.

Ok, I should have looked more closely into this maybe, sometimes I’m a bit over-eager to get going on things :wink: I see you are aware of the problems, and I would gladly offer my time for improving stuff in that regard, for starters!

Still, especially for the openssl example, where the version is critical from a security standpoint, it does not matter if binary or source, and the system integration / several parallel versions remarks still apply of course. There’s /etc/ssl/openssl.cnf, which belongs to 1.0.2.d on my system. Minor difference here, but you get my point.

I think providing source forms is already an improvement, but actually you want to stop supplying specific versions of third-party libs altogether. Integrating the library versions etc. is best left to the distributors in order to get a consistent configuration, and the software should be written such that it builds/runs against all versions of the libraries on which it depends (on the relevant target systems, that is).

Agreed in principle, but often at the bleeding edge, for example with GLFW, some of our source changes have not made it upstream … and the distributors are often way behind (e.g. debian variants).

Anyway – welcome to the community and looking forward to seeing what you make! In terms of getting involved, jump on the of-dev list, introduce yourself, check out the https://github.com/openframeworks/openFrameworks/blob/master/CONTRIBUTING.md and https://github.com/openframeworks/openFrameworks/blob/master/CODE_OF_CONDUCT.md, then check out our issues (we’ve got plenty) and we’d gladly welcome your contributions – particularly on testing / deployment across different linux distros as we move toward the 0.9.0 release :smile:

If you use modified source versions then there’s not much of an alternative, agreed! Well yeah Debian is a bit special there (don’t want to talk about RHEL/SL/CentOS ;))

Thank you, excited to become part of such a great project!