Using CEF with OF

smallfly · 2014-12-31 16:41:02 UTC

Hi,

I’m want to try to completely separate the backend from the frontend for some future projects. I would like to be able to use html, css and javascript for the front end (doing off screen rendering). The two main options to do this seems to be Awesomium and CEF. I quickly tried ofxAwesomium. It seems to be working quite nicely. I’m not sure it supports hidpi though, and it’s not open source. CEF looks like being a great option, but I’m having trouble getting it to work on OS X (even using ofxCEFClient). I did not manage to compile the example from the CEF source either. Then I found this repo, shared by Zach L. on the of-dev list. It works great: hidpi, webgl (I won’t need that for building UI though), etc…

CEF seems a bit tricky to ‘setup’ but I think it’s the best option, so before going further I wanted to know if anybody had some code/project of CEF working with OF to share (OS X and Windows) .

Thanks and happy new year!

bakercp · 2014-12-31 16:59:15 UTC

I’m currently trying to work out the same issue with CEF to use it (hopefully) with ofSketch.

I don’t have any good working code yet (other than modifications of these builds http://cefbuilds.com/), but it’s on my TODO list. Let’s collaborate!

zach · 2014-12-31 17:03:53 UTC

I’ve got some stuff around for the project generator, but also had mixed luck using CEF how awesomium is used, ie, rendered into a texture (and I also didn’t have luck with ofxCEFClient). I have had good luck using webkit via @admsyn’s fitc slides which sort of sits on top of OF in the window and I think there’s some different approaches to try there as well. both berkelium and awesomium seems to be dormant projects so getting CEF working similar to awesomium would be great. I’ve used awesomium in some commercial work and it was a huge help to be able to quickly get interfaces in via html / css and develop and test front ends in the browser.

one quick note is that there is a pretty good CEF syphon client which broadcasts webpages to syphon – I know that doesn’t really solve things but it could be useful code to look at?

bakercp · 2014-12-31 17:21:34 UTC

There are so many ways to use CEF – it’s pretty cool.

My main use is simply to create a consistent browser / gui experience for ofSketch. Right now we run it in the browser and things like key-board shortcuts are not consistent across browsers.

Other than creating a simple web browser, I basically just need a connection to 127.0.0:8888 (or whatever port) because the gui is all ofxJSONRPC based. We’d like to allow a user to still use their browser for remote coding (e.g. ofSketch running on a raspberry Pi).

Anyway, so I’m not going to be using CEF to do any sophisticated javscript callbacks, etc. (famous last words).

zach · 2014-12-31 17:22:38 UTC

also one of the things that I never totally understood is the relationship of CEF apps, in their examples they seem to have a “helper” application that gets embedded in a larger application.

I just took and I seem to have a project that’s broken but maybe (?) a useful starting point for figuring this out? It’s based on code from somewhere (maybe the ogre / CEF example?) that trys to render to a texture. It’s super wacky, I’ll get a repo up soon.

https://dl.dropboxusercontent.com/u/92337283/misc/cefWork.zip

it seems to have three main issues that it’s crashing out in:

LaunchProcess: failed to execvp:
/Users/zachlieberman/Documents/projects/OF/of_v0.8.4_osx_release/apps/myApps/cefWork/bin/cefWorkDebug.app/Contents/Frameworks/cefWorkDebug Helper.app/Contents/MacOS/cefWorkDebug Helper

1: it can’t find the helper app so it’s unhappy

ERROR: 0:3: Invalid qualifiers 'in' in global variable context
ERROR: 0:5: Invalid qualifiers 'out' in global variable context
ERROR: 0:10: Use of undeclared identifier 'outputColor'
ERROR: 0:10: Use of undeclared identifier 'Texcoord'
ERROR: 0:11: Use of undeclared identifier 'outputColor'

SHADER COMPILE ERROR

2: is has some shaders its trying to compile in RenderHandler that need some love

GetViewRect 
GetViewRect 
LaunchProcess: failed to execvp:
/Users/zachlieberman/Documents/projects/OF/of_v0.8.4_osx_release/apps/myApps/cefWork/bin/cefWorkDebug.app/Contents/Frameworks/cefWorkDebug Helper.app/Contents/MacOS/cefWorkDebug Helper

ok again, trying to launch this helper –

GetViewRect 
2014-12-31 12:06:12.145 cefWorkDebug[8057:303] _createMenuRef called with existing principal MenuRef already associated with menu
2014-12-31 12:06:12.199 cefWorkDebug[8057:303] (
	0   CoreFoundation                      0x91c39471 __raiseError + 193
	1   libobjc.A.dylib                     0x92469091 objc_exception_throw + 162
	2   CoreFoundation                      0x91c3938b +[NSException raise:format:] + 139
	3   AppKit                              0x92f1fc9c -[NSCarbonMenuImpl _createMenuRef] + 69
	4   AppKit                              0x92f1f49d -[NSCarbonMenuImpl _instantiateCarbonMenu] + 161
	5   AppKit                              0x92f1f3e8 -[NSCarbonMenuImpl setupCarbonMenuBar] + 40
	6   AppKit                              0x92f1d8af -[NSApplication finishLaunching] + 1357
	7   AppKit                              0x92f1cf49 -[NSApplication run] + 196
	8   Chromium Embedded Framework         0x00e380b0 _ZN4base24MessagePumpNSApplication5DoRunEPNS_11MessagePump8DelegateE + 368
	9   Chromium Embedded Framework         0x00e36db8 _ZN4base24MessagePumpCFRunLoopBase3RunEPNS_11MessagePump8DelegateE + 104
	10  Chromium Embedded Framework         0x00f34d0c _ZN4base11MessageLoop10RunHandlerEv + 300
	11  Chromium Embedded Framework         0x00f9cf93 _ZN4base7RunLoop3RunEv + 83
	12  Chromium Embedded Framework         0x00f9d25f _ZN4base7RunLoop12RunUntilIdleEv + 47
	13  Chromium Embedded Framework         0x00b8ea87 _ZN21CefBrowserMessageLoop22DoMessageLoopIterationEv + 55
	14  Chromium Embedded Framework         0x00bb6f79 _Z20CefDoMessageLoopWorkv + 505
	15  Chromium Embedded Framework         0x00a62cdf cef_do_message_loop_work + 31
	16  cefWorkDebug                        0x003a28ef _Z20CefDoMessageLoopWorkv + 31
	17  cefWorkDebug                        0x00006ec7 _ZN6Cefgui4drawEv + 23
	18  cefWorkDebug                        0x0000611a _ZN5ofApp4drawEv + 26
	19  cefWorkDebug                        0x0000584d _ZN9ofBaseApp4drawER11ofEventArgs + 29
	20  cefWorkDebug                        0x0042a0ff _ZN4Poco16PriorityDelegateI9ofBaseApp11ofEventArgsLb0EE6notifyEPKvRS2_ + 61
	21  cefWorkDebug                        0x0042f018 _ZN4Poco13AbstractEventI11ofEventArgsNS_16PriorityStrategyIS1_NS_24AbstractPriorityDelegateIS1_EEEES4_NS_9FastMutexEE6notifyEPKvRS1_ + 118
	22  cefWorkDebug                        0x0042d31f _Z13ofNotifyEventI7ofEventI11ofEventArgsES1_EvRT_RT0_ + 32
	23  cefWorkDebug                        0x0042cc2e _Z12ofNotifyDrawv + 51
	24  cefWorkDebug                        0x0046e38c _ZN15ofAppGLFWWindow7displayEv + 202
	25  cefWorkDebug                        0x0046e296 _ZN15ofAppGLFWWindow21runAppViaInfiniteLoopEP9ofBaseApp + 76
	26  cefWorkDebug                        0x00426f3b _Z8ofRunAppP9ofBaseApp + 1120
	27  cefWorkDebug                        0x0000511a main + 122
	28  cefWorkDebug                        0x00002d75 start + 53
)
[1231/120612:WARNING:channel.cc(188)] WriteMessage() while shutting down
[1231/120612:WARNING:message_in_transit_queue.cc(18)] Destroying nonempty message queue
[1231/120614:FATAL:gpu_process_transport_factory.cc(91)] Check failed: per_compositor_data_.empty(). 
0   Chromium Embedded Framework         0x00e6caef base::debug::StackTrace::StackTrace() + 63
1   Chromium Embedded Framework         0x00e6cb4b base::debug::StackTrace::StackTrace() + 43
2   Chromium Embedded Framework         0x00efaf72 logging::LogMessage::~LogMessage() + 82
3   Chromium Embedded Framework         0x00ef9cab logging::LogMessage::~LogMessage() + 43
4   Chromium Embedded Framework         0x0873b13a content::GpuProcessTransportFactory::~GpuProcessTransportFactory() + 298
5   Chromium Embedded Framework         0x0873b26b content::GpuProcessTransportFactory::~GpuProcessTransportFactory() + 43
6   Chromium Embedded Framework         0x0873b30e content::GpuProcessTransportFactory::~GpuProcessTransportFactory() + 46
7   Chromium Embedded Framework         0x0873b371 non-virtual thunk to content::GpuProcessTransportFactory::~GpuProcessTransportFactory() + 49
8   Chromium Embedded Framework         0x087516fa content::ImageTransportFactory::Terminate() + 58
9   Chromium Embedded Framework         0x07bf58fd content::BrowserMainLoop::ShutdownThreadsAndCleanUp() + 2397
10  Chromium Embedded Framework         0x07c05430 content::BrowserMainRunnerImpl::Shutdown() + 1520
11  Chromium Embedded Framework         0x00cbc9a9 CefMainDelegate::ShutdownBrowser() + 89
12  Chromium Embedded Framework         0x00bb7cfd CefContext::FinalizeShutdown() + 93
13  Chromium Embedded Framework         0x00bb6d56 CefContext::Shutdown() + 694
14  Chromium Embedded Framework         0x00bb8241 (anonymous namespace)::CefForceShutdown::~CefForceShutdown() + 65
15  Chromium Embedded Framework         0x00bb599b (anonymous namespace)::CefForceShutdown::~CefForceShutdown() + 43
16  libsystem_c.dylib                   0x911d3f0b __cxa_finalize + 183
17  libsystem_c.dylib                   0x911d4236 exit + 31
18  cefWorkDebug                        0x00002d7d start + 61

(lldb)

these seem like menu creation problem (we already have a menu) then a proper crash when we go to draw…

will dig around for other experiments I did in this area…

zach · 2014-12-31 17:24:55 UTC

@bakercp – also you might be fine just using CEF simple example and building off that with no OF if you don’t need it. I did a version of the project generator that way, and I think that’s how apps like spotify work (they are mostly html front end). Another option is node webkit, which might be a bit friendlier than CEF ?

bakercp · 2014-12-31 17:39:20 UTC

Indeed – regarding the helper app – if you check out “Architecture in 60 Seconds” here it explains the helper process architecture and why you need it (esp on OSX).

zach · 2014-12-31 17:51:22 UTC

thanks ! that’s helpful I probably should RTFM instead of just pounding at the keys

bakercp · 2014-12-31 17:58:32 UTC

Also, I had looked at node-webkit and (erroneously) concluded that it was for node based backends … but upon further inspection it looks like it could be a really good option for ofSketch and way less work … Thanks for the tip.

zach · 2014-12-31 18:14:23 UTC

another one to look at is atom-shell:

some comparisons

github.com

electron/electron/blob/master/docs/development/atom-shell-vs-node-webkit.md

# Technical Differences Between Electron and NW.js (formerly node-webkit)

__Note: Electron was previously named Atom Shell.__

Like NW.js, Electron provides a platform to write desktop applications
with JavaScript and HTML and has Node integration to grant access to the low
level system from web pages.

But there are also fundamental differences between the two projects that make
Electron a completely separate product from NW.js:

__1. Entry of Application__

In NW.js the main entry point of an application is a web page or a JS script. You specify a
html or js file in the `package.json` and it is opened in a browser window as
the application's main window (in case of an html entrypoint) or the script is executed.

In Electron, the entry point is a JavaScript script. Instead of
providing a URL directly, you manually create a browser window and load
an HTML file using the API. You also need to listen to window events

This file has been truncated. show original

http://clintberry.com/2013/html5-apps-desktop-2013/

& there’s also this one:

I don’t know how to compare any of these, just throwing them out there…

smallfly · 2014-12-31 19:04:09 UTC

Here comes the OF avalanche of information, help and sharing

@zach Thanks for the project, I will look at it and see if I can make it work.

I will of course post about my progress here.

pizthewiz · 2014-12-31 19:34:55 UTC

Yeah, over in Cinder-land Simon Geilfus went through a similar exploration flow where he started with CEF and migrated to node-webkit for a realtime Cinder scripting (Lua and AngelScript naturally) IDE of sorts – during his Gray Area workshop in August, I mentioned how ofSketch was doing something semi-similar on the OF side.

smallfly · 2014-12-31 21:43:57 UTC

I had seen that post on the Cinder forum. I sent a quick message to Simon G. to see if he has some inputs for us.

But, am I correct to think that CEF is the tool to implement a frontend UI in html/css/js? I mean Lua and AngelScript are totally different beasts.

silverbahamut · 2015-01-01 01:49:41 UTC

A word of caution.
Last time I checked offscreen rendering in CEF, it lacked HW acceleration support. So fancy css transform that rely on hardware to perform (or simply work) would be perfectly fine in chrome but super buggy in CEF (using offscreen rendering), so be careful when designing your front end, maybe test beforehand that the features you want to use work fine. (I know it kept bugging our web-developper who wanted to use the most advanced chrome features…).

admsyn · 2015-01-01 03:16:08 UTC

@smallfly if you want to give the Webkit / WebView approach a shot, it’s pretty easy to get up and running and see whether or not it bodes well for whatever you’re trying to accomplish. Add the WebKit framework to your project, change ofApp.cpp to .mm and:

#include "ofApp.h"
#include "ofAppGLFWWindow.h"
#import <Cocoa/Cocoa.h>
#import <WebKit/WebKit.h>

void ofApp::setup(){

  NSWindow * window = (NSWindow *)(ofAppGLFWWindow *)ofGetWindowPtr()->getCocoaWindow();
  WebView * webView = [[WebView alloc] initWithFrame:[window.contentView bounds]
                                           frameName:@"OF-frame"
                                           groupName:nil];
    
  [webView setWantsLayer:YES];
  [webView setAutoresizingMask:NSViewWidthSizable | NSViewHeightSizable]; // stick to the window's size
  [webView setDrawsBackground:NO]; // if you want OF to show up behind the web view
  [webView setCanDrawConcurrently:YES]; // may speed up rendering, worth profiling
  
  [webView setHostWindow:window];
  [window.contentView addSubview:webView];
}

loading a local URL:

NSURL * url = [NSURL fileURLWithPath:@"/some/path/to/page.html"];
NSURLRequest * req = [NSURLRequest requestWithURL:url];
[[webView mainFrame] loadRequest:req];

loading a web URL:

  NSURL * url = [NSURL URLWithString:@"http://openframeworks.cc"];
  NSURLRequest * req = [NSURLRequest requestWithURL:url];
  [[webView mainFrame] loadRequest:req];

There’s a bit more going on in the thing I whipped together that zach mentioned, but that’s the gist.

If Xcode freaks out about Obj-C things in a C++ file, sometimes you have to set the source type explicitly like this:

zach · 2015-01-01 15:06:44 UTC

I find embedded browser to be a bit of a rabbit hole w/ time

I’ve just posted my progress here:

it’s messy and basically just getting this to work:

I see the google doodle but it seems to animate at about 5-10 fps

note there is a framework that’s necessary to download since it’s so large and github was rejecting. I’ve thrown it on dropbox if anyone wants to see what I see. IMO CEF offscreen rendering seems slow – especially relative to awesomium while the regular simple examples are blazing fast, I think investigating some of the patches to CEF that do more opengl related things offscreen might be helpful.

I’ll keep poking around – I really did like using awesomium but really want retina support, etc. CEF when it’s in it’s own window looks amazing and crisp on an hdpi screen. having something like that offscreen would be amazing.

(also, just pointing out some useful extra conversation here regarding CEF: https://github.com/ofZach/cefWork/issues/1)

smallfly · 2015-01-02 16:49:02 UTC

I’m running the cefWork project. It’s true that the framerate is bad. If understand correctly the conversation (+ extra links) happening on GitHub, it seems it is a CEF issue (no hardware acceleration for off-screen rendering). But why is this far from being that bad on Windows? I mean, on that thread there is no mention that the issue is specific to OS X. And running ofxCEFClient on Windows give way better results than the poor ~10fps we get on OS X.

@zach thanks for putting this Xcode project together!

@admsyn well, I would like a solution that is cross platform as mush as possible. But thanks for the detailed example, it could become really helpful.

zach · 2015-01-02 16:57:52 UTC

from this video, it seems like it’s totally possible to get offscreen working well and fluid with CEF:

http://www.ogre3d.org/forums/viewtopic.php?f=11&t=79079

also, from my profiler, it doesn’t seem like it’s slow anywhere, just not updating all that fast… the app is running fine and it nothing seems really heavy. I’m currently working on getting the helper app compiling in the project and also placed in the right place (it’s mostly dylib errors now) but was hoping to try poking around on release vs debug and any other settings that might help. There’s also some patches to CEF that seem promising.

zach · 2015-01-02 17:20:27 UTC

I’ve made some progress on speed by using a more recent framework and compiling in release… just uploaded the git repo. I also feel like the CEF I was using prior was debug since it was so large?

note you have to build the helper app first

then build the main app. there is a build phase towards the end of the main app that moves this small app into the frameworks folder and does some renaming. I still get errors every so often, I am not 100% sure about this system…

zach · 2015-01-02 17:28:47 UTC

ps: I am seeing sometimes errors with EH helpers (https://code.google.com/p/chromiumembedded/wiki/GeneralUsage) I feel like I don’t have the helped stuff fully understood, gonna dig into some of the stuff like “make_more_helpers.sh”…