Gaze tracking & awesomium

Hey guys,

I am working on a rgb webcam bazed gaze tracker for a school project. I started writting it in bare openCV, but I turned to OF for easier GUIs and screen drawing. The gaze tracking is not as accurate as I initially wanted to, but it works. You can see a video here: https://vimeo.com/92533416 (the password is “of”).
As you can see I divided the screen with the use of a grid, and each rectangle gets a more intense red color as the user looks at it. Sort of like a blocky one-color heatmap.

The wikipedia page is just an image, for now. What I have in mind is to let the user browse a webpage and then save an image of the whole page with this “heatmap” data. I researched how to embed a browser into OF and ended up using https://github.com/WrongEntertainment/ofxAwesomium. However, I haven’t found a way to get the whole webpage image in awesomium. Does anyone have experience with this ?

I made some searches and found two possible answers: one is using an outdated version of awesomium and the other one is for the C# api, and I couldn’t get it going in C++. Practically the ExecuteJavascriptWithResult method returns something of the type Awesomium::FutureJSValue, and I haven’t found a way to convert that to a standard type.

So basically what I am asking for is a way to embed a browser in OF and be able to capture the whole page in an image to which I can apply the grid and the heatmap. Drawing to a frame buffer seems overkill. Also, awesomium was just a pick of mine, if you have other suggestions please let me know.

Thanks and sorry for the long post!

Best,
Andrei

PSL I’ll leave the links to the answers I was talking about in another post, because I can’t post more than two links in a post.

Here are the links to the answers I found and couldn’t post them above:
the first one
the second one

Have you tried CEF ? I think it’s mostly the same technology packaged differently.

The https://github.com/ddiakopoulos/ofxCEFClient should do what you want : If you use offscreen rendering, you’ll have a texture (or a buffer of pixels) of your webpage, and you can then create an ofImage out of it and save it on disk ( or do whatever you want). Keep it mind that :

  • Offscreen rendering is not going to perfom as well as a regular browser (it’s fine as long as you won’t need fancy css transform or play tons of video)
  • Saving images every frame is going to hurt the performances as well (if it’s what you’re trying to do)

As a side note, depending on what you do, you could also use a screen capture tool maybe ? (It’s not clear if you want to apply your transform in real time or only once in a while in post processing)

Thanks for the input. I will look into CEF, though it seems to be Windows only ( at least the OF wrapper) - I need something that works on osx.
I don’t want to save images every frame. The workflow that I have in mind goes like this:

  • load webpage, and get 1 full image of it. Compute the grid that you saw in the video for the said image.
  • then just build up on the ‘heat map’, depending where the user looks and eventually the scroll index, to determine which part of the webpage is visible.
  • save the image coupled with the heat map once the user jumps onto another page. Repeat.

In the samples for ofxAwesomium, is the loadURL sample do what you want in terms of web navigation ?

If so, you can extend this example; when you want to save your webpage you do something like :

      const Awesomium::RenderBuffer* renderBuffer = webView->render();
        if (renderBuffer) {
	ofImage im;
	im.setFromPixels(renderBuffer->buffer, webTexWidth, webTexHeight, ofImageType::OF_IMAGE_COLOR_ALPHA, false);
	im.saveImage("capture.png");

        }

It should give you a result close to what you want.

I have checked that and it works well. The issue is that I would like to get hold of the image of the full webpage, not just the viewable area, so that I can map that grid on it.
Anyway, I have managed to do this using the ExecuteJavascriptWithResult method. I’ll leave an outline of the process here:

First get the height of the full webpage:

std::wstring height = webView->executeJavascriptWithResult(JavaScriptCode).get().toString();

with this js code:

std::string JavaScriptCode = "(function() { var bodyElmnt = document.body; var html = document.documentElement; var height = Math.max( bodyElmnt.scrollHeight, bodyElmnt.offsetHeight, html.clientHeight, html.scrollHeight, html.offsetHeight ); return height; })();";

then resize the webview to the new height:

 webView->resize(ofGetWindowWidth(), ofToInt(convertWideToNarrow(height.c_str())));

finally we set an image from the render buffer, as you described in the previous post.

 image.setFromPixels(renderBuffer->buffer, renderBuffer->width, renderBuffer->height, OF_IMAGE_COLOR_ALPHA, false);

Thanks for the help!

1 Like

Nice! Thanks for sharing your findings! I assumed you’d already had the full page on your screen as it seemed much more painful to account for scrolling (as you’d have to translate your grid accordingly)