Improving performance of VideoGrabber Output

Hi, I’m trying to get a video input from a 1920x1080 @ 30 fps USB 3.0 camera and output it again at the same resolution and framerate running on an UpBoard (www.up-board.org). The way I’ve programmed it now is below but I only get about 15 fps and I wonder if there’s any improvements in the way I programmed to improve efficiency and performance.

Basically, I take the video from the camera, make background substraction (on backspace keypress) resp. substract the new image from the last saved one and then output the difference image again.

Thanks for any help on this, I’m quite new to C++ and Graphics programming.

myApp.h

#pragma once

#include "ofMain.h"
#include "hgDraw.h"
#include "ofxOpenCv.h"
#include "ofxCv.h"

class hgApp : public ofBaseApp {

	public:
		
		void setup();
		void update();
		void draw();
		ofImage *getOutputImagePointer();
		void keyPressed(int key);

	private:

		ofVideoGrabber vidGrabber;
		ofImage outputImage;
		ofPixels bgImage, diffImage;
		cv::Mat camMat, bgMat, diffMat;

		hgDraw drawManager;
		
};

myApp.cpp

#include "hgApp.h"

void hgApp::setup() {
	ofSetFrameRate(30);
	ofSetVerticalSync(true);
	vidGrabber.setDeviceID(0);
	vidGrabber.setDesiredFrameRate(30);
	vidGrabber.setup(1920, 1080);
	ofxCv::imitate(outputImage, vidGrabber);
	ofxCv::imitate(bgImage, vidGrabber);
	ofxCv::imitate(diffImage, vidGrabber);
	drawManager.setup(getOutputImagePointer());
}

void hgApp::update() {
	ofSetWindowTitle("Framerate: " + ofToString(ofGetFrameRate(), 2));
	vidGrabber.update();
	if(vidGrabber.isFrameNew()) {		
		camMat = ofxCv::toCv(vidGrabber);
		bgMat = ofxCv::toCv(bgImage);
		diffMat = ofxCv::toCv(diffImage);
		cv::absdiff(bgMat, camMat, diffMat);
		ofxCv::toOf(diffMat, outputImage);
		outputImage.update();
	}
}

void hgApp::draw() {
	drawManager.draw();
}

ofImage *hgApp::getOutputImagePointer() {
	ofImage *outputPointer = &outputImage;
	return outputPointer;
}

void hgApp::keyPressed(int key) {
	camMat.copyTo(bgMat);
}

myDrawer.h

#pragma once

#include "ofMain.h"
#include "ofxOpenCv.h"
#include "ofxCv.h"

class hgDraw {

	public:
		
		void setup(ofImage const *_playoutImage);
		void draw();

	private:

		ofImage const *playoutImage;
		
};

myDrawer.cpp

#include "hgDraw.h"

void hgDraw::setup(ofImage const *_playoutImage) {
    playoutImage = _playoutImage;
}

void hgDraw::draw() {
	ofBackground(0);
	ofSetColor(255);
	ofNoFill();
	playoutImage->draw(0, 0);
}

one thing you can do in linux to increase performance a lot is to use native mode when opening the camera.

by default openframeworks converts the output format to rgb which happens in the cpu and can be quite expensive at high resolutions.

to enable this, before opening the camera call:

vidGrabber.setPixelsFormat(OF_PIXELS_NATIVE);
vidGrabber.setDeviceID(0);
vidGrabber.setDesiredFrameRate(30);
vidGrabber.setup(1920, 1080);

that will return the pixels in whatever is the native format of the camera,usually some form of YUV and most of the time I420. you can query the exact format after initializing the grabber using vidGrabber.getPixels().getPixelFormat().

If the pixel format is some kind of YUV then usually the pixels won’t be interlaced as in rgb but there will be a first plane where the luminance information is, you can call:

camMat = ofxCv::toCv(vidGrabber.getPixels().getPlane(0));

which will directly return a grayscale image so you also avoid that step.

for drawing the camera image you’ll need to use the programmable renderer, which uses shaders to transform yuv -> rgb in the graphics card, otherwise all you will see would be the b&w plane, just set the opengles version to 2 when setting up the application in main like:

int main(){
    ofGLESWindowSettings settings;
    settings.setGLESVersion(2);
    ofCreateWindow(settings);
    return ofRunApp(new ofApp);
}

Thanks a lot @arturo! I will try this out right now and report back. I’ve just checked CPU/GPU usage and you can clearly see that almost all the work is on the CPU right now, with the GPU only being used 20% … so there’s a lot of potential in there still.

what is that application to meassure gpu usage?

@arturo it’s intel_gpu_top. For Arch Linux I found it in a package called intel_gpu_tools

I get a Segmentation Fault after switching the pixel format to native:

[notice ] ofGstVideoGrabber: Probing devices with udev...
[notice ] ofGstVideoGrabber: Found device 5986:0706, getting capabilities...
[notice ] ofGstVideoGrabber: detected v4l2 device: Integrated Camera
[notice ] ofGstVideoGrabber: driver: uvcvideo, version: 264708
[notice ] ofGstVideoGrabber: Capabilities: 0x84200001
[notice ] ofGstUtils: setPipelineWithSink(): gstreamer pipeline: v4l2src name=video_source device=/dev/video0 ! video/x-raw,framerate=30/1  ! appsink name=ofappsink enable-last-sample=0 caps="video/x-raw,format={RGBA,BGRA,RGB,BGR,RGB16,GRAY8,YV12,I420,NV12,NV21,YUY2}, width=1920, height=1080"
[1]    2291 segmentation fault (core dumped)  ./bin/test

Oh man sorry please ignore my last comment, the error was originating because I was doing

ofLog(OF_LOG_NOTICE, "%s", vidGrabber.getPixels().getPixelFormat());

[EDIT latest error message]

@arturo this is the output I get now. I’ve set the pixel format for the grabber and changed the opengles version to 2 as you suggested but it halts with this error (see bottom line):

[notice ] ofGstVideoGrabber: Probing devices with udev...
[notice ] ofGstVideoGrabber: Found device 5986:0706, getting capabilities...
[notice ] ofGstVideoGrabber: detected v4l2 device: Integrated Camera
[notice ] ofGstVideoGrabber: driver: uvcvideo, version: 264708
[notice ] ofGstVideoGrabber: Capabilities: 0x84200001
[notice ] ofGstUtils: setPipelineWithSink(): gstreamer pipeline: v4l2src name=video_source device=/dev/video0 ! video/x-raw,framerate=30/1  ! appsink name=ofappsink enable-last-sample=0 caps="video/x-raw,format={RGBA,BGRA,RGB,BGR,RGB16,GRAY8,YV12,I420,NV12,NV21,YUY2}, width=1920, height=1080"
[ error ] ofGstUtils: gstHandleMessage(): embedded video playback halted for plugin, module video_source  reported: Internal data stream error.

[EDIT Friday March 24th]
@arturo It seems my camera won’t accept OF_PIXELS_NATIVE so I’ve set the grabber to OF_PIXELS_YV12, as when set to the native constant it would return format={RGBA,BGRA,RGB,BGR,RGB16,GRAY8,YV12,I420,NV12,NV21,YUY2} which I presume are the supported formats. I tried YV12 and I420 which both work.

And voilà boom the performance is almost doubled! So indeed the bottleneck was clearly the automatic conversion of YUV to RGB on the CPU. I am not using the programmable renderer as you suggested, as I only need the B/W image in the end anyway resp. I don’t need to convert back to RGB at any point anyway. Is this ok or does it bring any other advantage to use the programmable renderer as you pointed out?

Also I am not using camMat = ofxCv::toCv(vidGrabber.getPixels().getPlane(0)); as it seems it works just fine the way I did it originally. Again is there any reason to do it like that if I only need the B/W image anyway?

My main goal is to get the most high quality B/W image stream from the camera in the least expensive way.

Thanks a lot for your help, I appreciate this a lot.

Best, Riccardo