[SOLVED] ofVideoGrabber with BAYER8 pixelformat

heyo,
I’m using the DFK 31AU03 camera by ImagingSource ( http://www.theimagingsource.com/de_DE/products/cameras/usb-ccd-color/dfk31au03/ ). The video stream is working, though the camera is using a BAYER8 format. What I end up is a greenish image like this

I can use the camera properly with other software, in ubuntu.
maybe somebody can give me a hint into a direction how I could figure this problem out.

best,
marcel

the problem is the RGB is swapped, this is pretty common, before uploading to the texture you can use ofPixels::swapRGB or just use BGR when doing loadData on the texture

I tried it, but it didnt’t do the job. I’m suspecting the videograbber is choosing a wrong format for the camera- is there a way to force use raw-bayer instead of raw-rgb? I’ve got some logs here:

[verbose] ofGstVideoGrabber: get_device_data(): device: DFx 31AU03(/dev/video0)
[verbose] ofGstVideoGrabber: add_video_format(): video/x-raw-yuv 1024x768 framerates:
[verbose] ofGstVideoGrabber: add_video_format(): video/x-raw-bayer 1024x768 framerates:
[verbose] ofGstVideoGrabber: add_video_format(): more similar framerate replacing existing format
[verbose] ofGstVideoGrabber: add_video_format(): video/x-raw-gray 1024x768 framerates:
[verbose] ofGstVideoGrabber: add_video_format(): already added, skipping
[verbose] ofGstVideoGrabber: add_video_format(): video/x-raw-yuv 1024x768 framerates:
[verbose] ofGstVideoGrabber: add_video_format(): non compressed format with same framerate, replacing existing format
[verbose] ofGstVideoGrabber: add_video_format(): video/x-raw-yuv 1024x768 framerates:
[verbose] ofGstVideoGrabber: add_video_format(): already added, skipping
[verbose] ofGstVideoGrabber: add_video_format(): video/x-raw-rgb 1024x768 framerates:
[verbose] ofGstVideoGrabber: add_video_format(): rgb format with same framerate as yuv, replacing existing format
[verbose] ofGstVideoGrabber: add_video_format(): video/x-raw-rgb 1024x768 framerates:

[notice ] ofGstVideoGrabber: initGrabber(): selected device: DFx 31AU03
[notice ] ofGstVideoGrabber: initGrabber(): selected format: 1024x768 video/x-raw-rgb framerate: 30/1

I still haven’t managed to get the correct camera image. I’ll be trying again and will update this post here. In the meanwhile, I would be very happy about any suggestions what else to do, if swapRGB doesn’t do it.

how are you using swapRGB? it should work, can you post some code?

ofApp.h:

#pragma once

#include "ofMain.h"
#include "ofxCv.h"

class ofApp : public ofBaseApp{

public:
	void setup();
	void update();
	void draw();

	ofVideoGrabber 		vidGrabber;
	ofImage image;
	ofPixels pix;
};

ofApp.cpp

#include "ofApp.h"

//--------------------------------------------------------------
void ofApp::setup(){
    vidGrabber.setVerbose( true );
    vidGrabber.setDeviceID(1);
	vidGrabber.setDesiredFrameRate(30);
    vidGrabber.setUseTexture( true );
    vidGrabber.setPixelFormat( OF_PIXELS_RGB );
    vidGrabber.initGrabber( 1024, 768, true);

    image.allocate( 1024, 768, OF_IMAGE_COLOR );

}

//--------------------------------------------------------------
void ofApp::update(){
    vidGrabber.update();
    ofPixelsRef pixels = vidGrabber.getPixelsRef();
    pixels.swapRgb();
    image.setFromPixels( pixels );
}

//--------------------------------------------------------------
void ofApp::draw(){
    vidGrabber.draw(0, 0);
    image.draw( 1024, 0 );
}

what do you see in the drawn image?

Exactly the same as in the original vidgrabber image.

can you try this:

//.h
ofTexture tex;

//setup
tex.allocate(w,h,GL_RGB);

//update
vidGrabber.update();
if(vidGrabber.isFrameNew()){
    tex.loadData(vidGrabber.getPixelsRef(),GL_BGR);
}

//draw
tex.draw(0,0);

you should only update the pixels when there’s a new frame or you’ll be swapping rgb back and forth every frame of the application, doing thi instead of using an ofImage should be faster since you don’t need to move the pixels in memory or make copies just upload the pixels as BGR to a texture

That doesn’t work either. I even tried to get the pixelsref, swaprgb it and then upload it as GL_RGB, too, but without changing the image in the slightest bit. It looks like the color channel swapping isnt applied at all.

that’s really weird you must see at least some change even if it’s not correct.

btw, which version of OF and ubuntu are you using?

I’m using 081 64bit with Ubuntu 12.04.4 LTS.

I’ve only been able to use the camera properly while using LibAVG, which supports bayer8 pixel format.

might be a problem with gstreamer then, latest versions of OF still work with gstreamer 0.10 (the one that comes with 12.04) but are more tested and work better with gstreamer 1.0, i would try with a computer with 14.04 if you don’t want to update the one you are using yet.

Okay, I’ve just installed gstreamer1.0 on my 12.04 machine, relaunched the install_* scripts and now I can’t seem to compile ofGstVideoGrabber/ofGstUtils anymore.

../../../libs/openFrameworksCompiled/lib/linux64/libopenFrameworksDebug.a(ofGstVideoGrabber.o)||In function `get_device_data':|
 /home/mrzl/devel/sco/0124_app/application/deps/of_v081_linux64_release/libs/openFrameworksCompiled/project/../../../libs/openFrameworks/video/ofGstVideoGrabber.cpp|563|undefined reference to `gst_caps_unref'|
../../../libs/openFrameworksCompiled/lib/linux64/libopenFrameworksDebug.a(ofGstUtils.o)||In function `on_new_buffer_from_source':|
/home/mrzl/devel/sco/0124_app/application/deps/of_v0.8.1_linux64_release/libs/openFrameworksCompiled/project/../../../libs/openFrameworks/video/ofGstUtils.cpp|45|undefined reference to `gst_app_sink_pull_buffer'|
../../../libs/openFrameworksCompiled/lib/linux64/libopenFrameworksDebug.a(ofGstUtils.o)||In function `ofGstUtils::setPipelineWithSink(_GstElement*, _GstElement*, bool)':|
/home/mrzl/devel/sco/0124_app/application/deps/of_v0.8.1_linux64_release/libs/openFrameworksCompiled/project/../../../libs/openFrameworks/video/ofGstUtils.cpp|186|undefined reference to `gst_plugin_feature_get_name'|
../../../libs/openFrameworksCompiled/lib/linux64/libopenFrameworksDebug.a(ofGstUtils.o)||In function `ofGstVideoUtils::update()':|
/home/mrzl/devel/sco/0124_app/application/deps/of_v0.8.1_linux64_release/libs/openFrameworksCompiled/project/../../../libs/openFrameworks/video/ofGstUtils.cpp|753|undefined reference to `gst_app_sink_pull_buffer'|
||=== Build finished: 4 errors, 0 warnings ===|

I will try on a 14.04 machine soon.

So, I’ve just tried on a 14.04 LTS machine and the problem is still there. Whenever somebody may have an idea, let me know. I will post my own updates here:

This actually swaps the red and green channel:

ofPixelsRef pixels = cam.getPixelsRef();
ofPixels rpix = pixels.getChannel(0);
ofPixels gpix = pixels.getChannel(1);
ofPixels bpix = pixels.getChannel(2);

pixels.setChannel(0, gpix);
pixels.setChannel(1, rpix);
pixels.setChannel(2, bpix);

image.setFromPixels( pixels );

Original image:

Without RG-swapping:

With RG-swapping:

This indicates that gstreamer doesn’t debayer the video image. Is there a way to manually force-set the video format gstreamer uses to bayer in order to get data from the camera that is interpreted in the right way? As I wrote in one of my first posts, when enabling verbose mode in VideoGrabber, you can see that ofGstVideoGrabber decides for x-raw-rgb, whereas x-raw-bayer or x-raw-yuv should be chosen, as those are the pixel formats of the camera.

you can try to create you own pipelne, in 14.04 it should be something like:

ofTexture tex;
ofGstVideoUtils gst;

//setup
gst.setPipeline("v4l2src ! video/x-bayer,format=bggr,width=640,height=480 ! bayer2rgb ! videoconvert", 24,false,640,480);
gst.startPipeline();
gst.play();
tex.allocate(640,480,GL_RGB);

//update
gst.update();
if(gst.isFrameNew()){
    tex.loadData(gst.getPixels());
}

//draw
tex.draw(0,0);

there’s several bayer formats, i’ve written the pipeline with the first one but if the image looks weird or it doesn’t work you can also try with bggr, grbg, gbrg, rggb

The app doesn’t crash and I don’t get a camera imagebut weird errors instead:

[ error ] ofGstUtils: setPipelineWithSink(): couldn't create pipeline: no element "videoconvert"


CRITICAL **: gst_v4l2src_fixate: assertation 'G_VALUE_TYPE (v) == GST_TYPE_LIST' failed


WARNING **: bayer2rgb0: size500 is not a multiple of unit size 786432

The Pipeline string I ended up using is this:

gst.setPipeline("v4l2src ! video/x-raw-bayer,format=bggr,width=1024,height=768 ! bayer2rgb ! videoconvert", 24, true, 1024, 768);

Using different debayering formats doesn’t do it.

PS: I’ve started thinking about using a debayer shader from here:


They’re from this paper: http://graphics.williams.edu/papers/BayerJGT09/bayer-jgt09.pdf

Do you think they’re compatible with of’s gl version?

I’m sorry, it ended up working with the solution you posted. Whoops. :~)

great, i think the camera you are using might be reporting different formats while only supporting bayer so you need to setup the pipeline yourself. anyway, glad it’s working