[SOLVED] ofVideoGrabber with BAYER8 pixelformat

MRZL · May 15, 2014, 2:44pm

heyo,
I’m using the DFK 31AU03 camera by ImagingSource ( http://www.theimagingsource.com/de_DE/products/cameras/usb-ccd-color/dfk31au03/ ). The video stream is working, though the camera is using a BAYER8 format. What I end up is a greenish image like this

I can use the camera properly with other software, in ubuntu.
maybe somebody can give me a hint into a direction how I could figure this problem out.

best,
marcel

arturo · May 15, 2014, 3:31pm

the problem is the RGB is swapped, this is pretty common, before uploading to the texture you can use ofPixels::swapRGB or just use BGR when doing loadData on the texture

MRZL · May 19, 2014, 2:06pm

I tried it, but it didnt’t do the job. I’m suspecting the videograbber is choosing a wrong format for the camera- is there a way to force use raw-bayer instead of raw-rgb? I’ve got some logs here:

[verbose] ofGstVideoGrabber: get_device_data(): device: DFx 31AU03(/dev/video0)
[verbose] ofGstVideoGrabber: add_video_format(): video/x-raw-yuv 1024x768 framerates:
[verbose] ofGstVideoGrabber: add_video_format(): video/x-raw-bayer 1024x768 framerates:
[verbose] ofGstVideoGrabber: add_video_format(): more similar framerate replacing existing format
[verbose] ofGstVideoGrabber: add_video_format(): video/x-raw-gray 1024x768 framerates:
[verbose] ofGstVideoGrabber: add_video_format(): already added, skipping
[verbose] ofGstVideoGrabber: add_video_format(): video/x-raw-yuv 1024x768 framerates:
[verbose] ofGstVideoGrabber: add_video_format(): non compressed format with same framerate, replacing existing format
[verbose] ofGstVideoGrabber: add_video_format(): video/x-raw-yuv 1024x768 framerates:
[verbose] ofGstVideoGrabber: add_video_format(): already added, skipping
[verbose] ofGstVideoGrabber: add_video_format(): video/x-raw-rgb 1024x768 framerates:
[verbose] ofGstVideoGrabber: add_video_format(): rgb format with same framerate as yuv, replacing existing format
[verbose] ofGstVideoGrabber: add_video_format(): video/x-raw-rgb 1024x768 framerates:

[notice ] ofGstVideoGrabber: initGrabber(): selected device: DFx 31AU03
[notice ] ofGstVideoGrabber: initGrabber(): selected format: 1024x768 video/x-raw-rgb framerate: 30/1

MRZL · June 10, 2014, 7:59am

I still haven’t managed to get the correct camera image. I’ll be trying again and will update this post here. In the meanwhile, I would be very happy about any suggestions what else to do, if swapRGB doesn’t do it.

arturo · June 10, 2014, 9:46am

how are you using swapRGB? it should work, can you post some code?

MRZL · June 10, 2014, 10:19am

ofApp.h:

#pragma once

#include "ofMain.h"
#include "ofxCv.h"

class ofApp : public ofBaseApp{

public:
	void setup();
	void update();
	void draw();

	ofVideoGrabber 		vidGrabber;
	ofImage image;
	ofPixels pix;
};

ofApp.cpp

#include "ofApp.h"

//--------------------------------------------------------------
void ofApp::setup(){
    vidGrabber.setVerbose( true );
    vidGrabber.setDeviceID(1);
	vidGrabber.setDesiredFrameRate(30);
    vidGrabber.setUseTexture( true );
    vidGrabber.setPixelFormat( OF_PIXELS_RGB );
    vidGrabber.initGrabber( 1024, 768, true);

    image.allocate( 1024, 768, OF_IMAGE_COLOR );

}

//--------------------------------------------------------------
void ofApp::update(){
    vidGrabber.update();
    ofPixelsRef pixels = vidGrabber.getPixelsRef();
    pixels.swapRgb();
    image.setFromPixels( pixels );
}

//--------------------------------------------------------------
void ofApp::draw(){
    vidGrabber.draw(0, 0);
    image.draw( 1024, 0 );
}

arturo · June 10, 2014, 11:00am

what do you see in the drawn image?

MRZL · June 10, 2014, 11:52am

Exactly the same as in the original vidgrabber image.

arturo · June 10, 2014, 12:37pm

can you try this:

//.h
ofTexture tex;

//setup
tex.allocate(w,h,GL_RGB);

//update
vidGrabber.update();
if(vidGrabber.isFrameNew()){
    tex.loadData(vidGrabber.getPixelsRef(),GL_BGR);
}

//draw
tex.draw(0,0);

you should only update the pixels when there’s a new frame or you’ll be swapping rgb back and forth every frame of the application, doing thi instead of using an ofImage should be faster since you don’t need to move the pixels in memory or make copies just upload the pixels as BGR to a texture

MRZL · June 10, 2014, 12:48pm

That doesn’t work either. I even tried to get the pixelsref, swaprgb it and then upload it as GL_RGB, too, but without changing the image in the slightest bit. It looks like the color channel swapping isnt applied at all.

arturo · June 10, 2014, 1:31pm

that’s really weird you must see at least some change even if it’s not correct.

btw, which version of OF and ubuntu are you using?

MRZL · June 10, 2014, 2:27pm

I’m using 081 64bit with Ubuntu 12.04.4 LTS.

I’ve only been able to use the camera properly while using LibAVG, which supports bayer8 pixel format.

arturo · June 10, 2014, 5:19pm

might be a problem with gstreamer then, latest versions of OF still work with gstreamer 0.10 (the one that comes with 12.04) but are more tested and work better with gstreamer 1.0, i would try with a computer with 14.04 if you don’t want to update the one you are using yet.

MRZL · June 11, 2014, 8:54am

Okay, I’ve just installed gstreamer1.0 on my 12.04 machine, relaunched the install_* scripts and now I can’t seem to compile ofGstVideoGrabber/ofGstUtils anymore.

../../../libs/openFrameworksCompiled/lib/linux64/libopenFrameworksDebug.a(ofGstVideoGrabber.o)||In function `get_device_data':|
 /home/mrzl/devel/sco/0124_app/application/deps/of_v081_linux64_release/libs/openFrameworksCompiled/project/../../../libs/openFrameworks/video/ofGstVideoGrabber.cpp|563|undefined reference to `gst_caps_unref'|
../../../libs/openFrameworksCompiled/lib/linux64/libopenFrameworksDebug.a(ofGstUtils.o)||In function `on_new_buffer_from_source':|
/home/mrzl/devel/sco/0124_app/application/deps/of_v0.8.1_linux64_release/libs/openFrameworksCompiled/project/../../../libs/openFrameworks/video/ofGstUtils.cpp|45|undefined reference to `gst_app_sink_pull_buffer'|
../../../libs/openFrameworksCompiled/lib/linux64/libopenFrameworksDebug.a(ofGstUtils.o)||In function `ofGstUtils::setPipelineWithSink(_GstElement*, _GstElement*, bool)':|
/home/mrzl/devel/sco/0124_app/application/deps/of_v0.8.1_linux64_release/libs/openFrameworksCompiled/project/../../../libs/openFrameworks/video/ofGstUtils.cpp|186|undefined reference to `gst_plugin_feature_get_name'|
../../../libs/openFrameworksCompiled/lib/linux64/libopenFrameworksDebug.a(ofGstUtils.o)||In function `ofGstVideoUtils::update()':|
/home/mrzl/devel/sco/0124_app/application/deps/of_v0.8.1_linux64_release/libs/openFrameworksCompiled/project/../../../libs/openFrameworks/video/ofGstUtils.cpp|753|undefined reference to `gst_app_sink_pull_buffer'|
||=== Build finished: 4 errors, 0 warnings ===|

I will try on a 14.04 machine soon.

MRZL · June 13, 2014, 3:40pm

So, I’ve just tried on a 14.04 LTS machine and the problem is still there. Whenever somebody may have an idea, let me know. I will post my own updates here:

This actually swaps the red and green channel:

ofPixelsRef pixels = cam.getPixelsRef();
ofPixels rpix = pixels.getChannel(0);
ofPixels gpix = pixels.getChannel(1);
ofPixels bpix = pixels.getChannel(2);

pixels.setChannel(0, gpix);
pixels.setChannel(1, rpix);
pixels.setChannel(2, bpix);

image.setFromPixels( pixels );

Original image:

Without RG-swapping:

With RG-swapping:

This indicates that gstreamer doesn’t debayer the video image. Is there a way to manually force-set the video format gstreamer uses to bayer in order to get data from the camera that is interpreted in the right way? As I wrote in one of my first posts, when enabling verbose mode in VideoGrabber, you can see that ofGstVideoGrabber decides for x-raw-rgb, whereas x-raw-bayer or x-raw-yuv should be chosen, as those are the pixel formats of the camera.

arturo · June 13, 2014, 5:33pm

you can try to create you own pipelne, in 14.04 it should be something like:

ofTexture tex;
ofGstVideoUtils gst;

//setup
gst.setPipeline("v4l2src ! video/x-bayer,format=bggr,width=640,height=480 ! bayer2rgb ! videoconvert", 24,false,640,480);
gst.startPipeline();
gst.play();
tex.allocate(640,480,GL_RGB);

//update
gst.update();
if(gst.isFrameNew()){
    tex.loadData(gst.getPixels());
}

//draw
tex.draw(0,0);

there’s several bayer formats, i’ve written the pipeline with the first one but if the image looks weird or it doesn’t work you can also try with bggr, grbg, gbrg, rggb

MRZL · June 16, 2014, 10:47am

The app doesn’t crash and I don’t get a camera imagebut weird errors instead:

[ error ] ofGstUtils: setPipelineWithSink(): couldn't create pipeline: no element "videoconvert"


CRITICAL **: gst_v4l2src_fixate: assertation 'G_VALUE_TYPE (v) == GST_TYPE_LIST' failed


WARNING **: bayer2rgb0: size500 is not a multiple of unit size 786432

The Pipeline string I ended up using is this:

gst.setPipeline("v4l2src ! video/x-raw-bayer,format=bggr,width=1024,height=768 ! bayer2rgb ! videoconvert", 24, true, 1024, 768);

Using different debayering formats doesn’t do it.

PS: I’ve started thinking about using a debayer shader from here:

github.com

olekristensen/LongingFastForward/blob/master/Experiments/lffCinderCapture/resources/debayer_frag.glsl

/** Monochrome RGBA or GL_LUMINANCE Bayer encoded texture.*/
uniform sampler2D       source;

varying vec4            center;
varying vec4            yCoord;
varying vec4            xCoord;

void main(void) {

    #define fetch(x, y) (texture2D(source, vec2(x, y)).r)

    float C = texture2D(source, center.xy).r; // ( 0, 0)
    const vec4 kC = vec4( 4.0,  6.0,  5.0,  5.0) / 8.0;

    // Determine which of four types of pixels we are on.
    vec2 alternate = mod(floor(center.zw), 2.0);

    vec4 Dvec = vec4(
        fetch(xCoord[1], yCoord[1]),  // (-1,-1)
        fetch(xCoord[1], yCoord[2]),  // (-1, 1)

This file has been truncated. show original

github.com

olekristensen/LongingFastForward/blob/master/Experiments/lffCinderCapture/resources/debayer_vert.glsl

/** (w,h,1/w,1/h) */
uniform vec4            sourceSize;

/** Pixel position of the first red pixel in the Bayer pattern.  [{0,1}, {0, 1}]*/
uniform vec2            firstRed;

/** .xy = Pixel being sampled in the fragment shader on the range [0, 1]
    .zw = ...on the range [0, sourceSize], offset by firstRed */
varying vec4            center;

/** center.x + (-2/w, -1/w, 1/w, 2/w); These are the x-positions of the adjacent pixels.*/
varying vec4            xCoord;

/** center.y + (-2/h, -1/h, 1/h, 2/h); These are the y-positions of the adjacent pixels.*/
varying vec4            yCoord;

void main(void) {

    center.xy = gl_MultiTexCoord0.xy;
    center.zw = gl_MultiTexCoord0.xy * sourceSize.xy + firstRed;

This file has been truncated. show original

They’re from this paper: http://graphics.williams.edu/papers/BayerJGT09/bayer-jgt09.pdf

Do you think they’re compatible with of’s gl version?

MRZL · June 17, 2014, 2:49pm

I’m sorry, it ended up working with the solution you posted. Whoops. :~)

arturo · June 17, 2014, 5:05pm

great, i think the camera you are using might be reporting different formats while only supporting bayer so you need to setup the pipeline yourself. anyway, glad it’s working