[SOLVED] Super Fast Exchange of video texture between Threads

Hey guys,

im having a problem. I want to grab video images and process them (optical flow farneback) in a different thread. My current solution sends pixels to the second thread (with camera.getPixels()) and everything is super fast! But im looking for the quickest logic possible and smallest latency.

Hence i was thinking if a Pixel Buffer Object upload would be faster and started to experiment with threadedPixelBufferExample. In this example the texture of an FBO is uploaded to a different thread via PBO.

But as soon as i draw a camera texture to the FBO - The camera images dont update anymore

if (camera.isFrameNew()) {
fbo.begin();
ofClear(0, 255);
ofSetColor(100);
ofSetColor(255);
camera.draw(0, 0);
fbo.end();
}

Now my question are:

  1. is PBO actually appropriate for my case? Would this speed up the exchange of pixel between the threads?

  2. What does the drawinig a camera texture have to do - with the uploaded FBO Texture. Why does it stop working as soon as i draw the texture in it.

To recreate the error: Simply open example threadedPixelBufferExample and draw a video grabber into the FBO - press Record

Any help - suggestion or written thought is highly appreciated.

thank you guys

mio

using pbo doesn’t make sense in your case since you already have the pixels in ram. the videograbber class (and the camera driver itself) grabs the pixel data from the camera and puts it in ram you can then send those pixels to a different thread and do the analisys there. until then the GPU hasn’t been involved at all yet.

ofVideoGrabber also gets those pixels and puts them in an ofTexture (memory in the graphics card) so it’s easy to draw the video.

what you have now goes something liek:

camera -> ram -> cpu

what you are trying to do by using pbo would be something like:

camera -> ram -> video memory -> ram -> cpu

which is way less efficient.

PBOs are meant to be used when you need to upload or download pixel info to or from the graphics card but in this case the pixels are already in ram.

Hey @arturo thx for the quick reply and explanation.

In order to explain my problem i tried to reduce the complexity but left out an important note i think.
Actually I want to pre process the image within an fbo and custom shaders (warp, scale, blur, erode ect)…before sending it to another thread calculating flow, hence they are already on a texture and not pixels anymore.
Wouldnt it be then the quickest version?

mmh, yes i’m seeing the same problem. not sure what’s going but will take a look and let you know

and yes then the threadedPixelBufferExample should do exatcly what you need. downloading pixels form the video memory to ram can be slow, even using PBOs, they just make it possible to do that asynchronously so the main thread is not blokced meanwhile.

if you are using this for computer vision you might want to check if doing those processes in the cpu is faster than using the gpu and then downloading the pixels to ram again.

So the problem is that the target the PBO is bound to is also the target used to copy a texture from a PBO. since we didn’t unbind the pbo before loading the video texture, the pixels where being loaded from the pbo instead of from the pixels in ram.

i’ve fixed this in master but you would need to unbind after mapping and then bind and unbind when unmapping too

thank you arturo for looking into it and your time. I will def give it a try right now…even though i still dont get why this relates to the content of the fbo. If there is a cube in the fbo - it works. If there is a camera image in the fbo - it doesnt.

Again thx ill try
Mio

One thing to be careful about if you do use the asynchronous reedback is that you will always be about two frames behind (this is my experience with satoru’s fast fbo reader addon at least)-- this can present problems if you are also doing things in ram on the frame data and comparing it to the reedback data.

I did a terrible hack on a recent project where I drew the frame number as a tiny rectangle into the image (i.e. Using ofsethexcolor ). It wasn’t pretty at all but helped me understand what frame this read back actually matched to.

1 Like

the problem is that the texture of the video grabber doesn’t get updated cause a pbo is bound to the origin where the texture will read the pixels from when calling loadData. when there’s no PBO bound the texture reads the pixels from ram but when there’s a pbo bound to GL_PIXELS_UNPACK it’ll read the pixels from that pbo. the problem is not that the fbo is not being updated but that the texture of the video is not getting the pixels but instead the content of the pbo so it always stays the same

1 Like

@arturo thx i think i finally understood the reason for it freezing/locking - thank you for taking the time.

@zach thx! i see…but this means that pbo is nice to enhance fps…but the latency seems horrible…

Thank you everyone

@princemio I found that you loose two frames with the ofxFastFbo reader (and you can set the number of buffers) which is generally common to loose in a projector or camera, so the latency wasn’t that bad. This means if it’s frame 1000, your readback of the fbo would give you frame 998…

In my case, I was matching data between cpu and gpu, so I basically stored the last n frames of data, and matched it up with the readback results. my problem was that I had to render alot of things in the fbo and I was kind of cpu constrained, so I had much worse hits if I rendered in ram (using cairo, opencv or something I rolled myself), etc and it helps me divide some of the work between the cpu and gpu and worked out pretty well.

Ok got it thx @zach — sounds very interesting. Thank you very much for the indepth look! i think i finally understood!