Receiving video frames from callback (best practices)

Hi all,

I just would like to know about best practices when receiving frames(long chunks of bytes) from callback functions. Could anyone please point me to any addon using callbacks. It would help to compare coding patterns…
I have a couple of questions that have been intriguing me for a while.

  1. The main issue here is that callback functions are unsynced with the ofMainLoop so i guess we need to use a mutex to prevent reading reading/writing from two different threads on the same memory address. So far I use one ofMutex per callback and everything “seems” to be working well. I use mutex.lock() inside the callback and mutex.try_lock() in my update so that we make sure we don-t block the main loop. How does that sound? Is there anything else I should care about?

  2. AFAIK Callback functions cant be member functions unless they are defined as static (something I tired but forced me to redefine to much code as static… and it produced similar results). It makes sense because callbacks shouldn’t be instantiated more than once. Therefore i have to place the callback out of the class and any objects used in the callback have to be also out of the class. I would like to know what is relevant to care about when using non member objects… I’m starting a videoRecorder and seem to hit many issues when storing frames, but not sure if that is related to the fact that they aren’t member objects.

Any comments appreciated.
Best,

So far my code looks like that:

//---- ofApp.h

class ofApp : public ofBaseApp{
    public:
        void setup();
        void update();
        void draw();
        Camera device;
        ofImage            liveImage;

}
ofMutex        LiveImageLock;
ofPixels        RawImage;
void ClCallback_Image(ImageStruct* frame, unsigned long dataSize);

//---- ofApp.cpp

void ofApp::update() {
    if (LiveImageLock.try_lock()) {        //----- Mutex begin
       liveImage.setFromPixels(RawImage);
        LiveImageLock.unlock();            //----- Mutex end
    }
}
void ofApp::draw(){
    if (liveImage.isAllocated()) {
        liveImage.draw(0, 0);
    }
}

void ClCallback_Image(ImageStruct* frame, unsigned long dataSize) {
    LiveImageLock.lock();        //----- Mutex begin
        CopyImage(&frame->bitmapTop, RawImage, frame->width, frame->height);
    LiveImageLock.unlock();        //----- Mutex end
}
  1. The way you are doing it you are copying the image in the main loop every frame no matter if there’s really a new frame or not. also try_lock is not a very good idea for this case, in a very extreme case it could happen that your mainloop and camera threads are in sync and the main loop always skips the frame, that won’t really happen but it’ll surely skip some frames here and there. what you want is to lock as few times as possible instead. one possiblity is to use doubel or triple buffering so you are only locking while swapping the buffers instead of while copying them, something like:
ofPixels frontPixels, middlePixels, backPixels;

//update
if(newBackFrame){
    lock()
    swap(frontPixels, middlePixels);
    newBackFrame = false;
    unlock();
    texture.loadData(frontPixels);
}


// callback
copyImage(cameraImage, backPixels);
lock();
swap(backPixels, middlePixels);
newBackFrame = true;
unlock();

another way to do it is to use an ofThreadChannel:

ofThreadChannel frontChannel, backChanel;
//setup
ofPixels pixels;
pixels.allocate(...);
backChanel.send(pixels); // sending more than 1 ofPixels effectivelty 
                         // creates a ringbuffer that helps hidding latency..

// update
ofPixels frontPixels;
if(frontChannel.tryReceive(pixels)){  // don't lock it there's not a new frame
    texture.loadData(pixels);
    backChannel.send(std::move(pixels));  // move avoids a copy
}

// calback
ofPixels pixels;
if(backChannel.receive(pixels)){ // wait until the main frame is done, this could also be tryReceive
    copyImage(cameraImage, pixels);
    backPixels.send(std::move(pixels));
}

There’s some other ways to do it but the basic idea is to have the main thread not lock waiting for a new frame and lock as few time as possible so none of the threads has to wait although the back thread waiting for the main thread to be done, like in the second example, can be good cause you avoid unnecessary copies in case the main thread slows down so much that it can keep up.

Some times there’s a way to tell the camera (or whatever video) API you are using to somehow lock one frame and not reuse it until you tell it in which case you could directly send the frame comming from the camera to the main thread using a channel for example, upload that directly to the texture and then return it to the camera thread where it’ll gets unlocked. that way do 0 copies which can speed up things with big image resolutions.

About 2, Usually there’s a way to pass a user data pointer when setting the callback so your call would turn into something like:

class ofApp: public ofBaseApp{
....

static void ClCallback_Image(ImageStruct* frame, unsigned long dataSize, void * userData);
void nonStaticCallback(ImageStruct* frame, unsigned long dataSize);
}

///cpp
void ofApp::ClCallback_Image(ImageStruct* frame, unsigned long dataSize, void * userData) {
    auto app = (ofApp*)userData;
    app->nonStaticCallback(frame, dataSize);
}

and then you do your stuff in the non static callback where you can access every instance variable.

2 Likes

Thank you SO MUCH! This post is awesome!
I will give it a try and follow up.

I tried double buffering and swapping last week. I guess I have to use the ofPixels.swap() in order to swap the entire object correctly, right?

no, swap(pixels1, pixels2) will work the same

1 Like