ofImage drawing a blank (includes a threading load)

hello OF folks

I’ve run into another apparently basic problem…

I’m using the threadedPixelBufferExample as a template to test something - basically I want to draw objects into an fbo; then buffer the fbo texture, send it to a thread, turn it into an image… but then importantly I want to call that image back from the thread, and display it on screen using ofImage.draw().

I know this seems pointless because I could just draw the fbo in the first place, but I have my reasons (another bigger project where I want to sample pixels etc from the image, and draw it too). i.e. I am looking to cut out the processing time that results when multiple very large Fbo textures are sequentially loaded into ofImage templates, and this is a test run before committing.

So at present I have the existing example, and I’ve tweaked it very slightly, so that on a keypress I take the resolved ofPixels from the thread, and load them back into an ofImage item in the main update section. Changes are as follows:

//--------------------------------------------------------------
// first, to send to the thread on a specific keystroke that triggered the record bool:
void ofApp::sendto_Thread(){
    
        if(!firstFrame){
            // wait for the thread to finish saving the
            // previous frame and then unmap it
            saverThread.waitReady();
            pixelBufferBack.unmap();
        }
        firstFrame = false;

        // copy the fbo texture to a buffer
        fbo.getTexture().copyTo(pixelBufferBack);

        // bind and map the buffer as PIXEL_UNPACK so it can be
        // accessed from a different thread  from the cpu
        // and send the memory address to the saver thread
        pixelBufferFront.bind(GL_PIXEL_UNPACK_BUFFER);
        unsigned char * p = pixelBufferFront.map<unsigned char>(GL_READ_ONLY);
        saverThread.save(p);

        // swap the front and back buffer so we are always
        // copying the texture to one buffer and reading
        // back from another to avoid stalls
        swap(pixelBufferBack,pixelBufferFront);
        calculating = true;
        record = false;
    }
// later, to retrieve from the thread, once, on a second keystroke:
void ofApp::getfromThread(){
    if(calculating && saverThread.imageReady){
        resultImage.setFromPixels(saverThread.pixels);
        resultImage.update();
        ofSaveImage(resultImage,"result.jpg");
        calculating = false;
        saverThread.imageReady = false;
    }}

Then in the draw() I just call:

    if(showImage){
        ofSetColor(255);
        resultImage.draw(ofGetMouseX(),ofGetMouseY(),width/2, height/2);
    }

But (and this is really doing my head in) whilst the ofSaveImage line works just fine, and saves a good .jpg file with the correct image drawing, the resultImage.draw() just gives me a blank, white, rectangle.

What am I missing? It must be something basic - Can someone put me right?
S

Hey.

I think the issue is that OpenGL is not designed for running across multiple threads.
I believe there are a few operations you can do in another thread but most OF calls that use openGL won’t work on a second thread.

That said if resultImage.setFromPixels(saverThread.pixels); is in the main thread ie: ofApp::getfromThread() is not running threaded then it should work as the OpenGL operations are in the main thread of the program.

For fast FBO read back to pixels I find that this addon accomplishes all my typical needs. You can easily get 60fps 1920x1080 read back to ofPixels. You could then thread the image saving if you notice any bottlenecks there.

Hope this is helpful.

Theo

Hey Theo, thanks for the reply!

I did some more tinkering and I still can’t get the image to draw… even though, yeah, the setFromPixels is in the main thread. So maybe it is something else. But then I realised anyway that the line:

fbo.getTexture().copyTo(pixelBufferBack);

Has the exact same bottleneck I am trying to get round - i.e. when I set the fbo massive (using glGetIntegerv(GL_MAX_TEXTURE_SIZE, &maxDimens); ) it takes about 500ms to do copyTo, and so I start to see stalls when I do several runs consecutively.

I’ll explore the add-on which sounds handy. Just in case anyone has any further advice, or can point out a simpler approach, it might help to explain the wider context…

I’m drawing shapes into an fbo at the largest possible size allocation, because I then send the fbo texture to an image and run ofxCV functions on the pixels; i.e. I want to trace the space of the drawn image as accurately as possible. At the moment, the way I’m doing this is:

   ofFbo floodCache;
   floodCache.allocate(width, height, GL_RGB8);

//later on....
//draw into the fbo
    floodCache.begin();
    ofClear(ofColor(255,0,255));
    ofSetColor(255);
    //draw things all in white here here, so later we trace the pink left overs as shapes
    floodCache.end();
    
//copy into a pixels object
    ofPixels tmpPixels;
    tmpPixels.clear();
//this next line is the only real bottleneck 
//it takes about 500ms at present on max size, same as in the above 
    floodCache.readToPixels(tmpPixels);
    
//make images from the pixels and translate to greyscale for tracing
    ofImage colorImg.setFromPixels(tmpPixels);
    ofxCvGrayscaleImage grayImage = colorImg;
    grayImage.threshold(170);

//then later send grayImage to trace outlines into polylines in an ofxCV thread

//tidy up my mess
    tmpPixels.clear();
    colorImg.clear();
    floodCache.clear();

This gives me a nice clean set of polyline shapes, it just has that one bottleneck left that adds up when I do the function multiple times in a row.

If anyone can tell me better method than the above, I’d love to hear it - and in the meantime I’ll check out the add-on, keeping everything crossed!

Thanks again
S

humm well, I’ve tried using the ofxFastFbo library, and it gives me a bit of speed up which is cool, but it also seems to lead to issues… some of the pixel info I am getting out no longer seems clear

in this, for instance, when the colorImg object is an ofxCvColorImage, I get an error on the later tracing exercise, which suggests the pixel info has been lost somewhere along the line:

    drawTo.begin();
//do some drawing to the fbo object here
    drawTo.end();
    
    ofPixels tmpPixels;
    tmpPixels.clear();

//old line
    //drawTo.readToPixels(tmpPixels);

//new line using ofxFastFbo reader object, which is about 200ms faster give or take
    reader.readToPixels(drawTo, tmpPixels);

    colorImg.setFromPixels(tmpPixels);
    tmpPixels.clear();

    grayImage = colorImg;
    grayImage.threshold(170);
    colorImg.clear();

I would make sure your FBO is RGB.
As the reader readToPixels call is assuming an RGB image.

If you need your FBO to be RGBA then you have to modify the last ( hidden argument ) of reader.readToPixels and then convert the pixels to RGB for ofxCvColorImage

tmpPixels.setImageType(OF_IMAGE_COLOR);

Also as you are using a temporary ofPixels object you should call.
reader.setAsync(false); in your setup.

Or add tmpPixels to ofApp.h

you may also want to double check if pixels has a width and a height – I could be totally mis-remembering this but there may be a few frame delay at startup before you start receiving data (I’ve used this in the past – https://github.com/satoruhiga/ofxFastFboReader) since it’s asynchronous…