Grabbing the screen in a thread?

I’m trying to capture the output of a simple OF program on a Raspberry Pi 3. I’ve looked in here and found a few different approaches, but none seems to work on this hardware, so I tried to grab the screen every time something changes, and later on I would put all the saved images together to a movie file with ffmpeg. Doing this in the main thread stalls the program until an image is saved, making the program not responsive at all (the program receives input from a keyboard and displays the output as strings).
I’m trying to do this with threads, following the third example from ofBook (ofBook - Threads). Here’s my class definition in threadedClass.h (please bear with me, my C++ knowledge is next to nothing):

class ImageSaver: public ofThread {
	public:
		ImageSaver() {
		this->saved = true;
		}

		void save(int width, int height, int ndx) {
			this->width = width;
			this->height = height;
			this->index = ndx;
			this->imgName = ofToString(index) + ".jpg";
			this->saved = false;
			startThread();
		}

		bool saved;

	private:
		void threadedFunction() {
			img.grabScreen(0, 0, width, height);
			img.saveImage(imgName);
			this->saved = true;
		}

		ofImage img;
		string imgName;
		int index;
		int width;
		int height;
};

Here’s how I import it in ofApp.h:

vector<unique_ptr<ImageSaver>> imgSavers;

And here’s how I’m trying to save the output in ofApp.cpp, whenever input is received from the keyboard:

frameCounter++;
if (captureScreen) {
	foundAvailableThread = false;
	for (int i = 0; i < imgSavers.size(); i++) {
		if (imgSavers[i].saved) {
			imgSaves[i].save(width, height, frameCounter);
			foundAvailableThread = true;
			break;
		}
	}
	if (!foundAvailableThread) {
		imgSavers.push_back(move(unique_ptr<ImageSaver>(new ImageSaver)));
	        imgSavers.back()->save(width, height, frameCounter);
	}
	captureScreen = false;
}

The above code is happening in draw() because the keyboard input is received via OSC instead of keyPressed() or keyRelease().

The program compiles and projects the text written with the keyboard fine. It also saves .jpg images with the frame number in bin/data, but they’re all black. I read in the ofBook chapter that all GL operations have to happen in the main thread, because openGL is single-threaded, but I guess there must be a way to do this, since addons like ofxVideoRecorder are threaded (though its code is too complicated for me to understand).

Any help appreciated.

Hey @alexandrosdrymonitis it looks like you have a vector of objects, where each object has its own thread. Maybe you could do something similar with just 1 ImageSaver thread, and a vector of ofImages for it to save. Your code seems like it would generate a new ImageSaver object (and hence start a new thread) every time it saves images. But there are no calls to ofThread::stopThread(), or calls to destroy ImageSaver objects that are no longer needed to save images.

A couple of relevant examples are: /input_output/imageSaverExample and /threads/threadExample.

Edit: Also here is a forum thread about recording video from the screen on the Pi; it may be helpful.

Thanks for the hints! I’ve changed the code of the ImageSaver class and stopThread() is called when an image is saved. Still, I get black images. Checking the threadExample from examples/threads, I saw that the draw() function is called from the main thread since it contains openGL calls. I guess this is the reason why I get black images. I tried to include the img.grabScreen() and img.save() functions in the save() function, and I do get snapshots of the screen, but the program is again stalled until the image has been saved.
Unfortunately I can’t get the ofxOMXRecorder to work, I get errors concerning Poco, which is not declared, but that should probably go in another forum post.

Question, calling stopThread() will actually kill the thread? If so, in ofApp.cpp I don’t need to run a loop and check if there is a thread running that is not currently saving an image and I can go straight to these two lines of code:

imgSavers.push_back(move(unique_ptr<ImageSaver>(new ImageSaver)));
imgSavers.back()->save(width, height, frameCounter);

Even if that’s the case, I first need to get saved snapshots without stalling the program.

Hey Alexandros, the “black images” issue could probably be solved by rendering into an ofFbo in ofApp::update(), instead of grabbing an image from the screen with an ImageSaver object. The ofFbo could then be used to create an ofImage which can be saved as a file, and it can be drawn in ofApp::draw(). Sometimes it helps to have a pair of ofFbo (one for reading and one for writing), and use std::swap() when you want to swap them.

I’m thinking that you just need 1 ImageSaver, and not a bunch of them as a std::vector<unique_ptr<ImageSaver>>.

An ofThread has a std::mutex inside of it that you can use to lock an object, which will keep multiple threads from simultaneously accessing or changing it. The threadExample uses this approach. In your oF app, your main thread could update an image in the ImagerSaver. Once an image has been updated, the ImagerSaver thread could then save the image to disk while the main thread works on the next cycle. The ImageSaver thread would then wait (sleep) until the ImageSaver has been updated with another new image for it to save.

ofThread::startThread() and ofThread::stopThread() will start or stop a thread without creating or destroying it. These functions incur some overhead each time they are called. Using an approach like the threadExample (with a std::mutex and a std::condition_variable) seems to incur fewer resources as a single thread continues to actively run or “sleep”, and it doesn’t have to start again after its been stopped.

I made it work! Though I’m not sure if this is the right way to do this, as I’m a C++ ignorant person, and deciphering the thread examples is not easy at all. Still, I’m saving images for the frames that change without stalling the main thread. There is room for improvement though.
Here’s my ImageSaver.h file:

#pragma once
#include "ofMain.h"

class ImageSaver: public ofThread {
        public:
                ImageSaver();
                ~ImageSaver();

                void update(ofPixels & pixels, unsigned long ndx);

        private:  
                void threadedFunction();
                ofThreadChannel<ofPixels> threadedPixels;
                ofImage image;
                string fileName;
                unsigned long index;
};

Here’s my ImageSaver.cpp file:

#include "ImageSaver.h"

ImageSaver::ImageSaver() {
        startThread();
}

ImageSaver::~ImageSaver() {
        threadedPixels.close();
        waitForThread(true);
}

void ImageSaver::threadedFunction() {
        ofPixels pixels;
        while(threadedPixels.receive(pixels)) {
                image.setFromPixels(pixels);
                image.save(fileName);
        }
}

void ImageSaver::update(ofPixels & pixels, unsigned long ndx) {
        this->index = ndx;
        this->fileName = "3dPdCodeFrame" + ofToString(ndx) + ".jpg";
        threadedPixels.send(pixels);
}

As you suggested, I’m doing all the drawing in an FBO in update() and in draw() I’m saving an image whenever a receive a new character from the keyboard via OSC. Here’s the draw() function:

void ofApp::draw(){
        // stuff for saving frames as images
        frameCounter++;
        if (captureScreen) {
                fbo.readToPixels(pixel);
                imgSaver.update(pixel, frameCounter);
                captureScreen = false;
        }
        fbo.draw(0, 0);
}

imgSaver is an object of the ImageSaver class (obviously, I guess).
It’s not optimal as some frames can be lost if frames change quickly, so I’m sure there’s something I’m not doing right. I guess it hass to do with the ofImage object inside the ImageSaver class, but I couldn’t create a ofThreadChannel<ofImage> threadedImages as I was getting errors with the setFromPixels() and save() methods.
This works for now, but I would appreciate any advice for optimizing it. Thanks for the help so far!

Hey fantastic that its working a bit better! And I think the ofThreadChannel is a nice choice for this project too! The ofFbo allows the rendering to become an object, so that the timing of grabbing the pixels from the screen isn’t as critical. And I like how you’re passing large objects by reference, like with ImageSaver::update() for instance.

Also thanks for posting your code! Its so helpful and makes sharing and testing so much easier.

Maybe have a look at the code for ofxThreadedImageLoader addon (included with oF in /addons) as it is somewhat similar. It loads images from a container (a std::map with instances of a struct), and it uses two ofThreadChannel to accomplish its tasks.

The ImageSaver might benefit from the ability to store multiple images for its thread to save, so that it doesn’t miss one (or more) incoming images when its busy saving the current one. Storing images in a std::vector (or std::deque) might work well, maybe with the help of another ofThreadChannel. And the image (or pixels) could become a struct (or a class), which would combine multiple data types into one object (ofPixles or ofImage, index, filename, etc). Thus each image would know its index value, file name, etc.

The slow step of the ImageSaver::threadedFunction() is saving the image. The ImageSaver thread doesn’t necessarily need to mange the ImageSaver image, or any of the other class members, within the threadedfunction(). So I’m thinking the main thread could use an ofThreadChannel to update a vector in the ImageSaver with a new image (struct), while the threadedFunction() is independently saving images from the same vector with another ofThreadChannel.

Hey this was fun to work on, and useful too! It works well for either a 1-off rendering, or for a sequence (every 3rd rendering in the code below). The ImageSaver thread starts and stops depending on if there are pixels for it to save.

A std::deque stores the pixels until the ImageSaver thread can save them. The deque (supposedly) doesn’t move around in memory like a std::vector can, which is a huge advantage. I’m thinking the deque should be relatively “thread-safe” as long as the two threads work on different elements. In the project below, the ImageSaver thread saves (then pops) the first element, while the main thread adds new pixels after the last element.

Condensed but complete code that should compile:
ofApp.h:

#pragma once
#include "ofMain.h"
//--------------------------------------------------------------
struct Pixels{
    ofPixels pixels;
    std::string fileName;
};
//--------------------------------------------------------------
class ImageSaver : public ofThread {
public:
    ~ImageSaver() {waitForThread(true);}

    void threadedFunction() {
        while(isThreadRunning()) {
            if(pixelsQueue.size() > 0) {
                ofSaveImage(pixelsQueue.at(0).pixels, pixelsQueue.at(0).fileName);
                pixelsQueue.pop_front();
            }
        }
    }

    void updateThreadStatus() {
        if(pixelsQueue.size() > 0 && !isThreadRunning()) {startThread();}
        else if(pixelsQueue.size() == 0 && isThreadRunning()) {stopThread();}
    }

    void pushBackPixelsQueue(Pixels& pix) {pixelsQueue.push_back(pix);}

protected:
    std::deque<Pixels> pixelsQueue;
};
//--------------------------------------------------------------
class ofApp : public ofBaseApp{
public:
    void setup();
    void update();
    void draw();
    void keyPressed(int key);

    ImageSaver imageSaver;
    ofFbo fbo;
    bool isImageSaverActive;
    const std::string filePath = "/home/chilina/oFImages/threadTest/";
    const std::string fileExtension = ".jpg";
};

ofApp.cpp

#include "ofApp.h"
//--------------------------------------------------------------
void ofApp::setup(){
    isImageSaverActive = false;
    fbo.allocate(ofGetWidth(), ofGetHeight(), GL_RGBA);
    fbo.begin();
    ofBackground(ofColor(0));
    fbo.end();
}
//--------------------------------------------------------------
void ofApp::update(){
    int x = ofRandom(fbo.getWidth());
    int y = ofRandom(fbo.getHeight());
    int length = ofRandom(10, 100);
    ofColor color = ofColor(ofRandom(255), 0, ofRandom(127), ofRandom(50, 200));

    fbo.begin();
    ofSetColor(color);
    ofDrawRectangle(x, y, length, length);
    fbo.end();

    // save every 3rd rendering
    if(isImageSaverActive && ofGetFrameNum() % 3 == 0)
    {
        Pixels pix;
        fbo.readToPixels(pix.pixels);
        pix.pixels.setImageType(OF_IMAGE_COLOR);
        pix.fileName = ofToString(ofGetFrameNum()) + ".jpg";

        imageSaver.pushBackPixelsQueue(pix);
        // uncomment to just save 1 image per keystroke
        // isImageSaverActive = false;
    }    

    imageSaver.updateThreadStatus();
}
//--------------------------------------------------------------
void ofApp::draw(){
    fbo.draw(0.f, 0.f);
    ofSetColor(ofColor(255));
    ofDrawBitmapString("frameRate:" + ofToString(ofGetFrameRate()), 20.f, 20.f);
    if(isImageSaverActive) {ofDrawBitmapString("'s' to stop the ImageSaver", 20.f, 40.f);}
    else {ofDrawBitmapString("'s' to start imageSaver", 20.f, 40.f);}
}
//--------------------------------------------------------------
void ofApp::keyPressed(int key){
    if(key == 's') {isImageSaverActive = !isImageSaverActive;}
}
1 Like

This works great! Thanks a lot for providing this code! I will adapt it to my project and come back here with results. It’s much better than my code!

Cheers!

Hey your code looked great! I tried a variation of it from scratch and used the ofThreadChannel in a similar way. The main thread slowed quite a bit when saving frames in rapid succession. So then I tried the deque as a buffer, but had no idea about the fact that it doesn’t move around in memory like a vector. Because of the Pi, you may want to add some checks and limits for max size of the deque, depending on how much memory you have to work with. Have fun!

I tried your code both on my laptop (which works fine) and the Pi. On the Pi I had to increase the GPU memory to 512 in order for your program to run without problems. When I plug your code into my project though, it crashes after a while, with one of the following messages:

[ fatal ] ofThreadErrorLogger::exception: std::bad_alloc
[ fatal ] ofThreadErrorLogger::exception: basic_string::_M_create

Running the program with gdb I got this:

[New Thread 0x711ff050 (LWP 3604)]
[ fatal ] ofThreadErrorLogger::exception: basic_string::_M_create
[New Thread 0x70821050 (LWP 3605)]
[Thread 0x711ff050 (LWP 3604) exited]

Thread 29 "3dPdModularCode" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x70821050 (LWP 3605)]
0x76fb9c1c in memcpy () from /usr/lib/arm-linux-gnueabihf/libarmmem-v7l.s

and the program crahsed. I’m not using your class in a different way than you do in your code. Here’s the bit from ofApp::update() that stores the images:

fbo.begin();
drawText();
fbo.end();

if (captureScreen) {
        Pixels pix;
        fbo.readToPixels(pix.pixels);
        pix.pixels.setImageType(OF_IMAGE_COLOR);
        pix.fileName = "3dPdModularCodeFrame" + ofToString(frameNum) + ".jpg";

        imgSaver.pushBackPixelsQueue(pix);
        captureScreen = false;
}
imgSaver.updateThreadStatus();

captureScreen is set to true from a received OSC message that contains the frame number, stored in frameNum. I don’t really know what’s wrong, and searching this forum or the internet doesn’t really help. I would appreciate if anyone can provide some information.

Oh dang! It didn’t crash on me, but I didn’t run it for very long, nor on a Pi.

Maybe see if it continues to crash if you remove the ImageSaver::updateThreadStatus() and related code. If it doesn’t, there’s probably a way to let the thread sleep when it doesn’t have anything to save from the queue. If it does, then its probably necessary to lock the deque (mutex or using ofThreadChannel())

The code you posted with ofThreadChannel() will work for saving individual images, but the main thread will probably slow with a rapid succession of images.

You could also try and modify the ofxThreadedImageSaver addon to save images instead of (or in addition to) loading them.

Also here is a new ImageSaver class to try. The thread may have been sleeping in the while loop after .stopThread() was called. I added a function stop() and a std::condition_variable to the class.

class ImageSaver : public ofThread {
public:
    ~ImageSaver() {waitForThread(true);}

    inline size_t getPixelsQueueSize() {return pixelsQueue.size();}

    void threadedFunction() {
        while(isThreadRunning()) {
            if(pixelsQueue.size() > 0) {
                ofSaveImage(pixelsQueue.at(0).pixels, pixelsQueue.at(0).fileName);
                pixelsQueue.pop_front();
            }
        }
    }

    void stop()
    {
        std::unique_lock<std::mutex> lock(mutex);
        stopThread();
        condition.notify_all();
    }

    void updateThreadStatus() {
        if(pixelsQueue.size() > 0 && !isThreadRunning()) {startThread();}
        else if(pixelsQueue.size() == 0 && isThreadRunning()) {stop();}
    }

    void pushBackPixelsQueue(Pixels& pix) {pixelsQueue.push_back(pix);}

protected:
    std::deque<Pixels> pixelsQueue;
    std::condition_variable condition;
};

This doesn’t work either. It’s way harder than I thought… ofxRPiCameraVideoGrabber looks like it can do something close to what I want, but it gets input from a camera connected to a Pi. I’m reading it’s source code trying to understand how I can edit the recording example, but I can’t really find it. If the video grabber gets pixels from the camera, then I should probably be able to load the pixels from the FBO to it, instead of a camera, and then call startRecording().
Thanks for the help so far!

,

Hey modifying something like ofxRPICameraVideoGrabber is a nice way to go if you can make it work! Sometimes people incorporate their successes into a new addon, or become contributors to the ones they’ve modified.

I had 1 more go and the code is below. I blew the dust off my RPi3 to test it (oF 0.11.2 / Buster / legacy driver / legacy oF compile / 128M gpu). It didn’t crash, but this code could use more extensive testing, and better code to start/stop the thread.

This version uses 2 deques. The main thread writes to one (“queued”), while the ImageSaver thread reads from the other (“threaded”) to save the files. The mutex locks both deques (I hope) when the ImageSaver thread needs more images to save. std::swap() is used to swap the deques. An int num determines the number of elements needed before the deques are swapped. The ImageSaver thread seems to start and stop correctly (via keyboard input).

Complete code that should compile and (hopefully) run without crashing on an RPi:

ofApp.h:

#pragma once
#include "ofMain.h"
//--------------------------------------------------------------
struct Pixels{
    ofPixels pixels;
    std::string fileName;
};
//--------------------------------------------------------------
class ImageSaver : public ofThread {
public:
    ~ImageSaver() {waitForThread(true);}

    inline void setNum(size_t i) {num = i;}
    inline void setIsQueuedActive(bool b) {isQueuedActive = b;}
    inline void setIsThreadedActive(bool b) {isThreadedActive = b;}
    inline bool getIsQueuedActive() {return isQueuedActive;}
    inline bool getIsThreadedActive() {return isThreadedActive;}
    inline size_t getQueuedSize() {return queued.size();}
    inline size_t getThreadedSize() {return threaded.size();}

    void threadedFunction() {
        while(isThreadRunning()) {
            if(threaded.size() > 0) {
                ofSaveImage(threaded.back().pixels, threaded.back().fileName);
                threaded.pop_back();
            }

            if(threaded.size() == 0 && queued.size() > num) {
                std::unique_lock<std::mutex> lock(mutex);
                std::swap(queued, threaded);
                condition.notify_all();
            }
        }
    }

    void stop() {
        std::unique_lock<std::mutex> lock(mutex);
        stopThread();
        condition.notify_all();
    }

    void pushBackQueued(Pixels& pix) {
        std::unique_lock<std::mutex> lock(mutex);
        queued.push_back(pix);
        condition.notify_all();
    }

    void checkThreadStatus() {
        if(isThreadedActive && !isThreadRunning()) {startThread();}
        else if(!isThreadedActive && isThreadRunning()) {stop();}
    }

protected:
    std::deque<Pixels> queued;
    std::deque<Pixels> threaded;
    size_t num;
    bool isQueuedActive;
    bool isThreadedActive;
    std::condition_variable condition;
};
//--------------------------------------------------------------
class ofApp : public ofBaseApp{
public:
    void setup();
    void update();
    void draw();
    void keyPressed(int key);

    ImageSaver imageSaver;
    ofFbo fbo;
    const std::string filePath = "";
    const std::string fileExtension = ".jpg";
};

ofApp.cpp:

#include "ofApp.h"

//--------------------------------------------------------------
void ofApp::setup(){
    imageSaver.setNum(8);
    imageSaver.setIsQueuedActive(false);
    imageSaver.setIsThreadedActive(false);
    fbo.allocate(ofGetWidth(), ofGetHeight(), GL_RGBA);
    fbo.begin();
    ofBackground(ofColor(0));
    fbo.end();
}
//--------------------------------------------------------------
void ofApp::update(){
    int x = ofRandom(fbo.getWidth());
    int y = ofRandom(fbo.getHeight());
    int length = ofRandom(10, 100);
    ofColor color = ofColor(ofRandom(255), 0, ofRandom(127), ofRandom(50, 200));

    fbo.begin();
    ofSetColor(color);
    ofDrawRectangle(x, y, length, length);
    fbo.end();

    // save every 6th rendering
    if(imageSaver.getIsQueuedActive() && ofGetFrameNum() % 6 == 0)
    {
        Pixels pix;
        fbo.readToPixels(pix.pixels);
        pix.pixels.setImageType(OF_IMAGE_COLOR);
        std::string fileName = filePath;
        fileName += ofToString(ofGetFrameNum());
        fileName += ".jpg";
        pix.fileName = fileName;

        imageSaver.pushBackQueued(pix);
    }    
}
//--------------------------------------------------------------
void ofApp::draw(){
    fbo.draw(0.f, 0.f);
    ofSetColor(ofColor(255));
    ofDrawBitmapString("frameRate:" + ofToString(ofGetFrameRate()), 20.f, 20.f);
    if(imageSaver.getIsQueuedActive()) {ofDrawBitmapString("'q' to stop queue", 20.f, 40.f);}
    else {ofDrawBitmapString("'q' to start queue", 20.f, 40.f);}
    if(imageSaver.getIsThreadedActive()) {ofDrawBitmapString("'t' to stop thread", 20.f, 60.f);}
    else {ofDrawBitmapString("'t' to start thread", 20.f, 60.f);}
    ofDrawBitmapString("queue size: " + ofToString(imageSaver.getQueuedSize()), 20.f, 80.f);
    ofDrawBitmapString("threaded size: " + ofToString(imageSaver.getThreadedSize()), 20.f, 100.f);
    ofDrawBitmapString("isThreadRunning: " + ofToString(imageSaver.isThreadRunning()), 20.f, 120.f);
}
//--------------------------------------------------------------
void ofApp::keyPressed(int key){
    if(key == 'q')
    {
        imageSaver.setIsQueuedActive(!imageSaver.getIsQueuedActive());
    }
    else if(key == 't')
    {
        imageSaver.setIsThreadedActive(!imageSaver.getIsThreadedActive());
        imageSaver.checkThreadStatus();
    }
}
1 Like

This does not stall or crash the program! Maybe some frames are lost when characters arrive fast, but that is sort of compensated by saving a later frame, which will make the resulting assembled video a bit jerky in some spots, but that’s really not a problem. Thank you very much for the help and code!