threaded image loader

[prisonerjohn]

Hello,

I'm building an app which loads images from a folder and displays them one at a time every minute, kind of like a slideshow, but there's a bunch of other stuff happening at the same time.

There's going to be about 50 large images, so I don't want to load them all in memory at startup. I built it so that the next image is loaded right before it should be displayed, but this freezes the entire application for half a second until ofImage.loadImage(...) is done.

So I thought of using the ofxThread addon to load the images, but this is causing my app to crash. I ran it through the debugger, and the problem is that when I call ofImage.loadImage(...), it calls ofTexture.clear(), which calls glDeleteTextures(...). I'm guessing that this is the problem since it's a GL call.

So............. what do I do? :)

Thanks

[memo]

Hi, yea the GL call is the problem. I had a similar situation with the videoGrabber.

If you want to load the image in a seperate thread you can call ofImage::setUseTexture(false) before loading the image (you can do this in the main thread, or the ofxThread - as long as you do it before the load). This will make sure the image doesn't get uploaded to the graphics card and an openGL texture will not be created. This means however, that you won't be able to draw the image! So after the image has finished loading, you need to create the texture yourself in the main thread. A method in the ofImage class would be nice... along the lines of ofImage::createTexture(), but you can create one quite easily.. I think you can just copy paste the code from the update() function...

[theo]

Zach and I did something similar for the liners project - where we needed to dynamically load videos while the app was running without any pause.
http://openframeworks.cc/liners/

As you suggested a loading image thread is a very good approach to solving this problem. It can still be a little tricky though as there are two parts that could cause a pause.

1) When ofImage is loading the pixels into memory from disk. - This can easily be threaded.

2) When the pixels are loaded into an openGL texture. This can't be threaded as openGL does not support multi threading.

I was working on a project that had to dynamically load aprox 900 10 megapixel images and the main problem was with 2) trying to get the data onto the graphics card without affecting the framerate. This can end up being very tricky, requiring splitting up the texture upload over multiple frames, so you are never sending too many pixels at once. It is also a massive pain.

I would suggest first finding out where the pause is occurring if it is in step 1 or step 2.

You could do this by first splitting up the loading of the image and the texture upload as memo suggests. You can use a seperate ofTexture object and load the pixels into it with ofImage::getPixels.

I would hook up the loading of the image to one key press and the texture upload to another key press - and then press them in turn while displaying the framerate or even better some sort of moving graphic. You should be able to easily see the effect of each, on your apps performance.

If the pause is mainly in 1) then I would go ahead with the threading approach.

If it is also in 2) then I would suggest trying to load all the images into memory when the app first launches. As long as you have a decent amount of RAM and a NVIDIA or ATI graphics card you should be able to load it all into memory. This might mean the app takes a few seconds to start but once it is loaded it should run solid and fast! If that is not possible I can point you towards some tricks for 'trickling' the images into a texture - but that is a massive pain and would be great to avoid!

Hope that helps,
theo

[zach]

A method in the ofImage class would be nice... along the lines of ofImage::createTexture(),

sure!  and since people are starting to do threading, it might be good to take a look at allowing advanced users to continue to create and use textures but called from the opengl thread.  Perhaps setUseTexture(false) could be replaced with something like:

Code:

setInternalTextureMode(OF_TEXTURE_MODE_MANUAL) setInternalTextureMode(OF_TEXTURE_MODE_NONE)

or something like that, so that thread folks can do the texture stuff on their own.  what do you think?

take care!
zach

[prisonerjohn]

Thanks for all the replies!

I turned off using textures for the ofImage and it doesn't crash anymore but i still get that freeze problem. I guess it's because of problem #2, but just to be sure, here's my code:

testApp.h

Code:

#ifndef _TEST_APP
#define _TEST_APP


#include "ofMain.h"
#include "imgLoader.h"

class testApp : public ofSimpleApp {
	
	public:		
		void setup();
		void update();
		void draw();
		
		void keyPressed  (int key);
		void mouseMoved(int x, int y );
		void mouseDragged(int x, int y, int button);
		void mousePressed(int x, int y, int button);
		void mouseReleased();
		
        imgLoader   loader;
        ofImage     currImg;
    
        float       currX;
        float       dirX;
        float       speedX;
};

#endif	

testApp.cpp

Code:

#include "testApp.h"

//--------------------------------------------------------------
void testApp::setup() {	 
	loader.start();
    ofSetVerticalSync(true);
    
    currX = 0;
    dirX = 1;
    speedX = 5;
}

//--------------------------------------------------------------
void testApp::update() {	
	ofBackground(0,0,0);   // black because threads are EVIL ;)
    
    if (dirX == 1) {
        currX += (ofGetWidth()-currX)/speedX;
    } else {
        currX -= (currX/speedX);
    }
    
    if (currX+5 > ofGetWidth())
        dirX = -1;
    if (currX < 5)
        dirX = 1;
    
    if (ofGetFrameNum()%300 == 0) {
        currImg = loader.getNextTexture();
        loader.goForIt();
    }
}

//--------------------------------------------------------------
void testApp::draw() {
    currImg.draw(0, 0, ofGetWidth(), ofGetHeight());
    
    ofCircle(currX, ofGetHeight()/2, 50);
}

//--------------------------------------------------------------
void testApp::keyPressed(int key) { 
    /*
    if (key == 'a'){
        TO.start();   
    } else if (key == 's'){
        TO.stop();
    }
    */
}

//--------------------------------------------------------------
void testApp::mouseMoved(int x, int y ) {
}

//--------------------------------------------------------------
void testApp::mouseDragged(int x, int y, int button) {
}

//--------------------------------------------------------------
void testApp::mousePressed(int x, int y, int button) {	
}

//--------------------------------------------------------------
void testApp::mouseReleased() {
}

imgLoader.h

Code:

#ifndef _TEXTURELOADER
#define _TEXTURELOADER

#include "ofMain.h"
#define OF_ADDON_USING_OFXDIRLIST
#define OF_ADDON_USING_OFXTHREAD
#include "ofAddons.h"

class imgLoader : public ofxThread {    
    public:
                imgLoader();
                ~imgLoader();
    
        void    start();
        void    stop();
        void    threadedFunction();
        
        void    fetchTextureNames();
        void    loadNextTexture();
        ofImage getNextTexture();
        void    goForIt();
        
    private:
        ofxDirList  dirList;
        string*     textures;
        int         numTextures;
        int         textureIndex;
        ofImage     nextTexture;
        bool        go;
    
};

#endif

imgLoader.cpp

Code:

#include "imgLoader.h"

//--------------------------------------------------------------
imgLoader::imgLoader() {
    go = false;
    nextTexture.setUseTexture(false);
    dirList.setVerbose(false);
    fetchTextureNames();
    
    start();
}

//--------------------------------------------------------------
imgLoader::~imgLoader() {	
    delete [] textures;
    textures = NULL;
}

//--------------------------------------------------------------
void imgLoader::fetchTextureNames() {
    // fetch all the texture file names
    dirList.reset();
    dirList.allowExt("jpg");
    numTextures = dirList.listDir("textures/");
    textures = new string[numTextures];
    for (int i=0; i < numTextures; i++) {
        textures[i] = "textures/"+dirList.getName(i);
    }
    
    // load the first texture
    textureIndex = -1;
    loadNextTexture();
}

//--------------------------------------------------------------
void imgLoader::loadNextTexture() {
    textureIndex = (textureIndex+1)%numTextures;
    nextTexture.loadImage(textures[textureIndex]);
    printf("Loading image %d: %s\n", textureIndex, textures[textureIndex].c_str());
    go = false;
}

//--------------------------------------------------------------
ofImage imgLoader::getNextTexture() {
    return nextTexture;
}

//--------------------------------------------------------------
void imgLoader::goForIt() {
    go = true;
}

//--------------------------------------------------------------
void imgLoader::start() {
    startThread(true, false);  // blocking, verbose
}

//--------------------------------------------------------------
void imgLoader::stop() {
    stopThread(); 
}

//--------------------------------------------------------------
void imgLoader::threadedFunction() {
    while (isThreadRunning() != 0) {
        if (lock()) {
            if (go) {
                loadNextTexture();
            }
            unlock();
            ofSleepMillis(1 * 1000);
        }
    }
}

Would you guys mind just having a quick look at it and making sure it's correct? I want to make sure it's not some dumb mistake before I go ahead and load all the images at startup (I will be loading about 60 4MB JPGs). I can also provide the project file (in Xcode) if it will make it easier...

Thanks!

[ding]

I don't mean to highjack the thread but I think this would be a great subject for a tutorial in the wiki.  I really don't understand it much and I think it is important when working with large amounts of data to understand how to fix your app from slowing down by using threading.

sorry for the highjack  :D

ding

[otherside]

2) When the pixels are loaded into an openGL texture. This can't be threaded as openGL does not support multi threading.

This isn't entirely true.  There's a technique for multithreading OpenGL using shared contexts that seems to work on all platforms I've tried (X-windows, windows, OSX).  The basics are:

1) Allocate a root opengl context
2) For each window, allocate shared "child" contexts
3) For each thread, allocate shared "child" contexts

When loading something like an image in another thread, use that thread's context to set the texture.  When done, the texture memory can be used directly in the main thread via the fact that the contexts are shared.

I'm not sure if you can do this in GLUT though.  Perhaps it's possible to get the root OS handles to the context and do the above steps.  I don't know.

[memo]

Hi,

Firstly I think you should definitely try what theo is suggesting. Forget about the the threads for now, and write two functions in the main thread. One that loads the image from disk with setUseTexture(false), and the other function creates an ofTexture out of the loaded image, then map them to the keyboard and see exactly how much of an effect each one has, that way any question marks will be erased as to what is the bottle neck. If the second function freezes the computer a bit - then you need to think of alternative solutions.  I'm guessing the 4MB disk loading will also take its toll as well so multithreading will still be useful for you.

Not having run your code, but looking at it, it looks correct - and the same solution I went for for my problem (having a flag in the threadedFunction loop to determine whether to run the thread update or not).

The freeze you're getting cannot be due to problem #2 because nowhere in your code are you actually creating a texture and sending it to the graphics card. In fact I'd be quite surprised if you are seeing your image onscreen as the currImg.draw () should just return doing nothing! Unless you modified it of course to create the texture and draw - in which case it could be the reason for the freeze (only way to be sure is the test mentioned above).


The tricky thing is going to be signaling the main thread that the image has finished loading and its ok to create the texture and display it. I ended up using posix condition variables - and was quite a nightmare understanding how it all links together - will post on that when I'm more confident.

But I think in your case the best solution, is for your main thread (the update loop) to check a loaded variable in the thread class every frame. But of course to be able to do that you need to put a lock on first from the main thread (the update() or draw() function) in order to check the variable in your imgLoader class. But if your loading thread is loading the image, it will have put a lock on, so your main thread will be blocked and waiting for the image to load - i.e. your main thread will be blocked and waiting for your loading thread to finish, defeating the whole point of your having a 2nd thread!

So in short you need to start the ofxThread non-blocked. So the if(lock()) statement will not pause the execution of your main thread, but will just return a fail - and thats what you want I think. I'm quite new to this whole multithreading thing too so I may be wrong! but thats my understanding of the situation!

[arturo]

Hi although your code should work, I think it's not very correct. lock and unlock should be used in more than one place, if not they are useless. When working with blocking (you are) and you call lock in one function, if any other function calls it, it gets sleeped since the first one calls unlock.

Also the ofSleepMillis(1 * 1000); in threadedFunction doesn't make sense.

For this, I think you can go without locks at all:

imgLoader.h:

Code:

ofImage * currTexture;
ofImage * nextTexture;
bool imgLoaded;

imgLoader.cpp:

Code:

imgLoader::imgLoader() {
    imgLoaded = false;
    dirList.setVerbose(false);
    fetchTextureNames();
    currTexture=NULL;
    nextTexture=NULL;
    start();
} 

void imgLoader::loadNextTexture() {
        textureIndex = (textureIndex+1)%numTextures;
        nextTexture = new ofImage();
        nextTexture->useTexture(false);
        nextTexture->loadImage(textures[textureIndex]);
        printf("Loading image %d: %s\n", textureIndex,  textures[textureIndex].c_str());
        imgLoaded=true;
} 

ofImage * imgLoader::getNextTexture() {
    if(imgLoaded){
        if(currTexture)
            delete currTexture;
        currTexture=nextTexture;
        imgLoaded=false
        return currTexture;
    }else{
        return NULL;
    }
}

//--------------------------------------------------------------
void imgLoader::threadedFunction() {
    while (isThreadRunning() != 0) {
          if (!imgLoaded) {
               loadNextTexture();
          }
    }
}
 

testApp.h:

Code:

ofImage * currImage;
bool          loadNextImg;

testApp.cpp:

Code:

void testApp::setup() {   
   loader.start();
    ofSetVerticalSync(true);
   
    currX = 0;
    dirX = 1;
    speedX = 5;
    currImg=NULL;
    loadNextImg=true;
} 

void testApp::update(){
...
if (ofGetFrameNum()%300 == 0) 
     loadNextImg=true;

if(loadNextImg){
        ofImage * nextImg = loader.getNextTexture();
        if(nextImg!=NULL){
              currImg=nextImg; 
              loadNextImg=false;
        }
}
}

void testApp::draw{
    if(currImg)
        currImg->draw();
    ...
}

I haven't tested it so perhaps it has some bug.

The problem with this solution is that the threadedFunction is running the loop although the texture is already loaded but is not very time consuming as it does almost nothing.

[CarlesGutierrez]

[CarlesGutierrez]

I've changed main form to show the images. Now using an array with all the images loadeds. There are a thread copying the images to this array, and the imageLoader, have this array as public , the variables textureIndext and numTexture too.

This modifications helps to control the amount of images loaded by the thread, and control then when it's possible to show them.

If anyone have interest of it , I can open another topic with this kind of "video streaming"

bye

[AlexandreRangel]

If anyone have interest of it , I can open another topic with this kind of "video streaming"

I would love to see that, as I´m trying to solve a freeze when I change directories on my app and have to load some 50 video files at once. I think threads would solve that.
I already have function to load the videos. All that´s needed, as I see, is the threads startup and finish code "around" it.

[CarlesGutierrez]

Hi Alexandre,
 
 I´m not doing yet changes of the data path, but try it, this solution should be enough to avoid the freeze...

Hope this help you. I´ve post it here

[AlexandreRangel]

Hi Carles, thank you! I commented it in there.