threaded image loader

Hello,

I’m building an app which loads images from a folder and displays them one at a time every minute, kind of like a slideshow, but there’s a bunch of other stuff happening at the same time.

There’s going to be about 50 large images, so I don’t want to load them all in memory at startup. I built it so that the next image is loaded right before it should be displayed, but this freezes the entire application for half a second until ofImage.loadImage(…) is done.

So I thought of using the ofxThread addon to load the images, but this is causing my app to crash. I ran it through the debugger, and the problem is that when I call ofImage.loadImage(…), it calls ofTexture.clear(), which calls glDeleteTextures(…). I’m guessing that this is the problem since it’s a GL call.

So… what do I do? :slight_smile:

Thanks

1 Like

Hi, yea the GL call is the problem. I had a similar situation with the videoGrabber.

If you want to load the image in a seperate thread you can call ofImage::setUseTexture(false) before loading the image (you can do this in the main thread, or the ofxThread - as long as you do it before the load). This will make sure the image doesn’t get uploaded to the graphics card and an openGL texture will not be created. This means however, that you won’t be able to draw the image! So after the image has finished loading, you need to create the texture yourself in the main thread. A method in the ofImage class would be nice… along the lines of ofImage::createTexture(), but you can create one quite easily… I think you can just copy paste the code from the update() function…

Zach and I did something similar for the liners project - where we needed to dynamically load videos while the app was running without any pause.
http://openframeworks.cc/liners/

As you suggested a loading image thread is a very good approach to solving this problem. It can still be a little tricky though as there are two parts that could cause a pause.

  1. When ofImage is loading the pixels into memory from disk. - This can easily be threaded.

  2. When the pixels are loaded into an openGL texture. This can’t be threaded as openGL does not support multi threading.

I was working on a project that had to dynamically load aprox 900 10 megapixel images and the main problem was with 2) trying to get the data onto the graphics card without affecting the framerate. This can end up being very tricky, requiring splitting up the texture upload over multiple frames, so you are never sending too many pixels at once. It is also a massive pain.

I would suggest first finding out where the pause is occurring if it is in step 1 or step 2.

You could do this by first splitting up the loading of the image and the texture upload as memo suggests. You can use a seperate ofTexture object and load the pixels into it with ofImage::getPixels.

I would hook up the loading of the image to one key press and the texture upload to another key press - and then press them in turn while displaying the framerate or even better some sort of moving graphic. You should be able to easily see the effect of each, on your apps performance.

If the pause is mainly in 1) then I would go ahead with the threading approach.

If it is also in 2) then I would suggest trying to load all the images into memory when the app first launches. As long as you have a decent amount of RAM and a NVIDIA or ATI graphics card you should be able to load it all into memory. This might mean the app takes a few seconds to start but once it is loaded it should run solid and fast! If that is not possible I can point you towards some tricks for ‘trickling’ the images into a texture - but that is a massive pain and would be great to avoid!

Hope that helps,
theo

A method in the ofImage class would be nice… along the lines of ofImage::createTexture(),

sure! and since people are starting to do threading, it might be good to take a look at allowing advanced users to continue to create and use textures but called from the opengl thread. Perhaps setUseTexture(false) could be replaced with something like:

  
  
setInternalTextureMode(OF_TEXTURE_MODE_MANUAL) setInternalTextureMode(OF_TEXTURE_MODE_NONE)  
  

or something like that, so that thread folks can do the texture stuff on their own. what do you think?

take care!
zach

Thanks for all the replies!

I turned off using textures for the ofImage and it doesn’t crash anymore but i still get that freeze problem. I guess it’s because of problem #2, but just to be sure, here’s my code:

testApp.h

  
  
#ifndef _TEST_APP  
#define _TEST_APP  
  
  
#include "ofMain.h"  
#include "imgLoader.h"  
  
class testApp : public ofSimpleApp {  
	  
	public:		  
		void setup();  
		void update();  
		void draw();  
		  
		void keyPressed  (int key);  
		void mouseMoved(int x, int y );  
		void mouseDragged(int x, int y, int button);  
		void mousePressed(int x, int y, int button);  
		void mouseReleased();  
		  
        imgLoader   loader;  
        ofImage     currImg;  
      
        float       currX;  
        float       dirX;  
        float       speedX;  
};  
  
#endif	  
  

testApp.cpp

  
  
#include "testApp.h"  
  
//--------------------------------------------------------------  
void testApp::setup() {	   
	loader.start();  
    ofSetVerticalSync(true);  
      
    currX = 0;  
    dirX = 1;  
    speedX = 5;  
}  
  
//--------------------------------------------------------------  
void testApp::update() {	  
	ofBackground(0,0,0);   // black because threads are EVIL ;)  
      
    if (dirX == 1) {  
        currX += (ofGetWidth()-currX)/speedX;  
    } else {  
        currX -= (currX/speedX);  
    }  
      
    if (currX+5 > ofGetWidth())  
        dirX = -1;  
    if (currX < 5)  
        dirX = 1;  
      
    if (ofGetFrameNum()%300 == 0) {  
        currImg = loader.getNextTexture();  
        loader.goForIt();  
    }  
}  
  
//--------------------------------------------------------------  
void testApp::draw() {  
    currImg.draw(0, 0, ofGetWidth(), ofGetHeight());  
      
    ofCircle(currX, ofGetHeight()/2, 50);  
}  
  
//--------------------------------------------------------------  
void testApp::keyPressed(int key) {   
    /*  
    if (key == 'a'){  
        TO.start();     
    } else if (key == 's'){  
        TO.stop();  
    }  
    */  
}  
  
//--------------------------------------------------------------  
void testApp::mouseMoved(int x, int y ) {  
}  
  
//--------------------------------------------------------------  
void testApp::mouseDragged(int x, int y, int button) {  
}  
  
//--------------------------------------------------------------  
void testApp::mousePressed(int x, int y, int button) {	  
}  
  
//--------------------------------------------------------------  
void testApp::mouseReleased() {  
}  
  

imgLoader.h

  
  
#ifndef _TEXTURELOADER  
#define _TEXTURELOADER  
  
#include "ofMain.h"  
#define OF_ADDON_USING_OFXDIRLIST  
#define OF_ADDON_USING_OFXTHREAD  
#include "ofAddons.h"  
  
class imgLoader : public ofxThread {      
    public:  
                imgLoader();  
                ~imgLoader();  
      
        void    start();  
        void    stop();  
        void    threadedFunction();  
          
        void    fetchTextureNames();  
        void    loadNextTexture();  
        ofImage getNextTexture();  
        void    goForIt();  
          
    private:  
        ofxDirList  dirList;  
        string*     textures;  
        int         numTextures;  
        int         textureIndex;  
        ofImage     nextTexture;  
        bool        go;  
      
};  
  
#endif  
  

imgLoader.cpp

  
  
#include "imgLoader.h"  
  
//--------------------------------------------------------------  
imgLoader::imgLoader() {  
    go = false;  
    nextTexture.setUseTexture(false);  
    dirList.setVerbose(false);  
    fetchTextureNames();  
      
    start();  
}  
  
//--------------------------------------------------------------  
imgLoader::~imgLoader() {	  
    delete [] textures;  
    textures = NULL;  
}  
  
//--------------------------------------------------------------  
void imgLoader::fetchTextureNames() {  
    // fetch all the texture file names  
    dirList.reset();  
    dirList.allowExt("jpg");  
    numTextures = dirList.listDir("textures/");  
    textures = new string[numTextures];  
    for (int i=0; i < numTextures; i++) {  
        textures[i] = "textures/"+dirList.getName(i);  
    }  
      
    // load the first texture  
    textureIndex = -1;  
    loadNextTexture();  
}  
  
//--------------------------------------------------------------  
void imgLoader::loadNextTexture() {  
    textureIndex = (textureIndex+1)%numTextures;  
    nextTexture.loadImage(textures[textureIndex]);  
    printf("Loading image %d: %s\n", textureIndex, textures[textureIndex].c_str());  
    go = false;  
}  
  
//--------------------------------------------------------------  
ofImage imgLoader::getNextTexture() {  
    return nextTexture;  
}  
  
//--------------------------------------------------------------  
void imgLoader::goForIt() {  
    go = true;  
}  
  
//--------------------------------------------------------------  
void imgLoader::start() {  
    startThread(true, false);  // blocking, verbose  
}  
  
//--------------------------------------------------------------  
void imgLoader::stop() {  
    stopThread();   
}  
  
//--------------------------------------------------------------  
void imgLoader::threadedFunction() {  
    while (isThreadRunning() != 0) {  
        if (lock()) {  
            if (go) {  
                loadNextTexture();  
            }  
            unlock();  
            ofSleepMillis(1 * 1000);  
        }  
    }  
}  
  

Would you guys mind just having a quick look at it and making sure it’s correct? I want to make sure it’s not some dumb mistake before I go ahead and load all the images at startup (I will be loading about 60 4MB JPGs). I can also provide the project file (in Xcode) if it will make it easier…

Thanks!

I don’t mean to highjack the thread but I think this would be a great subject for a tutorial in the wiki. I really don’t understand it much and I think it is important when working with large amounts of data to understand how to fix your app from slowing down by using threading.

sorry for the highjack :smiley:

ding

  1. When the pixels are loaded into an openGL texture. This can’t be threaded as openGL does not support multi threading.

This isn’t entirely true. There’s a technique for multithreading OpenGL using shared contexts that seems to work on all platforms I’ve tried (X-windows, windows, OSX). The basics are:

  1. Allocate a root opengl context
  2. For each window, allocate shared “child” contexts
  3. For each thread, allocate shared “child” contexts

When loading something like an image in another thread, use that thread’s context to set the texture. When done, the texture memory can be used directly in the main thread via the fact that the contexts are shared.

I’m not sure if you can do this in GLUT though. Perhaps it’s possible to get the root OS handles to the context and do the above steps. I don’t know.

Hi,

Firstly I think you should definitely try what theo is suggesting. Forget about the the threads for now, and write two functions in the main thread. One that loads the image from disk with setUseTexture(false), and the other function creates an ofTexture out of the loaded image, then map them to the keyboard and see exactly how much of an effect each one has, that way any question marks will be erased as to what is the bottle neck. If the second function freezes the computer a bit - then you need to think of alternative solutions. I’m guessing the 4MB disk loading will also take its toll as well so multithreading will still be useful for you.

Not having run your code, but looking at it, it looks correct - and the same solution I went for for my problem (having a flag in the threadedFunction loop to determine whether to run the thread update or not).

The freeze you’re getting cannot be due to problem #2 because nowhere in your code are you actually creating a texture and sending it to the graphics card. In fact I’d be quite surprised if you are seeing your image onscreen as the currImg.draw () should just return doing nothing! Unless you modified it of course to create the texture and draw - in which case it could be the reason for the freeze (only way to be sure is the test mentioned above).

The tricky thing is going to be signaling the main thread that the image has finished loading and its ok to create the texture and display it. I ended up using posix condition variables - and was quite a nightmare understanding how it all links together - will post on that when I’m more confident.

But I think in your case the best solution, is for your main thread (the update loop) to check a loaded variable in the thread class every frame. But of course to be able to do that you need to put a lock on first from the main thread (the update() or draw() function) in order to check the variable in your imgLoader class. But if your loading thread is loading the image, it will have put a lock on, so your main thread will be blocked and waiting for the image to load - i.e. your main thread will be blocked and waiting for your loading thread to finish, defeating the whole point of your having a 2nd thread!

So in short you need to start the ofxThread non-blocked. So the if(lock()) statement will not pause the execution of your main thread, but will just return a fail - and thats what you want I think. I’m quite new to this whole multithreading thing too so I may be wrong! but thats my understanding of the situation!

Hi although your code should work, I think it’s not very correct. lock and unlock should be used in more than one place, if not they are useless. When working with blocking (you are) and you call lock in one function, if any other function calls it, it gets sleeped since the first one calls unlock.

Also the ofSleepMillis(1 * 1000); in threadedFunction doesn’t make sense.

For this, I think you can go without locks at all:

imgLoader.h:

  
  
ofImage * currTexture;  
ofImage * nextTexture;  
bool imgLoaded;  
  

imgLoader.cpp:

  
  
imgLoader::imgLoader() {  
    imgLoaded = false;  
    dirList.setVerbose(false);  
    fetchTextureNames();  
    currTexture=NULL;  
    nextTexture=NULL;  
    start();  
}   
  
void imgLoader::loadNextTexture() {  
        textureIndex = (textureIndex+1)%numTextures;  
        nextTexture = new ofImage();  
        nextTexture->useTexture(false);  
        nextTexture->loadImage(textures[textureIndex]);  
        printf("Loading image %d: %s\n", textureIndex,  textures[textureIndex].c_str());  
        imgLoaded=true;  
}   
  
ofImage * imgLoader::getNextTexture() {  
    if(imgLoaded){  
        if(currTexture)  
            delete currTexture;  
        currTexture=nextTexture;  
        imgLoaded=false  
        return currTexture;  
    }else{  
        return NULL;  
    }  
}  
  
//--------------------------------------------------------------  
void imgLoader::threadedFunction() {  
    while (isThreadRunning() != 0) {  
          if (!imgLoaded) {  
               loadNextTexture();  
          }  
    }  
}  
   

testApp.h:

  
ofImage * currImage;  
bool          loadNextImg;  

testApp.cpp:

  
  
void testApp::setup() {     
   loader.start();  
    ofSetVerticalSync(true);  
     
    currX = 0;  
    dirX = 1;  
    speedX = 5;  
    currImg=NULL;  
    loadNextImg=true;  
}   
  
void testApp::update(){  
...  
if (ofGetFrameNum()%300 == 0)   
     loadNextImg=true;  
  
if(loadNextImg){  
        ofImage * nextImg = loader.getNextTexture();  
        if(nextImg!=NULL){  
              currImg=nextImg;   
              loadNextImg=false;  
        }  
}  
}  
  
void testApp::draw{  
    if(currImg)  
        currImg->draw();  
    ...  
}  
  

I haven’t tested it so perhaps it has some bug.

The problem with this solution is that the threadedFunction is running the loop although the texture is already loaded but is not very time consuming as it does almost nothing.

I’ve changed main form to show the images. Now using an array with all the images loadeds. There are a thread copying the images to this array, and the imageLoader, have this array as public , the variables textureIndext and numTexture too.

This modifications helps to control the amount of images loaded by the thread, and control then when it’s possible to show them.

If anyone have interest of it , I can open another topic with this kind of “video streaming”

bye

I would love to see that, as I´m trying to solve a freeze when I change directories on my app and have to load some 50 video files at once. I think threads would solve that.
I already have function to load the videos. All that´s needed, as I see, is the threads startup and finish code “around” it.

Hi Alexandre,

I´m not doing yet changes of the data path, but try it, this solution should be enough to avoid the freeze…

Hope this help you. I´ve post it here

Hi Carles, thank you! I commented it in there.

Hi,

After searching the OF forums for an answer, I thought this might be the best place to post my issue.

I’m trying to load 32 84x84 RGBA png images dynamically in realtime on iPhone. Ideally this needs to be done in realtime because otherwise I’d be storing hundreds of images in memory, which kills my loading time and prevents iOS from throwing memory warnings at me.

When I implemented this, it was taking way too long to load these images and stalling the main thread so I threaded the image loader (obviously using setTexture(false) to avoid GL issues) to load the images on an event (ideally within a few frames) and render them when the thread became unlocked. Unfortunately the threaded solution also took way too long to load the images; as mentioned, I need the loading to take no more than a few frames, not hundreds of milliseconds.

So, as a third approach, I created my images as 4 84x672 RGBA pngs in mind of using the ofImage cropFrom function to crop pixel from each 84x672 image into 8 84x84 images, four times (totalling my original 32 images). However, even loading these 4 images is taking roughly 300 milliseconds, plus an additional 200 milliseconds for the crop, which I presume is due to the internal allocations.

My motivation for going to the 4 image approach was to reduce the load time by loading less images (and they’re not particularly large ones at that) but it seems the time isn’t spent doing IO ops as I’d suspected but is purely limited by the combined image dimensions though, as I say, these images aren’t particularly large. I’m all out of ideas now to get the performance I need. Any suggestions on how to improve realtime image loading would be greatly appreciated.

Many thanks!

You might want to repost this in the IOS subsection.

Png’s are painfully slow on IOS. check here: http://www.cocoanetics.com/2011/10/avoiding-image-decompression-sickness/

jpgs decompress twice as fast, but lack the A channel. Ultimately I ended up using jpgs for the RGB and crunched 8-bit lossless PNGs (1/4th resolution, palette compression, using paint.net :D, ) for the A channel; manually pixel-copying them into an BGRA image. Took 2/3ths of the time on my ipad4 instead of loading simple PNGS.

Still not fast enough though.

In my final attempt I sacrificed disk space: just write out the RGBA bytes to a text file, and read in IOS directly to an allocated image of the same size. Took maybee 20% of the time of loading complete PNGS, and took about three times the disk space.

I believe there is an addon that does the above approach - http://ofxaddons.com/repos/762 - didn’t check it though.

Thanks for the info. I’ve tried jpg’s and it’s taking just as long to execute the loadImage function per texture; on average 120 milliseconds to load each 84x672 jpg :frowning: Seems if I use NSImage I might be able to get quite a speed up (http://forum.openframeworks.cc/t/slow-jpeg-loading…-trying-to-use-libjpeg-turbo/4755/0).

Also, if I do resolve the loading time I still get the cropFrom function stall on the main thread, which I’m not sure is avoidable? Seems I’m fighting a losing battle with this one.