ofxThreadedVideo

I’ve made a new threaded video addon called (groan) ofxThreadedVideo.

It can be found here: https://github.com/gameoverhack/ofxThreadedVideo

And replaces the previous goThreadedVideo/goVideoPlayer classes I was using (see index.php?topic=1335 and gameoverhack/goThreadedVideo and gameoverhack/goVideoPlayer

The old code worked but was getting long in the tooth (I made it for 006) and people had trouble on linux and some of the quirks of how I was stopping Quicktime from choking on simultaneous calls were annoying to egt around for end-users.

The new version:

* is of007 friendly and should work with an ofBaseVideoPlayer
* is both non-blocking on load AND truly threaded (ie., video update() calls are done in a thread)
* uses a static mutex between instances so that you can call loadMovie on many instances simultaneously and the mutex ensures they load in serial (this does mean that if you call loadMovie on a lot of instances the main thread will also block, but it does make using it to load a lot of movies across multiple instances simpler AND it does not block the main thread if you are only changing movies on one instance at a time!)
* you can either queue calls to loadMovie (ie., if an instance is busy loading it will queue the call), or you can ignore calls to loadMovie if an instance is already loading (this is the default; loadMovie both dispatches an event and returns false if you are using this mode; use setUseQueue to change the behaviour)
* implements a FastPause (basically just not calling update on the video instead of using Quicktime’s SetMovieRate(0) which causes video’s to jag when you pause/unpause)

I have not wrapped every method of ofVideoPlayer -> for now if you want to cal another function on the currently playing video you get a reference to the video with getVideo() and then make calls on that reference.

I’ll be adding the wrappers as I go and implementing things like FastScrub etc. I’ll post as I go.

There’s a very basic example on the repo which I’ll also improve on over time.

Think I’m finally getting the hang of this threading thang :wink:

1 Like

Oh forgot to say using this with quicktime, then ofQuicktimePlayer.cpp requires a bug fix in closeMovie() at line ~218:

  
width = height = 0;  

otherwise re-using the same instance of an ofVideoPlayer to load a new movie will fail because ofPixels and memGWorld do not get re-allocated (the allocation is only done in loadMovie() if width and height == 0)

I’ve made a pull request: pull/887

Sorry, wrong post.

Hello again ! :slight_smile:

Ok, I tested this quickly and came across this error:
C:\Program Files\OpenFrameworks_007\addons\ofxThreadedVideo\src\ofxThreadedVideo.cpp|143|error: ‘ofGetGLTypeFromPixelFormat’ was not declared in this scope|

Searching on the web I came across Joshua Noble’s update to:
https://github.com/openframeworks/openFrameworks/pull/823/files
libs/openFrameworks/gl/ofGLUtils.h
libs/openFrameworks/video/ofVideoPlayer.cpp

Starting with the ofGLUtils.h I compared his code with my current ofGLUtils.h (007) and I noticed that I do not have the code from line 142 to 152:

  
inline int ofGetGLTypeFromPixelFormat(ofPixelFormat pixelFormat){  
	switch(pixelFormat){  
	case OF_PIXELS_BGRA:  
		return GL_RGBA;  
	case OF_PIXELS_MONO:  
		return GL_LUMINANCE;  
	case OF_PIXELS_RGB:  
		return GL_RGB;  
	case OF_PIXELS_RGBA:  
		return GL_RGBA;  
	}  

I think I downloaded 007 in the summer of 2011 (??), so I’m assuming I need to do a complete update to 007 ?
If so, what master github should I pull from ??
(I don’t want to add this code to the existing file, I prefer to grab total sources (oF) new)

Gotta go…thanx & bye-bye :slight_smile:

ofxThreadedVideo will work with the ‘develop’ branch of openFrameworks: https://github.com/openframeworks/openFrameworks/tree/develop

My changes in pull request 887 as per above have been accepted - but then it turns out my approach was not the best one.

I’ve made a new pull request that deals with the issue better (tested now on mac and pc)…this approach is best for rapidly changing movies of the same size: pull/913

So you might need to make those changes manually or merge that pull request into your own branch/fork of oF until/if it’s accepted into the develop branch.

Some notes of interest: I’ve tested this a LOT more and now wrapped all the baseVideoPlayer functionality. A few functions (such as setSpeed) have not been optimized for threading yet, but will be over the coming days.

It works MUCH better than expected, particularly on windows. Previously I have not been able to get 6+ 1920x1080 movies to play smoothly even with a 16Tb RAID drive, oodles of Ram and CPU power. Now I can :slight_smile: On mac the performance increase is not so marked, as Quicktime is itself multithreaded natively, but still it works super well, particularly if you want to pause/unpause movies, scrub them back and forth all without jamming the main application thread or other videos that are playing!!

I’ve also added a getFrameRate (which shows the frameRate of texture updating) and am working on a getFrameRate for pixel updating (this is because pixels and textures are update in a thread and on the main application thread respectively). I’ll give them appropriate names and upload the changes soon…they are really handy to see if bottlenecks are due to CPU or harddrive/GPU…

Finally: beware trying to open more than 8 videos at the same time (particularly) on a PC -> it seems that the static mutex finds it hard to lock effectively with more than 8 movies at a time, causing the 9th or 10th movie to sometimes never load. If you want to open more than 8 movies you might need to use a method similiar to the old goThreadedVideo example of using the loaded event to make sure you load movies in serial rather than trying to force a mutex to make this happen for you.

I’ll try to make some examples to show this soon.

Have fun…

Hi Matt :slight_smile:

Thanx for your reply and detailed explanation! Since I’m still new/learning oF, I prefer to stick with the “master” branch at this point. Also because I need to stop trying various code and posting on forum…it’s time to get back to my art project and settle on a “working” idea since deadlines are a lurkin’, egads! [gulp] :wink:

***
I think what you’re offering to the community (and others who have contributed to the Addons), is really wonderful, so best of luck and keep me updated and maybe in a future project I’ll use it - if I ever I get through this 1st oF project, oh man…do {trial && error} while persistence(); :wink:

Hi Matt,

I’m using your ofxThreadedVideo addon and I’m trying to play more than 8 videos at the same time (on Mac with OF 0071). It seems to always end up crashing at some point in ofQuickTimePlayer at line 175 ----> MoviesTask(moviePtr,0); with an ‘EXC_BAD_ACCESS’.

I can read the following in your last post. What do you mean exactly? Should I check that I have not more of 8 instances of ofxThreadedVideo loading at the same time?

Finally: beware trying to open more than 8 videos at the same time (particularly) on a PC -> it seems that the static mutex finds it hard to lock effectively with more than 8 movies at a time, causing the 9th or 10th movie to sometimes never load. If you want to open more than 8 movies you might need to use a method similiar to the old goThreadedVideo example of using the loaded event to make sure you load movies in serial rather than trying to force a mutex to make this happen for you.

update:
In fact it crashes even when playing ‘just’ 8 videos (10sec - 240x320 - codec PhotoJpeg). Any advices would be great.

update #2:
It just crashed (EXC_BAD_ACCESS) in ‘ofQuickTimePlayer::start()’ —> line 387 - SetMovieRate(moviePtr, X2Fix(speed));

Thanks,

Hugues.

Hi Hugues

I’m just testing this on windows and mac. I made a new example called ofxThreadedVideoBenchmark (renamed the old example and changed the folder structure a bit too)…

I have an installation which I suspect has the same issue…but takes anywhere upto 5 hours to surface. Quite frustrating. For me it’s always been a case of crashing when allocating new pixels, rather than just during a normal update cycle…so maybe they are not related issues as I’m doing a mem swap and suspect that’s the issue…anyway, a few questions:

How often are you changing movies?

Are you doing any other changes to the properties of the movies (ie., changing speed, direction etc) whilst also trying to change movies?

Are you calling the setMovieRate function, or is that happening internally?

Could you post the full stack trace?

To be sure we are testing the same things, could you use the ofxThreadedVideoBenchmark example to test your machine/videos?

My reference to being wary about loading 8 movies is about loading them simultaneously (ie., calling loadMovie 8 times in a row in one function rather than waiting for each one to load one after another) – I found that sometimes the mutex’s would get deadlocked and not all instances would load…but often it’s fine…so nothing to do with having 8 movies running and then changing them whilst running.

Hi Matt,

Here are the answers…

How often are you changing movies?
The videos are 10sec long. They have a resolution of 240x320 and are encoded using PhotoJPEG. They are not necessarily started at the time. A maximum of 8 of them are playing at the same it. When a video ends I load a new one in the same instance of ofxThreadedVideo.

Are you doing any other changes to the properties of the movies (ie., changing speed, direction etc) whilst also trying to change movies?
I call setLoopState(OF_LOOP_NONE); and setBalance(_balance); when I receive the event VIDEO_EVENT_LOAD_OK

Are you calling the setMovieRate function, or is that happening internally?
I’m not calling this method

Could you post the full stack trace?
Running my project now. Will post the stack when it crashes

To be sure we are testing the same things, could you use the ofxThreadedVideoBenchmark example to test your machine/videos?
Will do.

update:
I draw the video using their texture (via getTextureReference()) on my ‘own’ GL_QUADS

update #2: Here a screenshot of the crash. I don’t know how to export the data.
[]()

Thanks,

Hugues

Ok I still get no crash on either windows or mac - been running since before I posted last…not conclusive by any means…but obviously something is different(ish) between my benchmark code, and your project.

The trick here will be to begin by running the ofxThreadedVideoBenchmark as is and see if it crashes on your machine or mine…it should run “forever” (ha!)…

…and then if you could slowly modify the benchmark example to work more and more like your app. eg., make it draw using getTextureRef() and custom Quads; make it change movie’s only when an end of file has been reached…etc etc…

…until it crashes in the benchmark consistently…

…then we can try to fix it :slight_smile:

Hey,

So running ‘ofxThreadedVideoBenchmark’ with the following modifications it crashed after about 1 hour.

main.cpp

  
  
#include "ofMain.h"  
#include "testApp.h"  
#include "ofAppGlutWindow.h"  
  
//========================================================================  
int main( ){  
  
    ofAppGlutWindow window;  
    window.setGlutDisplayString("rgba double samples>=4");  
	ofSetupOpenGL(&window, 960, 640, OF_WINDOW);			// <-------- setup the GL context  
  
	// this kicks off the running of my app  
	// can be OF_WINDOW or OF_FULLSCREEN  
	// pass in width and height too:  
	ofRunApp( new testApp());  
  
}  
  

testApp.cpp

  
  
//--------------------------------------------------------------  
void testApp::setup(){  
  
    ofSetLogLevel(OF_LOG_VERBOSE);  
      
    ofSetEscapeQuitsApp(true);  
    ofSetFrameRate(60);  
    ofSetVerticalSync(true);  
    ofEnableSmoothing();  
  
    files.allowExt("mov");  
    files.listDir("videosPhotoJPG_resized/"); // put a video path here with several video files in a folder  
  
    for(int i = 0; i < MAX_VIDEOS;i++){  
        videos[i].loadMovie(files.getPath(ofRandom(files.numFiles())));  
        ofAddListener(videos[i].threadedVideoEvent, this, &testApp::threadedVideoEvent);  
    }  
}  
  
//--------------------------------------------------------------  
void testApp::draw(){  
    ofSetColor(255, 255, 255);  
    for(int i = 0; i < MAX_VIDEOS;i++){  
        float xOffset;  
        float yOffset;  
        if(i < 4) {  
            xOffset = i*videos[i].getWidth();  
            yOffset = .0f;  
        } else {  
            xOffset = (i-4)*videos[i].getWidth();  
            yOffset = videos[i].getHeight();  
        }  
  
        //videos[i].draw(xOffset, yOffset);  
          
        glPushMatrix();  
        glTranslatef(xOffset, yOffset, .0f);  
          
            videos[i].getTextureReference().bind();  
              
            glBegin(GL_QUADS);  
              
            glTexCoord2f(.0f, .0f);  
            glVertex2f(.0f, .0f);  
              
            glTexCoord2f(videos[i].getWidth(), .0f);  
            glVertex2f(240.0f, .0f);  
              
            glTexCoord2f(videos[i].getWidth(), videos[i].getHeight());  
            glVertex2f(240.0f, 320.0f);  
              
            glTexCoord2f(.0f, videos[i].getHeight());  
            glVertex2f(.0f, 320.0f);  
          
            glEnd();  
              
            videos[i].getTextureReference().unbind();  
          
        glPopMatrix();  
    }  
  
    ofSetColor(0, 255, 0);  
    ofDrawBitmapString("FPS: " + ofToString(ofGetFrameRate()), 20, ofGetHeight() - 20);  
}  
  

Here are the sources - http://smallfly.com/of/ofxThreadedVideoBenchmark-crash-20120619-01.zip

Here is the sreenshot of xCode. Anyway better way to export the data?
http://smallfly.com/of/ofxThreadedVideo-crash-20120619-01.png

Hey there…thanks for this…

Realised things weren’t crashing for me on Mac because i was running in debug mode. With Release I can replicate the issue.

I’ve pushed a change to ofxThreadedVideo - not 100% sure it solves the problem, but has been running your code for 4+ hours so far…maybe you could try checking it out.

I think the problem had/has to do with calling the update method on ofQuicktimePlayer from within the threaded function during a load - MovieTask is not thread safe so it would make sense if that was causing the crash…I moved the update method into the main application thread update, so hopefully it solves the issue…

Let me know how you go and I’ll continue testing tonight.

Interestingly on the PC with the old code it’s been running for 28+ hours…go figure :slight_smile:

You are welcome. Thanks to you for looking into it.
Just pulled the new version. Will use it now on, and I will let you know how it goes.
But for sure what you found seems to be a good possibility.

With the old code, I have the problem even when running in debug mode.

So with the new version it’s running much better.
I have now started to use the setVolume method, and it just did crash though. After running for around 2.5hrs.

It crashed again in ofQuickTimePlayer at the line 474 - SetMovieVolume(moviePtr, volume);

Would have an idea why?
Can the setVolume() method of an ofxThreadedVideo be called at anytime?

Thanks.

Update
Just crashed in ofQuickTimePlayer::setPaused(). It seems that call setters methods is not safe.
[]()

I’m looking for a threaded videoplayer which will work on an iPad. I tried this but I keep getting “Could not load video” messages. Is it supposed to work on iOS?

No one?
Right now I actually get a video loaded message but nothing gets loaded. And before I get the video loaded message I get a ton of those:

  
objc[40887]: Object 0x82edba0 of class __NSCFData autoreleased with no pool in place - just leaking - break on objc_autoreleaseNoPool() to debug  
objc[40887]: Object 0x82edd00 of class __NSCFString autoreleased with no pool in place - just leaking - break on objc_autoreleaseNoPool() to debug  
objc[40887]: Object 0x82edd40 of class __NSCFString autoreleased with no pool in place - just leaking - break on objc_autoreleaseNoPool() to debug  
objc[40887]: Object 0x82edd80 of class __NSCFData autoreleased with no pool in place - just leaking - break on objc_autoreleaseNoPool() to debug  
objc[40887]: Object 0x82edfe0 of class __NSCFString autoreleased with no pool in place - just leaking - break on objc_autoreleaseNoPool() to debug  
objc[40887]: Object 0x82ee0b0 of class __NSCFString autoreleased with no pool in place - just leaking - break on objc_autoreleaseNoPool() to debug  
objc[40887]: Object 0x82ee0f0 of class __NSCFData autoreleased with no pool in place - just leaking - break on objc_autoreleaseNoPool() to debug  
objc[40887]: Object 0x81bf400 of class __NSCFString autoreleased with no pool in place - just leaking - break on objc_autoreleaseNoPool() to debug  
objc[40887]: Object 0x812fe50 of class __NSCFString autoreleased with no pool in place - just leaking - break on objc_autoreleaseNoPool() to debug  
objc[40887]: Object 0x8171740 of class __NSCFData autoreleased with no pool in place - just leaking - break on objc_autoreleaseNoPool() to debug  

Looks like the latest commit which moves update calls out of the thread breaks playback in windows. I am running the benchmark w/ the revision preceding this commit w/ no issues so far: https://github.com/gameoverhack/ofxThreadedVideo/commit/e89810778d89affa3618ef68c556bd7397ece909

Haven’t figured out why the new update location isn’t working, but will hopefully have a chance to dig in.

Thanks for the threaded video decoding; should be a default addon!

-k

edit: Windows 7, code::blocks, OF 0071 (same issues appear w/ VS2010 also)

I can confirm the issue on VS2010. Solution of ksimmulator works for me too.

Ok that’s strange…I’ll have a look into as soon as possible…

Hi, both example projects are crashing on startup for me at the same place. I think its happening at this line in ofxThreadedVideo.cpp:

if(videos[loadVideoID].loadMovie(paths[loadVideoID])){

i get this:
2012-10-09 23:29:43.772 ofxThreadedVideoExampleDebug[42256:5407] An uncaught exception was raised
2012-10-09 23:29:43.772 ofxThreadedVideoExampleDebug[42256:5407] QTMovie_QuickTime class must be initialized on the main thread.
2012-10-11 13:31:48.633 ofxThreadedVideoExample1Debug[1553:5007] *** Terminating app due to uncaught exception ‘QTMovieInitializedOnWrongThread’, reason: ‘QTMovie_QuickTime class must be initialized on the main thread.’

I’m on osx 10.8 and xcode 10.5., and the latest unstable OF

Any help is much appreciated.

Hi
did anybody test ofxThreadedVideo on linux? I’m running successfully an application with the old goThreadedVideo add-on. With the new add-on the movies don’t run, they’re standing still (like paused). I’m on the of-develop branch on a up to date ubuntu system. Although, the frames are counted correctly. Internally, the movies seems to run, but the texture is not updated.
Best
Urs Hofer