threaded video player

gameover · November 21, 2008, 11:26am

Finally have got some time to get back to playing with oF after the YCAM workshop.

I’m trying to port some of my VJ apps written in other languages to oF. At the moment I am trying to get video’s to load in a thread (trying to use really big HD movies).

I have read through many of the posts elsewhere about using vectors to store pointers to movies and loading images in threads, and have tried with some success to get these things to work: the vector approach works pretty fast, but still always jams, especially with larger (1080x1920) movies).

However I can’t seem to apply the logic used for threaded images successfully to video (http://forum.openframeworks.cc/t/threaded-image-loader/816/0).

I’ve also read the thread about how unsafe vectors are in threading, and the advice to move to arrays…is this what I should do? It’s pretty important to me to be able to load any number of movies into the application as I am constantly changing hard drives, making/deleting content etc so that’s why I started with the vector approach.

I think some of the code you used in liners would help me to better understand the process, but can’t seem to find it on the liners page.

Is it somewhere?

PS: In the end I want to be able to mix (ie., blend, subtract, multiply, blur, color, etc) two (or more) videos together - I have just assumed that the first step to getting this kind of application to work would be to be able to load the movies fast and without any frames jamming?

gameover · November 23, 2008, 2:45pm

ok so after a zillion crashes i have:

moviePlayer.cpp

  
#include "moviePlayer.h"   
  
//--------------------------------------------------------------   
moviePlayer::moviePlayer() {   
    goMovie = false;   
    dirList.setVerbose(false);   
}   
  
//--------------------------------------------------------------   
moviePlayer::~moviePlayer() {      
	//not sure how to kill the vector??      
}   
  
//--------------------------------------------------------------   
void moviePlayer::preLoadMovies(string path) {   
      
    goMovie = false;   
    start();   
      
    // get all the movie file names   
    dirList.reset();   
    dirList.allowExt("mov");   
    numberOfMovies = dirList.listDir(path);   
      
	//instantiate <vector> for ofVideoPlayer references   
	ofVideoPlayer *movie;   
      
	//load all the files into a vector   
	for(int i = 0; i < numberOfMovies; i++){   
		movie = new ofVideoPlayer();   
		movie->loadMovie(dirList.getPath(i));   
		movies.push_back(movie);   
		(*movies[i]).setUseTexture(false);   
		(*movies[i]).play();   
		(*movies[i]).setPaused(true);   
	}   
      
	currentMovie = 0;   
}   
  
//--------------------------------------------------------------   
void moviePlayer::cueMovie() {   
	  
	//reset the threading flag   
	goMovie = false;   
	  
	//if it's a different movie...   
	if(currentMovie != nextMovie) {   
		  
		//start the new movie rolling and pause the old one   
		(*movies[nextMovie]).setPaused(false);   
		(*movies[currentMovie]).setPaused(true);   
		  
		//update the current movie   
		currentMovie = nextMovie;   
		  
	}   
      
}   
  
  
//--------------------------------------------------------------   
unsigned char* moviePlayer::getMovie() {   
      
	//used in testApp::update() load pixels into an ofTexture   
	return ((*movies[currentMovie]).getPixels());   
      
}   
  
void moviePlayer::changeSpeed(float tSpeed) {   
	  
	//tried using mutex on this but made things crash more!!?!?   
	(*movies[currentMovie]).setSpeed(tSpeed);   
      
}   
  
//--------------------------------------------------------------   
void moviePlayer::doCue(int movieIndex) {   
	  
	if(lock()){   
		  
		nextMovie = movieIndex;   
		goMovie = true;   
		  
		unlock();   
		  
	}   
	  
}   
  
//--------------------------------------------------------------   
void moviePlayer::start() {   
      
    startThread(true, false);  // blocking, verbose   
	  
}   
  
//--------------------------------------------------------------   
void moviePlayer::stop() {   
	  
    stopThread();   
	  
}   
  
//--------------------------------------------------------------   
void moviePlayer::threadedFunction() {   
	  
    while (isThreadRunning() != 0) {   
          
		if(lock()){   
			  
			//if a key or midi note change movies   
			if(goMovie) {   
				cueMovie();   
			}   
			unlock();   
		}   
		  
    }   
      
}

…which works pretty well, but if i really hammer the keyboard or midi in (changing video really fast) it can still be crashed despite the mutex,

Any ideas?

Also not sure what happens if I load hundreds of movies - I’m assuming it’s going to be ok…

JGL · December 4, 2008, 5:16pm

See here.

I’ll be posting a large bit of threaded fun in the New Year, although I have to admit, I haven’t _actually_tried_it_with_movies_ so it could be completely useless.

BTW, did you find this link?

http://www.openframeworks.cc/liners/

The source is posted I believe, look down the page to the:

  
[b]Is it opensource? [/b]  
  
Hell yeah! Currently you can download the "screensaver" version of the software, (with movies - approx 85mb): pc | mac

Bit.

Cheers,

JGL

gameover · December 5, 2008, 1:59am

Thanks for the response.

I did indeed find the link to liners, but the source is not included in that download…it’s the app, movies and xml only…

I basically did the equivalent of the sleep method in the end (I had two performances last week and needed a quick fix) - since I was using midi input I blocked midi messages coming faster than 0.1 secs…

And I added an idleMovie…not entirely sure why this helps, but it does:

  
//--------------------------------------------------------------   
void moviePlayer::cueMovie() {   
	  
	//reset the threading flag   
	goMovie = false;   
	  
	//if it's a different movie...   
	if(currentMovie != nextMovie) {   
		//I dont' know why but the idleMovie here seems to help???  
		(movies[nextMovie]).idleMovie();  
		(movies[currentMovie]).idleMovie();  
		//start the new movie rolling and pause the old one   
		(movies[nextMovie]).setPaused(false);   
		(movies[currentMovie]).setPaused(true);   
		  
		//update the current movie   
		currentMovie = nextMovie;   
		  
	}   
      
}

It’s been pretty hard to debug the exact problem, since when it crashes generally it just hangs…occasionally however the Xcode debug points to (ofQtUtils.cpp):

  
  
...  
		//----- argb->rgb  
		for (int i = 0; i < h; i++){  
			pix24 * rgbPtr = (pix24 *) rgbPixels + ((i) * w);  
			for (int j = 0; j < w; j++){  
				rgbaStart = (unsigned char *)rgbaPtr;	  
				memcpy (rgbPtr, rgbaStart+1, sizeof(pix24));  
				rgbPtr++;  
				rgbaPtr++; // pointing to this line here...  
			}  
		}  
...

Which makes me think that the mutex is working fine on the flag I’m using to change movies, but the ofVideoPlayer is still doing this transformation from argb to rgb and then suddenly you change movies and the MoviePtr just isn’t where it was anymore…

…So I’m thinking that extending/making a different version of the ofVideoPlayer that either has a simple flag (that can be checked before trying to change videos), or more likely some event based callback method where it doesn’t try to change movies unless it’s finished processing a frame would be better…

I kind of think an event based ofVideoPlayer would be neat: it would work better for loading movies on the fly too, since you could have callbacks for when the movie was loaded, playing, stopped, etc.

I’ll be having a go and posting results…

M

damian · December 5, 2008, 9:04am

are you trying to change the movie on an already-setup() ofMoviePlayer() mid-stream? it might be better to delete the old one and construct a new ofMoviePlayer(), especially if you’re multithreading things.

d

charli_e · April 9, 2010, 4:35pm

Hi !

Im working on app that plays videos and make other high cpu cost process… then use threads for charge and play (idle) process shoul avoid freeze or slow down the main app, isn’t it?

Im going to try your code but looks was not finished or some doubs about setup, any news about this code?

all the best

gameover · April 12, 2010, 6:56am

Hey there

I did a *lot* more work on this last year and got threaded video’s to load using a getPixels() method - basically I was working on a VJ app so I wanted to ‘cue’ a bunch of videos and then access them fast using a keyboard etc. It worked pretty well but the getPixels() command is fairly cpu intensive…

Most recently I needed to access videos direct of a hard-drive, without cue-ing them first (just fast non-blocking hard drive loading) and also I wanted to maintain access to the texture information to minimize cpu overhead.

After a bit of mucking around I realised this was possible if I did the following basic steps:

Turn setUseTexture to false;
Load the video in a thread;
Immediately stop threading;
Wait until I’m sure it’s safe, and then ‘force’ texturing back on.

This last step requires a minor addition to ofVideoPlayer…(see the goVideoPlayer file)

Below is a A/B or flip-flop style threaded video player example - simply add the files to a 0.61 example src folder…I hastily wrapped access to a bunch of standard ofVideoPlayer functions that I was not using - so there will probably be bugs - let me know what you find.

I’ve been developing this for an application that runs 5 streams of HD video and uses a database of ~1600 videos, so my emphasis was on super reliable video loading - as a consequence I’ve implemented a few things to make sure it ‘never’ crashes - including blocking against multiple simultaneous loads…again, let me know how you go - happy to share my (somewhat painful) lessons in pushing quicktime hard…

I’ve tested this on Mac and PC (without the hastily wrapped bits - not sure about 'nix…

At some point I’ll reintegrate cue-ing (ie., pre-loading n-number of videos)…

Happy times.
M

threadedVideoLoader.zip

charli_e · April 12, 2010, 12:13pm

looks so great , thanks for Sharing!

I had to comment SetMovieAudioBalance(moviePtr, pan, 0);

Cheers

gameover · April 13, 2010, 1:47am

No problem

If you’re in Windows 0.61 is compiled against Quicktime 6.x … SetMovieAudioBalance requires 7.x

It’s here.

Sosolimited · February 1, 2011, 5:18pm

Hello,

Has anyone gotten this to work in 0.62?

EDIT: I just figured this out. The newest ofVideoPlayer::closeVideo() contains the following cleanup code that 0.61 did not.

void ofVideoPlayer::closeMovie(){

…

if(bLoaded){
tex.clear();

if(pixels){
delete[] pixels;
pixels = NULL;
}
}

…

}

You can either

use ofVideoPlayer.cpp & ofVideoPlayer.h from 0.61
make ofVideoPlayer::closeVideo() a virtual void - and re-add the function w/o the cleanup code to goVideoPlayer()
make new functions loadVideo2() and closeVideo2() for goVideoPlayer() w/o the cleanup code and then call those from goThreadedVideo.

cheers,
Justin
sosolimited.com

gameover · February 16, 2011, 8:36am

Hey Justin

Sorry for the delay in posting - been away from OF for a while…

I hadn’t noticed any problem with 0.062 cos I’ve been overriding the entire ofVideoPlayer class for a while now - looking for little memory leaks and the like…

I think I have something equivalent to the fix you mention above in my version of ofVideoPlayer

If it’s of any use I’ve started putting up code on github, and the threaded video classes I’m currently using in my projects are the first to go up (apologies if it’s a bit messy to begin with, just getting the hang of git

I’ve started looking into integrating this with the latest Git version of oF, but so far that looks like taking a while…will post when it’s working, but feel like there are some glitches with the base classes to begin with…testing…

You can find the 061/062 classes I’m using here: https://github.com/gameoverhack

Or download them here…

goThreadedAddons.zip

jlrouth · June 18, 2011, 12:32pm

Thanks for the video multi-threading code Gameover; I’ve found it extremely useful for getting multiple videos playing simultaneously.

For anyone else who is running on a rather slow machine (like me):

I found there were performance issues that I couldn’t resolve for some time, until recently realising that it was the sound part of the file that seemed to be slowing things down. Videos without sound played massively faster. I then played a simultaneous (seperate) mp3 file for each video playing, and the performance was still fine.

Thanks again.

TimS · June 18, 2011, 7:32pm

This is an old thread, but I just wanted to mention that I think this is a perfect scenario to use openMP. You dont havew to create any new threaded classes because openMP will just take a for loop and split it into a number of threads. for example:

  
  
#include <omp.h>  
...  
#pragma omp parallel for num_threads(omp_get_max_threads())  
for(int i=0; i<num_movies; i++) {  
  movies[i].loadMovie("whatever.mov");  
}

This will automatically detect the max number of threads your CPU supports and split the workload up amongst the cores/threads. It might even benefit to increase the number of threads on the load function to the number of movies because its more of a I/O bound task and not CPU bound.

You could do the same for the movie.idle(); function too.

The extra stuff would be adding -fopenmp to your linker and compiler options in gcc, or enabling it some other way in visual studio, I think its just a checkbox.

gameover · July 13, 2011, 2:24pm

Hey Tim

Have you tried doing this with openMP? It was the first thing I thought of doing when I read about openMP, but not sure it works with ofVideoPlayer … I get

  
Program received signal:  EXC_BAD_ACCESS.

I think that may be for the same reason I had to make goThreadedVideo: openGL is not thread safe, whether it be posix, win or openMP threads you will always get a Sigsev because of the tex getting instantiated inside ofVideoPlayer.

Interestingly I just tried using openMP with an array of pointers to goThreadedVideo objects and it does indeed seem to work, although I’m not sure how to benchmark the results…

It’s funny, I hadn’t actually contemplated goThreadedVideo being a way of getting better performance due to it’s threadedness - even though it’s kind of neat that it does that for people - I just wanted a non-blocking way to load video…[EDIT: just thought about performance and threadedness and I don’t think goThreadedVideo will provide any performance gains, other than at load, or via the ability to load video’s “in the background” so to speak => it’s only threaded in as much as it executes the loadMovie function in a thread; once that’s done it’s just a normal instance of ofVideoPlayer]

As far as I know openMP is blocking, but it might be possible to use a similar trick to goThreadedVideo if performance [EDIT: during loading] (rather than non-blockingness) is what you’re after: make a class that extends or duplicates ofVideoPlayer, make sure the tex is not in any way instantiated or used, do all the loading, make a way to check that loading is complete, when it is safely complete, force a texture instantiation and upload pixels…

Would be great to hear if anyone gets an openMP video player happening and to see what difference in performance can be achieved…

briangibson · August 1, 2011, 7:44pm

hey there. i’ve been testing out this player, and it seems to work way better than the built in class. the non-blockingness is awesome, however, i’ve got some concerns about not being able to load two movies at the same time- the goThreadedVideo class simply says ‘nope!’ any thoughts/suggestions about how i can get around this? my first thought was to build up a queue list where it

accepts the loadMovie call,
saves the request to a goThreadedVideo* loadMeObject
waits for the bool loading == false,
checks loadMeObject for null-ness (to see if there is a movie in the queue)
then loads the next movie.

EDIT:seems groovy…

so I call this on loadMovie when load is blocked and (!iAmLoading), to store my request in loadMeObject - iAmLoading is a bool I created to tell me when that specific instance is loading.

  
void goThreadedVideo::queueALoad(const string& _name){  
	if(loadMeObject != this){  
			cueVideo = videoRequests % 2;  
			name[cueVideo] = _name;  
			loadMeObject = this;  
	}  
  
}

and then every frame, check if (!loading) and call this function:

  
  
void goThreadedVideo::loadBack(){  
	if(loadMeObject != NULL){  
		if(loadMeObject->loadMovie(loadMeObject->name[loadMeObject->cueVideo]  ) ){  
			loadMeObject = NULL;  
		}  
	}	  
}

maximillion · August 2, 2011, 9:08pm

Hi,

Im trying to get this to work on win cb 0.062+ but im running into the follow errors

  
  
..\..\..\libs\openFrameworksCompiled\lib\win_cb\openFrameworksDebug.lib(ofVideoPlayer.o):ofVideoPlayer.cpp|| multiple definition of `createMovieFromPath(char*, MovieRecord*&)'|  
  
.....  
|first defined here|  
..\..\..\libs\openFrameworksCompiled\lib\win_cb\openFrameworksDebug.lib(ofVideoPlayer.o):ofVideoPlayer.cpp|| multiple definition of `createMovieFromURL(std::string, MovieRecord*&)'|

Im sure it maybe down to :

"4) Wait until I’m sure it’s safe, and then ‘force’ texturing back on.

This last step requires a minor addition to ofVideoPlayer…(see the goVideoPlayer file)"

but cant see any instruction in goVideoPlayer.cpp or goVideoPlayer.h for the minor addition. What do I need to do?

Thanks for your help

briangibson · August 3, 2011, 3:27am

brendan: first of all, it should never have to use ofVideoPlayer, so you’re doing something wrong there. goThreadedVideo uses goVideoPlayer instead, which is a modified ofVideoPlayer class. goVideoPlayer doesn’t inherit from ofVideoPlayer- just ofBaseVideo.

maximillion · August 4, 2011, 8:15am

cheers Brian,

I had an instance of ofVideoPlayer in the header, seperate from the goVideoPlayer instance and this was causing the error

briangibson · August 7, 2011, 8:06am

glad it worked for you. I added some modifications two posts up that I believe are useful.

klonyklon · August 15, 2011, 8:02pm

Has anyone had success migrating this addon to 0.07?