How to playback videos on the GPU

Hello,
I’m just beginning work on a project that involves playback of many video files simultaneously. This seems to be a relatively common thing on the forums, but I’m having trouble gleaning much from the threads that I’ve read. However, this one in particular is full of a lot of useful tips that I am trying to emulate:

Zach identifies QuickTime as a potential bottle neck, and therefore recommends pre-loading a bunch of videos as ofTextures so that they can be played back using the GPU rather than having to be decoded every frame by QuickTime. I’m currently trying to test this with one video file and a vector of ofTextures, but I can’t seem to work out the sequence of events that need to happen. From my understanding, shouldn’t I be able to:

  1. Load a video file into an ofVideoPlayer object
  2. Get the total number of frames contained in the video
  3. Iterate through each frame in the video file.
  4. On each iteration, get the pixels in the current frame, allocate sufficient memory in a new ofTexture, load the pixels into the texture, and then push the texture into the vector.

I’ve tried different methods of completing this task list, but I still can’t figure out how to iterate through the frames of the video. player.nextFrame() doesn’t seem to do anything when I call it in setup(). Here is my most recent attempt:

myApp.cpp:

#include "ofApp.h"

ofVideoPlayer player;
int numFrames;
int width;
int height;
ofTexture frame;
vector <ofTexture> frames;
ofPixels pixels;
//--------------------------------------------------------------
void ofApp::setup(){
    player.load("test.mov");
    numFrames = player.getTotalNumFrames();
    width = player.getWidth();
    height = player.getHeight();
    
    for(int i = 0; i < numFrames; i++){
        player.nextFrame();
        ofPixels framePix = player.getPixels();
        ofTexture newFrame;
        newFrame.allocate(framePix);
        newFrame.loadData(framePix);
        frames.push_back(newFrame);
    }
    
    
    
}

//--------------------------------------------------------------
void ofApp::update(){
    
}

//--------------------------------------------------------------
void ofApp::draw(){
    //Can we pull a random frame from the list and draw it?
    frames[100].draw(0,0);
}

//--------------------------------------------------------------
void ofApp::keyPressed(int key){

}

//--------------------------------------------------------------
void ofApp::keyReleased(int key){

}

//--------------------------------------------------------------
void ofApp::mouseMoved(int x, int y ){

}

//--------------------------------------------------------------
void ofApp::mouseDragged(int x, int y, int button){

}

//--------------------------------------------------------------
void ofApp::mousePressed(int x, int y, int button){

}

//--------------------------------------------------------------
void ofApp::mouseReleased(int x, int y, int button){

}

//--------------------------------------------------------------
void ofApp::mouseEntered(int x, int y){

}

//--------------------------------------------------------------
void ofApp::mouseExited(int x, int y){

}

//--------------------------------------------------------------
void ofApp::windowResized(int w, int h){

}

//--------------------------------------------------------------
void ofApp::gotMessage(ofMessage msg){

}

//--------------------------------------------------------------
void ofApp::dragEvent(ofDragInfo dragInfo){ 

}

Hi,
first of all, your GPU memory is limited and if you load all the frames of a video into the GPU it will fill up very soon, you will probably fill it up with less than a minute of video at 1920x1080 30fps.
So you have to read this data from the disc or from ram memory at least.
Btw, video playback , when drawn it happens in the GPU. Decompresion might also happen there.
If you want to access individual frames you need to use ofxGStremer.

If you really need fast access, What I’ve done before is to store the videos in raw format, by making an OF app that reads each frame and stores its binary (uncompressed) data, frame by Frame. And then load it back. This way no decompression has to be done.

Hi Roy, thanks for the advice. Yeah I’m not attached to the strategy, I just thought it would be the fastest way of handling it.
Is the process that you’ve described something that you do as part of a setup routine? Or do you store the uncompressed files on the disk and read them on the fly?
Either way, would you be able to point me in the right direction in the docs to figure out how to do that? I haven’t even managed to iterate through frames in a video yet.
Thanks again for your reply!

Hi @sdnorris,

I have also experienced slow reading speed in the ofVideoPlayer despite its efficiency - on 8K videos for example-.
Workaround for me was to decode the frames using FFMPEG and transfer the frames back to an ofTexture by using an ofThreadChannel . You can find good examples of how to adapt the code from reading mp4 files from disk here.

Hope this helps,

Best,

P

Hi @pierre_tardif00 , thanks for your reply.
This seems viable, but I’m not entirely sure how to make this work. Haven’t used threads in ofw before but I’m down to learn. Are you recommending that I decode the videos with this ffmpeg example in one thread, and then transfer them to another thread running the main ofw program? Apologies if this is redundant, just trying to make sure I understand.

i can try to help with it but I don’t habve much time at the mo.

Yeah, if you use a thread explicitly for a class, it will create a new thread on top of the main thread, to process it.

Well it’s not super simple to do, so woudl maybe recommend doing @roymacdonald method didn’t see it was a beginner’s question sorrry. I personnaly find it quite hard to deal with FFMPEG and requires streaming knowledge and in depth OF knowledge to know where bugs are coming from.

++

I’ve done multi-threading stuff in ChucK, which is an audio programming based on C++, so I could take a crack at it. I’ve done some programming before, I’m just very new to openFrameworks and OpenGL, and working with video at such a low-level.
But yeah I’ll start with @roymacdonald’s approach, and see how far I get, although I’m actually not sure how to get started doing that. I’m not sure what format the binary video frames would be stored as, nor how to load them back in to openFrameworks once they’re in that format.

this is a bit unrelated to the current discussion but have you looked at Hap or any GPU accelerated codecs

they seem to be designed for very fast playback (at the expense of large file sizes IIRC)

2 Likes

I think the problem with this code is that you are not updating the player object.

also in general, video player has some challenges in that it’s asynchronously updating so even if you say, “nextFrame” it sometimes has not advanced by the time you ask for pixels…

to do this accurately, it may make more sense to export the video as still frames and load those – but again it depends on the size and duration.

Hap is probably a good choice, but again, depends on what you are after…

1 Like

Hi Zach, thanks for the tips.
Haven’t even heard of HAP but seems like it’s definitely worth a shot.
I’ll also try exporting videos as still frames and loading all of the images into textures, see how that goes.
And I if nextFrame() is asynchronous, I could just maybe set up a start up loop that waits until isFrameNew() before trying to load it into a texture?
I’ll try all of these out and see what works best. It’s significantly less frustrating to work on this stuff when I actually have things to try out and I’m not just coding to no avail, so thanks again for the advice everyone, looking forward to testing stuff.

Hi, Give a try to HAP codec as zach mentions. It works really nicely. That might do the trick for you.
In order to iterate through the frames check what I posted here.

What I mention is not documented anywhere as there is no particular openframeworks way to do it, it is just saving raw memory into disk, and the loading it back.
This addon does such. It is a bit outdated so I am not sure if it works now, but it is quite simple, so take a look at the implementation of it so you make an idea on how to do such.

The idea is that you save your video files previously, and you do such only once. Make sure to do so on an SSD drive, so it is fast.
Then you just load each frame on the fly.

You can do this of just waiting for isFrameNew() but from my experience, it works slowly and with some odd results. OfxGstreamer is a lot more reliable in that sense.

1 Like

If you’re on windows, I’ve written a hardware accelerated media player for windows based on IMFMediaEngine and using WGL_NV_DX_Interop to share textures to OpenGL. It’s for cinder, but it should be trivial to port the cinder parts to their openFrameworks equivalents. It’s available here

1 Like

I vouch for Hap Video player too, i forgot about it but it’s really efficient, used it in the past and it has been an real improvement.

1 Like

Very good news, the HAP codec seems to have done the trick! I’m going to mark this post as the solution just so the info is all in once place, but many thanks to @zach, @roymacdonald and @pierre_tardif00 for your help. @lithium thanks for chiming in about your media player, but I’m working on a Mac.

For posterity (and for the sake of all future openFrameworks newbies who run into this same problem), here’s what I’ve done (aggregating all of the advice/links from this thread into one post):

  1. Install the HAP player add-on via its GitHub page.
  2. Convert the videos that you want to use into the HAP codec using ffmpeg by following the instructions on the HAP codec website
    2.1) If you don’t already have it, install ffmpeg with Homebrew. It may look a little intimidating because it uses a comand-line interface (CLI) but it’s much easier than it initially looks!
  3. Create a new openFrameworks with the projectGenerator, including the ofxHapPlayer addon as you do so, to test the installation.
  4. To deal with many ofxHapPlayers at a time, create a vector list of POINTERS, not objects, and deal with them as described in this stackoverflow thread.

My test project on my 2014 MacBook is now playing back an 8x8 grid of videos with no problem.

@zach and/or @roymacdonald , is it possible to change the title of this post? I’ve found a solution to my problem but the solution doesn’t really correspond to the title anymore.

Great to know that HAP worked for you.
I can not change the post tilte. Sorry, I am just a regular forum user.

As of the vector of pointers and the stackoverflow post you mention.
It is a 10 year old post, and I will most definitely recommend you to use smart pointers (unique_ptr or shared_ptr) instead of raw pointers (those that use the keyword new to allocate), as it might lead to a memory leak if you dont dealocate properly. Smart pointers do such for you, so you dont have to worry.


// declare in ofApp.h
vector < unique_ptr<ofxHapPlayer> > players;


//add elements to the vector, in ofApp::setup() most probably.
// the following line will add one player to the end of the vector. 
// Put this inside a for loop to create several of this.
players.emplace_back(std::make_unique<ofxHapPlayer>());
1 Like