How to use GPU as a decoder?

Hello,

i have a .mp4 video and I’m using ofVideoPlayer for the playback. I’ve installed K-Lite Codec as recommended.

The problem I have is that it takes too long to decode the 4 second video (like 15 seconds or so)in my opinnion and i wanted to ask if there are other better ways to do the playback for example GPU decoding?

In my project im reading everyframe and doing some video processing with them.
I already threaded the ofVideoPlayer and got some extra framerates but thats just my main-loop with the processing. the videodecoding is still as slow as before.

I hope I could make my point clear and looking forward to some tips.

Thanks in advance.

Hi @Phoenix03 - welcome!! :slight_smile:

If you just play the video without doing any processing ( ie just video.play(); video.update(); video.draw(); ) what sort of speed do you get?

It might be the way you are accessing the pixels is slowing things down.
If you can share your code it will make it easier to help you track down the slow down.

Also how big is the video?

Thanks!
Theo

Thanks theo,

and as u suggested i tried to draw the video without any processing and it’s still as slow as without processing.

ofApp.h:

#pragma once

#include "ofMain.h"

#include "ofxCv.h"
#include "ofxOpenCv.h"
#include "ofxGui.h"
#include "opencv.hpp"

#include "mosquitto.h"

#include "ofThreadedVideo.h" //own class

using namespace ofxCv;
using namespace cv;
using namespace std;

class ofApp : public ofBaseApp {

public:
	void setup();
	void update();
	void draw();

	void keyPressed(int key);
	void keyReleased(int key);
	void mouseMoved(int x, int y);
	void mouseDragged(int x, int y, int button);
	void mousePressed(int x, int y, int button);
	void mouseReleased(int x, int y, int button);
	void mouseEntered(int x, int y);
	void mouseExited(int x, int y);
	void windowResized(int w, int h);
	void dragEvent(ofDragInfo dragInfo);
	void gotMessage(ofMessage msg);
	void exit();


	ofThreadedVideo threadedCam;


	ofImage image;
	ofxCvGrayscaleImage grayscaleImg;

	ofPixels pixels;
};

In my setup() I’m calling the method of my own class:

threadedCam.startThread();

In my update() I’m getting the pixel from my own class and turn them into ofxGrayscaleImage (8 bit
Image):

if (threadedCam.pixel.isAllocated() && threadedCam.isThreadRunning())
{
image.setFromPixels(threadedCam.pixel);
image.setImageType(OF_IMAGE_GRAYSCALE);
grayscaleImg = image;
grayscaleImg.threshold(grayscaleTreshhold);
}

In my draw() I just draw the outcome of my processing…

To my own class ofThreadedVideo:

class ofThreadedVideo : public ofThread
{
public:
	ofPixels pixel;
	ofVideoPlayer video;

	void threadedFunction()
	{
		video.setUseTexture(false);
		video.setVolume(0.0);
		video.load("Video1.mp4");
		video.play();
		video.setPaused(true);

		while (isThreadRunning() && video.isInitialized())
		{
			video.update();

			if (video.isFrameNew())
			{
				lock();
				video.getPixels().rotate90To(pixel, 1);
				video.nextFrame();
				unlock();
			}
		}
	}
};

If I’m acessing the pixel of the ofVideoPlayer wrong than it must be in this class I made my own.

My Video is in .mp4 format and its like 4 second long (1,35MB, 720p)

Thanks in advance for any tips.

P.S.: I tried to draw threadedCam.video.draw(0,0) but i got nothing on my screen?! (I tried to skip the proces of copying pixel from one class to another).

I had to copy the pixel from the video to an ofImage and draw the image like so:

image = threadedCam.pixel;

image.draw(0,0);

Also i tried the addon ofxHapPlayer yesterday but the problem with it is, that you can’t access the pixel of the video because it uses textures. So that wasn’t a solution either… (But performance wise it was really good)

I searched for ways to still be able to access the pixels but it seems the only way to do so is with use of PBOs (Pixel Buffer Object). For that i found an addon ofxPBO from arturoc arturoc/ofxPBO: threaded PBO openframeworks addon (github.com) and have to try it out as a next step.

ofxHapPlayer uses the GPU the de/encode a hap videofile (.mov)

Hi @Phoenix03 , thanks for posting some code. I’d suggest that you start with a really simple project, and build more complexity into it only as needed, such as adding an ofThread for the video player, or stepping-thru the frames rather than letting the player .play().

A video should .play() at normal speed and not be slow, and still allow for some (lots) cpu and/or gpu processing. You could run the VideoPlayerExample in the /examples folder as a point of reference (bearing in mind the the mouse will change the speed in this example).

Shaders are super fast at processing images. If your processing step can be done with shaders (either all or in part), it will help make that step more efficient too. Again, maybe try to start with a simple approach and add complexity.

Hi TimChi,

thanks for your advise, but i think i managed to overcome my problem.

I went back to the ofxHapPlayer and cracked the code for accessing the pixels of my Video frame by frame. Next step is to look if i get much better performance.

	if (hapCam.isInitialized())
	{
		if (hapCam.isFrameNew())
		{
			texture.allocate(hapCam.getTexture()->getTextureData());
		}
	}

	pixels.allocate(texture.getWidth(), texture.getHeight(), 4);
	texture.readToPixels(pixels);

	image.setFromPixels(pixels);

And with this code I’m able to access the pixels of my video perfectly.

Hey glad you found a solution that works!

You could use an allocated class member for your ofPixels and recycle it each time, rather than creating a local one that goes out of scope. Or maybe just use the ofPixels in image.

Ah interesting.
I bet you could do this unthreaded and it would be faster.

Some notes:

  • ofPixels::rotate90To is not an optimized operation.
  • With threading in general you would want to avoid anything heavy in the lock/unlock that you could do in the main thread. A better approach would be to copy the pixels here to another ofPixels which you then access in the main thread. Convert to ofxCvGrayscaleImage in the main thread and then do rotate / thresholding etc. But again I don’t think threading is needed here.
  • A faster approach would be to draw the video into an FBO on the main thread ( draw it rotated into the fbo ) and then do readToPixels as @TimChi mentioned. Then pass those pixels to ofxCvGrayscaleImage for the threshold.
  • Or just set ofxCvGrayscaleImage directly from pixels. Do a threshold and draw rotated.

That is because the video.setUseTexture(false); which needs to be false as you can’t have OpenGL / texture calls in a thread.

Is the ultimate call that you just want to play a video rotated and thresholded?
As you could do that just with ofVideoPlayer and a single ofShader.

Happy to provide an example.
Theo

2 Likes

Thanks for your tips Theo and TimChi,

so i decided to do the video playback unthreaded because I’m using the ofxHapPlayer. I’m new to all of that stuff but now it works as fast as the video itself and I’m satisfied with the outcome.

One thing i noticed is that when im encoding my video to Hap, which you have to do in order to play it with the ofxHapPlayer, the videosize increases about 10 fold (i.e. 1,5MB → 20MB).

1 Like