VideoPlayback with Gstreamer and vaapi

Hey there,
do I have to do something special (custom pipeline) to enable Hardware Video Decoding via vaapi?
If I play a Full HD Video from OF i get at least one CPU Core with 100% load, whereas if i start a video from command line:

gst-launch-1.0 filesrc location=./bin/data/movies/jelly01_1920.mp4 ! qtdemux ! vaapidecode ! vaapisink

The CPU load is under 20%.
Also putting Of into Verbose Mode doesn’t give me any indication that vaapi is being used, whereas
the gst-launch command says something like

libva info: VA-API version 0.35.0
libva info: va_getDriverName() returns 0
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/i965_drv_video.so
libva info: Found init function __vaDriverInit_0_35
libva info: va_openDriver() returns 0
Setting pipeline to PAUSED …
Pipeline is PREROLLING …
Got context from element ‘vaapidecode0’: gst.vaapi.Display=context, display=(GstVaapiDisplay)NULL;
Pipeline is PREROLLED …
Setting pipeline to PLAYING …
New clock: GstSystemClock
Got EOS from element “pipeline0”.

Ok I have poked around in ofGstVideoPlayer and saw that OF uses playbin. When launching

gst-launch-1.0 -v playbin uri=file://jelly01_1920.mp4

I get

libva info: VA-API version 0.35.0
libva info: va_getDriverName() returns 0
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/i965_drv_video.so
libva info: Found init function __vaDriverInit_0_35
libva info: va_openDriver() returns 0

So this seems to work. So theoretically OF should also use the vaapi automagically through playbin, right? So why is there such a difference in CPU usage? @arturo do you have a hint on that?

right now, in ubuntu 14.10 (in general in any distribution using gst 1.4 or more) playbin seems to be using vaapi by default but OF doesn’t have support yet for gstreamer GL surfaces so it might be that the final element in the pipeline is downloading from the GPU to RAM to then upload again to the GPU in a texture which would be highly inefficient or that after checking vaapi internally playbin decides not to use it since the next element doesn’t have support for a GL surface. in the master version of OF in github there’s support for native formats by using:

player.setPixelFormat(OF_PIXELS_NATIVE);

which should make the CPU usage much lower by avoiding the colorspace conversion

I’m on 14.04 and playbin on GStreamer 1.2.4 uses vaapi, if present.
Setting

player.setPixelFormat(OF_PIXELS_NATIVE);

consumes much less CPU, but is plays without color, just grey values.
I tried Big Buck Bunny H264 and MP4.
The Color Code I420.
Should I open an Github Issue?

I have to check with my employer, but maybe I can work on GL Surface integration. Do you have any useful resources for that?
I would have to modify the appsink of OF to support

gst-inspect-1.0 vaapidecode

SRC template: 'src'
    Availability: Always
    Capabilities:
      video/x-raw(memory:VASurface)
                 format: { ENCODED, NV12, I420, YV12 }
                  width: [ 1, 2147483647 ]
                 height: [ 1, 2147483647 ]
              framerate: [ 0/1, 2147483647/1 ]
      video/x-raw(meta:GstVideoGLTextureUploadMeta)
                 format: RGBA
                  width: [ 1, 2147483647 ]
                 height: [ 1, 2147483647 ]
              framerate: [ 0/1, 2147483647/1 ]

one of these, right?!

yes OF_PIXELS_NATIVE returns pixels as the native format, usually I420 for videos. if you use the programmable renderer, by setting the GL version to more than 3 then the renderer will use a set of shaders internally to do the color conversion to rgb. the difference in cpu usage there is from avoiding the color conversion.

still using vaapi like this means that the piexls are being downloaded from the gpu which might be slow. i’ve been trying to use vaapi to upload directly to a GL texture but it’s still an unstable api in gstreamer and has changed for sure since 1.2.4. i’ve been using 1.4.3 in ubuntu 14.10 and it’s already slightly different.

anyway, my idea is to ifdef the direct GL upload so we can enable it conditionally to work on it until is more stable. i’ll upload my changes to github soon.

This would be super awesome. Do you have any idea when “soon” will be? No pressure! :smile:
For the sake of completeness:
On 14.04 there is vaapi Version 0.5.7 which is from 22-Nov-2013. I’ll have to check what 14.10 uses.
The freshest release is 0.5.10 from 03-Feb-2015.

i’ve just uploaded my changes to master. if you enable the GL define in ofGstUtils it’ll internally use a different pipeline and upload the pixels to a texture. i’ve only tried it on ubuntu 14.10 with gstreamer 1.4.3 and pretty sure it won’t work on anything else. also i haven’t managed yet to use vaapi + gl by now it’s only uploading the pixels to a texture internally and accessing that texture from OF which is not much but it’s the first time in years that i’ve managed to make that work so it’s a beginning.

also 1.4.3 is kind of buggy i’ve included a patch to make the videograbber work again and the pipeline will always block when closing waiting for an EOS event, 1.4.5 seems to have fixed most of those issues and also some things in the gl pipeline but haven’t tried yet

Some updates on this? I’m very very interested in using the vaapi.
Really needed for 4k video playback if you don’t want to use a power horse of a machine.

2 Likes

I was wondering what the current state is here. What is the current way to get the most efficent video decoding done on linux with vaapi?