Gstreamer Pipeline using v4l2 (Raspberry Pi)

i am trying to decode H264 video with OF 0.11.2 on a RPi4 with 32bit bullseye / full KMS drivers (on Pi4, but should be equivalent on Pi3).

The videoplayer exemple compiles and run, but as soon as i play fullscreen FullHD video, it’s very laggy.

I tried

gst-launch-1.0 playbin uri=file:///home/pi/media.mp4 video-sink="kmssink"

And it runs smoothly with almost no CPU usage.
So the installed Gstreamer is properly working with v4l2h264dec and properly leverage hardware accel.

I found a different pipeline that create the same laggy result i can see in OF:

gst-launch-1.0 filesrc location=media.mp4 ! qtdemux  ! h264parse ! v4l2h264dec ! videoconvert ! "video/x-raw,format=RGB" ! kmssink

So it seems what makes it laggy in OF is the RGB conversion.

The problem can be partially fixed by replacing videoconvert with v4l2convert:
it’s almost smooth, but not as smooth as the dry playbin version.

So my questions are:

  • Is there a way to dump the actual pipeline selected by gst-launch-1.0 playbin ?

  • How to enforce v4l2convert instead of videoconvert in OF ?
    (i tried tinkering with ofGstVideoPlayer::createPipeline without success…)

  • Or maybe we can avoid color conversion → but how to work with I420 in OF ?
    If i set movie.setPixelFormat(OF_PIXELS_I420); i end up with a green display and this error:
    ofGstVideoUtils: buffer_cb(): error on new buffer, buffer size: 3133440!= init size: 3110400

Thanks !

So to partially answers my questions:

  • Yes it is possible to dump the actual pipeline selected by playbin using dot files and generate pipeline graph

I can see that running gst-launch-1.0 playbin does use v4l2h264dec which output with I420 format, transmitted as-is to kmssink which supports this format. With this pipeline, we have perfectly smooth hardware accelerated h264 decode and display.

But OF seems to accepts only RGB appsink.

So two options:

  • Convert I420 to RGB: but when using OF, gstreamer uses videoconvert which is damn slow (memory copy i guess) instead of available v4l2convert which is almost good.
    => How can i enforce v4l2convert instead of videoconvert ?

  • Making OF work with I420 => any clue about that one ?

Thanks !

Seems it can handle I420, maybe you can try setPixelFormat(OF_PIXELS_NATIVE)