GStreamer git build.. how to properly configure for OF?

mgr37 · February 25, 2022, 8:52pm

Hi,
i am playing with OF on a RockPro64 (RK3399 / aarch64)

I managed to compile OF 0.11.2 by modifying the make config (creating an aarch64 target).
OF runs smoothly and i can achieve my goal which is decoding h264 video into a texture and apply shaders. I got a desktop-less distribution, and use xinit to start my script successfully (as suggested in posts about 0.11 on RPi around here)

OF uses Gstreamer to decode video files.
I am using an Armbian distribution tailored for the Rockpro64 with a standard 5.15 kernel which includes v4l2 h264dec uAPI for the board.
BUT the gstreamer version available in the repo is 1.18 and gstreamer support for this v4l2 h264dec feature only appears in 1.20.
So when i use OF to play video, it falls back to CPU decoding which works but is not ideal.

So now my problem is that i try to compile gstreamer from source in order to enable hardware decoding thru v4l2.
I downloaded gstreamer from official git, did run meson/ninja to configure/compile/install it.
I had to force some features to make it work with v4l2:

meson configure build -DFFmpeg:libv4l2=enabled -Dgst-plugins-bad:v4l2codecs=enabled -Dgst-plugins-good:v4l2=enabled -Dgst-plugins-good:v4l2-gudev=enabled

Now i have a working gstreamer 1.21 compiled and installed.
It works pretty well with hardware acceleration when i run

gst-launch-1.0 filesrc location="gravity.mp4" ! parsebin ! v4l2slh264dec ! autovideosink

But now when i run my OF app (after full clean recompile), the video doesn’t play and the app exits.
My issue is: the same OF app is working (without HW accel) with repository provided gstreamer 1.18, but fails with home compiled gstreamer 1.21 (while gst-launch for this version does handle playback with HW accel properly).

I guess i missed something in the gstreamer configuration to enable full OF support ?
Any advice to properly build Gstreamer for video decoding use in OF ?
Or maybe somewhere i can enforce this v4l2slh264dec option in OF sources ?

Thanks a lot !

PS 1:

i thought i might have missed the X11 part in gstreamer, so i tried to enable x11 features in gstreamer compilation with

meson configure build -Dgst-plugins-bad:x11=enabled -Dgst-plugins-base:x11=enabled -Dgst-plugins-base:xshm=enabled -Dgst-plugins-base:xvideo=enabled -Dgst-plugins-base:gl_winsys=x11

But no luck…

PS 2:

Here is my failing APP Verbose log:

xinit /root/OF11.2-aarch64/apps/37Player/bin/37Player

X.Org X Server 1.20.11
X Protocol Version 11, Revision 0
Build Operating System: linux Ubuntu
Current Operating System: Linux rockpro64 5.15.18-rockchip64 #trunk.0023 SMP PREEMPT Sat Jan 29 16:01:11 UTC 2022 aarch64
Kernel command line: root=UUID=ec4cde3b-89a5-4abf-b266-035e953a445b rootwait rootfstype=ext4 console=ttyS2,1500000 console=tty1 consoleblank=0 loglevel=1 ubootpart=92eb0cfb-01 usb-storage.quirks=0x2537:0x1066:u,0x2537:0x1068:u cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory swapaccount=1
Build Date: 14 December 2021 02:13:12PM
xorg-server 2:1.20.11-1ubuntu1.2 (For technical support please see http://www.ubuntu.com/support)
Current version of pixman: 0.40.0
Before reporting problems, check http://wiki.x.org
to make sure that you have the latest version.
Markers: (–) probed, (**) from config file, (==) default setting,
(++) from command line, (!!) notice, (II) informational,
(WW) warning, (EE) error, (NI) not implemented, (??) unknown.
(==) Log file: “/var/log/Xorg.0.log”, Time: Fri Feb 25 20:01:57 2022
(==) Using config directory: “/etc/X11/xorg.conf.d”
(==) Using system config directory “/usr/share/X11/xorg.conf.d”
(II) modeset(0): Initializing kms color map for depth 24, 8 bpc.
[verbose] GL Version:3.1 (Core Profile) Mesa 21.0.3
[verbose] ofShader: checkAndCreateProgram(): creating GLSL program
[verbose] ofShader: setupShaderFromSource(): GL_VERTEX_SHADER shader compiled
[verbose] ofShader: checkAndCreateProgram(): creating GLSL program
[verbose] ofShader: setupShaderFromSource(): GL_VERTEX_SHADER shader compiled
[verbose] ofShader: checkAndCreateProgram(): creating GLSL program
[verbose] ofShader: setupShaderFromSource(): GL_VERTEX_SHADER shader compiled
[verbose] ofShader: checkAndCreateProgram(): creating GLSL program
[verbose] ofShader: setupShaderFromSource(): GL_VERTEX_SHADER shader compiled
[verbose] ofShader: checkAndCreateProgram(): creating GLSL program
[verbose] ofShader: setupShaderFromSource(): GL_VERTEX_SHADER shader compiled
[verbose] ofShader: checkAndCreateProgram(): creating GLSL program
[verbose] ofShader: setupShaderFromSource(): GL_VERTEX_SHADER shader compiled
[verbose] ofShader: checkAndCreateProgram(): creating GLSL program
[verbose] ofShader: setupShaderFromSource(): GL_VERTEX_SHADER shader compiled
[verbose] ofShader: checkAndCreateProgram(): creating GLSL program
[verbose] ofShader: setupShaderFromSource(): GL_VERTEX_SHADER shader compiled
[verbose] ofShader: setupShaderFromSource(): GL_FRAGMENT_SHADER shader compiled
[verbose] ofShader: setupShaderFromSource(): GL_FRAGMENT_SHADER shader compiled
[verbose] ofShader: setupShaderFromSource(): GL_FRAGMENT_SHADER shader compiled
[verbose] ofShader: setupShaderFromSource(): GL_FRAGMENT_SHADER shader compiled
[verbose] ofShader: setupShaderFromSource(): GL_FRAGMENT_SHADER shader compiled
[verbose] ofShader: setupShaderFromSource(): GL_FRAGMENT_SHADER shader compiled
[verbose] ofShader: setupShaderFromSource(): GL_FRAGMENT_SHADER shader compiled
[verbose] ofShader: setupShaderFromSource(): GL_FRAGMENT_SHADER shader compiled
[verbose] ofShader: checkAndCreateProgram(): creating GLSL program
[verbose] ofShader: setupShaderFromSource(): GL_VERTEX_SHADER shader compiled
[verbose] ofShader: setupShaderFromSource(): GL_FRAGMENT_SHADER shader compiled
[verbose] ofShader: linkProgram(): attaching GL_FRAGMENT_SHADER shader to program 1
[verbose] ofShader: linkProgram(): attaching GL_VERTEX_SHADER shader to program 1
[verbose] ofShader: checkProgramLinkStatus(): program 1 linked
[verbose] ofShader: linkProgram(): attaching GL_FRAGMENT_SHADER shader to program 3
[verbose] ofShader: linkProgram(): attaching GL_VERTEX_SHADER shader to program 3
[verbose] ofShader: checkProgramLinkStatus(): program 3 linked
[verbose] ofShader: linkProgram(): attaching GL_FRAGMENT_SHADER shader to program 5
[verbose] ofShader: linkProgram(): attaching GL_VERTEX_SHADER shader to program 5
[verbose] ofShader: checkProgramLinkStatus(): program 5 linked
[verbose] ofShader: linkProgram(): attaching GL_FRAGMENT_SHADER shader to program 7
[verbose] ofShader: linkProgram(): attaching GL_VERTEX_SHADER shader to program 7
[verbose] ofShader: checkProgramLinkStatus(): program 7 linked
[verbose] ofShader: linkProgram(): attaching GL_FRAGMENT_SHADER shader to program 9
[verbose] ofShader: linkProgram(): attaching GL_VERTEX_SHADER shader to program 9
[verbose] ofShader: checkProgramLinkStatus(): program 9 linked
[verbose] ofShader: linkProgram(): attaching GL_FRAGMENT_SHADER shader to program 11
[verbose] ofShader: linkProgram(): attaching GL_VERTEX_SHADER shader to program 11
[verbose] ofShader: checkProgramLinkStatus(): program 11 linked
[verbose] ofShader: linkProgram(): attaching GL_FRAGMENT_SHADER shader to program 13
[verbose] ofShader: linkProgram(): attaching GL_VERTEX_SHADER shader to program 13
[verbose] ofShader: checkProgramLinkStatus(): program 13 linked
[verbose] ofShader: linkProgram(): attaching GL_FRAGMENT_SHADER shader to program 15
[verbose] ofShader: linkProgram(): attaching GL_VERTEX_SHADER shader to program 15
[verbose] ofShader: checkProgramLinkStatus(): program 15 linked
[verbose] ofShader: linkProgram(): attaching GL_FRAGMENT_SHADER shader to program 25
[verbose] ofShader: linkProgram(): attaching GL_VERTEX_SHADER shader to program 25
[verbose] ofShader: checkProgramLinkStatus(): program 25 linked
[verbose] ofxGLWarper setup: 0 0 1920 1080

[verbose] ofFbo: checkGLSupport(): maxColorAttachments: 8, maxDrawBuffers: 8, maxSamples: 4
[verbose] ofFbo: FRAMEBUFFER_COMPLETE - OK
[verbose] ofGstUtils: gstreamer inited
[verbose] ofGstVideoPlayer: loadMovie(): loading “file:///root/OF11.2-aarch64/apps/37Player/bin/data/movies/gravity.mp4”
[verbose] ofGstUtils: startPipeline(): attaching callbacks
[verbose] ofGstUtils: Stream Status: typefind 0
[verbose] Pipeline is PREROLLING
[verbose] ofGstUtils: Stream Status: typefind 1
[verbose] ofGstUtils: Stream Status: qtdemux0 0
[verbose] ofGstUtils: Stream Status: qtdemux0 1
[verbose] ofGstUtils: Stream Status: multiqueue0 0
[verbose] ofGstUtils: Stream Status: multiqueue0 1
[verbose] ofGstUtils: Stream Status: multiqueue0 0
[verbose] ofGstUtils: Stream Status: multiqueue0 1
[verbose] ofGstUtils: Stream Status: vqueue 0
[verbose] ofGstUtils: Stream Status: vqueue 1
[verbose] ofGstUtils: Stream Status: aqueue 0
[verbose] ofGstUtils: Stream Status: aqueue 1
xinit: connection to X server lost

waiting for X server to shut down (II) Server terminated successfully (0). Closing log file.

Here is the Gst-launch log that works properly (with 1.21 & HW accel):

Setting pipeline to PAUSED …
Pipeline is PREROLLING …
Redistribute latency…
Pipeline is PREROLLED …
Setting pipeline to PLAYING …
Redistribute latency…
New clock: GstSystemClock
Got EOS from element “pipeline0”.
Execution ended after 0:00:22.981724840
Setting pipeline to NULL …
Freeing pipeline …

mgr37 · February 25, 2022, 9:44pm

I guess there is something with the setPipelineWithSink method from ofGstUtils.h
and its usage in ofGstVideoPlayer.cpp line 153

Any clue how i should modify the pipeline configuration to insert this v4l2slh264dec option ?

Thanks

mgr37 · February 25, 2022, 10:30pm

Tried to add this in ofGstVideoPlayer.cpp line 125 (after pipeline creation)

	// Add v4l2 dec
	GstElement * h264dec = gst_element_factory_make("v4l2slh264dec", "hwdec");
	if (h264dec == NULL) g_error ("Could not create 'v4l2slh264dec' element");
	gst_bin_add_many(GST_BIN (gstPipeline), h264dec, NULL);

it doesn’t seems to work either…

But this works:

gst-launch-1.0 filesrc location="gravity.mp4" ! parsebin ! v4l2slh264dec ! appsink

I don’t get how to achieve the same kind of pipeline in OF…

mgr37 · February 25, 2022, 10:47pm

Also this works fine (with hardware accel) and seems to be close to what pipeline OF is using:

gst-launch-1.0 -v playbin uri=file:///root/OF11.2-aarch64/apps/37Player/bin/data/movies/gravity.mp4

It seems to fails while “Pipeline is PREROLLING” since the app exists before i get “Pipeline is PREROLLED”.