RPi seek to frame in video (sliced?)


#1

Hello everyone,
Goal:
vary the play speed of a video on the Raspberry Pi.

With ofxOMXPlayer, the speed cannot go higher than 1.6x, because it has to load all the frames in between, it doesn’t jump. I think I tried all the seeking methods but they all load every frame and so the limit is 1.6x.

Now I want to try slicing the video to pictures (1 per frame) and read it sequentially.
Has anyone tried that or has any advise ?
I didn’t find the right addon for reading those yet.


#2

You could have a vector<ofImage> frames and load all the images into it (this is presuming that the video is already converted into an image sequence and is present in the data folder)… but I’m pretty sure the rPi will not be able to handle a large number of image files this way.

Of course, you could load just a single frame at display time, with a just an int keeping track of the frame number… but then I doubt if it’ll be able to do this at an acceptable frame rate.

Or combining both the methods with a threaded image loader, with dynamically keeping however many number of frames the rPi CPU can handle from the current displayed frame, and depending on the speed, skipping those many intermediate frames to load in between…

Hmm, try them all!


#3

hey, thanks
that’s exactly the plan for now… will keep you updated :slight_smile:

Although I don’t know anything about OPENMAX optimisation (->RPi videogear)
why doesn’t the video players would work otherwise that what you describe,
and if I could simply integrate the legit omxPlayer into OF app that seems to propose this ?


#4

hi,

I got this working fine with dynamic frame seeking using ofvideo rather than OMX, variable rate forwards and backwards, after some rumbling and challenges on rpi
The source is on github if you want to look

The biggest issue I found was the speed of codec execution on the pi so how you encode video is crucial. It sounds like you have an issue with key frame seeming and frame reconstruction.

Have you tried tests encoding your video with more key frames (compared to difference frames) or a different codec - or all key frames - this will slow down overall performance (depending on the codec) but will make seeking and frame reconstruction much faster overall. and if you use all keyframes it makes bi-directional seekign and random seeking the same speed…


#5

Hey @danb ,
Thank you for your reply and experience, this is very informative and also encouraging ^^!
I’m working with ofxOMXPlayer for a while now, and I now stay with h264 every time, and also privilege everyframe=keyframe.
I will try ofVideoPlayer instead, I just always thought it would be slower due to the lack of openmax aceleration, but maybe not.
I’m on it right now, should keep you updated today.


#6

I’m still on it, it’s a lot of tests to make.
@danb were you finally using GLES 2 on the raspberrys ?
I see that it is commented in your main.ccp, but maybe it is for testing on computer ?
It look like the only way to get color out of the videos, right ?


#7

hi @CCx266
TBH i cannot remember now - you may be correct - i think the suggestion for GLES2 on Pi was from a thread from @arturo - I am travelling so cannot check on code discrepancies as I am away from my main dev machine and dont have any pi’s here. when encoding video remember that it will unpack more efficiently on 8 pixel boundaries also… in the end i got reasonable performance but did compromise on video size (width and height) to get unpacking (and file opening) working satisfactorily…


#8

Yes using GLES 2 allows the renderer to do the colorspace conversion in the gpu instead of the cpu. you then need to set the video mode to OF_PIXELS_NATIVE which tells the video player to not do any colorspace conversion and just return the pixels in yuv.

That makes a big difference in small hardware like the pi


#9

Hey @arturo, thanks,
But here on the pi, the GPU is totally under pressure, and CPU is fine, can even be overclocked. If I want to make the conversion by CPU, is

OF_PIXELS_YUV2 a good choice ?

@danb, you mean 8 bit per pixel max right ?


#10

if you want to do the conversion in the cpu you usually use OF_PIXELS_RGB which is the default since that’s what you need to upload to the opengl texture but doing the color conversion in the cpu will give you really low framerates in relatively big formats like 720 or 1080.


#11

Hey,
I think I’m getting away from using non openmax accelerated code, since I guess we cannot overtake that speed of rendering.

I will now investigate on modifying ofxOMXPlayer core to buffer specific frames,
I think I will have to wait jvcleave’s availability, because I feel close but overwhelmed on this.

Or I can call the legit popcornmix/omxplayer via a system call and then address it with d-bus, but I don’t know if that even give a good result on seeking to frames. I read somewhere that this seeking function was implemented but I can’t find it anymore. Also d-bus look very complex to put to work in the first place…


#12

the normal ofVideoPlayer also uses accelerated openmax code and as the omx player does the color conversion in the graphics card (when using pixels native and gles2). ofVideoPlayer is slightly slower but does have more control over playing speed…


#13

@arturo, when I make an app with ofxOMXPlayer, I don’t use gles2, but at least
ofSetupOpenGL(...)
Maybe I misunderstood your sentence.
I had also the idea of feeding ofVideoPlayer with a video that has an RGB colorspace, but it looks hard to encode it with h264 or mpeg-4.


#14

if you don’t use gles2 and use OF_PIXELS_NATIVE you’ll see a grayscale output. You need gles2 in order to use OF_PIXELS_NATIVE.

OF_PIXELS_NATIVE tells the video player to not do any colorspace conversion, which is a very cpu expensive process, and just return whatever the video’s colorspace is, usually some kind of yuv. Then the gl programmable renderer uses some shaders to convert to color in the gpu but shaders only work under gles 2 that’s why you need to enable it.

I’m almost sure there’s no way to encode rgb in h264


#15

Ok, I absolutely believe you but I really setup OpenGL with ofxOMXPlayer and there is colors. I don’t find how gles gets invoked anywhere in the sourcecode. Anyway,

I found a way for rgb h264 with ffmpeg:
ffmpeg -i "input file.mp4" -colorspace rgb -g 1 "output file.mp4"
-g is for gop.
I’ll try to see if there is improvements.
Also I would love to work with ofVideoPlayer, I think I saw precision issues that I would love to fix, and it is pretty powerful for sure.

I though someone could want the mediainfo of the output:

Video
ID                          : 1
Format                      : AVC
Format/Info                 : Advanced Video Codec
Format profile              : High Intra@L3.1
Format settings             : CABAC
Format settings, CABAC      : Yes
Format settings, GOP        : N=1
Codec ID                    : avc1
Codec ID/Info               : Advanced Video Coding
Duration                    : 35 s 840 ms
Bit rate mode               : Variable
Bit rate                    : 3 489 kb/s
Width                       : 1 280 pixels
Height                      : 720 pixels
Display aspect ratio        : 16:9
Frame rate mode             : Constant
Frame rate                  : 25.000 FPS
Color space                 : RGB
Chroma subsampling          : 4:2:0
Bit depth                   : 8 bits
Scan type                   : Progressive
Bits/(Pixel*Frame)          : 0.151
Stream size                 : 14.9 MiB (96%)
Writing library             : x264 core 157 r2932 303c484
Encoding settings           : cabac=1 / ref=1 / deblock=1:0:0 / analyse=0x3:0x113 / me=hex / subme=7 / psy=1 / psy_rd=1.00:0.00 / mixed_ref=0 / me_range=16 / chroma_me=1 / trellis=1 / 8x8dct=1 / cqm=0 / deadzone=21,11 / fast_pskip=1 / chroma_qp_offset=-2 / threads=12 / lookahead_threads=2 / sliced_threads=0 / nr=0 / decimate=1 / interlaced=0 / bluray_compat=0 / constrained_intra=0 / bframes=0 / weightp=0 / keyint=1 / keyint_min=1 / scenecut=40 / intra_refresh=0 / rc=crf / mbtree=0 / crf=23.0 / qcomp=0.60 / qpmin=0 / qpmax=69 / qpstep=4 / ip_ratio=1.40 / aq=1:1.00
Color range                 : Limited
Matrix coefficients         : Identity
Codec configuration box     : avcC

#16

oh, i see, that’s only needed for the plain ofVideoPlayer not for ofxOMXVideoPlayer


#17

@CCx266 unpacking (was/is) faster with image/video sizes that have pixel dimensions that are multiples of 8 - ie 512px, 1024 etc… - your mileage may vary…

can you post a link to the project/documentation when you are done?
d


#18

Exponential of 2 you mean ?
width*height = 2^x, x{integer}

Ok, that’s why I didn’t get it first time, I’ll try then, thanks


#19

OK. I needed this, too much to compare

GL ver OF_PIXELS_ Source video Cspace color, b/w FR (play) FR (seekFrame)
GLES NATIVE 1024*2048 RGB color 7 0
GLES NATIVE 1024*2048 YUV color 7 0
GLES NATIVE 1024*512 RGB color 30 16,5
GLES NATIVE 1024*512 YUV color 30 16,5
GLES RGB 1024*2048 RGB color 7 0
GLES RGB 1024*2048 YUV color 7 0
GLES RGB 1024*512 RGB color 30 0
GLES RGB 1024*512 YUV color 30 0
OpenGL NATIVE 1024*2048 RGB b/w 6 0
OpenGL NATIVE 1024*2048 YUV b/w 6 0
OpenGL NATIVE 1024*512 RGB b/w 30 15
OpenGL NATIVE 1024*512 YUV b/w 30 15
OpenGL RGB 1024*2048 RGB color 7 2
OpenGL RGB 1024*2048 YUV color 7 2
OpenGL RGB 1024*512 RGB color 30 8
OpenGL RGB 1024*512 YUV color 30 9

I uploaded the code because you asked, but it is a dirty extract of my project…