GPU jpg decoding - for media server project

Hi there,

I’m a french visual artist, vvvv user, and total newbie to OF…
I have a project in mind, to develop an open / unlimited output / media server project:
Basically, I want to use cheap/tiny PCs to stream a video output to a projector, and I need to read JPG sequences with high performances.

I found these 2 libraries:

I’d love to know if it seems feasible to integrate these with OF, to build a high performance player…
If so, I’d love to start an open project if some of you are interested…

Best
Joanie

1 Like

Hey Joanie, that’s something that I, and I’m sure thousands of other people have been thinking about doing for years, but never got round to :smile:

I haven’t looked at GPU jpg decoding, but there is HAP, developed by VidVox / VDMX and Tom ‘bangnoise’ Butterworth



If you just want playback, tom’s example should get you running.

I always wanted to do something like this with frame accurate syncing over the network, and running on embedded systems like raspberry pi, beaglebone etc. So for dirt cheap you can have nodes to play multi-channel video, but I have no experience in embedded systems unfortunately.

EDIT
Not sure if HAP could be made to run on embedded systems btw. But perhaps photojpg quicktime could work?

Well, JPG is my favorite option because:

  • you can replace parts of the content without re-encoding large files.
  • easy to distribute over network / replace frames, update parts…
  • easy to playback any chunk, loop, etc…
  • editing could be done via xml file (to rearrange frames, loop, duplicate etc…)

After trying several encoding options, JPG sequences seems to be the very best option (for me at least).

Each one of those are very valid arguments, I’m sold. + you could run it on raspberry pi too!

P.S. on http://www.ofxaddons.com/ you can find a bunch of image sequence addons which can act as a starter (using CPU jpg loading).
https://github.com/Flightphase/ofxImageSequence

then all that needs to be done is replace the image loader inside that class, to the GPU loader

rpi specific info that might be relevant:

http://www.raspberrypi.org/forums/viewtopic.php?f=33&t=57721

hi @JoanieLemercier !

Nice idea … as @memo said… something we’ve all dreamed off someday …

I would ask … if it’s a multi-projection project … should it be able to do :

· soft edge blending ? https://github.com/Flightphase/ofxProjectorBlend
· warping ? https://github.com/julapy/ofxQuadWarp

Could all this be running inside a Raspberry Pi ?

Regarding computer syncronization … that’s another nice issue to discuss about … There are many approaches and there are many options and tryout’s by different OFers … I was trying to approach it in a way that maybe it’s stupid, but it was using PTP (http://en.wikipedia.org/wiki/Precision_Time_Protocol) … i talked with some developpers on a PTPDaemon and they said it’s possible with a “normal” hardware to get quite syncronization times , very precise or precise enough for a “perfect” frame synch based on time on computers synched … Maybe it’s a unusefull approach, i never got time to go for it … but if it could work, it’s a method that just with a simple PTP Daemon running on the system it might work, avoiding to have to deal with network delays or message timestamping …

Go on Joannie !

e*

Thanks for the feedback guys !

@zach @eloi
I like the raspberry option, but getting high framerate with its hardware is tricky, so I would focus on small pc computers first (with 2 outputs per computer), and maybe add raspberry players as an option in the future.

@eloi
well soft edge and warping would be a nice bonus indeed, thanks for the links

Synch wise, I’m thinking about control over DMX, having a “frame_number” variable over network at 30fps shouldn’t be an issue.

I wish I had a clue on how to build things with OF, I guess I’m gonna try to find an good developer in Paris ! Any recommendation ?

Joanie,

It seems that CUDA approach would’nt work on a RaspberyPi (unless I misread the internet), but there’s some potential in your idea. Have you paid attention to the tables on the first gpujpeg project ? They seem to average on a 7.48ms (for a 90% quality) to decompress a HD frame and 20ms for 4K. With that you could get maybe 2 HD movies running at 60 fps, or more at 30 fps. I’m not sure how long the transfer from cuda memory to opengl would take though (but conveniently they do have a sample for that).
I’m not sure what they’ve been up to lately or if they still have oF people among them, but there’s this student vj association called barcotrip in Paris (well Gif sur Yvette) that you could try to approach for geeky visual project (they’re all engineering student, so if you’re lucky someone will know what they’re talking about…)

I think a challenge that could come would be the smoothness of the playback ; like taking into account the fact a frame might take longer to decode, or the DMX message that would arrive just a little bit late, etc. Moving from something ok most of the type, to something always perfect for massive projection mapping could be tricky.

@JoanieLemercier:

Sync is a complex matter. I have in my pipeline to approach to it in the same way that professional television equipment works. Also the network packets of Kinet (the Phillips propietary protocol to control lights) use the same approach.

Basically all the equipment syncs with the same clock (for example using SNTP) and the packets are sent before the triggering time (usually 100 to 200 ms before) with the action and a timestamp the defines when to execute it.

Phillips use that for the media facades to be completely in sync, no matter the amount of pixels.

I have been experimenting with raspberry pi for a artnet/DMX controlled media server. I made simple proto with artnet control thru ola and MSEX updates running on a raspberry pi.

The trouble I was with playing video…The basic requirement in my circles is playing 1080/50 HD video at 30fps… Couldn’t get it to run smoothly on raspberry pi…For single images it worked fine, but the video part didn’t work so good.

If anybody needs help with the DMX or MSEX code I’m willing to share :slight_smile: I spend some time digging into how that stuff works.

@Timoteus very interesting !
Well to be honnest I gave up with video files and codec: encoding is inconvenient, synching frames can be a pain, and codec performances and constraints don’t work for me…

Images sequences is basically the same as video, but without all the constraints…
It would be awesome to know the performances of jpg decoding with openMAX that @zach mentioned:
http://www.raspberrypi.org/forums/viewtopic.php?f=33&t=57721
http://www.raspberrypi.org/forums/viewtopic.php?t=15463

What do you think ?

@drakko very interesting indeed !

If all medias were stored locally, with a buffer (let’s say 10 preloaded frames) do you think that a “frame number” variable, over a fast network, wouldn’t be enough to have all clients synched at 30 fps ?

JPG sequences might offer better performance. Here is one blog with promising results http://my-cool-projects.blogspot.fi/2012/08/hardware-decoding-more-than-fast-enough.html

A simple process for transcoding media from video clips to jpg sequences is needed then. This probably has to happen on a separate computer because the pi might not have enough processing power?

Originally my idea with the raspberry pi was that I could bring video to theater shows that don’t have the budget to run a “real” media server. And have it integrated into the lighting design workflow.

I want to clarify this a bit. What I meant is that the signal is distributed thru a HDMI-SDI converts and possibly combined with camera signals, so it must be sent as 1080i/50hz. But the framerate that different video clips might be playing at must support at least 30hz. Currently with video clips I got somewhere around 15fps performance at 1920x1080 50hz

Theoretically yes, you have 33ms an local network connections tend to have a 1-7ms delivering packets.

But, as you are sending network packages (lets say UDP, for example), packets can be lost or arrive without sync. That creates shutters and lags, or frame loss in some instances. Its a problem that i had to teckle with a remote kinect sender, that some times 2-3 packets arrived to the osc queue in the same rendering frame.

One thing that may help, appart from the SNTP logic is to not sent a reference each frame, but a resync sgnal time to time. HW video players for digital signage do that, they receive the start command with a timestanp, to all start at the same time, but also get a resync order (also related to timestamp for playback) each 1-5 seconds. Those HW works in a master-slave mode, with one media player leading the rest trough the network.

no, you definitely need some kind of common clock if you want perfect sync, some details like fine lines… across 2 screens will look bad if there’s any slight desync and as @drakko explains you can’t really know how long a package is going to take to arrive to the different computers in the network, even if losses are almost non existent in a LAN.

gstreamer which is the default for video playback on the RPI has some utilities to do this, https://developer.gnome.org/gstreamer-libs/unstable/gstreamer-net.html i’ve used it before with small atom computers to play a video splitted in 4 parts with sync over the network: http://vimeo.com/44647112 it’s kind of tricky to setup though

gstreamer should also be able to playback jpg on the pi using openmax, something like:

ofGstUtils gst;
gst.setPixelFormat(OF_PIXELS_NATIVE);
gst.setPipeline("multifilesrc location=%03d.jpg ! decodebin");

should be enough to get hardware accelerated playback of jpeg sequences + gpu color conversion (on OF master)

i’ve also been wanting to do something like this on the rpi + edge blending, warping… for a while but never got to it

So I got a wisdom tooth pulled out yesterday and I have been looking into the jpeg sequence stuff…

Browsing thru some example code on the raspberry pi…Is there any other reason than file size that we couldn’t just skip the jpeg part and go straight to raw files… If the user has to convert video anyway to a image sequence…

There is a example called hello_triangle in /opt/vc/src/hello_pi

That would speed up the loading even more, because the image could be loaded as a texture straight without any decoding?

@arturo Funny timing! Just pushed a first version of ofxGstVideoSyncPlayer for playing back in sync over the network using gstreamer network clocks.

There are a few rough edges and unimplemented features like seeking but it should be a good starting point I think… Ideally more people with gstreamer experience and interest can get involved in order to refine and complete the addon!

It would be great to try it with the RPIs also and see how it performs there. If I find some time next week I will give it a try.

Thanks everyone for your feedback and advice… I have a much better understanding of all the options and requirements for the project.

I spent Sunday compiling my first OF projects with the help of @kikko who made a few prototypes to synch jpg sequences over OSC. Here are the next steps:

CHOICE OF FORMAT / CODEC
I’m currently running a series of benchmarks to fully understand performances on several resolution and hardware:

  • CPU jpg decoding (single and multi thread)
  • GPU jpg decoding (gpujpg & cudajpg)
  • HAP (also discussing codec evolution with @bangnoise.)

CLOCK and Synch

  • OSC: we’ll run some tests with fast network, to see if frame synch works with some content, expecting a 1-3ms delay. Skipping frames and using a buffer might help.
  • CLOCK: we’ll explore gstreamer clock options suggested by @arturo
    Thanks @drakko for th heads up !

HARDWARE:
I’m currently testing a variety of hardware options: i7, i5, i3, Celeron dual and single core, Rasp Pi, and various GPU options as well. Ideally I’d like to be able to mix hardware seemlessly (with specific software for each hardware solution).

I need help to make this happen, so if you’re interested in the project, feel free to get in touch !
I’m aiming at having something working by the end of February, I’ll keep you guys posted !

Hi Joanie !

How did you work it all out finally ? End of february is gone :wink:
Did you manage to build it all up ?
What are your reflexions / impressions / conclusion at this point ?

Would love to hear about your experience as your project sounded very promising …