multiple video at the same time. How to get better perf ?

advanced
#1

Hi,

I’m in the process of moving this app quickly done in as3, to c++/openFrameworks.
http://vimeo.com/11445376
Yes, I know … an other multitouch gallery.

Everything is going fine so far.
I’m just anticipating, the not so great performances I’m going to get when running several videos at the same time (just did a quick test using the ‘moviePlayerExample’, with something like 5 videos - some in HD).

I already found useful info on the forum. i.e. the video format/compression. I ran tests using h.264, mpeg4 and DV, But was not able to get ‘good’ perf yet.

Therefore I’m starting to look for information about the best way to do this.
And clearly understand the limitations too.

Thanks,

-h

#2

hi smallfly,

all the compression methods you mention (even DV) are still fairly processor heavy. try using motion-jpeg (but then your disk access becomes the bottleneck, unless you can put the videos on an enormous ramdrive).

but you might just be trying to push too many pixels to the graphics card. try using scaled-down movies in the gallery (when you have to draw 5 at once) and then only switch to HD (crossfading, keeping the frames in sync) when they go full-screen.

#3

How many videos? And what hardware?

For most of my multi video applications I use either JPEG or Animation Codec:

JPEG - smaller file size + data rate (ie., less HD bandwidth), but greater decompression (ie., greater CPU bandwidth)

Animation - larger file size + data rate (ie., greater HD bandwidth), but much lower decompression (ie., lower CPU bandwidth)

Obviously using lower compression amounts and lower resolutions in either codec give better frame rates…

In the end “performance” or smooth frame rates for multiple videos depends mostly on balancing HD performance against CPU utilization.

For low(er) quality/smaller videos I get decent playback on a mac laptop using JPEG/50% quality. For installations I use Animation Codec (usually 50% quality to keep file sizes down) and RAID 0 hard drive configurations…

Hope that helps some

#4

Thanks damian and gameover.
Really useful information.

I tried both JPEG and Animation codec, and I get ‘better’ performances.

  • JPEG is the best on my current computer (mac book pro unibody 2.8Ghz - 4GB ram). Both HD and CPU bandwidth are able to cope with 5 videos (not super smooth yet). CPU use is at ~65%
  • with ‘Animation’ the files are way bigger and the HD can’t provide enough bandwidth. The CPU use goes down to 12%. I will have to try with faster HD.

I was also looking at the QTKitPlayer sample code available on the apple developer website. The same h264 HD video played using this QTKitPlayer uses a lot less CPU than played with ofVideoPlayer.
Could ‘merging’ this in my actual project, be a way to get better performance ? If so would I still be able to manipulate (ie rotate) the video/texture.

I’m trying to understand why with this QTKitPlayer or even the flash player I’m able to get better perf. (sorry for the dumb question)

hugues

#5

re qtkitplayer, there are all sorts of possible reasons. the quicktime carbon code that we user in of is quite old, and qtkit will be full of vectorisation optimisations, take advantage of 64 bit processing, and could even be multithreaded.

#6

What is your opinion on those RAID 0 external HDs:
I would go with this one. It’s more in my range of price - 1TB but the only 5200rpm. The specs says 120+ MB/second transfer rate. Would it be enough for feeding video file compressed with the Animation codec. The following one is better but too expensive - 4TB at 7200rpm - 200MB/s

For the QTKitPlayer:
Would ‘merging’ this code with OF be possible and if so is it a big job ?
I have quickly tried to merge it and to make a obj-c pointer but without success yet.
If I succeed will I be even able to manipulate (i.e. rotate) the video/texture like with the ofMoviePlayer ?

#7

i think there is an ofxQtKitVideoPlayer or similar add-on available if you search the forums for ‘qtkit video’.

the reason it’s staying out for now is that AFAIK qtkit isn’t available on OSX 10.4 (or maybe even 10.5), so an OF based on qtkit wouldn’t work with 10.4 any more; and i think would require 64 bit builds of all the other libs - which is a bit of nightmare in itself.

#8

I saw a presentation on libavg at FMX in Stuttgart, and it seems to have an impressive video engine, using hardware acceleration where available, and SSE2 ASM otherwise. Could be worth to take a look at the source.

#9

@robotfunk - Thanks. I will have a look to libavg.

@damian - I had a look at ofxQTKitVideoGrabber. Really useful in order to understand how to mix obj-c and c++.
I put together a simple video player in obj-c (that I understand :slight_smile: ). It uses a QTMovieView and a QTMovie and no InterfaceBuilder resources (everything done by code).

MyMediaPlayer.h

  
  
#import <Cocoa/Cocoa.h>  
#import <QTKit/QTKit.h>  
  
@interface MyMediaPlayer : NSObject <NSApplicationDelegate> {  
    NSWindow *window;  
}  
  
@property (assign) IBOutlet NSWindow *window;  
  
- (void)addMovie;  
  
@end

MyMediaPlayer.mm

  
  
  
#import "MyMediaPlayer.h"  
  
@implementation MyMediaPlayer  
  
@synthesize window;  
  
- (void)applicationDidFinishLaunching:(NSNotification *)aNotification {  
	[self addMovie];	  
}  
  
- (void)addMovie {  
  
 QTMovieView     *aMovieView;  
 QTMovie         *aMovie;  
 NSRect          contentFrame;  
   
 contentFrame = [[window contentView] frame];  
   
 aMovieView = [[QTMovieView alloc] initWithFrame:contentFrame];  
  
 [aMovieView setWantsLayer:YES];  
 [aMovieView setControllerVisible:NO];  
 [aMovieView setPreservesAspectRatio:YES];  
   
 aMovie = [[QTMovie alloc] initWithFile:@"video_HD.mov" error:nil];  
 [aMovieView setMovie:aMovie];  
 [[window contentView] addSubview:aMovieView];  
 [aMovie play];  
}  
   
-(void)dealloc {  
    [super dealloc];  
}  
  
@end

Now I’m trying to implement this in a c++/OF project.
I’m starting from ofxQTKitVideoGrabber. Just added to it the obj-c for the QTMovie. The project compiles.
Now I’m stuck with how to draw the video - how to replace/implement the QTMovieView ?
I did not find much info about that.

I was thinking to copy each frame of the video, or to copy the pixels of each frame to an ofTexture (as ofxQTKitVideoGrabber does with the capture device). But by doing this I would loose the increase in performance I get.

Any suggestions on how to do this would help.

#10

I have an experimental version of ofxQTKitMoviePlayer i’m using in a current project. I plan on cleaning it up afterwords and sharing it. In the meantime, here is what I did to get it working…

QTMovieView uses only Core Image under the hood, and doesn’t make it easy to get at the openGL textures which is what we need to connect to OF. But quicktime has a “visual context” object that you can attach to QTMovies that then renders their frames to OpenGL textures

Look at the QTCoreVideo102 sample code, my class is totally based off of this. The only leap was how to create a visual context in OF.

in their code they do this:

  
  
CGLContextObj      cglContext     = (CGLContextObj)[theOpenGLContext CGLContextObj];  
CGLPixelFormatObj  cglPixelFormat = (CGLPixelFormatObj)[theOpenGLPixelFormat CGLPixelFormatObj];  
CFDictionaryRef    dictAttribs    = NULL;  
  
OSStatus err = QTOpenGLTextureContextCreate(	allocator,  cglContext,	 cglPixelFormat, dictAttribs, &visualContext->context );

but we don’t have a an NSOpenGLContext or NSOpenGLPixelFormat in OF since we dont have an NSOpenGLView. But luckily we can do this:

  
  
QTOpenGLTextureContextCreate(NULL, CGLGetCurrentContext(), CGLGetPixelFormat(CGLGetCurrentContext()), NULL, &visualContext);

then add that visual context to the QTKit movie via:
[movie setVisualContext:visualContext];

also if you are running Snow Leopard, you can get extra hardware acceleration through setting QTMovieOpenForPlaybackAttribute to yes when opening the movie (opts into QuicktimeX)

I get decent framerates at HD+ quality movies, but have seen some slow down playing multiple movies at once. so this is something I want to work through before publishing the code.

Hope this helps a little!

#11

Hey there

For some reason I’m not getting notifications for some threads so missed a lot of these developments!

My five cents about getting better performance:

Platform
I don’t have a MacPro to really benchmark these things, but I consistently get better video performance on PCs than Macs, at a lower cost - I don’t really like having to use PC’s, although Windows 7 makes me like it a little more…

RAID
If you’re getting a RADI get the fastest RAID you can afford. Definetly use at least 7200rpm drives. I’ve been getting really good results from Hitachi 7200rpm 2Tb drives. Often it’s best/economic to buy the RAID enclosure seperate, and then put your own drives in. Since I’m doing a lot of video editing/effects I recently went down the path of investing in a RAID control card and an external RAID unit - definetly not cheap but amazing speed. For installations I’ve been using on-board RAID controllers (ICH9/10 or even nForce) to setup the disks - 2 drives in RAID 0 give me ~185Mb/s.

ofxQtKitVideoPlayer
Been looking into QTCoreVideo102 examples and trying/failing to integrate with oF - objective-C is totally new to me, so no surprises there, but I’ll keep on plugging…

@obviousjim: what kind of improvements in quality and/or framerates are you seeing over ofVideoPlayer?

Quality vs Multiplicity
I don’t really want to divert this thread, but one thing I’ve spent a lot of time trying to figure out is why I can’t even get a single HD video clip to play back frame-accurately in any openGL environment - even with super beefy computers and RAID setups. After a fair bit of research it seems this is to do with the fact that really smooth video playback is predicated on having some kind of gen-lock on the output source.

The web-lore goes that since video cards are not gen-locked and their refresh frequencies do not match the video frame rates, there will always be some tearing.

There are plenty of threads dedicated to this on Processing and Max/Jitter forums too. One thing that does interest me about the idea of integrating aspects of QTKit/CoreVideo is the prospect of creating something like Final Cuts print-to-video function - whereby RT-effects (really just openGL?) are able to be pushed out via a camera/video card/Matrox-MXO to give us flicker-free, frame accurate output…wish wish wish??

Thoughts?

M

#12

You could try my ofxQTKitVideoTexture, it’s pretty rough around the edges, but it might help you out. It’s a drop in replacement for ofVideoPlayer, except you call update() instead of idleMovie()

http://code.google.com/p/ofxmarek/downl-…-exture.zip

I’m getting much better framerates with it at 1920x1080.

Mac only. Only tested with SDK 10.6 too.

#13

Thanks all for the great replies.

I have also found those two posts:
http://forum.openframeworks.cc/t/alpha-channel-in-.mov/954/2 - arrays of ofTextures (not sure this would be the way to go for what I currently doing, as my videos are quite hi-res)

I also realized that I won’t be able to try with RAID0 setup right away, as my development computer is a mac book pro unibody and apple had the great idea to replace the express card 34 slot by and SD reader.

-h

#14

Platform
I don’t have a MacPro to really benchmark these things, but I consistently get better video performance on PCs than Macs, at a lower cost - I don’t really like having to use PC’s, although Windows 7 makes me like it a little more…

Im also working with a mount of videos at the same time, first wirking on windows (works nice) , now going to MAC OS X -> It has gone down extremely…
I don’t know why, but if it loads all the videos at memory, the rest of OF program is blocked for each second… Checking the frame rate (by OF command) it’s normal (around 400) because im not uploading and drawing anything. But as I commented before the “general” frame rate is too lower, step by step each one second more or less…

About higher video performance
What about gstreamer (at least from windows) I’ve got really nice result playing all video formats and big avi videos (1Gb)

Using Ram-Disk as solution, it goes faster but still step by step…

#15

HI marek

Im trying to compiled simple videoPLayer with this ofxQTKitVideoTexture , but I don’t know how to

  • add QTKit.framework and CoreVideo.framework to your project.

Im novice using this…
I’ve tried to go all files into the project, and Im with this kind of error

“_CVOpenGLTextureRelease”, referenced from:
ofxQTKitVideoTexture::update() in ofxQTKitVideoTexture.o

#16

Hi Charlie,

One way to do it :

Then select the frameworks from the list.

Hugues

#17

Hey, a little late here, but finally got a spare moment to put together a more complete add on for playing videos through QTKit in openFrameworks

I’ve published it to my git: https://github.com/obviousjim/FlightPha-…-ideoPlayer

To get better performance I made it possible to open the videos in different “modes”. If you just want to draw the video you can open it in texture only, and it’ll be really fast for big / multiple videos. This is how Marek’s addon works too.

But if you want pixel access you can open it in pixel mode, and it’ll still be pretty quick and lets you play videos with Alpha too.

The git has an example project that shows the different ways to use it.

Thanks to everyone who provided code snippets and advice along the way, this was a tricky thing to get right.

Hope it’s a useful addition =)

#18

Awesome!!! I have been waiting for this.
Thanks!
Tim

#19

I can’t seem to be able to get the setLoopState(…) or getIsMovieDone() methods to work.

Does anyone else have this problem?

#20

@sebben - bummer!

it could be an operating system thing. I only tested this on OS 10.6.5

what os version are you running?

the example project you can hit “l” to change loop state, and it should print it out in the on-screen display.