A DirectShow video player for openFrameworks

Just got this hooked up today.

It is video player that uses the native DirectShow engine in Windows.
It should offer better performance than the default QuickTime based player, and will help with a 64bit release of OF for VS and CB.

I still have some things to add to it, but it has pretty much 1:1 parity with the current player.
Note: grab the K-Lite Mega Codec Pack 10.2 to allow it to play back QuickTime mov files.

More info and download here:

Cheers!
Theo

6 Likes

This is great, something that I’ve been chatting with folks at Microsoft about anyways, really cool to see that you’ve just gone ahead and gotten it started. I’d give it a test but I just left my windows machine at home, any significant speed boost with larger (HD-ish) videos?

ah - nevermind, I see you just answered on the mailing list :slight_smile: anyhow, great work!!

Thanks!
:slight_smile:

For the benefit of the forum:

Currently it loads videos about 30% faster the QuickTime and it seems to play them back at about 5-10 times the rate.
( a 1080p mp4 video was unplayable on my 2009 MBP with the QT player and audio was out of sync, ofDirectShowPlayer had a decent fps and synchronized audio ).

Since the fast post I switched the capture to a callback based solution that should be faster than the original commit.

1 Like

Thanks for this meaningful contribution !

Out of curiosity, why would you use Direct Show rather than Media Foundation ?
A couple of weeks ago I was posting about hardware acceleration, by processing the samples on the CPU side you’re not taking advantage of hardware acceleration such as DXVA or HW decoder, am I correct ? (ie for instance do you have a big cpu spike if you play some h264 hd video ?)

Thanks Theo!!

This has come just in time, as we are about to launch an installation that uses a video player on windows where we have definitely been having some pains.

Josh and I have also been chatting with the MS folks about Media Foundation. The challenge there is that Media Foundation is pretty closely connected to Direct X, and seems like it’ll be tricky to get the pixels or textures into openFrameworks’ openGL context. While DirectShow is apparently deprecated, it seems like a great alternative to QuickTime.

I think Direct Show and Media foundation are equally close to DirectX. (but well clearly if it was openGL it would make everything so much easier).

Getting the pixels using WMF is probably the same level of complexity as what Theo did with direct show (you use a sample grabber sink instead (http://msdn.microsoft.com/en-us/library/windows/desktop/hh184779(v=vs.85).aspx)).

As for getting the texture in oF I had to work for that on a project. An approach is to build a custom EVR and then use the nvidia_interop extension to share the texture between a D3D surface and your openGL context. It’s not fun but can work well. I’m pretty the same can be done with Direct Show…

1 Like

That’s really promising to hear!

I’m definitely speaking in the blind with a lot of this - based on conversations not actually coding. If you could post an example of WMF working like Theo’s direct show, or an OF example of the NVDIA interop extension that would go a long way to see the way forward with video on windows

btw, what happened to using gstreamer on windows? wouldn’t that work well, too?

Hey Silverbahamut, James!

The truth is I was half way through writing the code before I even realized that Media Foundation was the new way to handle this stuff. : )

In terms of the CPU/GPU stuff, my original goal was to get something fairly uncomplicated working first.
The current way the player is working is that we tell it that our grabber requires RGB_24 pixels. DirectShow handles that conversion internally, so it might be hardware optimized. But then when we get the buffer back, we have to flip the bytes ( it returns BGR ) and flip the video vertically, so that adds a little overhead.
However even with this approach decoding, on a 1080p video its is a LOT faster than the current implementation.

Thanks for posting the Media Foundation stuff, it looks very familiar, much like DS but less nasty!
I’m going to check it out. DirectShow is one of my less favorite APIs, so it can’t be worse!

@bilderbuchi I think there is a working Gstreamer on Windows, the issue is it is quite a large library and adds a lot of bloat to an app. Whats nice with my approach is its just two classes and no additional libraries being added to the linker.

Theo

1 Like

@theo,this is super.btw,for gstreamer on windows,i’m using it quite a lot,the library its large,but only need 14 libs to complied unless you really need doing some advanced stuff.

To be honest I feel like MS api are gross but I really love media foundation. How the topology work is plain magic to me. I was able to play a couple of HD video next to each other nearly frame synced and that’s really an impressive work from Microsoft I think.

@obviousjim : I will share samples on how to mix directx texture with oF. I hope this way it will help further the work on hardware acceleration and integrate ms technology to oF!

1 Like

Thanks @silverbahamut !
Any code would be helpful, I wouldn’t want to duplicate your existing efforts.

The texture stuff still seems a little iffy to me ( from reading about its support on AMD/ATI based cards )
But it could be a nice option, with a non-hardware optimized grabbing solution as an alternative.

Thanks!
Theo

AMD says they support it (my vaio laptop with a 2 year old graphic card do support it actually… or at least it says it does, I’ m still having a few problem having it running smoothly)

But apart from that, great news, I was talking for myself in my previous posts, but after a chat with my boss I’ll be able to opensource the work I did on windows media foundation. It will need some cleaning up first but should come out in a couple of days :slight_smile:

1 Like

Thanks Theo this is great. Slowing my development has shifted away from OSX and towards windows and this really helps.

I was wondering if any has got alpha video running yet?
When I try I get an error on

 Line 587 m_pGrabber->GetConnectedMediaType(&mt);

Also, I had to install the KLite codec pack to get MP4 encoded videos to play also.

Cheers
Az

Thanks to Josh and James for letting me know about this thread. I am from Microsoft and can provide some insight into the media pipelines available on Windows. As James stated, DirectShow is legacy while it is available for desktop development it has not been updated for a long time. The SampleGrabber IS deprecated and shouldn’t be used. It is based on DirectX Editing Services, there hasn’t seen an update since Windows XP. If you use this, you should test the long run scenarios fully before deploying.

If you just want the simple Sample Grabber type functionality, really consider Media Foundation and the MF Source Reader. I have given James some sample code to show how easy it is to use.
Using the Source Reader to Process Media Data

The benefits of this are you can configure the output to pull the uncompressed NV12 or have it include the hardware transform to convert to ARGB. I will agree that if you need the full media pipeline is can get a bit complex to setup the full media session/event/EVR, but if the intent is to get the resulting frame you don’t need it. Also understand, using Sample Grabber or for that matter MFSourceReader you loose the pipeline keep things in-sync (audio/video). With the SourceReader, you get the advantage of configuring both audio and video output samples, but presenting this against a presentation clock would fall to the framework.

1 Like

Thanks @carmines

From the link it seems fairly straightforward and definitely easier than what I know from DirectShow.
Seems like it would be a good idea to make a MF version of this player.

Would that same logic apply to camera capture too?

OF is using my VideoInput library which is also based on DirectShow ( developed in 2006-2009 ).
Can MF access all the same cameras / devices as a DirectShow based solution? Does it also expose the same camera settings ( like shutter/gain/format etc )

Thanks!!!
Theo

Thanks Theo for this very promising advance,

Sorry to interrupt this interesting conversation with a perhaps trivial question. Should this code work under CB or is it specific to VS?

I’ve tried to compile with CB and I get the following errors:

||=== videoPlayerExample, release ===|
c:\program files (x86)\codeblocks\mingw\bin…\lib\gcc\mingw32\4.7.1…\include\strmif.h|661|error: ‘IUnknown’ does not name a type|
c:\program files (x86)\codeblocks\mingw\bin…\lib\gcc\mingw32\4.7.1…\include\strmif.h|991|error: ‘LPDDPIXELFORMAT’ does not name a type|
c:\program files (x86)\codeblocks\mingw\bin…\lib\gcc\mingw32\4.7.1…\include\strmif.h|1073|error: expected class-name before ‘{’ token|
c:\program files (x86)\codeblocks\mingw\bin…\lib\gcc\mingw32\4.7.1…\include\strmif.h|1090|error: expected class-name before ‘{’ token|
c:\program files (x86)\codeblocks\mingw\bin…\lib\gcc\mingw32\4.7.1…\include\strmif.h|1114|error: expected class-name before ‘{’ token|
c:\program files (x86)\codeblocks\mingw\bin…\lib\gcc\mingw32\4.7.1…\include\strmif.h|1123|error: expected class-name before ‘{’ token|
c:\program files (x86)\codeblocks\mingw\bin…\lib\gcc\mingw32\4.7.1…\include\strmif.h|1133|error: expected class-name before ‘{’ token|
c:\program files (x86)\codeblocks\mingw\bin…\lib\gcc\mingw32\4.7.1…\include\strmif.h|1145|error: expected class-name before ‘{’ token|
c:\program files (x86)\codeblocks\mingw\bin…\lib\gcc\mingw32\4.7.1…\include\dsound.h|230|error: ‘WAVEFORMATEX’ does not name a type|
c:\program files (x86)\codeblocks\mingw\bin…\lib\gcc\mingw32\4.7.1…\include\dsound.h|349|error: ‘LPWAVEFORMATEX’ does not name a type|
c:\program files (x86)\codeblocks\mingw\bin…\lib\gcc\mingw32\4.7.1…\include\dsound.h|365|error: ‘LPWAVEFORMATEX’ does not name a type|
c:\program files (x86)\codeblocks\mingw\bin…\lib\gcc\mingw32\4.7.1…\include\dsound.h|416|error: ‘LPWAVEFORMATEX’ does not name a type|
c:\program files (x86)\codeblocks\mingw\bin…\lib\gcc\mingw32\4.7.1…\include\dsound.h|425|error: ‘LPWAVEFORMATEX’ does not name a type|
c:\program files (x86)\codeblocks\mingw\bin…\lib\gcc\mingw32\4.7.1…\include\dsound.h|686|error: ‘LPWAVEFORMATEX’ has not been declared|
c:\program files (x86)\codeblocks\mingw\bin…\lib\gcc\mingw32\4.7.1…\include\dsound.h|696|error: ‘LPCWAVEFORMATEX’ has not been declared|
c:\program files (x86)\codeblocks\mingw\bin…\lib\gcc\mingw32\4.7.1…\include\dsound.h|770|error: ‘LPWAVEFORMATEX’ has not been declared|
c:\program files (x86)\codeblocks\mingw\bin…\lib\gcc\mingw32\4.7.1…\include\dsound.h|780|error: ‘LPCWAVEFORMATEX’ has not been declared|
c:\program files (x86)\codeblocks\mingw\bin…\lib\gcc\mingw32\4.7.1…\include\dsound.h|1041|error: ‘LPWAVEFORMATEX’ has not been declared|
c:\program files (x86)\codeblocks\mingw\bin…\lib\gcc\mingw32\4.7.1…\include\dsound.h|1099|error: ‘LPWAVEFORMATEX’ has not been declared|
c:\program files (x86)\codeblocks\mingw\bin…\lib\gcc\mingw32\4.7.1…\include\ddraw.h|259|error: conflicting declaration ‘typedef struct _DDCOLORKEY DDCOLORKEY’|
c:\program files (x86)\codeblocks\mingw\bin…\lib\gcc\mingw32\4.7.1…\include\strmif.h|737|error: ‘DDCOLORKEY’ has a previous declaration as ‘typedef struct DDCOLORKEY DDCOLORKEY’|
c:\program files (x86)\codeblocks\mingw\bin…\lib\gcc\mingw32\4.7.1…\include\amvideo.h|56|error: expected ‘;’ at end of member declaration|
c:\program files (x86)\codeblocks\mingw\bin…\lib\gcc\mingw32\4.7.1…\include\amvideo.h|56|error: ‘rcTarget’ does not name a type|
c:\program files (x86)\codeblocks\mingw\bin…\lib\gcc\mingw32\4.7.1…\include\mmsystem.h|1570|error: ‘LPCWAVEFORMATEX’ has not been declared|
c:\program files (x86)\codeblocks\mingw\bin…\lib\gcc\mingw32\4.7.1…\include\mmsystem.h|1591|error: ‘LPCWAVEFORMATEX’ has not been declared|
C:\Users\gatopardo\Documents\of_v0.8.0_win_cb_release\apps\myApps\videoPlayerExample\src\ofDirectShowPlayer.cpp|38|error: ‘IMediaSample’ has not been declared|
C:\Users\gatopardo\Documents\of_v0.8.0_win_cb_release\apps\myApps\videoPlayerExample\src\ofDirectShowPlayer.cpp|68|error: ‘IMediaSample’ has not been declared|
C:\Users\gatopardo\Documents\of_v0.8.0_win_cb_release\apps\myApps\videoPlayerExample\src\ofDirectShowPlayer.cpp|84|error: ‘IPin’ has not been declared|
C:\Users\gatopardo\Documents\of_v0.8.0_win_cb_release\apps\myApps\videoPlayerExample\src\ofDirectShowPlayer.cpp||In function ‘long int GetUnconnectedPin(IBaseFilter*, PIN_DIRECTION, int**)’:expressionless:
C:\Users\gatopardo\Documents\of_v0.8.0_win_cb_release\apps\myApps\videoPlayerExample\src\ofDirectShowPlayer.cpp|87|error: ‘IEnumPins’ was not declared in this scope|
C:\Users\gatopardo\Documents\of_v0.8.0_win_cb_release\apps\myApps\videoPlayerExample\src\ofDirectShowPlayer.cpp|87|error: ‘pEnum’ was not declared in this scope|
C:\Users\gatopardo\Documents\of_v0.8.0_win_cb_release\apps\myApps\videoPlayerExample\src\ofDirectShowPlayer.cpp|88|error: ‘IPin’ was not declared in this scope|
C:\Users\gatopardo\Documents\of_v0.8.0_win_cb_release\apps\myApps\videoPlayerExample\src\ofDirectShowPlayer.cpp|88|error: ‘pPin’ was not declared in this scope|
C:\Users\gatopardo\Documents\of_v0.8.0_win_cb_release\apps\myApps\videoPlayerExample\src\ofDirectShowPlayer.cpp|89|error: invalid use of incomplete type ‘struct IBaseFilter’|
…\libs\videoInput\include\videoInput.h|176|error: forward declaration of ‘struct IBaseFilter’|
C:\Users\gatopardo\Documents\of_v0.8.0_win_cb_release\apps\myApps\videoPlayerExample\src\ofDirectShowPlayer.cpp|100|error: ‘pTmp’ was not declared in this scope|
C:\Users\gatopardo\Documents\of_v0.8.0_win_cb_release\apps\myApps\videoPlayerExample\src\ofDirectShowPlayer.cpp||In function ‘long int DisconnectPins(IBaseFilter*)’:expressionless:
C:\Users\gatopardo\Documents\of_v0.8.0_win_cb_release\apps\myApps\videoPlayerExample\src\ofDirectShowPlayer.cpp|123|error: ‘IEnumPins’ was not declared in this scope|
C:\Users\gatopardo\Documents\of_v0.8.0_win_cb_release\apps\myApps\videoPlayerExample\src\ofDirectShowPlayer.cpp|123|error: ‘pEnum’ was not declared in this scope|
C:\Users\gatopardo\Documents\of_v0.8.0_win_cb_release\apps\myApps\videoPlayerExample\src\ofDirectShowPlayer.cpp|124|error: ‘IPin’ was not declared in this scope|
C:\Users\gatopardo\Documents\of_v0.8.0_win_cb_release\apps\myApps\videoPlayerExample\src\ofDirectShowPlayer.cpp|124|error: ‘pPin’ was not declared in this scope|
C:\Users\gatopardo\Documents\of_v0.8.0_win_cb_release\apps\myApps\videoPlayerExample\src\ofDirectShowPlayer.cpp|125|error: invalid use of incomplete type ‘struct IBaseFilter’|
…\libs\videoInput\include\videoInput.h|176|error: forward declaration of ‘struct IBaseFilter’|
C:\Users\gatopardo\Documents\of_v0.8.0_win_cb_release\apps\myApps\videoPlayerExample\src\ofDirectShowPlayer.cpp|146|error: ‘IPin’ has not been declared|
C:\Users\gatopardo\Documents\of_v0.8.0_win_cb_release\apps\myApps\videoPlayerExample\src\ofDirectShowPlayer.cpp||In function ‘long int ConnectFilters(IGraphBuilder*, int*, IBaseFilter*)’:expressionless:
C:\Users\gatopardo\Documents\of_v0.8.0_win_cb_release\apps\myApps\videoPlayerExample\src\ofDirectShowPlayer.cpp|160|error: ‘IPin’ was not declared in this scope|
C:\Users\gatopardo\Documents\of_v0.8.0_win_cb_release\apps\myApps\videoPlayerExample\src\ofDirectShowPlayer.cpp|160|error: ‘pIn’ was not declared in this scope|
C:\Users\gatopardo\Documents\of_v0.8.0_win_cb_release\apps\myApps\videoPlayerExample\src\ofDirectShowPlayer.cpp|167|error: invalid use of incomplete type ‘struct IGraphBuilder’|
…\libs\videoInput\include\videoInput.h|175|error: forward declaration of ‘struct IGraphBuilder’|
C:\Users\gatopardo\Documents\of_v0.8.0_win_cb_release\apps\myApps\videoPlayerExample\src\ofDirectShowPlayer.cpp||In function ‘long int ConnectFilters(IGraphBuilder*, IBaseFilter*, IBaseFilter*)’:expressionless:
C:\Users\gatopardo\Documents\of_v0.8.0_win_cb_release\apps\myApps\videoPlayerExample\src\ofDirectShowPlayer.cpp|187|error: ‘IPin’ was not declared in this scope|
C:\Users\gatopardo\Documents\of_v0.8.0_win_cb_release\apps\myApps\videoPlayerExample\src\ofDirectShowPlayer.cpp|187|error: ‘pOut’ was not declared in this scope|
C:\Users\gatopardo\Documents\of_v0.8.0_win_cb_release\apps\myApps\videoPlayerExample\src\ofDirectShowPlayer.cpp||In function ‘void LocalFreeMediaType(AM_MEDIA_TYPE&)’:expressionless:
C:\Users\gatopardo\Documents\of_v0.8.0_win_cb_release\apps\myApps\videoPlayerExample\src\ofDirectShowPlayer.cpp|208|error: ‘AM_MEDIA_TYPE’ has no member named ‘pUnk’|
C:\Users\gatopardo\Documents\of_v0.8.0_win_cb_release\apps\myApps\videoPlayerExample\src\ofDirectShowPlayer.cpp|211|error: ‘AM_MEDIA_TYPE’ has no member named ‘pUnk’|
||More errors follow but not being shown.|
||Edit the max errors limit in compiler options…|
||=== Build finished: 50 errors, 0 warnings (0 minutes, 14 seconds) ===|

I’ve tried to download the Windows SDK and replace the strmif.h that comes with mingw32 with the file in the SDK but it hasn’t worked.

Thanks.

Yes, with a caveat. Getting capture to work in Windows 7 takes a bit more setup and would recommend Windows 8 for capture. Windows 7 focused on simplifying playback/trancode, and Windows 8 added better support for capture scenarios with the capture engine.

@theo I’m cleaning the work I’ve done with WMF and will release the code something like today or tomorrow, if you don’t want to waste to much time you can wait a little bit and potentially build upon what I’ll release.

@carmines I have a question about playing multiple video in sync within WMF. The best I was able to achieve was mostly in sync with from time to time a frame off (using the Sequencer Source). Do you feel it’s the best that can be achieved ? (I was mostly following what topoedit does when multiple sources are present). Our problem was to play a H264 video that was too big for Win7 decoder, so we tried to split it in two and then framesync it… (we then switched to wmv which was more tolerant on the file size)

The sequencer source was meant to provide a playlist sequencer support for a media session. The idea here is the pipeline can pre-load the next source to support tapeless playback.

If you are trying to pull MFSamples from multiple sources then you want to create an Aggregate Source`