export audio and/or video using ffmpeg

hey all!


FIRST OF ALL: sorry for the long post, this is rather cause i tried be comprehensive, not becasue it’s crazy complicated.
Download (with sample code) is at the end of this post.

i’m working on a/v export for soundythingie which resulted in this little addon that renders audio/video to a file. for portability reasons i’ve decided to go for ffmpeg instead of quicktime so this should work on most platforms.

the package contains the source files of the main class “FFMovieMaker.mm” (just rename to .cpp if you’re on windows) as well as a very minimal ffmpeg universal binary that is compiled for newer macos and armv7 (iphone4/ipad). if those binaries aren’t suitable for you please compile your own (i’ll get into that later).

how to export video (optionally with audio)

first somewhere in setup you’ll want to create your moviemaker and adjust the options:

  
  
// output video will be 640x480. if your app runs at a different    
// resolution you can pass in an additional two params for the   
// source width/height  
movieMaker = new FFMovieMaker( "my-filename.mpg", 640, 480 );  
  
// now go nuts on the options, check FFMovieMaker.h for the   
// struct definition with short explanations  
movieMaker->config.v_destFrameRate = 25;   
  
// those are only needed if you have audio as well  
// again, check the defaults in the struct,   
// maybe you don't need to change anything!   
movieMaker->config.a_srcSampleRate = SAMPLE_RATE;   
movieMaker->config.a_srcChannels = NR_CHANNELS;   
  
// all options set? go!   
movieMaker->begin();   
  

once you’ve done this you’ll want to add frames into the video (presumably inside draw), the easiest way of doing so is to write

  
  
movieMaker->addScreen();   
  

you can also pass in an pixels array directly. there’s a function movieMaker->addPixels( int * pixels ); look at the source of addScreen() to see how to use it.

if you don’t need audio skip this paragraph. but if you also want to record audio it gets a bit more complicated. the easiest way to accomplish it is by disabling audio output during record. i do that as follows:

  
  
void audioRequested( float * data, int buffSize, int channels ){  
  if( isYourThingRecording )  
     // uh... don't generate audio!   
     for( int i = 0; i < buffSize*channels; i++ ) data[i] = 0;   
  else{  
     calculateAudio( data, buffSize, channels );   
}  
  
void calculateAudio( float * data, int buffSize, int nChannels ){  
  // your original audioRequested code goes here!  
}  
  
  
// finally, inside draw add this:   
if( movieMaker->wantsAudio() ){  
	float temp[BUFFER_SIZE*CHANNELS];   
	calculateAudio( temp, BUFFER_SIZE, CHANNELS );   
	movieMaker->addAudio( temp, BUFFER_SIZE*CHANNELS );   
}  
  

when you recorded enough material call:

  
  
movieMaker->finish();   
  

DON’T call any function on the movieMaker after that, your app will crash!

how to export just audio
that is much easier, setting the FFMovieMaker up is the same, just pass in 0 for the width and the height

  
movieMaker = new FFMovieMaker( "my-file.mp2", 0, 0 );   

and no video stream will be added to the output.

now, because there are no threading issues, you can just forward the audio generated inside audioRequested using

  
  
void audioRequested( float * output, int bufferSize, int nChannels ){  
	// fill the buffer... and then:  	  
	movieMaker->addAudio( output, BUFFER_SIZE*CHANNELS );   
}  
  

when you’re done don’t forget to call movieMaker->finish();

where to get the proper ffmpeg binaries
as i said above, this download contains a very very minimal set of ffmpeg binaries (only compiled for macos+armv7, only mpeg1-video and mpeg2-audio, everything else excluded from the build).

if you want to build for windows/linux/mac os it should be straight forward,
download the ffmpeg source, then run the usual ./configure [your options] && make
this is the minimum set of configure options you’ll need:
–enable-swscale --enable-encoder=mpeg1video --enable-encoder=mp2 --enable-encoder=pcm_s16le --enable-muxer=mpeg1video

if you want to compile your own iphone version (e.g. armv6) i point you to this great project:
https://github.com/gabriel/ffmpeg-iphone-build
it’s a bit of a mess and needs some tweeking, but with some love you can get it working.

another thing you can do is look around for pre-built binaries. i’ve seen a few movie players based on ffmpeg in this forum, if you’re lucky there’s one for your platform and they have the right stuff compiled into it.

there are many more things this addon could support (like choosing codecs) but i just don’t have the time.
please paste back here if you have fixes, ffmpeg binaries, examples or whatnot.

allright, hope this is a good starting point.
enjoy…
best, hansi.

p.s. here’s the download:
ofxFFMovieMaker v2.10764 + minimal ffmpeg binaries for iphone/mac os + sample code for recording audio and video together.
http://asdfg.me/up/ofxFFMovieMaker-with-sample.zip

also look for posts by “plong0” below, he posted a very simple example for exporting just video.

p.p.s. Troubleshooting:
whatever kind of problem you run into, first you should enable debugging output, i.e.

  
  
movieMaker = new FFMovieMaker( ... );   
av_log_set_level( AV_LOG_INFO ); // this needs to be done _after_ creating the FFMovieMaker instance. by default it's set to AV_LOG_FATAL.   
  

this should give you a whole lot of additional output.

This is fantastic! Can’t wait to check it out.

FYI: I’m working on debugging this right now, but if you are running oF0062, there is a conflict:

/System/Library/Frameworks/QuickTime.framework/Headers/ImageCompression.h:318: error: conflicting declaration ‘typedef OSType AVMediaType’
…/…/…/addons/ofxFFMovieMaker/include/libavutil/avutil.h:79: error: ‘enum AVMediaType’ has a previous declaration as ‘enum AVMediaType’

Any ideas?

==edit==
Also, could you provide an example of the fillAudioBuffer function you mention? I am trying to make just a minimal example using audioOutputExample as a base.

uh, that’s not good. i’m using 0061, not 0062.

it means that the AVMediaType enum exists in quicktime, as well as in ffmpeg.
i asked around and it _might_ work to just do a search-and-replace.
replace “AVMediaType” with “FFMPEG_AVMediaType” in all files (.h, .mm, .cpp) inside:

  • ofxFFMovieMaker/src/**
  • ofxFFMovieMaker/include/**

really hope that works… if i have time i’ll download 0062 and try myself.


sorry i don’t have time right now to test myself, but here’s an alternative plan you can try:

in ofxFFMovieMaker/src/FFMovieMaker.mm there are a bunch of clals to ofGetSystemTime, it’s always a bit like
cout << “encode: " << ( ofGetSystemTime() - a ) << " ms” << endl;
a = ofGetSystemTime();

remove all those, they’re not needed anyways.
then open FFMovieMaker and wrap the ffmpeg includes in a namespace, so it looks like this:

[code=cpp]

#ifdef __cplusplus
namespace ffmpeg{
extern “C”{
#endif
#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#include “libavcodec/avcodec.h”
#include “libavformat/avformat.h”
#include “libswscale/swscale.h”
#include “libavutil/log.h”
#include “libavutil/opt.h”
#ifdef __cplusplus
}
}
#endif

(maybe you need to drop the extern “C” now… dunno).
also remove the include of “ofMain.h”, it’s not needed anymore.

finally go back to the FFMovieMaker.mm file and

[code=cpp]
using namespace ffmpeg;

[code=cpp]

somewhere at the top.

Actually, I tracked it down to avcodec.h, line 406

  
#define CodecType AVMediaType  

The problem was that “CodecType” is also used in Quicktime, so it was being replaced with AVMediaType. So I just changed it to

  
#define FFCodecType AVMediaType  

I did a quick search/replace, and I couldn’t find anywhere else in the FFMPEG headers that the macro was used, and it seems to compile fine…

[quote author=“jefftimesten”]Actually, I tracked it down to avcodec.h, line 406

  
#define FFCodecType AVMediaType  

I did a quick search/replace, and I couldn’t find anywhere else in the FFMPEG headers that the macro was used, and it seems to compile fine…[/quote]

i solved this now using namespaces, it’s in the download in the original post along with an example xcode project for 0062.

Hello,

I think this is just what I need. I am wanting to capture video from a webcam or even the kinect RGB to a video file I can save and playback.

I am working to see if I can get this going win 7 VS 2008. A little over my head but plowing forward.

Currently I am getting this: libavutil\common.h(31) : fatal error C1083: Cannot open include file: ‘inttypes.h’: No such file or directory

I have found no inttypes.h in the files. I expected to find it in the libavutil folder.

Worth saying I have had enough trouble that I am checking in to see if anyone has tackled this or has a better suggestion for capturing video.

Thanks for any time!

[quote author=“jaredrawk”]Hello,

I think this is just what I need. I am wanting to capture video from a webcam or even the kinect RGB to a video file I can save and playback.

I am working to see if I can get this going win 7 VS 2008. A little over my head but plowing forward.

Currently I am getting this: libavutil\common.h(31) : fatal error C1083: Cannot open include file: ‘inttypes.h’: No such file or directory
[/quote]

hey!

you should replace the entire lib and include folder with windows versions of ffmpeg (the download only contains an iphone/mac version).

i searched around for a bit and this seems to be the route to takes:

[list type=decimal]

  • download ffmpeg binaries from here: http://ffmpeg.arrozcru.org/autobuilds/
    make sure you get a dev-package from the very bottom of the page for the appropriate platform (e.g. win, 32bits, dev). if you downloaded the right thing it will contain a lib and an include folder.

  • trash the lib and include folders from ofxFFMovieMaker and replace them with the ones you just downloaded.

  • add addons/ofxFFMovieMaker/include to your include path. it depens on the IDE you’re using how that actually works. however, ensure you add the path recursively, if you’re IDE doesn’t have that option add the ofxFFMovieMaker/include folder and also all it’s subfolders individually.

  • somehow somewhere you need to add the lib folder to the linker path. in xcode this is done by just adding the files to the project, in other IDEs you might need to find a “linker path” setting somewhere in your project config.
    [/list]

this should do the trick,
if it works (or doesn’t) please let me know so i can add the info to the initial post.

best, hansi.
p.s. if all you want to do is get a movie out quickly here are two other options:

  • use a quicktime library instead of ffmpeg (search the forum, i remember someone wrote a movieexport)
  • even simpler: just create a new image in setup, then use image.grabScreen() and image.save() at the end of each draw() to store loads of individual images to a folder. once you’re done use the ffmpeg executable or quicktime pro or some other image2video converter. if you go with ffmpeg you’ll need the executables which you can find using the link from above (get a normal package without the “-dev” at the end). here are some example commands to do the conversion: http://www.catswhocode.com/blog/19-ffmp-…–all-needs

@kritzikratzi

Thank You for the post and the encouragement. Just having the link and order of setup will help me a lot. My IDE is not helping, VS2008 is a lesser used IDE in the OF world and has the problem of the compiler not being supported by ffmpeg. Your direction to the binaries should help me crack this. I may also switch over to CodeBlocks. OSX is not an option for me at the moment.

I will continue to chip away at this and if I succeed post the progress.

The promises of FFmpeg is good enough for me to pursue over the other options for now.

This is all I needed. I have tried to download file but couldn’t get it. Is the file broken?

http://asdfg.me/ofxFFMovieMaker-with-sample.zip

[quote author=“thedofl”]This is all I needed. I have tried to download file but couldn’t get it. Is the file broken?

http://asdfg.me/ofxFFMovieMaker-with-sample.zip[/quote]
hey!

oh, sorry… i cleand up my server yesterday and moved all those files in the root directory somewhere else.

the new location is:
http://asdfg.me/up/ofxFFMovieMaker-with-sample.zip

my apologies…

best, hansi.

is it possible to create a video without an audio track?

I disabled the audio this way:

  
  
for (int i = 0; i < BUFFER_SIZE*NUM_CHANNELS; i++)  
		audioBuffer[i] = 0.0;  
  

I am having some troubles when I try to use the addon in a project besides the example though… As soon as I try to start recording the video, I get this in my console:

  
  
> src width: 1024  
> src height: 768  
> dest width: 1024  
> dest height: 768  
adding video stream  
adding audio stream  
Output #0, mpeg, to '/Users/plong0/Documents/Work/oF/of_preRelease_v0062_osxSL_FAT/apps/SpiralSense/LiveArt/bin/data/test.mpg':  
    Stream #0.0: Video: mpeg1video (hq), yuv420p, 1024x768, q=2-31, 10485 kb/s, 90k tbn, 25 tbc  
    Stream #0.1: Audio: mp2, 44100 Hz, 2 channels, s16, 192 kb/s  
SIZE===1179648  
S2=1179648  
S3=3  
S4=6  
SIZE===3145728  
S2=3145728  
S3=4  
S4=16  
KKK!  
YUV-LS-0=1024  
RGB-LS-0=4096  
YUV-LS-1=512  
RGB-LS-1=0  
YUV-LS-2=512  
RGB-LS-2=0  
YUV-LS-3=0  
RGB-LS-3=0  
CHANNELS===2  
A-FRAME_SIZE===1152  
[mpeg @ 0x9aa600] VBV buffer size not set, muxing may fail  
Program received signal:  “EXC_BAD_ACCESS”.  
  

It looks identical to the output from the working example, up until the crash at the EXC_BAD_ACCESS

I am wondering… is there anything special that I might be missing in the Project settings?

[quote author=“plong0”]I disabled the audio this way:

  
  
for (int i = 0; i < BUFFER_SIZE*NUM_CHANNELS; i++)  
		audioBuffer[i] = 0.0;  
  

[/quote]
it should be sufficient to just not call movieMaker->addAudio( … );

I am having some troubles when I try to use the addon in a project besides the example though…

Well, if the example works that’s already a good thing.
Is it possible you’re calling moviemaker functions from more than one thread? (not sure how this whole event-thing works on windows, but e.g. the event-thread could be different from the drawing thread). calling ffmpeg from various threads at a time is one of the easiest ways to generate crashes.

As soon as I try to start recording the video, I get this in my console:

  
>       
>     > src width: 1024  
>     > src height: 768  
>     > dest width: 1024  
>     > dest height: 768  
>     adding video stream  
>     adding audio stream  
>     Output #0, mpeg, to '/Users/plong0/Documents/Work/oF/of_preRelease_v0062_osxSL_FAT/apps/SpiralSense/LiveArt/bin/data/test.mpg':  
>         Stream #0.0: Video: mpeg1video (hq), yuv420p, 1024x768, q=2-31, 10485 kb/s, 90k tbn, 25 tbc  
>         Stream #0.1: Audio: mp2, 44100 Hz, 2 channels, s16, 192 kb/s  
>     SIZE===1179648  
>     S2=1179648  
>     S3=3  
>     S4=6  
>     SIZE===3145728  
>     S2=3145728  
>     S3=4  
>     S4=16  
>     KKK!  
>     YUV-LS-0=1024  
>     RGB-LS-0=4096  
>     YUV-LS-1=512  
>     RGB-LS-1=0  
>     YUV-LS-2=512  
>     RGB-LS-2=0  
>     YUV-LS-3=0  
>     RGB-LS-3=0  
>     CHANNELS===2  
>     A-FRAME_SIZE===1152  
>     [mpeg @ 0x9aa600] VBV buffer size not set, muxing may fail  
>     Program received signal:  “EXC_BAD_ACCESS”.  
>       
>     

It looks identical to the output from the working example, up until the crash at the EXC_BAD_ACCESS

I am wondering… is there anything special that I might be missing in the Project settings?

shit, where does that “kkk!” come from? hopefully not from my lib… :confused:
one k makes quite a difference!

hmm, I didn’t put the KKK there lol

but I did a test with a brand new, clean project… and it seems to be working now. Must have been something conflicting in the project I was using before. Strange brew. I will keep working with the other one, and see if I find the source.

I do get a lot of crazy output from the video recording though (hoping there is a way to suppress it?)…

during recording (each frame):

  
  
FFMovieMaker: addScreen() only works with config.v_srcFormat=PIX_FMT_RGBA!  
scale: 8 ms  
OUTSIZE=614400  
encode: 4 ms  
other crazy shit: 0 ms  
  

when finish()'ing:

  
  
[mpeg @ 0x188a800] buffer underflow i=0 bufi=2002 size=29785  
[mpeg @ 0x188a800] buffer underflow i=0 bufi=4043 size=29785  
[mpeg @ 0x188a800] buffer underflow i=0 bufi=6084 size=29785  
[mpeg @ 0x188a800] buffer underflow i=0 bufi=8125 size=29785  
[mpeg @ 0x188a800] buffer underflow i=0 bufi=10166 size=29785  
  

I’ve attached the source for my very simple example of recording the screen.

ofxFFMovieMakerSimpleVideoExample_src.zip

[quote author=“plong0”]hmm, I didn’t put the KKK there lol

but I did a test with a brand new, clean project… and it seems to be working now. Must have been something conflicting in the project I was using before. Strange brew. I will keep working with the other one, and see if I find the source.

I do get a lot of crazy output from the video recording though (hoping there is a way to suppress it?)…

I’ve attached the source for my very simple example of recording the screen.[/quote]

hey!

thanks for the example!

for the “strange output” thing: i’ve uploaded a new version of the library to my original post which surpresses that output.

best, hansi.
p.s. i also got rid of that uncalled third k.

Thanks! :slight_smile: And thank you for providing the addon to us! It is very useful and very much appreciated.

Hey this is quite exciting!

Recording video & sound with sync is a big missing factor in several projects I’ve wanted to make in OF and this goes a long way towards solving the problem…

Ages ago I started in on FFMPEG and didn’t have time to finish, so big ups for making it happen!

Here’s a little example I just whipped up to test recording a web cam and mic input…fails sometimes and loses sync often…but it’s an awesome start…

I’m also interested to see if we could use ffmpeg for video streaming (RTMP, RTSP, etc)…will start looking into it :wink:

M

WebCamandMicrophoneRecording_Example.zip