Recording Audio & Video in Sync SOLVED (Mac only)

extend
#1

No issues just pure sync 8)

This old chestnut is finally solved, at least for Mac OSX 10.6+

Basically it takes QTKit to do it -> no mucking round with timestamps, recording audio and video separately, recombining them with QT7 SDK, FFMPEG, a blunt sword, etc etc…

Just hit record and it works…

I got this happening on it’s own over here, by implementing this, but decided to extend James George’s most excellent work on QTKit Video Playing and Grabbing over here (or on his-github).

So I present you with a beefed up ofxQTVideoGrabber that:

* Records video and audio in sync (you can turn the audio off if you just want the video, but I didn’t make it do the reverse -> let me know if that would be good)
* Adds setting devices by name (overloaded so you can still use ID numbers)
* Adds setting codec by name (didn’t do it by ID -> let me know if that would be good)
* You can init the grabber without any preview or textures etc if you want to just have recording functionality without the draw

I’ve tested it with my built in iSight and with a HVX-202 and it did fine with 1080i (!!) footage and my built-in mic (though it seemed to choke on the DVCPro HD Audio input…mhmm)

One thing to note: the only sizes available for the recorded videos are the native size of the capture device…no matter what you set the width and height of the preview to it will NOT change the video output. Only changing codec types affects the size of the video output (downwards not upwards)…this is a limitation of the method (for more info see here).

addon and example src are attached below OR you can head on over to my fork of Jame’s addons on my-github.

Either way, may the force be with you.

M

PS: thanks to Sam Fox at HydraPoesis for supporting the pursuit of this evil…and perhaps now we should delve into the even greater axis of e: DirectShow and the amcap example that comes with win sdk -> perhaps that way doth lie windoze sync’d recording???

ofxQTKitVideoGrabber.zip

Record Quicktime video + audio, "succesful" with some issues
ofxQTKitVideoGrabber QTKit & CoreVideo + OF
oF sub-section champions
#2

Hello,

Has anyone tested this code with a facetime camera on a 2011 mac? When I run the example all I get in the app window is screen artifacts. When I record, I get a quicktime file of the correct length that plays, but with no audio or video.

Thanks!

#3

Hey there

I tested this on a late 2010 Macbook Pro, so no facetime HD cam…

But just to be sure of some things: what version of Xcode? Which model of Mac? Which OS version?

And maybe this is too obvious but did you change this…

  
	vidGrabber.setVideoDeviceID("Built-in iSight");							// can set deviceID's by string  
	vidGrabber.setAudioDeviceID("Built-in Microphone");

To the correct name for your cam and sound input?

Maybe could you post your console output? Mine looks like:

  
2011-06-19 00:54:19.039 ofxQTKitVideoGrabberDebug[24556:a0f] ofxQTKitVideoGrabber listing video devices  
2011-06-19 00:54:21.787 ofxQTKitVideoGrabberDebug[24556:a0f] 0 - Built-in iSight  
2011-06-19 00:54:21.788 ofxQTKitVideoGrabberDebug[24556:a0f] 1 - ManyCam Virtual Webcam (Y'CbCr)  
2011-06-19 00:54:21.789 ofxQTKitVideoGrabberDebug[24556:a0f] 2 - ManyCam Virtual Webcam (RGB)  
2011-06-19 00:54:21.789 ofxQTKitVideoGrabberDebug[24556:a0f]   
2011-06-19 00:54:21.790 ofxQTKitVideoGrabberDebug[24556:a0f] ofxQTKitVideoGrabber listing audio devices  
2011-06-19 00:54:21.790 ofxQTKitVideoGrabberDebug[24556:a0f] 0 - Pro Tools Aggregate I/O  
2011-06-19 00:54:21.790 ofxQTKitVideoGrabberDebug[24556:a0f] 1 - Built-in Input  
2011-06-19 00:54:21.791 ofxQTKitVideoGrabberDebug[24556:a0f] 2 - Built-in Microphone  
2011-06-19 00:54:21.791 ofxQTKitVideoGrabberDebug[24556:a0f] 3 - Soundbooth 3.0  
2011-06-19 00:54:21.792 ofxQTKitVideoGrabberDebug[24556:a0f]   
2011-06-19 00:54:21.792 ofxQTKitVideoGrabberDebug[24556:a0f] ofxQTKitVideoGrabber listing video compression options  
2011-06-19 00:54:21.843 ofxQTKitVideoGrabberDebug[24556:a0f] 0 - QTCompressionOptionsLosslessAppleIntermediateVideo  
2011-06-19 00:54:21.843 ofxQTKitVideoGrabberDebug[24556:a0f] 1 - QTCompressionOptionsLosslessAnimationVideo  
2011-06-19 00:54:21.844 ofxQTKitVideoGrabberDebug[24556:a0f] 2 - QTCompressionOptionsSD480SizeH264Video  
2011-06-19 00:54:21.844 ofxQTKitVideoGrabberDebug[24556:a0f] 3 - QTCompressionOptions240SizeH264Video  
2011-06-19 00:54:21.845 ofxQTKitVideoGrabberDebug[24556:a0f] 4 - QTCompressionOptions120SizeH264Video  
2011-06-19 00:54:21.845 ofxQTKitVideoGrabberDebug[24556:a0f] 5 - QTCompressionOptionsSD480SizeMPEG4Video  
2011-06-19 00:54:21.846 ofxQTKitVideoGrabberDebug[24556:a0f] 6 - QTCompressionOptions240SizeMPEG4Video  
2011-06-19 00:54:21.846 ofxQTKitVideoGrabberDebug[24556:a0f] 7 - QTCompressionOptions120SizeMPEG4Video  
2011-06-19 00:54:21.847 ofxQTKitVideoGrabberDebug[24556:a0f] 8 - QTCompressionOptionsJPEGVideo  
2011-06-19 00:54:21.847 ofxQTKitVideoGrabberDebug[24556:a0f]   
2011-06-19 00:54:21.848 ofxQTKitVideoGrabberDebug[24556:a0f] ofxQTKitVideoGrabber listing audio compression options  
2011-06-19 00:54:21.850 ofxQTKitVideoGrabberDebug[24556:a0f] 0 - QTCompressionOptionsHighQualityAACAudio  
2011-06-19 00:54:21.853 ofxQTKitVideoGrabberDebug[24556:a0f] 1 - QTCompressionOptionsVoiceQualityAACAudio  
2011-06-19 00:54:21.854 ofxQTKitVideoGrabberDebug[24556:a0f] 2 - QTCompressionOptionsLosslessALACAudio  
2011-06-19 00:54:21.854 ofxQTKitVideoGrabberDebug[24556:a0f]   
2011-06-19 00:54:21.938 ofxQTKitVideoGrabberDebug[24556:a0f] Audio Input Format: Linear PCM, 16 bit little-endian signed integer, 2 channels, 44100 Hz  
2011-06-19 00:54:21.939 ofxQTKitVideoGrabberDebug[24556:a0f] starting video session
#4

Hey gameover, this is fantastic for grabbing video straight off the camera/mic, but I have been struggling to try to capture and modify the video before saving it.
I’m working on a simple photo booth app and it seems ofxQtVideoSaver is great for saving what’s on screen but not with audio sync. Is QTCaptureSession writable at all? It is possible to read and manipulate the grabbed camera pixels, but how write? Using QTMovie I suspect will take me back to the start… https://developer.apple.com/library/mac/#samplecode/StillMotion/Listings/MyDocument-m.html

Cheerio

#5

We are looking into the same think - how can we manipulate the pixels (say combine two videos) and then save the result with the audio to a file. Anyone have advise? Our goal is to automate an upload to youtube if possible.

#6

hi all !

looking for something similar … anyone got a solution to record frames to video with audio in sync ?
if i understood well james+gameover addon is to record something captured from a capture device, not the screen ?

thanks !

e*

#7

I’ve been working on a solution for the mac here: https://github.com/sethismyfriend/ofxQTKitAVScreenRecorder- let me know how it works for you if you try it.

#8

thanks seh4b !! looks promising ! i’ll try it out and let you know !!

e*

#9

Yes, the example should work for you I think. I transfered the code to another project where I push and pull a glMatrix and it fails, but it works for simple fbo operations. I’m still looking into that error.

What are you planning to do with it?

#10

hi seh4b !

I would like to try to record to file a “audiovisual” buffer from our “audiovisual sampling engine” …
https://github.com/eloimaduell/ofxPlaymodes

This is a “toy” we’ve been working on for some years as a divertimento and when we have the chance …
Basically it’s a sampling engine where you can capture audio and video in realtime and then “play” with it in many ways … in the past we made an interactive installation with something similar and a theater show …

http://vimeo.com/5081638
http://vimeo.com/1665563

Now we’re working on the new version (on github) where all the audio part is inside OF app now (before we made all the audio part in Reaktor) … Now all the audio part is done by a Pd patch in ofxPd … we’re working on it with the idea of making crazy sampling performances … it’s super simple tool but there are many uses to have fun with it…

The idea is that we can “save” what you’ve been doing … that would be amazing !!
Still on development, we’ll make a proper addon asap as we make it run smooth and perfect …

I’m doing some stuff inside the Fbo … you mean i can’t call glPushMatrix() ?

e*

#11

I see - I love this concept by the way. It fits nicely with this project we have been divertido (ing) http://www.perspectum.com/chronographer/index.htm

The addon I made was for another project where I wanted to record video and audio in sync, but the video was generated by my program, so I worked on the following addon as a proof of concept that you could do that. Once I got the example working I made it into an addon and tried adding alpha png files and it worked!

When I moved the addon into another program with a lot of textures being rendered in the fbo occationally I got this error:

ofxQTKitRecorder.mm line 185

“Inconsistent set of values to create NSBitmapImageRep”
then: GLUT Fatal Error: internal error: NSRangeException, reason: *** -[__NSArrayI objectAtIndex:]: index 0 beyond bounds for empty array

if you change: hasAlpha:YES it fixes the problem, but then the frame is black. Still working on this…

#12

@gameover

Hey what is the status of this in .73? Is it the same as the new video recorder? I made this post [url]http://forum.openframeworks.cc/index.php/topic,11921.0.html/url]

As I am trying to get the some good fast capturing.

Is there any difference in the current implementation and your version? I dont get very good sync and it changes with different devices.

As an aside, what is the status of the goVideoPlayer with 73- I could not get it to work but maybe I missed something there.

Cheers