ofxVideoRecorder

Hi all, I’ve written a fast video recorder addon, here it is on github:

https://github.com/timscaffidi/ofxVideoRecorder

UPDATE
Things have changed significantly since this was originally posted. The whole system has been majorly overhauled to increase functionality and performance.

Changes:

  • Implemented lock-free queues to boost performance.

  • Encoding is now done on-the-fly via FIFO pipes to a running ffmpeg process. This eliminates the need to rely on unoptimized jpeg encoding on the OF side of things. Playing around with the output codec can change the speed/size ratio.

  • On-the-fly encoding also brings support for streaming. Just use a custom output string and you can use any streaming format that ffmpeg supports.

  • Audio+Video recording is now supported. A lot of effort was done to get this working, but I’m happy to say it works now and should maintain perfect synchronization with the video stream. More testing is needed but for now it seems like everything is working (at least on mac and linux). Special thanks to Ben and Helios Interactive for the incentive to finally add this feature.

  • Implemented auto creation/management of FIFO pipes on mac and linux (never tested on windows, not even sure what the implementation would be). The system uses a std::set of pipe IDs which can be checked out and retired, this should in theory allow as many video streams as you want to be recorded at once (using multiple recorder objects), as well as be able to start and stop them whenever you want with no overlap or confusion of the pipe files. Pipe IDs will be reused when possible to keep from cluttering your system with pipe files.

Please test the new features, especially audio+video recording! The main issue I was fighting during development was orchestrating the creation of two FIFO pipes, then getting them to become emptied at the same time when it is time to close the stream. I think I have finally fixed this issue by moving all of the file operations into the audio/video threads. Opening, writing and closing the pipe files is all handled in a thread. The other fix was to enable non blocking writes during cleanup so that if ffmpeg didn’t want to consume any more data the write() calls would not block and make the system hang.

Also, if someone wants to figure out windows support, I’d be much obliged.

Cheers,

Tim

/UPDATE

The recorder works by adding each frame to be saved into a queue which is constantly saved to disk in a separate thread. If the thread cant save frames in realtime, the queue will grow, but I’ve achieved 640x480@60fps in near realtime using this system. Each frame is compressed to jpeg and appended to the video file, so as to reduce the overhead spent on opening and closing individual jpg files. The last step is to re-encode the mjpeg with ffmpeg to generate proper headers for the container. I may add in functionality to save a traditional image sequence in the future, but this is the fastest method I’ve come up with.

included is an example project for 007, though the example will freeze if the frame queue is too large on exit due to oF unloading FreeImage too early. There is a simple modification to fix this though, here’s the bug report: https://github.com/openframeworks/openFrameworks/issues/770

Here is a discussion that that lead to it’s development and release.
http://forum.openframeworks.cc/t/screenrecording/7090/0

Thanks to Arturo for providing many useful improvements to the original design.

1 Like

I copy the files in an Xcode project and I get multiple compile errors…

the main error is that it doesn’ t recognize

public ofThread

as sth valid, so the whole class is not created.

am I missing some dependency ??

Hey Tim

was looking at the code, and there’s something that can be problematic. The way you are starting and stopping the thread from different threads can lead to trying to start the thread twice because of some problem in the way ofThread works. It’s better to not stop the thread at all. If you want it to not consume cpu cycles while there’s no frames to save, you can use a pthread condition. Take a look at ofURLLoader, to see how the threadedFunction waits on a condition for new requests to be available.

The way you are locking can also be problematic since you are starting the thread as non-locking, but in addFrame you are calling:

  
  
lock();  
frames.push(buffer);  
unlock();  
  

which means that at some point you can be modifying the queue from both threads in case this lock fails. Also in that case you could be unlocking the thread twice which in osx at least, leads to all the mutexes in the process getting corrupt.

Also you can do the same you are doing when saving the image to a file in threadedFunction with:

  
  
ofSaveImage(pixels,buffer);  
file << buffer;  
  

where buffer is an ofBuffer. Although probably you are avoiding a memory copy the way you are doing it.

You may be using an older version of oF. ofThread used to be an addon ofxThread. It should work with version 007.

Arturo, thanks for the pointers. I think I’ve fixed the locking now, and I’ve changed it to never stop the thread from within itself. As far as the mutex conditions, I can’t seem to get that to work, I’m not sure what I did wrong, but it just waits indefinitely, even if I put condition.signal() inside of the addFrame function. I’m using ofSleepMillis(1.0/(float)frameRate) instead when the queue reaches zero.

This also gives an added boost to capture speed also because it’s no longer creating and destroying threads all of the time, cool!

  
  
ofSaveImage(pixels,buffer);    
file << buffer;   
  

There is a significant performance penalty from using this method compared to direct access of the FreeImage functions. With the added speed of not starting/stopping the thread, I can consistently record 640x480@60 in realtime, but when using ofSaveImage I get a queue growth rate of around 3-5 frames per second. Pretty good but I think the added speed trumps elegance in this situation.

Thanks again for these suggestions, I’ve updated the git. let me know what you think!

i’ve sent you a pull request, feel free to get the changes you want.

There was still a thread potential problem cause you were querying frames.empty() outside the lock.

Then i’ve changed the rgb -> bgr to use ofPixels::swapRgb which should be slightly faster cause it uses std::swap instead of tmp = a; a = b; b = tmp; which in modern implementations should be faster (probably unnoticeable)

And i’ve added the condition. Probably your problem with the condition was that you were using the same mutex. This can actually be a little slower since the condition is locking and unlocking the mutex but really compared to saving to disk or the operations with memory everything else should not be an issue

Thanks, Arturo, I merged your changes. I was using swapRgb() earlier but in testing it seemed a slight bit slower than the other way. It could have just been chaotic HDD performance though, as I’m not noticing any difference now.

Hello,

first thank you for the ofxVideoRecorder it works great on Mac OS, except that you need to run ffmpeg manually.

Recently I’ve used ofxVideoRecorder & ofxKinect and this results in a lots of crashes in the 8th thread (somewhere around FreeMemory). I ended up using Memo’s ofxThreadedImageSaver for now.

Is there a way to limit the number of threads which ofxVideoRecorder is creating, because I’ve red somewhere in the forum, that this could solve the problem.

I’m really new to Threads in general, but willing to read and learn :slight_smile:

ofxVideoRecorder only uses one extra thread, disabling that thread would make it run slower and potentially remove most of the benefits that this system provides.

I’m curious why you need to run ffmpeg externally? I’ve not tested this on MacOS so I can’t provide an explanation for that.

Yes, it uses one extra thread, but can I limit the thread simultaneous running to 4. If there is any reason in my words. I know that there will be decrease in the performance but otherwise I’ve got dropped frames from the Kinect.

As for the ffmpeg, I don’t know why but it seems that the system command is not executed, so I’m running it from the Terminal and I have no problem with that :slight_smile:

Not sure if this is what’s causing the confusion, but when you’re seeing lots of extra threads running in the debug view, some of those are from the IDE itself pausing processes and from other execution threads. ofxVideoRecorder only uses one thread, so your app has 2: regular OF application thread, and then another thread from ofxVideoRecorder.

@Joshua Yes, that cause my confusion.

I ended up using http://github.com/jamezilla/ofxQTVideoSaver - no crashes and it saves directly in .mov. Works OK for 640x480 frames.

lots of updates to this system, see the original post. Highlights include audio+video in sync and streaming support.

Hi, I have had a lot of great results form this but I have one question, I tried to find out from ffmpeg documentation but I could not find it.

After some experiments I found I had a lot better quality video using prores as the codec. The result is a larger file but amazing quality.

I have only one problem with this, the speed of the files is wrong. If is set my framerate (in the recorder.setup function) to 12 fps I get the correct speed. If I make it 25 the files are played back double speed. When I look at the console output from the app it says the frames are being grabbed at the correct rate (the first 2 frames are always a bit slow but then it catches up). I don’t get any errors about data overflow/underflow but I still dont get the correct playback rate.

When I play the movies at half speed they are almost at the right speed but the motion is not smooth.

Here is the output.

Output #0, mov, to ‘/Users/fredrodrigues/Dropbox/Code/OF_0.8.0/openFrameworks/apps/myApps/LikeMinds/bin/data/movies/2013-11-04-15-20-13-673.mov’:
Metadata:
encoder : Lavf54.29.104
Stream #0:0: Video: prores (apcn) (apcn / 0x6E637061), yuv422p10le, 1920x1080, q=2-31, 10 kb/s, 25 tbn, 25 tbc
Stream mapping:
Stream #0:0 -> #0:0 (rawvideo -> prores)
Press [q] to stop, [?] for help
frame= 9 fps=0.0 q=0.0 size= 5513kB time=00:00:00.24 bitrate=188175.1kbits/s
frame= 16 fps= 14 q=0.0 size= 11926kB time=00:00:00.52 bitrate=187873.1kbits/s
frame= 23 fps= 14 q=0.0 size= 18304kB time=00:00:00.80 bitrate=187437.4kbits/s
frame= 30 fps= 13 q=0.0 size= 24686kB time=00:00:01.08 bitrate=187250.5kbits/s
frame= 37 fps= 13 q=0.0 size= 31075kB time=00:00:01.36 bitrate=187183.3kbits/s
frame= 44 fps= 13 q=0.0 size= 37465kB time=00:00:01.64 bitrate=187143.3kbits/s
recording message false
frame= 48 fps= 12 q=0.0 Lsize= 43850kB time=00:00:01.92 bitrate=187092.7kbits/s
video:43849kB audio:0kB subtitle:0 global headers:0kB muxing overhead 0.002441%
recording finished

Cheers

Hi Fred,

frame= 30 fps= 13 q=0.0 size= 24686kB time=00:00:01.08 bitrate=187250.5kbits/s

see the part in the ffmpeg output that says fps=13? This means that ffmpeg is not encoding fast enough (only 12-13 frames per second) to keep up with your desired 25 fps, and explains why 12 fps is close to real time.

I would guess that prores codec is a bit too processor intensive to use for real time encoding on your machine. I suggest using something with a very light overhead, like 100% quality mjpeg, which actually produces very nice results. Or, if you can spare the hard drive space and have a fast enough one to keep up, you can use the rawvideo codec which shouldn’t do any processing on the video stream and just dump it to a file to be encoded later on.

Hope this helps,

Hi Tim,

Great addon. Works really smoothly and well on OSX. One thing in your example though:
I had to remove the vidGrabber.isFrameNew() condition in the update function in order to match the right framerate.
I’m not sure if this is related to your previous post, or if it works differently on other platforms.

FYI:
I know its not windows ready yet, but just so you know, I’ve tried compiling in CB on Windows, and the threads don’t work. (Its always the threads) Am still trying to see if I can build a workaround for it.

cheers,
cherdd

Hmm you shouldn’t need to remove that line. There are a few things to keep in mind in regards to framerate however. There are three different rates that you need to care about in this case:

  1. app framerate
  2. camera framerate
  3. video file framerate

generally #1 will be much higher than 2 and 3 unless you are doing some serious computation or rendering. ideally #2 and #3 will be as close as possible to avoid any issues in playback.

This can usually be boiled down to just source framerate and output framerate, assuming your app can keep up with #2 and #3.

If you are recording audio with the video, then the addon will attempt to keep these in sync and adhere to your chosen output framerate by doing two things:

  • Repeat frames if the video source is too slow to keep up with the input audio source
  • Skip frames if the video input rate is too high for the chosen output rate, for example if you are saving at 30fps but your video source is producing frames at 60fps

This process works very well and you will always get a video that plays back at the proper rate, not slow or fast motion.

The other scenario is if you are not recording audio, then the video has nothing to keep pace with, and will simply add whatever frame you give it no matter what, and never skip or repeat frames. Since the output framerate is fix at lets say 30fps, then if you are adding frames at 20fps due to a slow camera, then you will end up with a video that plays back too fast, and vice versa.

I could potentially add a feature to make it keep time by checking against the system clock, and would allow it to skip/repeat frames without audio being necessary.

So in conclusion, you may just have a mismatch with your video source framerate and your output framerate.

As far as windows goes, I have still never tried, but I would assume the threads would not be able to properly communcate with ffmpeg via the FIFO pipes that unix supports. I’m not sure if cygwin adds support for this? I’ve never had a real burning desire to make it work in windows, but if you can figure it out I will gladly merge your code!

I have looked around for a similiar solution just to capture the screen of the OpenGL window, and much of what seems to be needed to make it really smooth is already in your extension. Could you perhaps add the functionality to record whats being displayed on the window and saved to disk?

Could you perhaps add the functionality to record whats being displayed on the window and saved to disk?

This is already possible, but requires you to do the screen grabbing part manually. I tried to keep this addon as use case agnostic as possible by allowing you to pass ofPixels into it. It’s up to you where those pixels come from.

The simplest way to save whatever is being displayed would be to use the ofImage::grabScreen() method and would require little to no reworking of the application you want to save.

A potentially better performing method would be to draw your entire scene into an FBO and use the ofFbo::readToPixels() method.

Well the problem is that grabScreen uses glReadPixels which is way too slow to get a good realtime stream.
And my hopes would be that what you are suggesting on the bottom could be handled by ofxvideorecorder :slight_smile:

Correct, the glReadPixels method is not ideal, and is usually too slow to be feasible.

Unfortunately, the FBO is not handled by ofxVideoRecorder, but it is pretty simple to set up.

you would allocate the FBO and pixels in setup, for whatever your desired output resolution is, for this example 720p:

void myApp::setup() { myFbo.allocate(1280,720,GL_RGB); myPixels.allocate(1280,720,OF_IMAGE_COLOR); }

Then your draw step would look something like this:

void myApp::draw() { myFbo.begin(); // whatever draw routines you are currently doing myFbo.end()

// draw the frame
myFbo.draw(0,0);

// save the FBO frame to the recorder
myFbo.readToPixels(myPixels);
myRecorder.add(myPixels);
}

1 Like