ofxAvCodec - ffmpeg wrapper for reading/writing audio files

kritzikratzi · December 18, 2015, 6:46pm

hey.

I’ve been writing an ffmpeg wrapper to read/write audio files (and metadata) using FFmpeg. I think it’s reached a point where it could be useful for other people.

Reads virtually any file format (including audio contained in videos, aac support disabled)
Writes a lot of formats (not aac, not mp3)
Resampling built into the player and file writer
Methods to read/write/update metadata
Utility to generate an ofPath representing the waveform from an audio file
Contains a readme file that explains how ffmpeg was compiled (must be included with your project if you distribute it publicly)
Has the funny name of ofxAvCodec, because it took me forever to understand the difference between FFmpeg, libav and libavcodec.

Here’s a quick example (more are in the github readme):

#include "ofxAvAudioPlayer.h"

class ofApp : public ofBaseApp{
    // ...
    ofSoundStream soundStream;
    ofxAvAudioPlayer player;
    void audioOut( float * output, int bufferSize, int nChannels );
};

//--------------------------------------------------------------
void ofApp::setup(){
    soundStream.setup(this, 2, 0, 44100, 512, 4);
    player.setupAudioOut(2, 44100); // set up resampling 
    player.loadSound(ofToDataPath("testo.flac"));
    map<string,string> metadata = player.getMetadata(); 
}

void ofApp::audioOut( float * output, int bufferSize, int nChannels ){
    player.audioOut(output, bufferSize, nChannels); 
}

For this to work you’ll also need the FFmpeg shared libraries. You can either compile them yourself (follow this guide: https://github.com/kritzikratzi/ofxAvCodec/blob/master/ffmpeg_src/readme.md or the ffmpeg compilation guide: https://trac.ffmpeg.org/wiki/CompilationGuide)

or instead take the libs i compiled here: https://www.wetransfer.com/downloads/81f26bbceec1d5a783fe8bc36de50b6420151218183950/bdd690f0b76d1f6fc5caafd89287710920151218183950/b6aeb9 contains source, linux32/64, win32/64, osx32, but i can’t guarantee they’ll work, i only tested half of them.

fakob · April 15, 2016, 7:07am

Hi Kritzikratzi
I have not yet tried out your addon, but wondered if it supports also video and could possibly solve two problems I currently have.

Read video files with frame accuracy, so that I can move through a video frame by frame
This is something I now have lots of problems with using the ofAVFoundationPlayer
Multi platform support, so that I am being able to deploy my app also for Windows and Linux systems

Two years ago I had developed an OSX app called MoviePrint where you could make customised screenshots of movies. Now I would like to update it so it can read more codecs, is more reliable when it comes to frame navigation and also deploy Windows and Linux versions.
Future plans are to create a more modular video analysis and archive tool, but first things first.

Check out my old version if you like - www.fakob.com/2014/movieprint-an-osx-tool/

By the way, I like your “Oh” installation. As I am planning to finally attend Ars Electronica this year again, I wondered if it would still be up then.

Best regards
Jakob

jvcleave · April 15, 2016, 5:07pm

Typically this a limitation of the container/codec (especially with .h264/.mp4)

kritzikratzi · April 15, 2016, 5:35pm

hey!

i worked on this today actually. it’s possible with the latest version on the ‘video’ branch.
(under heavy heavy development for at least another week).

for now precise frame seeking works, but is slow. it does this by skipping to the previous keyframe, and then advancing until the correct frame is found. i’m using a 20 frame cache, the current implementation could be a lot smarter by utilizing for skipping also and not just for playback.

also, after seeking the audio buffers aren’t perfectly aligned. the video won’t go out of sync, but there will be a small time jump in the video when going from paused+frame seeking to playback again. this is something that isn’t important for my project, so i probably won’t fix it for now.

feel free to dive into the mess. i’m happily taking pull requests.

best, hansi.

fakob · April 16, 2016, 8:34am

Hi

That sounds great. As soon as I have time I will check it out.

Cheers
Jakob

fakob · April 22, 2016, 6:22pm

Hi

I was now able to do a first test with the video branch trying to play a video file

Compiling of FFmpeg worked great - thanks for the thorough explanation
As I am using Qt Creator I had to manually copy the dylib files into the Contents/MacOS folder
The included example app runs fine
I was able to load a video file using ofxAvVideoPlayer
I was able to draw something using myMovie.getTexture().draw(20,20);

Problems

I could not get the video to play or the texture to update - do you have an idea what I am doing wrong?

ofApp.h

ofxAvVideoPlayer    myMovie;

ofApp.cpp

void ofApp::setup(){
    myMovie.load("movies/movieFile.mp4");
    myMovie.setLoop(true);
    myMovie.play();
}

void ofApp::update(){
    myMovie.update();
}

void ofApp::draw(){
    myMovie.getTexture().draw(20,20);
}

I also wondered how I could help to get some of the functions of the standard videoplayer to work. E.g. nextFrame() or the like. As mentioned, I am only an amateur, but if you could provide me with some hints, I would love to help out.

I am using of_v0.9.2_osx_release on OSX 10.11 with Qt Creator

kritzikratzi · April 22, 2016, 6:59pm

currently the whole thing syncs up to the audio track.

basically you need to get an audio stream going in your app.
something along the lines of:

class ofApp{
    //...
    ofSoundStream soundStream; 
}


void ofApp::setup(){
    soundStream.setup(this, 2, 0, 44100, 512, 4);

    // do this before loading the movie! 
    myMovie.setupAudioOut(2, 44100); // resample to stereo, 44khz
    myMovie.load("movies/movieFile.mp4");
    myMovie.setLoop(true);
    myMovie.play();


}

void ofApp::update(){
    myMovie.update();
}

void ofApp::draw(){
    myMovie.getTexture().draw(20,20);
}

void ofApp::audioOut( float * buffer, int nFrames, int nChannels ){
    myMovie.audioOut( buffer, nFrames, nChannels ); 
}

(hope i didn’t forget anything).

i have time-synced support planned too, but before that i have to fix some other issues (e.g. still some a/v sync issues when audio and video are too many packets apart, when keyframes are too far apart, and so on).

kritzikratzi · April 22, 2016, 7:01pm

ps. did you see that there are precompiled ffmpeg binaries in the releases tab?

i’m glad you managed to compile them yourself though

best, hansi.

fakob · April 23, 2016, 8:01am

Yes, thanks I had seen the precompiled ones, I just wanted to try it out myself.

Thanks for the explanation. I have added the soundstream like you said and it now works.
Does that mean that a video without sound will not work?

Btw, when I tried to use getFrameNumber() my app crashed.
I also noticed that the video hangs every now and then. When I have more time, I will look into it and try to give you a better description on what happens. I had not had time to debug it properly.

kritzikratzi · April 23, 2016, 9:09am

hey.

thanks for all the testing. i’ve created some bug reports.

Does that mean that a video without sound will not work?

for now, yes. (should be fixed next week)
https://github.com/kritzikratzi/ofxAvCodec/issues/2

Btw, when I tried to use getFrameNumber() my app crashed.

https://github.com/kritzikratzi/ofxAvCodec/issues/1

I also noticed that the video hangs every now and then.

this is a known issue (and a non issue for me). it’s quite easy to understand if you know what’s going on:

the audio and video frames are serialized into a data stream. often it looks something like this:

A = audio data (e.g. 1024 frames)
V = video data (one I/B/P frame)
AAAAAAAAAAAVVVVVAAAAAAAAAVVAVAVAAAAAVVVAAAAAAVV

depending on how the video was compressed, the audio lags behind, or the video lags behind, and it is up to the library (=me) to sync things up.
my current approach doesn’t work well when the audio sections are too long (>.3 seconds). the best way to fix this is in ofxVideoPlayer::update(). a variable needsFrame=true/false is calculated here, that figures out if we will need another video frame or not. if this variable is also set to true if the next frame wasn’t decoded yet, then this bug should go away.
https://github.com/kritzikratzi/ofxAvCodec/issues/3

i’m happy to accept pull requests (use the video branch!), but you can also just wait a few days.

kritzikratzi · April 23, 2016, 10:40am

hey!

I also noticed that the video hangs every now and then.

can you check if this is better with the latest commit? if you could subscribe to the bugs and comment there it would be easier for me.

fakob · May 2, 2016, 7:14pm

Hi kritzikratzi

Do you have an idea on how to setup a getPixels() method? What I have tried so far, did not work.

ofPixels & ofxAvVideoPlayer::getPixels(){
    if(duration>0){
        texture.readToPixels(pixels);
        return & pixels;
    }
}

I also wanted to ask why you are using av_seek_frame opposed to avformat_seek_file? It is said that avformat_seek_file supposed to improve frame accuracy.

Btw, I have found this tutorial where they claim to be quite frame accurate:
http://dranger.com/ffmpeg/tutorial07.html

kritzikratzi · May 3, 2016, 9:56am

hey!

the pixels for ~15frames are always cached in the video_buffers vector.
in the future this might have to change to store the undecoded video frames, but for it’s fine.

you could basically copy the update() method and instead of doing videoTex.loadData(…) you just return the pixels. see https://github.com/kritzikratzi/ofxAvCodec/blob/master/src/ofxAvVideoPlayer.cpp#L827

ie.

// instead of 
texture.loadData(data->video_dst_data[0], width, height, GL_RGB);
// you do: 
return data->video_dst_data[0];

for now this will always be rgb (no alpha). in principle some videos might contain an alpha channel, but at the moment it’s discarded upon decoding.

Btw, I have found this tutorial where they claim to be quite frame accurate:
i’ve used this tutorial to get some clues, unfortunately it’s rather old and only kept up to date so it compiles,
but apparently many internal things changed that should be done differently now.

i honestly don’t even remember the current seek implemention i’m using, but i’ve tried lots of different things from that tutorial and from mailinglist tips, plus theres (i think) some uncomitted code still on my computer.

the last few days i was working on another part of the application, but as it looks now i’m revisiting the video player starting thursday.

best, hansi.

fakob · May 9, 2016, 10:12pm

Hi Hansi

I am trying to get getPixels() to work, but failed.
In your suggestion, I had to replace data with video_buffers[0] as there is no data reference. This is what I came up with, but it does not compile/work.

ofPixels & ofxAvVideoPlayer::getPixels(){
    if(duration>0){
        return video_buffers[0]->video_dst_data[0];
    }
}

This version does compile, but also does not work.

ofPixels & ofxAvVideoPlayer::getPixels(){
    if(duration>0){
        ofPixels pixels;
        pixels.setFromPixels(video_buffers[0]->video_dst_data, width, height, OF_IMAGE_COLOR);
        return pixels;
    }
}

Sorry to bother you again, but do you have a tip for me?
Thanks
Jakob

kritzikratzi · May 9, 2016, 11:26pm

hey!

ah, forgot about this post. i finally got back to working on this today actually (but on color corrections, no pixel/playback stuff at the moment).

you need to also take the few lines above the line that i linked, all of this:

github.com

kritzikratzi/ofxAvCodec/blob/master/src/ofxAvVideoPlayer.cpp#L779-L827


		}
	}
	
	if( !video_pixels.isAllocated() || video_pixels.getWidth() != width || video_pixels.getHeight() != height ){
		video_pixels.allocate(width, height, OF_IMAGE_COLOR);
	}
	
	AVL_MEASURE(uint64_t sws_start = ofGetSystemTime();)
	int linesize[4] = {3*width,0,0,0};
	
	uint8_t * video_pixel_data = video_pixels.getData();
	uint8_t * dst[4] = {video_pixel_data, 0, 0, 0};
	
	sws_scale(sws_context,
			  // src slice / src stride
			  data->video_dst_data, data->video_dst_linesize,
			  // src slice y / src slice h
			  0, height,
			  // destinatin / destination stride
			  dst, linesize );

This file has been truncated. show original

thats where the data* pointer is coming from.
if you have compilation errors: whenever there is no data you can do a
return ofPixels();

to return an empty pixels object.

sorry that this project is such a mess at the moment …

fakob · May 11, 2016, 5:47pm

Ah, ok I had missed that one. Then I will give it another try and report back.
Don’t be sorry, I am sorry that I can not really contribute more as my skills are limited.

kritzikratzi · May 15, 2016, 12:29pm

there is a getPixels() method now on the video branch (still needs merging with master).
haven’t used it myself, but it should work.

also there are some more improvements with seeking, but it can take up to a few frames until getPixels() returns the correct frame. (i might add blocking version of setPositionMS() to solve that. but currently it’s not an issue for me).

fakob · May 30, 2016, 10:02pm

Thanks so much for the getPixels() function. I had now time to try it and as you said, it sometimes returns the wrong pixels. You had talked about, that it takes a few frames until getPixels() returns the correct frame. How would I set that up? Currently I have the following procedure.

I set the position, update, wait and then getPixels()

gmMovie.setPositionMS(_frame*frameLength);
gmMovie.update();
ofSleepMillis(500);
grabbedFrame[i].gsImage.setFromPixels(gmMovie.getPixels());

Would I have to setPositionMS a couple of times, and approach the specific frame? Do you have a suggestion on how to do that?

Also I had tested to run the grabbing of specific frames in an extra thread, but without any error the app crashes. Not sure what I am doing wrong this time. Did you ever try it in an extra thread using ofThread

Here the video I am testing frame accuracy with - transfer.fakob.com/movieprint/FrameTestMovie_v001.mov

kritzikratzi · May 31, 2016, 6:04am

put another update after ofSleepMillis().

no, just call setPositionMS once, it won’t get better with the second call.
(can take a while for a 4k video).

running it on a separate thread will be problematic, but only because it’s writing to texture memory.
if there would a bUseTexture flag like with ofImage, then it should be no problem.

jon_erik · January 19, 2017, 1:28am

Hi @kritzikratzi

I’m not sure if you are still actively maintaining this but I’m trying to get it to work with oF 0.9.0, vs2015, win10.

I’ve followed the instructions you’ve written on github and I’m using your compiled libraries and I’ve tried this with win32 and 64 but I’m getting the following error:

invalid or corrupt file: cannot read at 0x390 avcodec-56.dll

Any ideas what is wrong?

Thanks!