I’ve been writing an ffmpeg wrapper to read/write audio files (and metadata) using FFmpeg. I think it’s reached a point where it could be useful for other people.
Reads virtually any file format (including audio contained in videos, aac support disabled)
Writes a lot of formats (not aac, not mp3)
Resampling built into the player and file writer
Methods to read/write/update metadata
Utility to generate an ofPath representing the waveform from an audio file
Contains a readme file that explains how ffmpeg was compiled (must be included with your project if you distribute it publicly)
Has the funny name of ofxAvCodec, because it took me forever to understand the difference between FFmpeg, libav and libavcodec.
Here’s a quick example (more are in the github readme):
#include "ofxAvAudioPlayer.h"
class ofApp : public ofBaseApp{
// ...
ofSoundStream soundStream;
ofxAvAudioPlayer player;
void audioOut( float * output, int bufferSize, int nChannels );
};
//--------------------------------------------------------------
void ofApp::setup(){
soundStream.setup(this, 2, 0, 44100, 512, 4);
player.setupAudioOut(2, 44100); // set up resampling
player.loadSound(ofToDataPath("testo.flac"));
map<string,string> metadata = player.getMetadata();
}
void ofApp::audioOut( float * output, int bufferSize, int nChannels ){
player.audioOut(output, bufferSize, nChannels);
}
Hi Kritzikratzi
I have not yet tried out your addon, but wondered if it supports also video and could possibly solve two problems I currently have.
Read video files with frame accuracy, so that I can move through a video frame by frame
This is something I now have lots of problems with using the ofAVFoundationPlayer
Multi platform support, so that I am being able to deploy my app also for Windows and Linux systems
Two years ago I had developed an OSX app called MoviePrint where you could make customised screenshots of movies. Now I would like to update it so it can read more codecs, is more reliable when it comes to frame navigation and also deploy Windows and Linux versions.
Future plans are to create a more modular video analysis and archive tool, but first things first.
By the way, I like your “Oh” installation. As I am planning to finally attend Ars Electronica this year again, I wondered if it would still be up then.
i worked on this today actually. it’s possible with the latest version on the ‘video’ branch.
(under heavy heavy development for at least another week).
for now precise frame seeking works, but is slow. it does this by skipping to the previous keyframe, and then advancing until the correct frame is found. i’m using a 20 frame cache, the current implementation could be a lot smarter by utilizing for skipping also and not just for playback.
also, after seeking the audio buffers aren’t perfectly aligned. the video won’t go out of sync, but there will be a small time jump in the video when going from paused+frame seeking to playback again. this is something that isn’t important for my project, so i probably won’t fix it for now.
feel free to dive into the mess. i’m happily taking pull requests.
I also wondered how I could help to get some of the functions of the standard videoplayer to work. E.g. nextFrame() or the like. As mentioned, I am only an amateur, but if you could provide me with some hints, I would love to help out.
I am using of_v0.9.2_osx_release on OSX 10.11 with Qt Creator
currently the whole thing syncs up to the audio track.
basically you need to get an audio stream going in your app.
something along the lines of:
class ofApp{
//...
ofSoundStream soundStream;
}
void ofApp::setup(){
soundStream.setup(this, 2, 0, 44100, 512, 4);
// do this before loading the movie!
myMovie.setupAudioOut(2, 44100); // resample to stereo, 44khz
myMovie.load("movies/movieFile.mp4");
myMovie.setLoop(true);
myMovie.play();
}
void ofApp::update(){
myMovie.update();
}
void ofApp::draw(){
myMovie.getTexture().draw(20,20);
}
void ofApp::audioOut( float * buffer, int nFrames, int nChannels ){
myMovie.audioOut( buffer, nFrames, nChannels );
}
(hope i didn’t forget anything).
i have time-synced support planned too, but before that i have to fix some other issues (e.g. still some a/v sync issues when audio and video are too many packets apart, when keyframes are too far apart, and so on).
Yes, thanks I had seen the precompiled ones, I just wanted to try it out myself.
Thanks for the explanation. I have added the soundstream like you said and it now works.
Does that mean that a video without sound will not work?
Btw, when I tried to use getFrameNumber() my app crashed.
I also noticed that the video hangs every now and then. When I have more time, I will look into it and try to give you a better description on what happens. I had not had time to debug it properly.
I also noticed that the video hangs every now and then.
this is a known issue (and a non issue for me). it’s quite easy to understand if you know what’s going on:
the audio and video frames are serialized into a data stream. often it looks something like this:
A = audio data (e.g. 1024 frames)
V = video data (one I/B/P frame)
AAAAAAAAAAAVVVVVAAAAAAAAAVVAVAVAAAAAVVVAAAAAAVV
depending on how the video was compressed, the audio lags behind, or the video lags behind, and it is up to the library (=me) to sync things up.
my current approach doesn’t work well when the audio sections are too long (>.3 seconds). the best way to fix this is in ofxVideoPlayer::update(). a variable needsFrame=true/false is calculated here, that figures out if we will need another video frame or not. if this variable is also set to true if the next frame wasn’t decoded yet, then this bug should go away. https://github.com/kritzikratzi/ofxAvCodec/issues/3
i’m happy to accept pull requests (use the video branch!), but you can also just wait a few days.
I also wanted to ask why you are using av_seek_frame opposed to avformat_seek_file? It is said that avformat_seek_file supposed to improve frame accuracy.
the pixels for ~15frames are always cached in the video_buffers vector.
in the future this might have to change to store the undecoded video frames, but for it’s fine.
// instead of
texture.loadData(data->video_dst_data[0], width, height, GL_RGB);
// you do:
return data->video_dst_data[0];
for now this will always be rgb (no alpha). in principle some videos might contain an alpha channel, but at the moment it’s discarded upon decoding.
Btw, I have found this tutorial where they claim to be quite frame accurate:
i’ve used this tutorial to get some clues, unfortunately it’s rather old and only kept up to date so it compiles,
but apparently many internal things changed that should be done differently now.
i honestly don’t even remember the current seek implemention i’m using, but i’ve tried lots of different things from that tutorial and from mailinglist tips, plus theres (i think) some uncomitted code still on my computer.
the last few days i was working on another part of the application, but as it looks now i’m revisiting the video player starting thursday.
I am trying to get getPixels() to work, but failed.
In your suggestion, I had to replace data with video_buffers[0] as there is no data reference. This is what I came up with, but it does not compile/work.
Ah, ok I had missed that one. Then I will give it another try and report back.
Don’t be sorry, I am sorry that I can not really contribute more as my skills are limited.
there is a getPixels() method now on the video branch (still needs merging with master).
haven’t used it myself, but it should work.
also there are some more improvements with seeking, but it can take up to a few frames until getPixels() returns the correct frame. (i might add blocking version of setPositionMS() to solve that. but currently it’s not an issue for me).
Thanks so much for the getPixels() function. I had now time to try it and as you said, it sometimes returns the wrong pixels. You had talked about, that it takes a few frames until getPixels() returns the correct frame. How would I set that up? Currently I have the following procedure.
I set the position, update, wait and then getPixels()
Would I have to setPositionMS a couple of times, and approach the specific frame? Do you have a suggestion on how to do that?
Also I had tested to run the grabbing of specific frames in an extra thread, but without any error the app crashes. Not sure what I am doing wrong this time. Did you ever try it in an extra thread using ofThread
no, just call setPositionMS once, it won’t get better with the second call.
(can take a while for a 4k video).
running it on a separate thread will be problematic, but only because it’s writing to texture memory.
if there would a bUseTexture flag like with ofImage, then it should be no problem.
I’m not sure if you are still actively maintaining this but I’m trying to get it to work with oF 0.9.0, vs2015, win10.
I’ve followed the instructions you’ve written on github and I’m using your compiled libraries and I’ve tried this with win32 and 64 but I’m getting the following error:
invalid or corrupt file: cannot read at 0x390 avcodec-56.dll