Recording audio with ofxFFmpegRecorder

It seems I can use this addon to record in hap format in Windows, which is very nice, but what’s up with the audio? The example project seems set up to only record video or audio, not both at the same time… audio records at twice as slow/deep as the input. And recording audio very often makes the program crash:

@NickHardeman Any tips for me? If I understand correctly you press ‘a’ to start recording audio and then ‘a’ again to finish recording.

Hi @s_e_p I don’t think I have ever used this add-on to record audio :upside_down_face:
Have you tried using ffmpeg directly via command line?
I tried something like the following but still didn’t seem correct with audio
osx/ffmpeg -f avfoundation -t 6.000000 -framerate 30 -thread_queue_size 100 -i "0:0" -fps_mode vfr -b:v 8000k out.mp4
Are you using my fork?

Ohhh yeah, I was using the @Furkanzmc version, which has audio functionality (that I was just having trouble with) but apparently isn’t being maintained anymore.

Your fork has a setRecordAudio(bool) method, but I guess it doesn’t do anything?

I want to record in conjunction with other code, so running FFmpeg from the command prompt is not ideal unless I do some weird setup with oF calling batch files or something.

I’ll look at @roymacdonald 's ofxSoundObjects to record the sound and try to use it in conjunction with your fork!

==========================================
edit: @roymacdonald , I tried running your example-soundRecorder project, but get the error

|Severity|Code|Description|Project|File|Line|Suppression State|
|---|---|---|---|---|---|---|
|Error|C1083|Cannot open source file: '..\..\..\addons\ofxSoundObjects\src\ofxSoundPlayerObject.cpp': No such file or directory|example-soundRecorder|C:\Users\selli\openFrameworks\addons\ofxSoundObjects\example-soundRecorder\c1xx|1||

and indeed, there is no ofxSoundPlayerObject.cpp in that folder (though there is ofxSoundPlayerObject.h). What can I do about this?

hi. try updating the project with project generator.

1 Like

you need to use AddBuffer method for that.
whats your FFmpeg version? have you spoecified the correct audio input for OF and audio codecs flags for FFmpeg?
I haven’t used it in a while but can try to help, :).You are best to use FFmpeg instead of 2 different outputs source in my opinion.

Hi. It looks like Mr. Hardeman’s fork removes that method, but I’ll try your fork out if you’re offering support!

My ffmpeg version is 4.4.1.

Your example project is similar to that of the Furkanzmc version and just like that one I’m able to record video (hap) fine. The wave form graphic is responding properly to my microphone input but the app crashes shortly after I press ‘a’ to record the audio stream.

Two questions:

  1. I’m sure this error is due to me not setting up the audio codecs properly as you said. How can I do this? There’s no setAudioCodec method…

  2. The example looks like it’s set up to either record video or audio (with spacebar and ‘a’ respectively), since the same method is used to set the output path (ofxFFmpegRecord::setOutputPath with “.avi” or “.mp3”). How do I go about recording both simultaneously? Do I need two ofxFFmpegRecord objects- one for video and one for audio?

hello,

ok for 1), you don’t need to do anything sorry. As it’s mp3 by default, so the flags are set here, but I wouldn’t touch it tbh.

  1. need to dig into it a little bit more, probably this evenin. That’s correct actually it’s one or the other. But you can use 2 streams inputs in the command line, so should be quite quick to add that in ofxFFmpegRecoder.
    Will keep you posted.

Did you manage to record audio?

Hey @pierre_tardif00 ; thanks for the swift reply.

Just like with the Furkanzmc version, I get exceptions thrown when recording audio, like in the screenshot of my previous post- the abort usually occurs when I press ‘a’ a second time to end recording. However, it does actually record audio and the audio is twice as (s)low as it should be. This is fixed by not setting the ofSoundStreamSettings::numBuffers value and by setting InputChannels to 1, but there’s a substantial amount of crackling in the sound.

Here’s my cpp file- do you see anything off? The main changes are the audio settings in setup() and setting the video codec to hap.

void ofApp::setup() {

    ofSoundStreamSettings settings;
    soundStream.printDeviceList();
   
    int bufferSize = 1024;
    auto devices = soundStream.getDeviceList(ofSoundDevice::Api::MS_DS);
    auto inDevice = devices[8]; 
    settings.setInDevice(inDevice);

    settings.setInListener(this);
    //settings.numBuffers = 4; //I forget what this setting does,. '4' has worked in other apps, but here it contributes to the octave drop.
    settings.sampleRate = 44100;
    settings.numOutputChannels = 0;
    settings.numInputChannels = 1; //I think this is a stereo mic, but setting this to 2 leads to the octave drop, even without setting numBuffers.
    settings.bufferSize = bufferSize;
    soundStream.setup(settings);
    
    m_Grabber.setDeviceID(1);
    m_Grabber.setup(800,448);
    mCapFbo.allocate( m_Grabber.getWidth(), m_Grabber.getHeight(), GL_RGB ); //this is the wrong color setting for hap, but apparently it gets auto-corrected

    m_Recorder.setup(true, false, glm::vec2(m_Grabber.getWidth(), m_Grabber.getHeight()) );
    m_Recorder.setAudioConfig(1024,44100);
    m_Recorder.setOverWrite(true);  
    m_Recorder.setFFmpegPath(ofToDataPath("C:/ffmpeg/bin/ffmpeg.exe", true));
    m_Recorder.setVideoCodec("hap");
    m_Recorder.setRecordAudio(true);

    isRecordingVideo = false;
    isRecordingAudio = false;

    audioFPS = 0.0f;
    audioCounter    = 0;
    bufferCounter	= 0;
    lastAudioTimeReset = ofGetElapsedTimeMillis();
}

void ofApp::update() {
    m_Grabber.update();    
	
	if (m_Recorder.isRecording()) {
		if (isRecordingVideo) {
			// curious to learn how ofxFastFboReader can be used to speed this up!
			mCapFbo.readToPixels(mPix);
			if (mPix.getWidth() > 0 && mPix.getHeight() > 0) {
				m_Recorder.addFrame(mPix);
			}
		}
	}
}

void ofApp::draw() {
    
    mCapFbo.begin(); {
        ofBackground(0);
        ofSetColor( 255 );
        m_Grabber.draw(0, 0, mCapFbo.getWidth(), mCapFbo.getHeight() );

    } mCapFbo.end();
    
    mCapFbo.draw(0,0);

    ofSetColor(255,255,220);
    waveform.draw();
    
    if(isRecordingVideo){
        ofDrawBitmapStringHighlight(std::to_string(m_Recorder.getRecordedDuration()), 40, 45);
    }else if(isRecordingAudio){
        ofDrawBitmapStringHighlight(std::to_string(m_Recorder.getRecordedAudioDuration(audioFPS)), 40, 45);
    }else{
        ofDrawBitmapStringHighlight(std::to_string(m_Recorder.getRecordedDuration()), 40, 45);
    }
    ofDrawBitmapStringHighlight("FPS: " + std::to_string(ofGetFrameRate()), 10, 16);
    ofDrawBitmapStringHighlight("AFPS: " + std::to_string(ofGetFrameRate()), 140, 16);

    ofPushStyle(); {
        if (m_Recorder.isPaused() && m_Recorder.isRecording()) {
            ofSetColor(ofColor::yellow);
        }
        else if (m_Recorder.isRecording()) {
            ofSetColor(ofColor::red);
        }
        else {
            ofSetColor(ofColor::green);
        }
        ofDrawCircle(ofPoint(10, 40), 10);

        // Draw the information
        ofSetColor(ofColor::green);
        ofDrawBitmapStringHighlight("Press spacebar to toggle video record."
                                    "\nPress (t) to save thumbnail."
                                    "\nPress (a) to save audio stream."
                                    "\nPress (s) to start rtp video streaming.",
                                    10, ofGetHeight() - 200 );
    }
    ofPopStyle();
}

void ofApp::keyReleased(int key) {
    if (key == ' ') {
        
        if( m_Recorder.isRecording() ) {
            // stop
            m_Recorder.stop();
            isRecordingVideo = false;
        } else {
#if defined(TARGET_OSX)
            m_Recorder.setOutputPath( ofToDataPath(ofGetTimestampString() + ".mp4", true ));
#else
            m_Recorder.setOutputPath( ofToDataPath(ofGetTimestampString() + ".avi", true ));
#endif

	   m_Recorder.setVideoCodec("hap");
            m_Recorder.setBitRate(8000);

            isRecordingVideo = true;
            m_Recorder.startCustomRecord();
        }
    } else if (key == 't') {
        m_Recorder.saveThumbnail(0, 0, 2, "thumbnail.png", ofVec2f(0, 0), ofRectangle(0, 0, 500, 400));
    }else if(key == 's'){
        m_Recorder.startCustomStreaming();
    }else if(key == 'a'){
        if( m_Recorder.isRecording() ) {
            // stop
            m_Recorder.stop();
            isRecordingAudio = false;
        } else if(audioFPS != 0) {
            m_Recorder.setOutputPath( ofToDataPath(ofGetTimestampString() + ".mp3", true ));
            isRecordingAudio = true;
            m_Recorder.startCustomAudioRecord();
        }
    }
}

//--------------------------------------------------------------
void ofApp::audioIn(ofSoundBuffer & input){

    if(ofGetElapsedTimeMillis()-lastAudioTimeReset >= 1000){
        lastAudioTimeReset = ofGetElapsedTimeMillis();
        audioFPS = audioCounter;
        audioCounter = 0;
    }else{
        audioCounter++;
    }

    if(m_Recorder.isRecording() && isRecordingAudio){
        m_Recorder.addBuffer(input,audioFPS);
    }

    waveform.clear();
    for(size_t i = 0; i < input.getNumFrames(); i++) {
        float sample = input.getSample(i,0);
        float x = ofMap(i, 0, input.getNumFrames(), 0, ofGetWidth());
        float y = ofMap(sample, -1, 1, 0, ofGetHeight());
        waveform.addVertex(x, y);
    }

    bufferCounter++;

}

hey!

ok so recording audio with 2 channels is working fine if you specify 2 channels input to ofSoundStreamSettings:
for me :


    settings.setInDevice(devices[0]);
    settings.setInListener(this);
    settings.sampleRate = 44100;
    settings.numOutputChannels = 0;
    settings.numInputChannels = 2;
    settings.bufferSize = 1024;
    soundStream.setup(settings);

Beyond that, setting up AV recording isn’t done yet, as libmp3lame and flags required / structure I’m using isn’t right:
this is as far as I got(adding the manual re compilation with libmp3lame) then recording AV tests which outputs in the console:


ffmpeg.exe -hide_banner  -y -framerate 30.000000 -s 1280x720 -f rawvideo -pix_fmt rgb24 -vcodec rawvideo -i - -f f32le -ac 1 -i - -vcodec h264 -b:v 8000k -r 30.000000 -framerate 30.000000 -acodec libmp3lame -b:a 320k C:\Users\pierr\Documents\DEV\OF\of_\apps\ofxffmpegrecorder\audioExample\bin\data\2022-09-30-00-54-51-393.mp4
Input #0, rawvideo, from 'pipe:':
  Duration: N/A, start: 0.000000, bitrate: 663552 kb/s
  Stream #0:0: Video: rawvideo (RGB[24] / 0x18424752), rgb24, 1280x720, 663552 kb/s, 30 tbr, 30 tbn
Guessed Channel Layout for Input Stream #1.0 : mono
Input #1, f32le, from 'pipe:':
  Duration: N/A, bitrate: 1411 kb/s
  Stream #1:0: Audio: pcm_f32le, 44100 Hz, mono, flt, 1411 kb/s
Stream mapping:
  Stream #0:0 -> #0:0 (rawvideo (native) -> h264 (libx264))
  Stream #1:0 -> #0:1 (pcm_f32le (native) -> mp3 (libmp3lame))
[libx264 @ 000001cb86029940] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2
[libx264 @ 000001cb86029940] profile High 4:4:4 Predictive, level 3.1, 4:4:4, 8-bit
[libx264 @ 000001cb86029940] 264 - core 164 r3099 e067ab0 - H.264/MPEG-4 AVC codec - Copyleft 2003-2022 - http://www.videolan.org/x264.html - options: cabac=1 ref=3 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=7 psy=1 psy_rd=1.00:0.00 mixed_ref=1 me_range=16 chroma_me=1 trellis=1 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=4 threads=22 lookahead_threads=3 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1 b_bias=0 direct=1 weightb=1 open_gop=0 weightp=2 keyint=250 keyint_min=25 scenecut=40 intra_refresh=0 rc_lookahead=40 rc=abr mbtree=1 bitrate=8000 ratetol=1.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00
[f32le @ 000001cb85fb48c0] Thread message queue blocking; consider raising the thread_queue_size option (current value: 8)
Output #0, mp4, to 'C:\Users\pierr\Documents\DEV\OF\of_\apps\ofxffmpegrecorder\audioExample\bin\data\2022-09-30-00-54-51-393.mp4':
  Metadata:
    encoder         : Lavf59.27.100
  Stream #0:0: Video: h264 (avc1 / 0x31637661), yuv444p(tv, progressive), 1280x720, q=2-31, 8000 kb/s, 30 fps, 15360 tbn
    Metadata:
      encoder         : Lavc59.37.100 libx264
    Side data:
      cpb: bitrate max/min/avg: 0/0/8000000 buffer size: 0 vbv_delay: N/A
  Stream #0:1: Audio: mp3 (mp4a / 0x6134706D), 44100 Hz, mono, fltp, 320 kb/s
    Metadata:
      encoder         : Lavc59.37.100 libmp3lame
Assertion failed: el >= 0, file ../../lame-3.100/libmp3lame/psymodel.c, line 5760x

Well, you can see my code in my previous post- setting numInputChannels to 2 drops the recorded output an octave. Below is the console FFmpeg-related output up until the crash. See anything revealing? Is the version alright? Between the crackling, speed change of the recording and the crash, it looks like I’m not going to be able to use this addon to record audio, unless you have an idea of what’s wrong.

ffmpeg version 4.4.1-full_build-www.gyan.dev Copyright (c) 2000-2021 the FFmpeg developers
built with gcc 11.2.0 (Rev1, Built by MSYS2 project)
configuration: --enable-gpl --enable-version3 --enable-shared --disable-w32threads --disable-autodetect --enable-fontconfig --enable-iconv --enable-gnutls --enable-libxml2 --enable-gmp --enable-lzma --enable-libsnappy --enable-zlib --enable-librist --enable-libsrt --enable-libssh --enable-libzmq --enable-avisynth --enable-libbluray --enable-libcaca --enable-sdl2 --enable-libdav1d --enable-libzvbi --enable-librav1e --enable-libsvtav1 --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxvid --enable-libaom --enable-libopenjpeg --enable-libvpx --enable-libass --enable-frei0r --enable-libfreetype --enable-libfribidi --enable-libvidstab --enable-libvmaf --enable-libzimg --enable-amf --enable-cuda-llvm --enable-cuvid --enable-ffnvcodec --enable-nvdec --enable-nvenc --enable-d3d11va --enable-dxva2 --enable-libmfx --enable-libglslang --enable-vulkan --enable-opencl --enable-libcdio --enable-libgme --enable-libmodplug --enable-libopenmpt --enable-libopencore-amrwb --enable-libmp3lame --enable-libshine --enable-libtheora --enable-libtwolame --enable-libvo-amrwbenc --enable-libilbc --enable-libgsm --enable-libopencore-amrnb --enable-libopus --enable-libspeex --enable-libvorbis --enable-ladspa --enable-libbs2b --enable-libflite --enable-libmysofa --enable-librubberband --enable-libsoxr --enable-chromaprint
libavutil 56. 70.100 / 56. 70.100
libavcodec 58.134.100 / 58.134.100
libavformat 58. 76.100 / 58. 76.100
libavdevice 58. 13.100 / 58. 13.100
libavfilter 7.110.100 / 7.110.100
libswscale 5. 9.100 / 5. 9.100
libswresample 3. 9.100 / 3. 9.100
libpostproc 55. 9.100 / 55. 9.100
Guessed Channel Layout for Input Stream #0.0 : mono
Input #0, f32le, from ‘pipe:’:
Duration: N/A, bitrate: 1411 kb/s
Stream #0:0: Audio: pcm_f32le, 44100 Hz, mono, flt, 1411 kb/s
Stream mapping:
Stream #0:0#0:0 (pcm_f32le (native) → mp3 (libmp3lame))
Output #0, mp3, to ‘C:\Users\selli\openFrameworks\addons\ofxFFmpegRecorder\example\bin\data\2022-09-30-21-41-50-287.mp3’:
Metadata:
TSSE : Lavf58.76.100
Stream #0:0: Audio: mp3, 44100 Hz, mono, fltp, 320 kb/s
Metadata:
encoder : Lavc58.134.100 libmp3lame
size= 178kB time=00:00:04.46 bitrate= 325.6kbits/s speed=1.33x
video:0kB audio:177kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.602433%

@NickHardeman I saw some other forum posts recommend your fork specifically. Is it superior in some way? I might still use it if I can’t get audio with the other forks anyway…

I added some functionality regarding pixel input and output types but didn’t change too much. And I haven’t given much attention to audio support. Not quite sure how it might compare to other forks.

hey sorry will have alook later in the afternoon.
Yeah maybe try something else.

I’d say to test how to make it work would be to stream from an input .mov, and input .mp3, into an oputput .mp4 only in the command line, to test the flags, then use the addon with same flags in. This requires a good knowledge of the codecs you use and their associated flags/options in relation to the FFmpeg version as well, but not impossible :).

Will try this route later on, and report here.

Best,

P

1 Like

Hey @roymacdonald

I’ve got playing and recording in one app working. I see that it’s necessary to connect the input to the output to record and to connect the player to the output to play. It seems you can’t do both of these simultaneously. (I wasn’t able to end my recording while player was connected to output).

Is that correct? I’d like to record to file A while recording to file B- is there a way to do that?

EDIT: playing a sound with ofFmodPlayer while recording with your objects works, but is probably not what you had in mind.

Hi @s_e_p, you are right. the input must be connected to the output because the audio data gets “pulled” from the output.
You can simply add a mixer (check the examples) connect it to the output and connect the recorder and player to the mixer.
let me know if this works for you

Before I try that, one other question: I may want to record two audio streams at once (overlapping). Is this possible with your objects?

you mean two audio streams from different sound devices?
otherwise you can connect two recorders one after the other and record with these to different files at different moments, these can record simultaneusly.

input.connectTo(recorder1).connectTo(recorder2).connectTo(output);

1 Like