How would you go about manipulating videos' audio in realtime?

Not sure how how to do that with the desktop app, but with Emscripten it was possible to route the audio nodes from the video into OFs audio input and from there to ofxPd.

Hey @steve.m, how’s your experience with ofxAvCodec going? Does it require you to recode your videos using their codec? If there’s a simple way to do so, I’d really like to be able to have simple dynamic speed-manipulation of both audio and video. It looks like this is possible with ofxHapPlayer, but I need to convert my videos to hap codec.

Great! If I was going to try this out today I would split audio and video, like having a wav or aiff file and doing the manipulations all there.
having the video track without audio in a performant random access codec like hap, ofxHapPlayer and trying to make it follow audio position.
if your video is not huge (duration and frame size) you can even export them in image sequence and load all the frames in memory to have a faster access

Heya @s_e_p I haven’t got to it yet: some refactoring and rework to do before I hit it. But I will get there.

ofxHapPlayer doesn’t suit my usecase very well: I need access to the frame pixels to feed OpenCV, so I think that rules out that approach.

I’ll try to push ahead with ofxAvCodec in the coming days/weeks. I’m trying to build a tool for other artists to use, so if I can just say “feed this app a video file with a soundtrack” then that’s better than asking non-tech users to split the audio into a separate file.

@steve.m , yeah, that’s a good point. I know how to split audio from video and I don’t even want to do it. I’m not savvy enough of a coder to write a script to do it for me, so I’d take each of the 20 videos and put them in a DAW or video-editing software to do it…

I’ve posted a question about hap audio on this thread, so please reply if you’ve any ideas…

OK, after talking to myself in this thread, I finally got hap files playing. A little more digging on my own paid off and I was able to get ofxDSHapPlayer working. However, changing the speed in that also leads to image&audio stuttering, so it looks like I’ll need to follow your advice and separate audio & video.

Yeah audio manipulation is real good in OF. as each video frame (30fps) has 33ms it is a lot from an audio buffer, about 1470 samples (in 44100). so maybe easier to make the video follows the audio.
you can use ffmpeg to script video and audio splits

I was not sure if a video playing backwards could reverse the individual audio buffers or just play them normally and jumping

@dimitre Ohh, that sounds way more convenient than manually exporting the audio in some other software. How do I use ffmpeg in oF? Do I need a specific addon?

sorry I mean you can use ffmpeg to split audio & video

Hey, I got it working! I split the audio from the video, and now the speed change works much better.

To split the audio I typed

ffmpeg -i IMG_0151.avi -map 0:v -c copy video/IMG_0151.avi -map 0:a:0 -c copy IMG_0151.wav

in the Windows Command Prompt. Is there a way to batch split a bunch of videos in the Command Prompt? Or do I need to run ffmpeg from a script in some framework? A quick Google search seems to indicate I need something like a .bat file…

Yeah you can script in bat, python, even edit each command in a text editor
and there are some guis out there to batch convert using ffmpeg and some scripts too.

I’ve been trying to use ofFmodSoundPlayer so I can load and play sounds, destroy them while saving the audioPosition and then load them again and start from the new position. The following always produces audio cracks: even if I keep the amplitude parameter at 0.0 the whole time.

class ofApp : public ofBaseApp{

	public:
		//generic oF stuff
		void toggleSound(bool on);
		
		std::shared_ptr<ofFmodSoundPlayer> sound;

		ofParameter<float> amplitude;
		ofParameter<bool> on;
		ofxPanel gui;
		float ap;
};
#include "ofApp.h"

//--------------------------------------------------------------
void ofApp::setup(){
	gui.setup();
	gui.add(amplitude.set("amplitude", 0.0, 0.0, 1.0));
	gui.add(on.set("on/off", false, false, true));
}

//--------------------------------------------------------------
void ofApp::update(){

	if (sound != nullptr)
	{
		sound->setVolume(amplitude);
	}
}

//--------------------------------------------------------------
void ofApp::draw(){
	gui.draw();
}

void ofApp::toggleSound(bool on)
{
	if (on)
	{
		sound = make_shared<ofFmodSoundPlayer>();
		sound->load("IMG_0145.wav");
		sound->play();
		sound->setPosition(ap);
		return;
	}
	ap = sound->getPosition();
	ap = ap >= 1.0f ? 0.0f : ap;
	sound = nullptr;
}
//--------------------------------------------------------------
void ofApp::keyPressed(int key){

	if (key == ' ')
	{
		on = !on;
		toggleSound(on);
	}
}

Now I’ve tried reordering the methods called in toggleSound(), but the result is always one of two things:

  1. The setPosition() doesn’t work and the sound plays from the beginning. No cracking sound.
  2. setPosition() works. Cracking sound.

@dimitre , @roymacdonald , somebody help!

EDIT: I’ve tried putting a setPaused(true) after the setPosition(), waiting as much as 60 frames and then setPaused(false), and the crack comes when the setPaused(false) is called.

Hi, why are you specifically using ofFmodSoundPlayer?
Have you tried out the sound player in ofxSoundObjects? I certainly can do what you are looking for.

@roymacdonald Because I couldn’t do it with ofSoundPlayer and someone suggested ofFmodPlayer as a native solution. But I’ll try ofxSoundObjects tonight. Is there an example project for what I need?

There are several examples that can be useful.

Hey, I got example-audioOutput and example-SoundPlayerObject to work… but the latter has some odd behavior. When I lower the play speed below 1.0 the sound freezes, and when I raise it even a little over 1.0 (eg. 1.05) it already sounds 3-4 times faster. (raising it to 8.0) causes the app to crash). And I get the repeated warning:

[warning] ofRtAudioSoundStream: stream over/underflow detected

It might be worth noting my soundStream settings (I get “unavailable hardware” errors if I don’t change anything):

auto devices = stream.getDeviceList(ofSoundDevice::Api::MS_DS);
	ofSoundStreamSettings settings;
	settings.setInDevice(devices[7]);
	settings.setOutDevice(devices[1]);
	settings.setInListener(this);
	settings.sampleRate = player.getSoundFile().getSampleRate();
	settings.numOutputChannels = 2;
	settings.numInputChannels = 2;
	settings.bufferSize = 256;
	stream.setup(settings);

I’m also having trouble with example-soundMixer. I get the same over/underflow error and no sound:

	auto devices = stream.getDeviceList(ofSoundDevice::Api::MS_DS);
	ofSoundStreamSettings settings;
	settings.setInDevice(devices[7]);
	settings.setOutDevice(devices[1]);
	settings.setInListener(this);
	settings.sampleRate = 48000;
	settings.numOutputChannels = 2;
	settings.numInputChannels = 2;
	settings.bufferSize = 256;
	settings.numBuffers = 1;
	stream.setup(settings);

I tried a simple mixer setup in my main project, which plays both sounds, but the ofxSoundPlayerObject’s sound is highly distorted


	m_soundStream.printDeviceList();

	int bufferSize = 256;

	ofSoundStreamSettings settings;

	auto devices = m_soundStream.getDeviceList(ofSoundDevice::Api::MS_DS);
		
	auto inDevice = devices[7];
	auto outDevice = devices[1];
	settings.setInDevice(inDevice);
	settings.setOutDevice(outDevice);

	settings.setInListener(this);
	settings.sampleRate = 44100;
	settings.numOutputChannels = 2;
	settings.numInputChannels = 2;
	settings.bufferSize = bufferSize;
	m_soundStream.setup(settings);

	m_soundStream.setOutput(m_mixer);
	d_sound.connectTo(m_mixer);
	sine.setup(440);
	sine.connectTo(m_mixer);
	sine.freq = 100.0f;
	for (int i = 0; i < 2; i++) {
	m_mixer.setChannelVolume(i, 0.2f);	

    d_sound.load(D:\\IMG_0144.wav, true);
    d_sound.play();

Hi, yeah. that is a problem with the audio resampling algorithm in ofSoundBuffer.
try the experimental branch of ofxSoundObjects and add to you project (using project generator) ofxSampleRate. By doing so it will automatically switch to use ofxSampleRate and the resampling, thus change of playback speed will work with no problems.

For a Windows user this is a lot of set up… ofxSoundObjects + ofxSampleRate + libsamplerate + cmake… and I’m sure even then all of my problems won’t be taken care of.

The performance is Sunday so I think I’ll just put up with the clicking and use PD for some other sounds via OSC… and let my sound card work the mixing out, lol. After Sunday I’ll probably try looking to ofxPD. Hopefully that’s easier to get working…

(I forgot to press the send button. I replied the following yesterday)
if you cloned ofxSoundObjects using git you can easily get the experimental branch.
in terminal:

cd your_openframeworks_folder/addons
cd ofxSoundObjects
git checkout -b experimental origin/experimental

Make sure you have ofxSampleRate in your addons folder as well.

Generate the project for example-SoundPLayerObject with project generator and remember to add ofxSampleRate. You will be able to change speed with no issues.

Oh. I am sorry. I did not add the windows libraries. Compiling it on windows does not sound that complicated, if I was you I would give it a try. I might have access to a windows computer later and will try to compile and add to the addon. I will let you know.