How would you go about manipulating videos' audio in realtime?

I’d like to activate various videos via different MIDI signals from my MIDI-violin, and be able to shift their pitch/playback-speed up and down with the MIDI’s bend value (slide/glissando/vibrato) in real-time. Is this kind of manipulation of video audio even possible with oF’s native video tools? Or am I going to need some kind of special video add-on?

@s_e_p I would use ofxPd for manipulating videos audio in realtime (because I am used to Pure Data).
You can also control ofxPd with midi (ofxMidi). Maybe there is also a way without using addons, I am not sure about that.

1 Like

Hey, thanks for your answer, @Jona . I’ve been using PD directly and linking it to oF via osc, but if using ofxPd let’s you grab video audio easily that’d be great. Are there examples of this in the ofxPd directory?

I’m interested in something similar: I want to react to both movement and audio from a video stream, but finding it difficult to extract the audio track from an ofxVideoPlayer/ofxVideoGrabber into a ofSoundBuffer etc.

However, I just found https://github.com/kritzikratzi/ofxAvCodec from the prolific @kritzikratzi which appears to fit the bill. Hopefully!

I haven’t tried it yet: need to build an ARM ffmpeg for my M1 mac but it looks promising.

Ohh, let us know how it goes!

1 Like

Not sure how how to do that with the desktop app, but with Emscripten it was possible to route the audio nodes from the video into OFs audio input and from there to ofxPd.

Hey @steve.m, how’s your experience with ofxAvCodec going? Does it require you to recode your videos using their codec? If there’s a simple way to do so, I’d really like to be able to have simple dynamic speed-manipulation of both audio and video. It looks like this is possible with ofxHapPlayer, but I need to convert my videos to hap codec.

Great! If I was going to try this out today I would split audio and video, like having a wav or aiff file and doing the manipulations all there.
having the video track without audio in a performant random access codec like hap, ofxHapPlayer and trying to make it follow audio position.
if your video is not huge (duration and frame size) you can even export them in image sequence and load all the frames in memory to have a faster access

Heya @s_e_p I haven’t got to it yet: some refactoring and rework to do before I hit it. But I will get there.

ofxHapPlayer doesn’t suit my usecase very well: I need access to the frame pixels to feed OpenCV, so I think that rules out that approach.

I’ll try to push ahead with ofxAvCodec in the coming days/weeks. I’m trying to build a tool for other artists to use, so if I can just say “feed this app a video file with a soundtrack” then that’s better than asking non-tech users to split the audio into a separate file.

@steve.m , yeah, that’s a good point. I know how to split audio from video and I don’t even want to do it. I’m not savvy enough of a coder to write a script to do it for me, so I’d take each of the 20 videos and put them in a DAW or video-editing software to do it…

I’ve posted a question about hap audio on this thread, so please reply if you’ve any ideas…

OK, after talking to myself in this thread, I finally got hap files playing. A little more digging on my own paid off and I was able to get ofxDSHapPlayer working. However, changing the speed in that also leads to image&audio stuttering, so it looks like I’ll need to follow your advice and separate audio & video.

Yeah audio manipulation is real good in OF. as each video frame (30fps) has 33ms it is a lot from an audio buffer, about 1470 samples (in 44100). so maybe easier to make the video follows the audio.
you can use ffmpeg to script video and audio splits

I was not sure if a video playing backwards could reverse the individual audio buffers or just play them normally and jumping

@dimitre Ohh, that sounds way more convenient than manually exporting the audio in some other software. How do I use ffmpeg in oF? Do I need a specific addon?

sorry I mean you can use ffmpeg to split audio & video

Hey, I got it working! I split the audio from the video, and now the speed change works much better.

To split the audio I typed

ffmpeg -i IMG_0151.avi -map 0:v -c copy video/IMG_0151.avi -map 0:a:0 -c copy IMG_0151.wav

in the Windows Command Prompt. Is there a way to batch split a bunch of videos in the Command Prompt? Or do I need to run ffmpeg from a script in some framework? A quick Google search seems to indicate I need something like a .bat file…

Yeah you can script in bat, python, even edit each command in a text editor
and there are some guis out there to batch convert using ffmpeg and some scripts too.

I’ve been trying to use ofFmodSoundPlayer so I can load and play sounds, destroy them while saving the audioPosition and then load them again and start from the new position. The following always produces audio cracks: even if I keep the amplitude parameter at 0.0 the whole time.

class ofApp : public ofBaseApp{

	public:
		//generic oF stuff
		void toggleSound(bool on);
		
		std::shared_ptr<ofFmodSoundPlayer> sound;

		ofParameter<float> amplitude;
		ofParameter<bool> on;
		ofxPanel gui;
		float ap;
};
#include "ofApp.h"

//--------------------------------------------------------------
void ofApp::setup(){
	gui.setup();
	gui.add(amplitude.set("amplitude", 0.0, 0.0, 1.0));
	gui.add(on.set("on/off", false, false, true));
}

//--------------------------------------------------------------
void ofApp::update(){

	if (sound != nullptr)
	{
		sound->setVolume(amplitude);
	}
}

//--------------------------------------------------------------
void ofApp::draw(){
	gui.draw();
}

void ofApp::toggleSound(bool on)
{
	if (on)
	{
		sound = make_shared<ofFmodSoundPlayer>();
		sound->load("IMG_0145.wav");
		sound->play();
		sound->setPosition(ap);
		return;
	}
	ap = sound->getPosition();
	ap = ap >= 1.0f ? 0.0f : ap;
	sound = nullptr;
}
//--------------------------------------------------------------
void ofApp::keyPressed(int key){

	if (key == ' ')
	{
		on = !on;
		toggleSound(on);
	}
}

Now I’ve tried reordering the methods called in toggleSound(), but the result is always one of two things:

  1. The setPosition() doesn’t work and the sound plays from the beginning. No cracking sound.
  2. setPosition() works. Cracking sound.

@dimitre , @roymacdonald , somebody help!

EDIT: I’ve tried putting a setPaused(true) after the setPosition(), waiting as much as 60 frames and then setPaused(false), and the crack comes when the setPaused(false) is called.

Hi, why are you specifically using ofFmodSoundPlayer?
Have you tried out the sound player in ofxSoundObjects? I certainly can do what you are looking for.

@roymacdonald Because I couldn’t do it with ofSoundPlayer and someone suggested ofFmodPlayer as a native solution. But I’ll try ofxSoundObjects tonight. Is there an example project for what I need?

There are several examples that can be useful.