How would you go about manipulating videos' audio in realtime?

I’d like to activate various videos via different MIDI signals from my MIDI-violin, and be able to shift their pitch/playback-speed up and down with the MIDI’s bend value (slide/glissando/vibrato) in real-time. Is this kind of manipulation of video audio even possible with oF’s native video tools? Or am I going to need some kind of special video add-on?

@s_e_p I would use ofxPd for manipulating videos audio in realtime (because I am used to Pure Data).
You can also control ofxPd with midi (ofxMidi). Maybe there is also a way without using addons, I am not sure about that.

1 Like

Hey, thanks for your answer, @Jona . I’ve been using PD directly and linking it to oF via osc, but if using ofxPd let’s you grab video audio easily that’d be great. Are there examples of this in the ofxPd directory?

I’m interested in something similar: I want to react to both movement and audio from a video stream, but finding it difficult to extract the audio track from an ofxVideoPlayer/ofxVideoGrabber into a ofSoundBuffer etc.

However, I just found https://github.com/kritzikratzi/ofxAvCodec from the prolific @kritzikratzi which appears to fit the bill. Hopefully!

I haven’t tried it yet: need to build an ARM ffmpeg for my M1 mac but it looks promising.

Ohh, let us know how it goes!

1 Like

Not sure how how to do that with the desktop app, but with Emscripten it was possible to route the audio nodes from the video into OFs audio input and from there to ofxPd.

Hey @steve.m, how’s your experience with ofxAvCodec going? Does it require you to recode your videos using their codec? If there’s a simple way to do so, I’d really like to be able to have simple dynamic speed-manipulation of both audio and video. It looks like this is possible with ofxHapPlayer, but I need to convert my videos to hap codec.

Great! If I was going to try this out today I would split audio and video, like having a wav or aiff file and doing the manipulations all there.
having the video track without audio in a performant random access codec like hap, ofxHapPlayer and trying to make it follow audio position.
if your video is not huge (duration and frame size) you can even export them in image sequence and load all the frames in memory to have a faster access

Heya @s_e_p I haven’t got to it yet: some refactoring and rework to do before I hit it. But I will get there.

ofxHapPlayer doesn’t suit my usecase very well: I need access to the frame pixels to feed OpenCV, so I think that rules out that approach.

I’ll try to push ahead with ofxAvCodec in the coming days/weeks. I’m trying to build a tool for other artists to use, so if I can just say “feed this app a video file with a soundtrack” then that’s better than asking non-tech users to split the audio into a separate file.

@steve.m , yeah, that’s a good point. I know how to split audio from video and I don’t even want to do it. I’m not savvy enough of a coder to write a script to do it for me, so I’d take each of the 20 videos and put them in a DAW or video-editing software to do it…

I’ve posted a question about hap audio on this thread, so please reply if you’ve any ideas…