audio analysis on video...

beginners
#1

Hello,

I am thinking about some sort of VJ tool kind of software where I want video (mov mp4, like that) be the base of the app. So I would like to do audio analysis on a video containing audio.

I am aware of the ofSoundGetSpectrum(…) for audio but cant find a way of doing something similar on the audio part of a video.

I thought about ‘splitting’ the audio from the video and directly analyse that and use it, but since it will be a ‘live’ thing that will be pretty hard to do without delays I assume…

anyone any idea’s to get me started?

thanks!

Jarno

1 Like
#2

Old topic, but i’m also looking for a feature to analyse the audio from an input video file with fft.

Any tips would be greatly appreciated!

#3

How about stripping audio file from video and load/play separately?

#4

Yeah if it isn’t possible that will be my fallback solution. (Or otherwise I could maybe use the approximate light in the video as a variable)

But I’m a bit reluctant to go into the direction of using multiple files, because the app is going to be used by someone to generate promotional material from it. And I want to keep the app stupid simple, video in, add my layers, video out.

Maybe there are some other c++ libraries that have the capability of reading the audio of a video file or I could use some commandline tools to separate the file in the background, but if there is any OF way of doing this that would definitely be great!

#5

OF doesn’t support this right away but using gstreamer you can access the audio stream by setting your own pipeline and in the audio part using a tee to send the audio both to the normal audio sink and to for example an appsink so you can read the audio stream from the application or even use the spectrum plugin https://gstreamer.freedesktop.org/documentation/spectrum/index.html?gi-language=c + appsink to directly get the fft analisys. it’s not very straightforward but is possible to do it

1 Like
#6

Ah awesome, i came across your ofxGStreamer addon looking for solutions.
I definitely need to read into the GStreamer framework though :wink: but it looks comprehensive and worth getting into!

#7

I managed to get ofxGstreamer on macOS 10.15.7 working. My source is an IP camera that also sends audio.

 m_Player.load("rtsp://root:password@198.168.0.122/axis-media/media.amp?resolution="+ofToString(vidWidth)+"x"+ofToString(vidHeight)+"&latency=0&audio=1");
    m_Player.play();

I have a Axis P1343 with build in mic and I can hear the audio through my computer speakers well.
I also have a P3245 with attached external mic which only plays audio for the first 2 seconds and then the audio stream get noisy.

question 1:
is there anything in terms of audio sampleRate that can be set for the gStreamer?

question 2:
just like the previous posters I am interested in accessing the audio stream, in order to record them along with the video. Since the gStreamer does not show up as an audio device I think can’t use ofSoundStream to pipe the audio in to my ofxFFmpegRecorder.

Thanks for any advice.

1 Like
#8

Here some experiments with the Audio Input Encoding settings via the cameras browser portal:

P3245:
Good for 1 sec then very stuttery and distorted
AAC - 32 hz, 32 kbit/s
AAC - 32 hz, 16 kbit/s

Good for 1 sec then stutter and distortion come in
G711 - 8 kHz, 64 bkit/s (underwater like quality)
AAC - 16 kHz, 12 kbits/s

Better but has clicks and pops
AAC - 32 kHz, 24 kbit/s
AAC - 16 kHz, 64 kbit/s

G726- 8 kHz, 32 kbit/s (underwater like quality but consistent)
G726- 8 kHz, 24 kbit/s (very bad, lots of pops)

AAC - 16 kHz, 32 kbit/s
AAC - 16 kHz, 24 kbit/s

P1343
pretty good and consistent
AAC - 16 hz, 32 kbit/s

When opening a RTSP stream in VLC I do not get any of these audio issues, but the stream is 2 seconds delayed.
So, maybe setting the latency in OF to 0 as mentioned here https://github.com/arturoc/ofxGStreamer/issues/31#issuecomment-1089036757 causes the audio to not be happy.

1 Like