Read Volume of Video Frame and/or Filter Frames based on Volume

I have video files that were made with a circuit as the sound source. I want to extract the frames with a volume over a certain threshold. I looked at the utils and addons and did not see one that would allow me to do this.

I think there are two approaches possible:

1.) Read each frame’s volume level and if it is over a certain threshold, save the image.

2.) Filter all frames over the volume threshold (ffmpeg? libav?) - not sure how this works exactly.

In either case, I don’t know what library, add-on, etc. I should be using. None of my searches have showed me a direct way to do this (that’s a lie - I found a way in MATLAB but I don’t want to learn it just for this project).

Any guidance much appreciated!

you can look at the addon i’m currently developing. it’s a bit fragile, and you’ll have to be experienced to set it all up. https://github.com/kritzikratzi/ofxAvCodec

you’ll make your own life easier if you preprocess a bit:

  1. separate the audio into it’s own file, extract individual frames into a tiff sequence (both is easy with ffmpeg command line)
  2. write your OF app so that it reads the wav on startup and loads the frames accordingly.

Thanks @kritzikratzi. I had downloaded your addon but had not considered separating the audio and video. Thanks for the response!

you can do it with my addon, but it’s also some work. roughly the code would go like this:
(this is completely untested, i just typed it here, but should give you some rough idea)

void setup(){
    ofxAvVideoPlayer player; 
    player.setupAudioOut(1,22050); // you only want to get the amplitude, 22kHz is good enough! 
    player.setLoop(false); 
    player.play(); // this starts buffering, the timing is controlled by the audioOut() function
    
    // the player is made for realtime and doesn't have a "sync" mode yet
    // give it some time to buffer. 
    ofSleepMillis(1000); 
    
    int bufferSize = 1024; // 1024/22050 = 0,046, roughly 50ms 
    float * buffer = new float[bufferSize]; 
    
    
    int numSamplesRead = 0; 
    while(player.isPlaying()){
        // read up to bufferSize
        if(numSamplesRead<bufferSize){
            // read some more 
            player.audioOut(buffer+numSamplesRead, bufferSize-numSamplesRead, 1); 
        }
        
        // update the video, this keeps audio and video in sync
        player.update(); 
        
        if(numSamplesRead == bufferSize){
            // yes, we have a full frame. this is great
            float volume = yourFunctionThatCalculatesVolume(buffer, bufferSize); 
            if( volume > yourThreshold ){
                yourFunctionThatStoresThePixels(player.getPixels()); 
            }
        }
        
        // let it breath a bit
        ofSleepMillis(1); 
    }
}