Adjust video speed to sync to bpm onsets

hi
i am trying to create a video player with syphon output (got that working) that adjusts its speed to an incoming bpm plus onset signal.

i currently have: incoming osc messages with system millis timestamps of next onset plus a tempo given in bpm.

all my videos are edited so they have 60 bpm and a length that matches the bpm plus a 60fps framerate.
(this means that a video with a guy hitting a nail with a hammer might have ten hits, each hit taking one second which gives the video a length of 10 * (60bpm/60) == 10 seconds)

lets imagine the video position is at 00m01s05ms and the incoming bpm+onset-ms tells us that there is a beat in 00m00s200ms then i wanna adjust the video speed so that the video will be at position 00m02s00ms in 00m00s200ms from now.

my problem is that i am not strong in maths and i cant formulate the problem so that i am able to find standard solutions (which i guess there exists)

i guess this problem is similar to PID control as i am gonna modify the speed of the video so that it will eventually be having the video position increase by a second every time an incoming beat is received.

anybody able to elaborate and help me understand the issue at hand? :slight_smile:

i have created a simple solution:


    int now = ofGetSystemTimeMillis();
    if (now >= nextBeatMS)
    {
        
        float position = fingerMovie.getPosition() * fingerMovie.getDuration();
        
        float nextposition = floor(position + 1.0);
        if (nextposition > fingerMovie.getDuration()){
            nextposition - fingerMovie.getDuration();
        }

        fingerMovie.setPosition(nextposition);
        
        fingerMovie.setSpeed(bpm/60.0);

       
        nextBeatMS = now + (60 * 1000 / bpm);
    }

this means:
on every received bpm + next-beat-in-system-time
calculate new video speed: bpm/60 (as all input videos are in 60bpm)
calculate next beat position : this is just the next video second, so if the video position currently is 00m01s200ms then the next beat position is 00m02s000ms
remember to wrap the next position around the video duration (kinda modulus) in order to find the real next position.

then:
set the video position to the nextposition
and
set the speed to the new speed

1 Like

Hi just thinking out loud here

Tight syncing, to the millis, or to the nanos, you are using OSC and Syphon, that’s already not good, overheads & timing/disk access errors, etc. You already did the hard work, and have the maths, so I would adjust access strategy.

Remember the infrastructure of your videos, codecs are important, some on time, others on space. Some don’t like you touching their speeds too often. They themselves have to find the surrounding keyframes (or other mechanisms), decode the cyphered datum, fill out your framebuffer. Others abolish keyframe compression & boilds down to spatial frame decomprssion. You want these nanos controlled and kept to a minimum overhead.

Then you can stop the video. And manually sync each desired framed, from a linear or curved interpolation between your desired timestamps to the nano, translated into a pair of surrounding frames to your videos.

Another thing I would try would be not using instant speed, low pass it to your needs. Stop the videos. Normalize the lenghts. Interpolate. You already have the maths in place.

I can confirm you don’t need network to synchronize studff. You may only use time. It’s what we only have. Since I can’t compile at the moment, I cannot veryfy these assumptions, though I guess they are very helpful, at least they are, to me

My 2 cents

1 Like

Almost forgot. Check out this bug report addon. Probably can help you out,.

https://forum.openframeworks.cc/t/line-129-title-std-chronos-feb-aug-seth22/40305/8

@opseidon i got around making it work!
here is a video: VID 20220912 115801829 - YouTube

here is the beat tracker: beatape GitHub - sloev/beatape: beat tracker with osc publisher