Scrub through video without stutter

I’m using ofxQTKitVideoPlayer to play a video and I want to be able to scrub through it without stutters.
What I found so far is that using the setFrame() method works way smoother than setPosition() and fast forward works better than scrubbing backwards.
Also I’m using some sort of acceleration detection (with a Griffin PowerMate). Slow changes result in linear movement, basically just fast forward/backward - which works fine. But when the wheel is turned fast the acceleration kicks in and I’m making big jumps in the video. This is where the whole thing stutters.

In testApp::setup() I’m loading the video and calling video.play() - the video playing doesn’t change during runtime. In update() I call idleMovie() and I draw it in draw(). The scrubbing takes place whenever I get a keypress (the Griffin wheel is mapped to send keystrokes).
The playback speed is never changed.

So inside keyPressed I would call something like:

  
highResVideo.setFrame(highResVideo.getCurrentFrame() + 1 * ofClamp(accWheel,1,20));   

for jumping forward and

  
highResVideo.setFrame(highResVideo.getCurrentFrame() - 1 * ofClamp(accWheel,1,20));   

for jumping backward. With accWheel being the wheel acceleration (1.0f with slow movements, higher if wheel is turned faster than a certain threshold)

Do you have any hints about how I could get rid of the stutters?
I also tried lower resolutions but with really big jumps in the video even tiny videos stutter.

And if it’s not possible to get rid of the stutters completely it would be great if the stuttering video wouldn’t affect my main program. Because now the whole program hangs until the video resumes playback. Could I avoid that using ofxThreadedVideo?

1 Like

what codec is your video? Try the jpeg or animation-codec, as these codecs are much friendlier when jumping back and forth.

If you want to fast-forward/fast-rewind your video why not adjust the speed of the video via setSpeed?

cheers,
Stephan

That behaviour is related to the way the video you are playing is codified.

Codecs use two different ways to compress the videos:

  • Intraframe compression. Based on redundancies in a single frame.
  • Interframe compression. Based on redundancies in consecutive frames.

For interframe compression the codec sets some “key-frames” and then codify the next frames based on differences with that key-frame.

When you “scrub forward” the player decodes normally, but if you “scrub backwards” the player needs to look for the first previous key-frame to the one you are jumping to, and decode following the changes from that key frame.

The same happens with big forward jumps, as the player needs to decode from the first previous key-frame to the one you are jumping to.

If you want to play a video backwards smoothly the best is to use a video with no compression or only intraframe compression (you can use mjpeg or in the codec options force every frame to be a key-frame).

2 Likes

+1 to what sth and eduardo said. mjpeg is much better suited for scrubbing.

Thanks for the hints. Currently I’m using motion jpg with every frame as a keyframe. So the keyframes shouldn’t be the problem.
As for the setSpeed() thing: I’m using the acceleration so that it’s possible to make fine adjustments with slow wheel movements but when the wheel is turned fast I want the video to actually jump quite a bit. I just don’t want it to take too long to resume playing after the jump.
Does ofxThreadedVideo have any advantages when I don’t need to load videos at runtime?

Hi hrb

if you’re already using mjpeg then your problem might be disk i/o speed. try to put the video file on the fastest hard drive you can, an internal SSD is best followed by an SSD on an eSATA, USB2 or firewire connection. if you don’t have an SSD use a hard drive with a 7200RPM platter speed if you can. if you have enough memory, create a RAM disk and then load the video file into that (although you might have performance hits if the OS has to do page swapping).

Thanks damian,
I tried the RAM disk approach and created a 1GB RAM disk on a 4GB ram iMac. It actually performs worse than reading from disk. That’s probably due to page swapping. I have to try it on a machine with more RAM.

yeah, ramdisk performance can be odd sometimes. in theory it’s supposed to work, but it doesn’t always.

if your CPU is not bogged down with too many other tasks, MJPEG might actually be overkill. as @eduardo said you basically want to reduce the number of B/P-frames and increase the number of I-frames. MJPEG is 100% I-frames but it creates heavy disk i/o for high quality. you could increase the amount of JPEG compression but then you’ll end up with JPEG artifacts. if you want to retain quality you could try using a more compressed but CPU-heavy format like MPEG-2 or even H.264, but encode the video to have many more I-frames, like once every 2 or 3 frames (the default for most encoders is around once every 12-24 frames). something like this is very achievable using FFMPEG/X264 as they allow very fine grained control over encoder parameters. this stackexchange question might help: http://avp.stackexchange.com/questions/3164/which-decoder-allows-for-fast-seeking-or-how-to-configure-k-lite-codec-pack

Ok, after restarting the project it turns out the RAM disk solution works really smooth. Way better than I expected. But: it takes a couple of attempts. So after the third or fourth start of the project it’s really great. As if it takes a while to actually load the video.

You should be able to load the movie completely into the RAM via quicktime. I don’t know if OpenFrameworks exposes the method “LoadMovieIntoRam” from quicktime.

This is what i am using in my own framework.

  
  
void preload(float startTime = 0, float duration=-1)   
{  
    if (duration < 0) duration = getMovieDuration();  
    LoadMovieIntoRam(_movie, (TimeValue) (startTime*_timescale), (TimeValue)  (duration*_timescale), keepInRam);  
}  
  

cheers,
Stephan

I set decodemode to OF_QTKIT_DECODE_PIXELS_ONLY and put setFrame() on another thread, and feed movie pixels in that threadedFunction().

#include "ofMain.h"

class Movie : public ofThread
{
public:
    
    void setup()
    {
        stopThread();
        
        decodeMode = OF_QTKIT_DECODE_PIXELS_ONLY;
        player = ofPtr<ofQTKitPlayer>(new ofQTKitPlayer);
        player->setPixelFormat(OF_PIXELS_RGB);
        player->loadMovie("movs/0.MP4", decodeMode);
        player->stop();
        
        state = FEED;
        
        startThread();
    }
    
    void threadedFunction()
    {
        while (isThreadRunning())
        {
            if (state == FEED)
            {
                movPx.clear();
                
                player->setFrame(frame);
                player->update();
                
                if (player->isFrameNew())
                {
                    movPx = player->getPixelsRef();
                    state = DRAW;
                }
            }
        }
    }
    
    void update()
    {
        if (state == DRAW)
        {
            movImg.setFromPixels(movPx);
            state = FEED;
        }
        
        frame = ofMap(ofGetMouseX(), 0, ofGetScreenWidth(), 0.0, player->getTotalNumFrames(), true);
    }
    
    void draw()
    {
        if (movImg.isAllocated())
            movImg.draw(ofGetWidth()/2 - movRes.x/2, ofGetHeight()/2 - movRes.y/2,
                        movRes.x, movRes.y);
        
        rot += 0.5;
        ofSetRectMode(OF_RECTMODE_CENTER);
        ofPushStyle();
        ofSetColor(ofColor::cyan);
        ofPushMatrix();
        ofTranslate(ofGetWidth()/2, ofGetHeight()/2);
        ofRotate(rot, 0, 0, 1);
        ofRect(0, 0, 100, 100);
        ofPopMatrix();
        ofPopStyle();
        ofSetRectMode(OF_RECTMODE_CORNER);
    }
    
    void exit()
    {
        if (isThreadRunning())
        {
            stopThread();
            ofSleepMillis(200);
        }
        
        player->closeMovie();
        player->close();
        player.reset();
    }
    
    ofPtr<ofQTKitPlayer> getPlayerPtr()
    {
        return player;
    }
    
private:
    
    enum STATE
    {
        FEED,
        DRAW
    };
    STATE state;
    
    const float scale = 0.8;
    const ofVec2f movRes = ofVec2f(1920 * scale, 1080 * scale);
    
    int frame;
    ofPixels movPx;
    ofImage movImg;
    
    ofPtr<ofQTKitPlayer> player;
    ofQTKitDecodeMode decodeMode;
    
    // debug
    float rot;
};

Scrubbing h264 is still jaggy (HAP codec is quite smooth), but does not affect main thread.

A

I’d suggest the HAP codec too, as well as SSD as much as possible.

I found more nice solution!

[_player seekToTime:CMTimeMakeWithSeconds(time, _duration.timescale)
 toleranceBefore:kCMTimeZero
 toleranceAfter:kCMTimeZero];

It allows more sensitive scrubbing like Trim on QuickTime Player. I tested it with fullHD h264 and do not see any shutter.
Added setExactPosition(float pct) here::

void ofApp::setup()
{
    player.loadMovie("movs/ComputerEntertainment/0.MP4");
    player.stop();
    rot = 0.0;
}

void ofApp::update()
{
    float pct = ofMap(ofGetMouseX(), 0, ofGetScreenWidth(), 0.0, 1.0, true);
    player.setExactPosition(pct);
    player.update();
}

void ofApp::draw()
{
    player.draw(0, 0);
    
    rot += 0.5;
    ofSetRectMode(OF_RECTMODE_CENTER);
    ofPushStyle();
    ofSetColor(ofColor::cyan);
    ofPushMatrix();
    ofTranslate(ofGetWidth()/2, ofGetHeight()/2);
    ofRotate(rot, 0, 0, 1);
    ofRect(0, 0, 100, 100);
    ofPopMatrix();
    ofPopStyle();
    ofSetRectMode(OF_RECTMODE_CORNER);
}
4 Likes