Only update moving pixels

Hi all!

I’d like to write a program that will only update the moving pixels in a video, and leave the static pixels as a trail without updating the background.

For example, I shot this video of wind turbines. I’d like to run it through this program so that the arms of the turbine would leave a trail as they spin, rather than the sky updating behind them.

https://vimeo.com/72711874

Is there a name for this type of filtering. I can think of it as similar to how a long exposure photograph would work, or when you use setBackgroundAuto(false); or almost a variant of a slitscan algorithm.

What would the best approach be? I’ve been trying to do some research into what the best way to do this might be, but without too much luck. Is this something that frame differencing or background subtraction would be good for? Or would I need to store cached pixels in a FBO?

Any pushes in the right direction or examples of similar projects would be most appreciated! If this topic has come up before, are there any old threads I can take a look at? I couldn’t seem to find anything or didn’t have the right words for my search.

Many thanks!

You’re looking for pixel differencing and there’s a ton of different ways that you can do it, but I have an example at https://github.com/joshuajnoble/Programming-Interactivity-Code/tree/master/ch09/0908 that does exactly that. Hopefully that points you in the right direction :slight_smile:

Thank you Joshua! This is absolutely what I needed to see. I had your book on my desk, the answer was right in front of my face >_<

I was able to get something going, but I’m kind of cheating in a way by manually taking a screenshot first and loading the image and placing those pixels into the unchanged spots. I tried just capturing the pixels of the first frame, but I couldn’t figure out how to do that and update my movement image at the same time. Here’s what I’ve got working. It’s an extra annoying step, but it works, although only with video shot from a static point.

I’m running the ofQTKitPlayer for a performance boost since it was exceedingly slow running a 1080p video file.

.h file

  
  
#pragma once  
  
#include "ofMain.h"  
  
class testApp : public ofBaseApp{  
	public:  
		void setup();  
		void update();  
		void draw();  
		  
		void keyPressed(int key);  
		void keyReleased(int key);  
		void mouseMoved(int x, int y);  
		void mouseDragged(int x, int y, int button);  
		void mousePressed(int x, int y, int button);  
		void mouseReleased(int x, int y, int button);  
		void windowResized(int w, int h);  
		void dragEvent(ofDragInfo dragInfo);  
		void gotMessage(ofMessage msg);  
          
        ofImage firstImage;  
        unsigned char * firstFrame;   
        ofVideoGrabber videoIn;  
        ofImage lastFrame;  
        ofImage movement;  
        ofQTKitPlayer movie;  
        int totalPixels;  
      
        int w ;  
        int h ;  
};  
  

and the main app

  
  
#include "testApp.h"  
  
//--------------------------------------------------------------  
void testApp::setup(){  
    w = 1920;  
    h = 1080;  
  
      
    ofSetWindowShape(w, h);  
    totalPixels = w * h * 3;  
    //videoIn.initGrabber(1920 ,1080);  
    lastFrame.allocate(w ,h, OF_IMAGE_COLOR);  
    movement.allocate(w ,h, OF_IMAGE_COLOR);  
    //ofClear(0);  
    firstImage.loadImage("frame1.png");  
    firstImage.draw(0,0);  
    firstFrame = firstImage.getPixels();  
      
      
    movie.setPixelFormat(OF_PIXELS_RGB);  
    ofQTKitDecodeMode decodeMode = OF_QTKIT_DECODE_PIXELS_AND_TEXTURE;  
    movie.loadMovie("windFarm.mp4", decodeMode);  
    movie.play();  
}  
  
//--------------------------------------------------------------  
void testApp::update(){  
      
    movie.update();  
      
    if(movie.isFrameNew())  
    {  
        unsigned char * lastFramePix = lastFrame.getPixels();  
        unsigned char * videoPix = movie.getPixels();  
        unsigned char * movementPix = movement.getPixels();  
          
        for (int i = 0; i < totalPixels; i+=3)  
        {  
            unsigned char r = abs(lastFramePix[i] - videoPix[i]);  
            unsigned char g = abs(lastFramePix[i+1] - videoPix[i+1]);  
            unsigned char b = abs(lastFramePix[i+2] - videoPix[i+2]);  
              
			int diff = r+g+b;  
            if (diff > 25) {  
                movementPix[i] = lastFramePix[i];  
                movementPix[i+1] = lastFramePix[i+1];  
                movementPix[i+2] = lastFramePix[i+2];  
            } else if(movementPix[i] == 0){  
                movementPix[i] = firstFrame[i];  
                movementPix[i+1] = firstFrame[i+1];  
                movementPix[i+2] = firstFrame[i+2];  
            }  
        }  
          
        movement.setFromPixels(movementPix, w, h, OF_IMAGE_COLOR);  
        lastFrame.setFromPixels(videoPix, w, h, OF_IMAGE_COLOR);  
    }  
       
}  
  
//--------------------------------------------------------------  
void testApp::draw(){  
    movement.draw(0, 0);  
}  
  

Doing it this way seems to produce a lot of jagged edges, I wonder if running some sort of edge detection and blur might clean it up a bit. There is also a definite sweet spot with the difference threshold. Setting it really low doesn’t really work as it appears that even pixels that seem like they should be static still have a slight flicker or noise to them.

You mentioned that there are a number of different ways to do this, out of curiosity, what are some of the other methods?

Oh yes and my first test!
http://25.media.tumblr.com/1d99add993e3ae692b2d8fb410ba1952/tumblr-mso48xa3Ak1qzhwpro1-1280.png

The other way that comes to mind is to use a shader that looks at the two different textures (last texture and current texture) and outputs a differencing image. I think that might actually work better wtih the QTKit video player since that really wants to directly upload textures to the card (hence the decodeMode stuff). Somthing like:

  
uniform sampler2D tex;  
uniform sampler2D tex2;  
  
void main()  
{  
	vec3 rgb = texture2D(tex, gl_TexCoord[0].xy).rgb - (texture2D(tex2, gl_TexCoord[0].xy)).rgb;  
	gl_FragColor = vec4(rgb, 1.0);  
  
}  

should work fine.

I’m having a little bit of trouble getting your shader to work. I’m relatively new to glsl so I might be making a really obvious error.

diff.vert

  
  
void main() {  
    gl_Position	= gl_ModelViewProjectionMatrix * gl_Vertex;   
    gl_FrontColor = gl_Color;  
    gl_TexCoord[0] = gl_MultiTexCoord0;  
}  
  

diff.frag

  
  
uniform sampler2D tex;    
uniform sampler2D tex2;    
uniform float thresh;  
  
void main()    
{    
    vec3 rgb = texture2D(tex, gl_TexCoord[0].xy).rgb - (texture2D(tex2, gl_TexCoord[0].xy)).rgb;    
    if(rgb.x>thresh){  
        gl_FragColor = vec4(rgb, 1.0);  
    }  
    else{  
        gl_FragColor = vec4(1.0,0.0,0.0,1.0);  
    }  
      
}  
  

testApp.cpp

  
  
void testApp::setup(){  
    w = 720;  
    h = 560;  
  
    shader.load("diff.vert", "diff.frag");  
      
    ofSetWindowShape(w, h);  
    totalPixels = w * h * 3;  
      
    videoIn.setDeviceID(7);  
    videoIn.initGrabber(w ,h);  
    videoIn.setUseTexture(true);  
      
    lastFrame.allocate(w ,h, OF_IMAGE_COLOR);  
      
}  
  
//--------------------------------------------------------------  
void testApp::update(){  
 videoIn.update();      
}  
  
//--------------------------------------------------------------  
void testApp::draw(){  
    shader.begin();      
      
    videoIn.draw(0,0);  
    shader.setUniformTexture("tex", videoIn, 0);  
    shader.setUniformTexture("tex2", lastFrame, 1);  
    shader.setUniform1f("thresh", (float)mouseX/ofGetWidth());  
  
    shader.end();  
      
    unsigned char * videoPix = videoIn.getPixels();  
    lastFrame.setFromPixels(videoPix, w, h, OF_IMAGE_COLOR);  
}  
  

It seems like it should be pretty straightforward, but nothing is drawing to the screen. I can change the gl_FragColor to be something like 1.0,0.0,0.0,1.0 to turn everything red so I know that it is working, it just doesn’t seem like it’s receiving the pixel data.

It didn’t work without calling video.draw from within the shader. I was referring to the chapter in your book about ofShader for reference and the vert shader with the chromakey example you use. But you do not call draw on your video source. Also, is it necessary to have a vertex shader in this case? It seemed to throw an error when I did not include one.

I tried to put a little threshold test in to see if it was working, but it only swaps when i go below zero with my mouse leading me to believe I’m incorrectly testing for difference or that all of my pixels are marked as 0(black). My other thought was that I’m not pulling the last frame properly, so even if the shader was working the frame is the same meaning the difference would always be zero. However, I’m not sure this is the case because I plugged in a different image for the tex2 and got the same result.

Any suggestions or pointers? Thanks!

Ah, my bad. OF uses ARB textures, which means that the fragment shader should look like this:

  
  
uniform sampler2DRect tex;      
uniform sampler2DRect tex2;    
  

and the call to texture2D should be texture2DRect:

  
 vec3 rgb = texture2DRect(tex, gl_TexCoord[0].xy).rgb - (texture2D(tex2, gl_TexCoord[0].xy)).rgb;   

Aha! Thank you Joshua, works like a charm now :slight_smile:

Hi!

First of all: Thank you all for sharing this great information!

My system only supports glsl from version 1.5. gl_MultiTexCoord0 and gl_TexCoord seem to be deprecated in 1.5.
In other forums people say you should use a custom attribute for texture coordinates. I really couldn’t figure out how to do that. Does ofx already set them? Or how would i get them from the videoplayer? Im a beginner with glsl and find really hard to understand where does a variable, attribute or uniform come from or how i need to set them.
I’d highly appreciate if somebody could help me out on that.

Thanks a lot!

In short, if you’re using the programmable renderer (ofProgrammableRenderer) then yes. This tutorial hopefully will help http://www.openframeworks.cc/tutorials/graphics/shaders.html

Hi again,

I just want to clarify how the last frame’s pixels are being obtained. In the example that Joshua links to it looks like they are loaded using setFromPixels() after everything else in the update loop has been called. Is this generally the best way to get previous frames? What if I want to get the pixels/texture of a frame that is more than 1 frame back? Is this where it would be handy to start saving things into a fbo? Does it make a difference if the pixels are coming from a webcam vs. a movie file (with the movie file we can know ahead of time which past pixels we want rather than the continuous stream from the cam)?

Many thanks for the help and patience!

You can always create multiple ofPixels instances and store pixel values in them. You could also store it in an ofFbo or an ofTexture. It all depends on what you’re trying to do.

Thank you for the quick reply Joshua. I should have been more direct with my question. Say I am trying to compare the current frame with one that is three seconds back. How do I access the three seconds back frame?

What’s confusing to me is how the two timeframes get staggered in the first place. In your example, it kind of makes sense, because lastPixels isn’t updated until the camera has updated, so there is an inherent delay in that little cascade. But if it’s not the immediately previous frame, I’m not sure I understand how it can be retrieved.

You can’t retrieve it, but you can store it. The thing is, if you want 3 seconds of frames, you’re going to store 90 frames in a vector of ofPixels and then look back to the beginning. I know that there’s an example of this floating around somewhere on the forum but I can’t seem to dig it up right now.

Ok, that makes more sense. I’d love to see it if you ever happen to stumble across it again!

I was trying to test a code similar to the one you wrote to make sure I was looking at two distinct frames by holding a stopwatch on my phone up to the camera and drawing both current and past images to the screen. I saved the images, but they are exactly the same, even the microsecond counter. I lowered the framerate to 5fps thinking that maybe the frames were updating faster than the micro seconds but still the same. Maybe I’m wrong in thinking that this is a good way to test for the last frame?

For my purposes I need a grayscale image, but I wouldn’t think that should affect the outcome
Here is what I did:

  
  
void testApp::setup(){  
    w = 640;  
    h = 480;  
          
    cam.initGrabber(w, h, true);  
    ofSetWindowShape(w*2, h);  
    ofSetFrameRate(5);  
      
    grayTex.allocate(w,h,GL_LUMINANCE);  
    lastTex.allocate(w,h,GL_LUMINANCE);  
  
    fbo.allocate(w,h,GL_RGB);  
  
    fbo.begin();  
    ofClear(0,0,0);  
    fbo.end();  
  
    fboPix.allocate(w, h, 3);  
}  
  
void testApp::update(){  
    cam.update();  
  
    if(cam.isFrameNew()){       
         grayTex.loadData(cam.getPixelsRef()); //convert to grayscale  
        
         fbo.begin();  
         grayTex.draw(0,0);  
         fbo.end();  
  
         fbo.readToPixels(fboPix);  
         lastTex.loadData(fboPix);  
     }  
}  
  
void testApp::draw(){  
     grayTex.draw(0, 0);  
     lastTex.draw(w,0);  
}  
  

Thanks again for taking the time to answer my questions, I really appreciate it!

That’s because you’re using the same data each time. Here’s what you’re doing:

  
  
// load all your current pixels into grayTex  
         grayTex.loadData(cam.getPixelsRef()); //convert to grayscale    
  
// draw grayTex into your FBO (meaning that your FBO is only going to show grayTex)  
         fbo.begin();    
         grayTex.draw(0,0);    
         fbo.end();    
    
// read all the pixels from your FBO (which only shows grayTex) into fboPix  
         fbo.readToPixels(fboPix);    
// load all the pixels from your fboPix (which only represent what's in fbo (which only shows grayTex)) into lastTex  
         lastTex.loadData(fboPix);  
  

you want:

  
  
// store last thing that was in grayTex.  
fbo.begin();    
grayTex.draw(0,0);    
fbo.end();    
  
// put new data in grayTex  
grayTex.loadData(cam.getPixelsRef()); //convert to grayscale    
    
  

now just draw the two and you should see a *very slight* difference.

Ah that works. I’m still not sure I understand why, but thank you!