OpenCV Error: Assertion failed while using ofxCv::FlowPyrLK

advanced
#1

Hi all,

I’m working on a project where the user can load a video file and have it processed by filters controlled by optical flow. I decided to use Kyle McDonald’s ofxCv because of its simplicity and tight integration with OF. Here is my code:

  
void testApp::update() {  
if (videoLoaded && video.isFrameNew()) {  
            pyrLk.calcOpticalFlow(video);  
}  
}

This works perfectly until I load a different size video at which point the following assertion fails:

OpenCV Error: Assertion failed (prevImg.size() == nextImg.size() && prevImg.type() == nextImg.type()) in calcOpticalFlowPyrLK, file /Users/theo/Downloads/OpenCV-2.3.1/modules/video/src/lkpyramid.cpp, line 515

This makes sense since calcOpticalFlow() is storing the last image, so the the first time its called after I load a new video, the two image sizes are not the same.

To solve the problem, I wrote a new function in the ofxCv::FlowPyrLk class to clear the image held in the FlowPyrLk object so the next time calcOpticalFlow() is called, it should start from scratch. Instead, another assertion fails!

Here’s the function I wrote:

  
  void FlowPyrLK::clearLast() {  
        last.clear();  
        curr.clear();  
        prevPts.clear();  
        nextPts.clear();  
        hasFlow = false;  
    }

and here is the failed assertion:

OpenCV Error: Assertion failed ((npoints = prevPtsMat.checkVector(2, CV_32F, true)) >= 0) in calcOpticalFlowPyrLK, file /Users/theo/Downloads/OpenCV-2.3.1/modules/video/src/lkpyramid.cpp, line 518

As far as I know this assertion checks to see if the input to the openCV function is a set of points. So i don’t understand why it is triggered in this case. Originally the function didn’t have these lines: prevPts.clear(); nextPts.clear();. I added them in response to the failed assertion.

Moving along, I created a three level if statement. If the video is new, store the image in an ofImage called “last”, if the first frame of the video has been stored, calculate the optical flow with the current frame and “last”, finally return to using pyrLk.calcOpticalFlow(video). Here’s the code:

  
        if (videoLoaded && video.isFrameNew()) {  
            if (flowStage == 0) {  
                last.setFromPixels(video.getPixels(), video.width, video.height, OF_IMAGE_COLOR);  
                flowStage = 1;  
            }  
            else if (flowStage == 1) {  
                ofImage curr;  
                curr.setFromPixels(video.getPixels(), video.width, video.height, OF_IMAGE_COLOR);  
                pyrLk.calcOpticalFlow(last, curr);  
                flowStage = 2;  
            }  
            else if (flowStage == 2){  
                pyrLk.calcOpticalFlow(video);

When I run this, the first assertion(size and type of the two image are the same) fails again. When I debug the code, I found that the assertion fails on step two. This led me to test the optical flow function that takes to arguments on its own. Not only does it trigger the failed assertion when I use two images that I know are the same size and type… it fails when I call the function using the SAME image in both arguments!

So now I know I’m missing something. Has anyone had success using void Flow::calcOpticalFlow(ofBaseHasPixels& lastImage, ofBaseHasPixels& currentImage) ? The example that comes with the addon only uses** void Flow::calcOpticalFlow(ofBaseHasPixels& nextImage)** or maybe I’m missing an obvious flaw in my own code?

Any insight is greatly appreciated,
Jason

#2

hi Jason, i had a similar issues because i used OF 007 version instead OF 0071 with my old pc … i think this will help you:
https://github.com/kylemcdonald/ofxCv/issues/39
http://forum.openframeworks.cc/t/issue-with-ofxcv-in-ubuntu-11.04-and-of007-solved/9897/0
Cheers
Walter

#3

Hi Walter,

Thanks for the reply. I’m using OF 0071 so I don’t think that is the problem.

I was recently told that any CV object should only be used uniquely for one image size. Since my project involves changing the source image often, I’m testing out a method using dynamically allocated memory: When a new video is loaded, the current optical flow object is deleted and a new one is created. So far, no success. :frowning:

If you or anyone has tried something similar with success, hit me up!

Thanks,
Jason

#4

I was going through the exact same situation in a very similar scenario.

This is how I solved it:
I wrote a wrapper class to easily swap between video-sources (grabber/player) with different frame pixel types and/or size. I overloaded [tt]void update()[/tt] to increase a “frame counter” variable on each call and a [tt]bool isReady()[/tt] function returning true only when video-source is ready frame was >= 1, being this a condition to call [tt]ofxCv::FlowPyrLK::update[/tt]

in the main app’s code:

  
  
...  
void testApp::update() {  
  video.update();  
  if (video.isReady() && !opFlow.enabled) opFlow.enable();  
  if (video.isFrameNew()) {  
    frame.setFromPixels(video.getPixelsRef());  
    opFlow.calcOpticalFlow(frame);  
  }  
}

in my wrapper class:

  
  
void reset() { _frame = 0; } // call on image change and/or initialization  
  
void VideoSource::update() {  
  _frame++;  
  if (_frame == 0) _frame++;  // avoid being 0 on integer overflow  
  usingCamera ? camera.update() : player.update();  
}  
...  
bool VideoSource::isReady() {  
  return _frame > 0 && (usingCamera ? camera.isInitialized() : player.isLoaded());  
}