Depth and color tracking

So, I’ve been hitting my head against this program for the better part of 3 weeks without much success. I’ve been going through ofxCv and its pretty impressive and think its the key to solving my issue.

I’m trying to accurately detect when a ball is thrown at a wall and what color that ball is. To make things harder, there is a game projected onto the wall with many changing colors. I am using a kinect to get depth and color data. The kinect is mounted right next to the projector so the balls that the kinect sees are seeing the full image of the projector, unfortunately.

What I’ve tried with decent success depth-wise is doing a simple absdiff on the background, thresholding it with the current kinect depth image and then taking whatever values fit and flattening them to 0 or 255.

With that thresholded depth image I run an ofxCvContourFinder with some simple min/max size constraints. From this point, I could just send the message that the ball hit the wall and depth wise it would work alright. But unfortunately, I need color too!

So if I found a contour using the OF contour finder then take the thresholded depth image and use it as a stencil to cut out the blob from the kinect color camera, leaving everything I don’t want black. I run an ofxCv::contourFinder (The ofxCv one) for the 4 different colors of the balls I’m looking for. If I find a contour (which generally I do as the OF contour finder found one) then I use a ROI to go through all the pixels and find out which color the blob is most similar to.

With this process I would say it works about 80% of the time after a whole bunch of tweaking.

I think the biggest issue is that I’m not using any frame-by-frame tracking, but I think that the kinect depth data isn’t good enough to rely on it solely. If I save the depth image out every frame I see that when things move fast the depth data of the ball is spotty and not uniformly circular. I attempted to use color background differencing but didn’t have much success with that. My other issue is that the tracker in ofxCv looks like it would be super useful but it confuses me in that I don’t know how to tell how many labels were tracked (so I know when to send the message. If I could get it to accurately detect the velocity of the ball, that would be amazing).

I also attempted Hough circle tracking using openCv but it doesn’t seem super accurate and its rather laggy.

Do you have any suggestions on how I should tackle it? Any help would be so so greatly appreciated!

I haven’t thoroughly investigated about applying on Kinect depth stream but the tracker does supports labeling and you can adjust the time to forget the label and the maximum speed to follow ( https://github.com/kylemcdonald/ofxCv/blob/master/example-contours-tracking/src/ofApp.cpp ):

    // wait for half a frame before forgetting something
    contourFinder.getTracker().setPersistence(15);
    // an object can move up to 32 pixels per frame
    contourFinder.getTracker().setMaximumDistance(32);

I agree that the tracker API is really a bad design (this is due to not Kyle’s but original OpenCV), and you need to get labels disappeared by tracker.getDeadLabels().

Here are the images I’m currently getting. Its taking an snapshot every frame the kinect updates. I’m recording the kinect depth data, the thresholded depth image, and the cutout color image.

just a note, @micuat the tracker is actually written from scratch, if you have API suggestions i’d be happy to hear!

@kylemcdonald sorry about that! I thought playing with getDeadLabels() is non-OOP idea and hard to manage the labels. Instead, each tracked object should be assigned with an instance with x,y and isDead members, but it seems like there is already a Follower class that takes care of each object, which I didn’t notice for a long time =P

@micuat that’s correct, there are essentially two ways to use the Tracker: basic (non-OOP) and advanced (OOP).

Btw, @zack, have you tried installing the Kinect orthogonal to the projector so that it has less effect from projection?

Hi @kylemcdonald is there a file I am missing where OOP and non-OOP are explained?
I have not grasped the difference in which these differ.
Or is OOP just the notion of using the follower class?