best way to contourfind coloured balls

Hi,

I’m in the process of creating an installation the motion and position of various colored balls (you know the kind you get in ball ponds) are rolled around by participants and the computer tracks position, speed and collisions to determine sounds.

In the past I’ve used hsv color tracking for such things but I’m finding that spherical shapes are quite tricky as the shading changes across the shape. I’ve been poking around for a better technique but I wonder if anyone has done this before and can point me in the right direction.

Things I have tried so far…

  • hsv color tracking
  • background differencing combined with color tracking (the best so far)
  • hough circle detection - (didn’t work due to the shading problems)
  • camShift (i’m still trying to understand this but the results didn’t seem that much better than simple color tracking anyway).
  • template matching (produced pretty erratic results)

thanks

Simon

template matching is a demanding beast: it needs that the object you’re looking for is exactly the same size and orientation of the template; so it’s very cool if you’re looking for something like an icon on your desktop, but it’s too fragile for most real world / real time scenarios.

imho background subtraction + color tracking can be a robust solution and there are a few tricks you could use to improve your result:

  • using a color threshold when looking for a color: ie if you’re looking for a value of 120, keep all the values inside 120-th<120<120+th where th is a threshold value you can calibrate;

  • cvAdaptiveThreshold(): the adaptive threshold is pretty good at adapting to changing lighting conditions and that’s exactly what happens to colored balls;

  • histogram comparison: also this technique is pretty good at finding balls and/or objects withlight variations; I successfully used it to find a certain feature inside a pool of candidates;

  • otsu thresold: it’s a kind of adaptive threshold that uses histograms.

Thanks for the advice,

I’d already done the kind of thresholding you described and it didn’t solve my shadow problems so I had a fiddle with adaptive thresholding and histograms.

This is my plan.

  1. use Open Cv to calculate hue and saturation histograms for my selected colored balls
  2. do a backproject on extracted hue and sat images
  3. the adaptive thresholding on resultant image/images ?
  4. contourfind on that

I’ll use bg differencing to prevent false objects from appearing and also to detect obscuring objects such as hands etc…

I’m wondering if there is a way to do this without using separate images for Hue & Sat … I note that openCv can create 2D hue sat histograms.

http://www.seas.upenn.edu/~bensapp/open-…-ref-cv.htm

(see under CalcHist)

but there’s no example of using an HSV image to do a back project from that ?
Just using hue as in an example of camShift that I have produce less reliable results.

How did your plan work out? I’m trying to use camshift to track colored markers - and am doing so in a similar way to you. Please let me know if you have any insights.

Thanks!

It worked great thanks.

I found the best way was to use a meanShift algorithm with an HSV backproject for each colour. I used Bckgrnd Averaging to segment (a little more sophisticated than absDiff as it has higher and lower thresholds which I managed to use to cut out shadows) and 2D HS histograms with some some thresholding on the V channel.

You can accommodate for various light conditions and the variable shade of the balls by doing several back projects for each colour ball. Even with 4 colour balls and 5 or 6 backprojects for each ball, I found my fps stayed around 30.

I’ve done this installation in several indoor and outdoor locations and it seems pretty robust.

I’ll post some footage soon.

S

It worked great thanks.

I found the best way was to use a meanShift algorithm with an HSV backproject for each colour. I used Bckgrnd Averaging to segment (a little more sophisticated than absDiff as it has higher and lower thresholds which I managed to use to cut out shadows) and 2D HS histograms with some some thresholding on the V channel.

You can accommodate for various light conditions and the variable shade of the balls by doing several back projects for each colour ball. Even with 4 colour balls and 5 or 6 backprojects for each ball, I found my fps stayed around 30.

I’ve done this installation in several indoor and outdoor locations and it seems pretty robust.

I’ll post some footage soon.

S

That helps a lot although how to implement this in OF is a bit fuzzy, hard to find examples.

How do you calculate the histogram for your back projections? Do you have a calibration routine in your software - or do you have existing histograms? In different lighting conditions my software has different HSV properties, so I recalibrate every 2 seconds based on a fiducial, but this is not optimal.

The different back projects for each ball makes sense - but once you get the backprojection patches do you add them into a single mask and do a contour find on that mask? I have five colors - I’m looking for on different parts of the HSV pie…

For the background - I have been using cvCreateGaussianBGModel - with a window of 4000 but the problem is that if a person stays in the foreground, eventually they become part of the model, because of the naive pixel averaging. But I assume this is what you mean by background averaging?

If you have an portion of what you do in the opencv, I’d love to see it so that I can get this right. My objects are moving towards and away from the camera, so I’m going to use camshift, I’m assuming in your installation you have a camera above the balls so they stay relatively the same size. thats smart. .

That sounds a quite complicated solution. I took my histogram samples from captured images through the webcam. Once I’d done a few and added the different back project masks together I found that was plenty for a mean shift algorithm. I think I used cvAdd but would need to check.

re: masks . yes contour find on the combined mask for each colour and use that to set the algorithm.

Actually the area I was tracking was too large for overhead shots so I manually aligned the corners of the square I wanted to track with a rotated and transformed rectangle and use glUnproject to get the 3D coordinates. I found that despite the change in size to perspective mean shift was more reliable than camshift as where two balls were together the camshift algorithm with get confused. I had to write a fair amount of code to prevent double and false allocations of meanshift windows to preexisting contours.

I think I used something simpler than CvCreateGaussianBGModel. It’s a method that takes the average change in each channel over a number of frames (I used 20). The method is in this book

http://oreilly.com/catalog/9780596516130 - which is quite good

Not sure if all this is the best way but it worked for me.

I’ve just switched from linux to mac and will be porting this project in the nex few weeks. I can send you some screen grabs then which will make it clearer. The code is pretty extensive so I’ve just posted the most relevant bit here. hope it’s helpful in anyway

s

  
  
capture::capture()  
{  
  
    width = 320;  
    height = 240;  
    learningBg = 30;  
  
  
    vidGrabber.setVerbose(true);  
    vidGrabber.setDeviceID(0);  
    vidGrabber.initGrabber(width,height);  
  
    vidImg.allocate(width,height);  
    selectImg.allocate(width,height);  
    zoomSelectImg.allocate(width,height);  
  
    bgSeg.AllocateImages(cvSize(width, height));  
    bgSeg.dilate = false;  
  
    for(int col =0; col < 4; col ++){  
  
        colChannels[col].allocateImages(cvSize(width,height));  
        colChannels[col].colIndex = col;  
        colChannels[col].erode = 0;  
        colChannels[col].enabled = true;  
  
    }  
  
    vidImage = cvCreateImage( cvGetSize(vidImg.getCvImage()), 8, 3 );  
	vidImage->origin = vidImg.getCvImage()->origin;  
  
	vidhsv = cvCreateImage( cvGetSize(vidImg.getCvImage()), 8, 3 );  
	segImg = cvCreateImage( cvGetSize(selectImg.getCvImage()), 8, 3 );  
	selImage = cvCreateImage( cvGetSize(selectImg.getCvImage()), 8, 3 );  
  
    masker.setAsRectangle(cvRect(20,20,280,200));  
    maskedImg = cvCreateImage( cvGetSize(vidImg.getCvImage()), 8, 3 );  
  
   	minBlob = 40;  
	maxBlob = 1000;  
    db_thresh = 10;  
  
	selection = cvRect(100,100,10,10);  
  
	selectChanged = true;  
	learnBg = true;  
	learnImage = true;  
	zoomSel = false;  
  
	selectedColour = 0;  
	selectedView = 0;  
	zoom = 5;  
	zx = 0;  
	zy = 0;  
	sample_view = 0;  
  
  
  
}  
  
void capture::createZimg(){  
  
if(!zoomSel){  
    selectImg.setROI(zx,zy,320/zoom,240/zoom);  
    zoomSelectImg.clear();  
    zoomSelectImg.allocate(320/zoom, 240/zoom);  
    zoomSelectImg.setFromPixels(selectImg.getRoiPixels(),320/zoom,240/zoom);  
    zoomSelectImg.resize(320,240);  
    selectImg.resetROI();  
}else{  
zoomSelectImg = selectImg;  
}  
}  
  
  
  
//--------------------------------------------------------------  
bool capture::update(trackingObject t_objs[][10] , int size){  
  
ofBackground(100,100,100);  
  
    vidGrabber.grabFrame();  
    bNewFrame = vidGrabber.isFrameNew();  
  
        if(learnImage){  
            if(bNewFrame){  
            selectImg.setFromPixels(vidGrabber.getPixels(), 320,240);  
            //cvEqualizeHist(selectImg, selectImg);  
            createZimg();  
            learnImage = false;  
            }  
  
        }  
  
        cvCopy(zoomSelectImg.getCvImage(), selImage);  
        if(newZbp)colChannels[selectedColour].recalcSampleImages(selImage);  
        if(selectChanged)colChannels[selectedColour].setHistFromSample(selImage, selection);  
  
        selectChanged = false;  
        newZbp = false;  
  
  
        if(bNewFrame){  
  
        if(learningBg > 0)learningBg-=1;  
  
        vidImg.setFromPixels(vidGrabber.getPixels(), 320,240);  
        maskedImg = masker.getMask(vidImg.getCvImage());  
  
        if(learnBg){  
        bgSeg.learnNewBackground();  
        learnBg = false;  
        learningBg = 40;  
        }  
  
        cvCvtColor(maskedImg, vidhsv, CV_BGR2HSV );  
        segImg = bgSeg.segment(vidhsv);  
  
        //for each colour  
  
            for(int colour = 0; colour < 4; colour ++)  
            {  
  
                //do the backproject  
  
                colChannels[colour].backProject(segImg);  
  
                //meanshift any exisiting blobs  
                for(int blob = 0; blob < 10; blob++){  
  
                        if(t_objs[colour][blob].assigned){  
  
                             cvMeanShift( colChannels[colour].get_vid_bp(), t_objs[colour][blob].bounding_cv,  
                cvTermCriteria( CV_TERMCRIT_EPS | CV_TERMCRIT_ITER, 10, 1 ),  
                    &t_objs[colour][blob].cont_cv);  
  
                        //check for db_allcts in camshifts  
  
                        float distance = 100;  
  
                        ofxVec2f ct_point;  
  
                        ct_point.set(t_objs[colour][blob].cont_cv.rect.x +  
                                     t_objs[colour][blob].cont_cv.rect.width/2,  
                                     t_objs[colour][blob].cont_cv.rect.y +  
                                     t_objs[colour][blob].cont_cv.rect.height/2);  
  
                        for(int sb = 0; sb < 10; sb++){  
  
                            if(t_objs[colour][sb].assigned){  
                                if(sb != blob){  
                                ofPoint sb_point;  
                                sb_point.set(t_objs[colour][sb].bounding_cv.x +  
                                         t_objs[colour][sb].bounding_cv.width/2 ,  
                                         t_objs[colour][sb].bounding_cv.y +  
                                         t_objs[colour][sb].bounding_cv.height/2);  
  
                                distance = min(distance, ct_point.distance(sb_point));  
                                }  
                            }  
  
                        }  
  
                        if(distance > db_thresh && t_objs[colour][blob].cont_cv.area > 0){  
                        t_objs[colour][blob].bounding_cv = t_objs[colour][blob].cont_cv.rect;  
                        t_objs[colour][blob].ghostReset();  
  
                        }else if(distance < 1){  
                        t_objs[colour][blob].cont_cv.area = 0;  
                        }  
                    }  
  
                }  
  
                //delete any lost blobs;  
                for(int sb = 0; sb < 10; sb++){  
  
                    if(t_objs[colour][sb].cont_cv.area == 0 && t_objs[colour][sb].ghost > 5)  
                    {  
                        t_objs[colour][sb].assigned = false;  
                        t_objs[colour][sb].present = false;  
                        t_objs[colour][sb].reset();  
  
                    }else if(t_objs[colour][sb].assigned && t_objs[colour][sb].cont_cv.area == 0){  
  
                        t_objs[colour][sb].ghostIncrement();  
                       // printf("%i \n",t_objs[colour][sb].ghost);  
  
                    }  
  
                }  
  
                //search for new blobs  
                contFinder[colour].findContours(  
                                          colChannels[colour].get_vid_bp(),  
                                           minBlob,  
                                           maxBlob,  
                                            10,  
                                            false,  
                                            0);  
  
                //check that there's no double allocation  
                for(int cb = 0; cb < contFinder[colour].myblobs.size(); cb++)  
                {  
  
                    bool db_allct = false;  
                    ofxVec2f cb_point;  
                    cb_point.set(contFinder[colour].myblobs[cb].centroid);  
  
                    if(((cb_point.x < 3) || (cb_point.x > width -3)) ||  
                       ((cb_point.y < 3) || (cb_point.x > height -3)) )db_allct = true;  
  
                    for(int sb = 0; sb < 10; sb++){  
  
                        if(t_objs[colour][sb].assigned){  
                            ofxMyVec2f sb_point;  
                            sb_point.set(t_objs[colour][sb].bounding_cv.x +  
                                     t_objs[colour][sb].bounding_cv.width/2 ,  
                                     t_objs[colour][sb].bounding_cv.y +  
                                     t_objs[colour][sb].bounding_cv.height/2);  
  
                            if(cb_point.distance(sb_point) < db_thresh*2)db_allct = true;  
  
                            if(sb_point.pointPolyTest(&contFinder[colour].myblobs[cb].pts) == 1)db_allct = true;  
                        }  
                    }  
  
                    //assign new blobs  
                    if(!db_allct){  
  
                    vector<distListItem> distList;  
  
                    //first assign for the nearest ghost  
                        for(int blob = 0; blob < 10; blob++){  
                            if(t_objs[colour][blob].assigned && t_objs[colour][blob].ghost > 0){  
                                distListItem temp;  
                                ofxVec2f cp;  
                                cp.set(contFinder[colour].myblobs[cb].centroid);  
  
                                temp.dist = cp.distance(t_objs[colour][blob].centroid);  
                                temp.fp = blob;  
                                temp.np = cb;  
  
                                distList.push_back(temp);  
                            }  
                        }  
  
                     if(distList.size() > 1)sort(distList.begin(),distList.end(), comparebyDistance);  
  
                     if(distList.size() > 0){  
                        t_objs[colour][distList[0].fp].bounding_cv = contFinder[colour].myblobs[cb].bounding_cv;  
                        t_objs[colour][distList[0].fp].assigned = true;  
                        t_objs[colour][distList[0].fp].ghostReset();  
  
                     }else{  
  
                        //just assign to the first empty slot  
  
                        for(int blob = 0; blob < 10; blob++){  
                            if(!t_objs[colour][blob].assigned){  
                            t_objs[colour][blob].bounding_cv = contFinder[colour].myblobs[cb].bounding_cv;  
                            t_objs[colour][blob].assigned = true;  
                            break;  
                            }  
                        }  
  
                     }  
  
                    }  
  
                    //set the centroids  
                    for(int sb = 0; sb < 10; sb++){  
                        if(t_objs[colour][sb].assigned){  
                        t_objs[colour][sb].centroid.set(t_objs[colour][sb].bounding_cv.x +  
                                 t_objs[colour][sb].bounding_cv.width/2 ,  
                                 t_objs[colour][sb].bounding_cv.y +  
                                 t_objs[colour][sb].bounding_cv.height/2);  
                        }  
  
                    }  
  
                }  
  
  
            }  
  
        }  
  
    if(learningBg > 0)bNewFrame = false;  
    return bNewFrame;  
  
}