hand tracking

Hi OFpeople!

I would want use **pdp_opencv_contours_convexity **& **pix_opencv_contours_convexity ** , to make some tracking of fingers… See the visual example below of that.

pdp_opencv_contours_convexity&pix_opencv_contours_convexity
This software is for pd finds defects of convexity of biggest detected contour?

See http://hangar.org/wikis/lab/doku.php?id=start:puredata-opencv

Any addon example? if there aren’t , i could try to use the source of this opencv for pd.

by :slight_smile:

I have written some plugins for pd and a couple for GEM (for my own purposes) it would not be hard to use the source code to make an of addon. If you need help I could probably be able to help/guide you if you where to give it a go.

ding

Looking at the code, the bulk of it it here:

  
CvSeq* seqhull;  
    CvSeq* defects;  
    CvSeq* contours;  
    int* hull;  
    int hullsize;  
    CvPoint* PointArray;  
    CvConvexityDefect* defectArray;  
    CvMemStorage* stor02;  
    CvMemStorage* stor03;  
    stor02 = cvCreateMemStorage(0);  
    stor03 = cvCreateMemStorage(0);  
  
  
    cvFindContours( gray, stor02, &contours, sizeof(CvContour), CV_RETR_EXTERNAL, CV_CHAIN_APPROX_SIMPLE, cvPoint(0,0) );  
    if (contours) contours = cvApproxPoly( contours, sizeof(CvContour), stor02, CV_POLY_APPROX_DP, 3, 1 );  
    
    int i = 0;   
    int area = 0;  
    int selected = -1;  
      
    //busquem el contorn mes gran  
    CvSeq* first_contour;  
    first_contour = contours;  
    for( ; contours != 0; contours = contours->h_next )  
        {  
        CvRect rect;  
	int count = contours->total;   
	rect = cvContourBoundingRect(contours, 1);  
	if  ( (rect.width*rect.height) > area )   
		{  
		selected = i;  
		area = rect.width*rect.height;  
		}  
	i++;  
	}  
  
    contours = first_contour;  
	  
    int k = 0;  
    for( ; contours != 0; contours = contours->h_next )  
        {  
        int i;                   // Indicator of cycles.  
        int count = contours->total; // This is number point in contour  
        CvPoint center;  
        CvSize size;  
        CvRect rect;  
  
	rect = cvContourBoundingRect( contours, 1);  
	if ( (k==selected) ) {  
          
          
    //fprintf(stderr,"malloc\n");  
        // Alloc memory for contour point set.      
        PointArray = (CvPoint*)malloc( count*sizeof(CvPoint) );  
                  
        // Alloc memory for indices of convex hull vertices.  
        hull = (int*)malloc(sizeof(int)*count);  
          
        // Get contour point set.  
    //fprintf(stderr,"cvCvtSeqToArray\n");  
        cvCvtSeqToArray(contours, PointArray, CV_WHOLE_SEQ);  
          
                  
        // Find convex hull for curent contour.  
    //fprintf(stderr,"cvConvexHull\n");  
        cvConvexHull( PointArray,  
                      count,  
                      NULL,  
                      CV_COUNTER_CLOCKWISE,  
                      hull,  
                      &hullsize);  
          
        // Find convex hull for current contour.  
        // This required for cvConvexityDefects().  
    //fprintf(stderr,"cvConvexHull2\n");  
        seqhull = cvConvexHull2( contours,0,  
                             CV_COUNTER_CLOCKWISE,  
                             0);  
          
        // This required for cvConvexityDefects().  
        // Otherwise cvConvexityDefects() falled.  
        if( hullsize < 4 )  
            continue;  
           
        // Find defects of convexity of current contours.                          
    //fprintf(stderr,"cvConvexityDefects\n");  
        defects = cvConvexityDefects( contours,  
                            seqhull,  
                            stor03);  
        int j=0;  
        // This cycle marks all defects of convexity of current contours.  
        for(;defects;defects = defects->h_next)  
        {  
            int nomdef = defects->total; // defect amount  
    	    outlet_float( m_nomdef, nomdef );  
              
            if(nomdef == 0)  
                continue;  
               
            // Alloc memory for defect set.     
    //fprintf(stderr,"malloc\n");  
            defectArray = (CvConvexityDefect*)malloc(sizeof(CvConvexityDefect)*nomdef);  
              
            // Get defect set.  
    //fprintf(stderr,"cvCvtSeqToArray\n");  
            cvCvtSeqToArray(defects,defectArray, CV_WHOLE_SEQ);  
              
  
            // Draw marks for all defects.  
            for(i=0; i<nomdef; i++)  
            {  
                cvLine(rgb, *(defectArray[i].start), *(defectArray[i].depth_point),CV_RGB(0,0,255),1, CV_AA, 0 );  
                cvCircle( rgb, *(defectArray[i].depth_point), 5, CV_RGB(0,255,0), -1, 8,0);  
                cvCircle( rgb, *(defectArray[i].start), 5, CV_RGB(0,255,0), -1, 8,0);  
                cvLine(rgb, *(defectArray[i].depth_point), *(defectArray[i].end),CV_RGB(0,0,255),1, CV_AA, 0 );  
    	    t_atom rlist[7];  
            SETFLOAT(&rlist[0], i);  
            SETFLOAT(&rlist[1], defectArray[i].start->x);  
            SETFLOAT(&rlist[2], defectArray[i].start->y);  
            SETFLOAT(&rlist[3], defectArray[i].depth_point->x);  
            SETFLOAT(&rlist[4], defectArray[i].depth_point->y);  
            SETFLOAT(&rlist[5], defectArray[i].end->x);  
            SETFLOAT(&rlist[6], defectArray[i].end->y);  
    	    outlet_list( m_dataout, 0, 7, rlist );  
            }  
  
	    j++;  
               
            // Free memory.         
            free(defectArray);  
        }  
          
        // Draw current contour.  
        //cvDrawContours(x->cnt_img,contours,CV_RGB(255,255,255),CV_RGB(255,255,255),0,1, 8);  
        cvDrawContours( rgb, contours, CV_RGB(255,0,0), CV_RGB(0,255,0), 2, 2, CV_AA, cvPoint(0,0)  );  
          
        // Draw convex hull for current contour.          
        for(i=0; i<hullsize-1; i++)  
        {  
            cvLine(rgb, PointArray[hull[i]],   
                            PointArray[hull[i+1]],CV_RGB(255,255,255),1, CV_AA, 0 );  
        }  
        cvLine(rgb, PointArray[hull[hullsize-1]],  
                             PointArray[hull[0]],CV_RGB(255,255,255),1, CV_AA, 0 );  
          
            
        // Free memory.            
        free(PointArray);  
        free(hull);  
            /* replace CV_FILLED with 1 to see the outlines */  
            //cvDrawContours( x->cnt_img, contours, CV_RGB(255,0,0), CV_RGB(0,255,0), x->levels, 3, CV_AA, cvPoint(0,0)  );  
	    //cvConvexityDefects( contours, cvConvexHull2( contours, 0, CV_CLOCKWISE, 0 ), stor022 );  
        }  
	    k++;  
        }  
  
    cvReleaseMemStorage( &stor03 );  
    cvReleaseMemStorage( &stor02 );  
    //if (defects) cvClearSeq(defects);  
    //if (seqhull) cvClearSeq(seqhull);  
  
    cvCvtColor(rgb, gray, CV_RGB2GRAY);  
  
    //copy back the processed frame to image  
    memcpy( image.data, gray->imageData, image.xsize*image.ysize );  
}  

The SETFLOAT stuff is just outputting the data thru a pd outlet. It is pretty well documented. Always a plus.

[edit] this is for a grayscale input so you would have to use/convert to ofCvGrayscaleImage.

ding

nice ! I’m going to try. thanks

Forgot to mention that you are also going to need these vars:

  
  
	/////////  
	// IplImage needed  
    	IplImage 	*rgb, *orig, *gray;  

ding

Hi ,

Well , jeje some time later… during the SummerLab Gijón [August08] I was doing some tests and I got an demo aplication running. This is the source code:cvConvexity-class , TestApp

Now the results are the middle points between fingers, see this picture.

But there are a memory bug that I do not resolved. A constant raise of memory of stor03 and stor02, vars used by Opencv to store the main info. For resolve this I think is necessary realase the memory like in the example, but I can not find where apply it for this Addon, because for Draw the points I’ve prefered make a separated function.

Those are the realase lines that It-s necessary apply to avoid increase the memory each new frame.

  
    cvReleaseMemStorage( &stor03 );  
    cvReleaseMemStorage( &stor02 );  

Well , If anyone wants to help …

Hi ,

I’ve fixed the memory bug.

There was one var that i was forgotten to clear (seqhull), and I’ve read a littel of how use opencv memory storage and I’ve discovered how to empty a memory storage without de-allocate it: cvClearMemStorage(). :wink:

  
cvClearMemStorage( stor03 );  
    cvClearMemStorage( stor02 );  
    if (seqhull)cvClearSeq(seqhull);  

Opencv-Addon-class&TestApp

The points have been drawn are the convex points, but more
interest is get fingers points…

hi,

i get this example of the convex hull working, but i cant figure how to find the tips of the fingers instead of the holes :?

Thnak you

[quote author=“charli_e”]Hi ,

I’ve fixed the memory bug.

There was one var that i was forgotten to clear (seqhull), and I’ve read a littel of how use opencv memory storage and I’ve discovered how to empty a memory storage without de-allocate it: cvClearMemStorage(). :wink:

  
cvClearMemStorage( stor03 );  
    cvClearMemStorage( stor02 );  
    if (seqhull)cvClearSeq(seqhull);  

Opencv-Addon-class&TestApp

The points have been drawn are the convex points, but more
interest is get fingers points…[/quote]

I’ve managed to get your example to work with code::blocks. The testApp.h was missing and you have to modify some other files:
opencvConvexHull

Hi ,
We should explore how to get this points looking into the defect structure, that mainly is composed by three points, “the hole” and 2 other points more that are at left and right of the hole, as defined in the next Opencv explantion.

  
  
CvConvexityDefect  
Structure describing a single contour convexity detect  
  
typedef struct CvConvexityDefect  
{  
    CvPoint* start; /* point of the contour where the defect begins */  
    CvPoint* end; /* point of the contour where the defect ends */  
    CvPoint* depth_point; /* the farthest from the convex hull point within the defect */  
    float depth; /* distance between the farthest point and the convex hull */  
} CvConvexityDefect;  
  


Convexity defects of hand contour [pure data finger detection example]

pd: Thanks for remember and upload this src.

I think have the previous post working but do not get the image that carles shows in the very last post. I never get hayspuntos >0. What settings did you use?

Hey,

I can not figure now what presets I got, but the most important is localize a clear polygonal structure in the image.

I can recommend you check gamuza software that use a GUI for easy object detection by colors and thersholds and finger detection also.

Finger detection commonly is unstable, so try to use it for a simple interaction, or try to improve it :slight_smile: with a multitracking for example.

I’ve uploaded the convexhull addon again.

Hope that helps… next step would be extract that points, store them and search how to find finguers

Cheers

ConvexHullAddonExample.zip

hello Charli_e:

Great job! I´m using your addon with ofxKinect and works excellent. I want to catch if the hand it´s open or close and where are the points of the fingers.

For the moment I just get the depressions between the fingers and I notice it´s too sensitive… it´s possible to make threshold of the level of convect accidents?

How can I get the top of the fingers? I´m ready to work on it… so if you need a second hand on this one count on me! :wink:

Thanks for the work

Patricio

Hello

I found this video at youtube ( http://www.youtube.com/watch?v=lCuItHQEgEQ&feature=related )and decide to mix the code with yours

This is the result:

  
   
  
vector<ofPoint> testApp::getFingerTips( ofxCvGrayscaleImage input) {  
	CvMemStorage*	storage = cvCreateMemStorage(0);  
	CvSeq*			contours;  
	CvPoint*		PointArray;  
	  
	int*			hull;  
    int				hullsize;  
	  
	vector<ofPoint> fingerTips;  
	  
    //START TO FIND THE HULL POINTS  
    cvFindContours( input.getCvImage(), storage, &contours, sizeof(CvContour), CV_RETR_EXTERNAL, CV_CHAIN_APPROX_SIMPLE, cvPoint(0,0));  
	  
	// If there is a contour it´ll make it more simple  
    if (contours)  
		contours = cvApproxPoly(contours, sizeof(CvContour), storage, CV_POLY_APPROX_DP, 20, 1 );  
	  
	  
    int i = 0;  
    int area = 0;  
    int selected = -1;  
	  
    CvSeq* first_contour = contours; // Remember the first contour address  
    for( ; contours != 0; contours = contours->h_next ){	// Search for the bigger countour  
		CvRect rect;  
		int count = contours->total;  
		rect = cvContourBoundingRect(contours, 1);  
		if  ( (rect.width*rect.height) > area ){  
			selected = i;  
			area = rect.width*rect.height;  
		}  
		i++;  
	}  
	  
    contours = first_contour;		// Go again to the first contour  
    int k = 0;   
    for( ; contours != 0; contours = contours->h_next ){  
        int i; // Indicator of cycles.  
        int count = contours->total; // This is number point in contour  
        CvPoint center;  
        CvSize size;  
        CvRect rect;  
		  
        rect = cvContourBoundingRect( contours, 1);  
		  
        if ( (k==selected) ){		// Analize the bigger contour  
			palmCenter.x = rect.x + rect.width/2;  
			palmCenter.y = rect.y + rect.height/2;  
			  
			fingerTips.clear();  
			  
            PointArray = (CvPoint*)malloc( count*sizeof(CvPoint) ); // Alloc memory for contour point set.  
            hull = (int*)malloc(sizeof(int)*count);	// Alloc memory for indices of convex hull vertices.  
			  
            cvCvtSeqToArray(contours, PointArray, CV_WHOLE_SEQ); // Get contour point set.  
			  
            // Find convex hull for curent contour.  
            cvConvexHull( PointArray,  
						count,  
						 NULL,  
						 CV_COUNTER_CLOCKWISE,  
						 hull,  
						 &hullsize);  
			  
			int upper = 640, lower = 0;  
			for	(int j=0; j<hullsize; j++) {  
				int idx = hull[j]; // corner index  
				if (PointArray[idx].y < upper) upper = PointArray[idx].y;  
				if (PointArray[idx].y > lower) lower = PointArray[idx].y;  
			}  
			float cutoff = lower - (lower - upper) * 0.1f;  
			  
			// find interior angles of hull corners  
			for (int j=0; j<hullsize; j++) {  
				int idx = hull[j]; // corner index  
				int pdx = idx == 0 ? count - 1 : idx - 1; //  predecessor of idx  
				int sdx = idx == count - 1 ? 0 : idx + 1; // successor of idx  
				  
				cv::Point v1 = cv::Point(PointArray[sdx].x - PointArray[idx].x, PointArray[sdx].y - PointArray[idx].y);  
				cv::Point v2 = cv::Point(PointArray[pdx].x - PointArray[idx].x, PointArray[pdx].y - PointArray[idx].y);  
				  
				float angle = acos( (v1.x*v2.x + v1.y*v2.y) / (norm(v1) * norm(v2)) );  
				  
				// low interior angle + within upper 90% of region -> we got a finger  
				if (angle < 1 && PointArray[idx].y < cutoff) {  
					int u = PointArray[idx].x;  
					int v = PointArray[idx].y;  
					  
					fingerTips.push_back(ofPoint(u,v));  
				}  
			}  
			  
            // Free memory.  
            free(PointArray);  
            free(hull);  
			  
        }  
	    k++;  
    }  
	  
	  
    cvClearMemStorage( storage );  
   // if (seqhull)  
	//	cvClearSeq(seqhull);  
	  
	  
	return fingerTips;  
}  
  

For drawing the lines and circules i´m using:

  
  
        ofPushMatrix();  
	ofTranslate(10, 320, 0);  
	ofScale(0.6, 0.6, 0);  
	for(int i = 0; i < fingers.size(); i++){  
		ofSetColor(255,0,0);  
		ofSetLineWidth(3);  
		ofCircle(fingers[i].x, fingers[i].y,5);  
		ofLine(palmCenter.x, palmCenter.y, fingers[i].x, fingers[i].y);  
		ofSetLineWidth(0);  
	}  
	ofPopMatrix();  
  

It´s quite stable and accurate. But I´m not sure if it´s enough efficient. I want to use this app

  
[http://www.patriciogonzalezvivo.com/blog/?p=289](http://www.patriciogonzalezvivo.com/blog/?p=289)   

for knowing the gestures of the hands in order to send the position through TUIO or not. Pointing hand sending TUIO… open hand… or complety close not… something like that. Please if you notice a memory leak tellme.
Thanks!

I tried your code as listed with minor fixes to get through compile but the result wasn’t as good as the YouTube video.
In fact it detected number of fingers incorrectly … don’t know what I did incorrectly.
What’s the reason for choosing an angle that is <1?
Thanks.

hi Patricio ! Great Job Man!

Following this kind of concept, If you need to detect hands with or without fingers tips open, you can apply this or similar algorithm to the hall body contour, and extract where should be the hands.
Here there is an example: http://forum.openframeworks.cc/t/hand-and-fingers-detection/1916/0
It work pretty nice but has some issues with the hands up…

But … kinect skeleton also give hand information, isn’t it? Just need have the hall body to learn the skeleton…

Also If your work research can go far away take in consider this other way because seems to use more 3d information useful for this kind of purposes.
https://github.com/pkmital/KinectHandAnalysis form http://pkmital.com

As you can see there are many ways to get the fingers tips of a contour hand.
If I can help you in that order tell me

Cheers!

Thanks Charli_e. You are right there are millons of ways.
That the best about code!

Here I have one example very similar of what you said. Using the skeleton information to take the center of the hands and only take the pixels of them… and then do a traditional finger tracking using convex hull. It´s not so state forward but works

http://vimeo.com/23654510

thank you for sharing the code. It works quite well.
by the way, how can I detect fingertips of two hands at the same time?

Thank you for sharing the code. This works really well with the kinect. There is a little trouble when the palm is not facing the camera. I doubled the size of the hand and then eroded a bit and it works much better. Any other tips?