Best way to track multiple colored gloves(currently two)

I have used the example-contours-color example and the tracking example to write a code which finds the contour for blue and red colors. Then, I find the largest area contour for each color and use the center of that contour as my reference point for the glove. I want to ask whether this is a good approach or there is a different more stable approach.
I am currently getting a lot of contours for say red colour but I want a single contour for the whole glove so that the detection is more reliable. I am reducing that to one by taking the largest area contour so that is where I need help. Is there any method to limit the contour detection to the most significant one.