Hand movement tracking / hand gestures

Dear artists and coders,

I’m looking for a way to track hand movements. I will use this to program a musical instrument where pitch and volume are controlled by moving the hands in the air.

I have a Kinect model 1414 and I thought the best way to track the position of the hands is to use ofxOpenNI. However, the project seems abandoned for years and I’m not sure if this is the way to go in 2017/2018. Another project which seems interesting is openpose (https://github.com/CMU-Perceptual-Computing-Lab/openpose). However, it comes with some disadvantages. You need a Nvidia graphics card, it does not run on macOS and there is no ofxOpenPose addon (yet).

Does anyone have a suggestion? Or tips how to get ofxOpenNI working in of 0.9.8 on macOS? Is there another project which does hand position tracking by normal webcam or kinect?

Many thanks in advance,
Laurent

Hi,
ofxOpenNi could work if you get to install all the right libs and make some changes here and there.
OpenPose is great but you will need a beast of computer to run in something close to realtime. Why do you say it doesn’t work on macOS? afaik it does.

I’d suggest you to use a kinect v2 and the microsoft SDK and use windows. You can run the kinect on a different computer and then send the hand position values through OSC to another computer using macOS if you dont feel comfortable with windows. Microsoft’s sdk works so much better than openNI and has some really handy stuff.
hope this helps.
best

1 Like

Hello Roy,

Many thanks for the kind answer. I will try again to get ofxOpenNI to work on my Mac. It is good to know that it should work. At the moment I don’t have a second computer at hand where I can install windows on. If I give up on ofxOpenNI, I might go for your second suggestion and buy a cheap Windows machine.

Laurent

Is using the Leap Motion an option?

HI, well you can install windows on a mac using bootcamp and just make everything run on windows, thus avoiding the second computer.
cheers

Be careful about the computer you choose, the Kinect V2 is very picky when it comes to the USB3 interface it needs, in my experience it is not so easy to find one that works. This is especially true with laptops, but desktop PCs may also need an extra PCI Express USB3 card with an Intel or Renesans based controller.

If you can try it before you buy it :slight_smile:

Hi Roy,
@roymacdonald

I am working on a hand tracking project with Kinect and just saw this post! And wonder maybe you can give some advice!

I tried with contourFinder and ofMesh first, but the image can’t project on hand palm accurately since I can only find and use the centroid of the contour.
(The idea of the work is to set up the kinect and projector on ceiling and facing the floor, to track people’s hands and project some 3D mesh-like model on people’s hands.)
Here are some hint for the visual (https://www.instagram.com/p/Bf4G6IRnJZ5/?taken-by=amozeng)

Then I saw your post and think kinect v2 and microsoft SDK on Windows is worth to try!

Since I don’t have much experience with Windows, I am wondering, do you recommend to

  • use ofxKinectWindows2 on Visual Studio (but it seems like does not have the hand tracking function)
  • just play with microsoft SDK directly, if so, do you have any recommend tutorial?

Also I have a question, which is that, I was using ofMesh to project the 3d Model on hand, which is a function only in openFrameworks. If I just mess with microsoft SDK, is there any similar function as ofMesh I can use for my project?

Thanks so much!
Amo

Hi!
Have you tried to calibrate first the kinect and the projector?


This will help a lot!

Check at ofxaddons.com for other kinect addons. there are a lot.
If you dont have experience with Windows I wouldnt recommend to go and play directly with the MS SDK.

cheers

Hi!

Thanks for the quick reply!
ofxKinectProjectorTookitV2 looks like a great tool to project image on bodies, thanks for recommending this to me!

I looked all over the addons in the link. However, there is only one related to “hand tracking” – ofxHandTracker (https://github.com/blazm/ofxHandTracker) It requires ofxOpenNI, is ofxOpenNi only work with Kinect V1?

I also found another addon ofxHandDetect (https://github.com/jonasfehr/ofxHandDetect), which works with Kinect V2. I tried with Kinect V2, but never able to make it work. It requires ofxDepthCamera, ofxMultiKinectV2 and other addons. I installed all of them and tried but stuck with a bitwise_and function, which seems like a openCV problem…

Do you have any suggestion to achieve this project?
(project image specifically on hand palms, exclude other part of body )

Thank you so much!!

Best,
Amo

Hey @amoz
ofxHandDetect seems to work by using some cv stuff to filter and threshold, and seems that it would work on a setting where you put your arms over a table. I just speculating from a quick overview of the code. I think that ofxHandTracker simply uses openNI skeleton and simply tracks the hand of that skeleton, which woudn’t be useful on your situation.
I’m not sure if MS SDK’s hand tracking would work if you place the kinect above looking downwards.
If you install MS SDK you’ll get an app that is called “kinect viewer” (if I am not wrong) which will show you if it is tracking hands. Try it and see what you get. if it works try to get that data into OF, otherwise you’ll need to look for a different solution.
cheers

1 Like

Here is good information about hand movement tracking or hand gestures. I really love the approach and it will be sound good for me.

Thanks for the reply! Getting Data from kinect viewer to OF is a great idea! But I have zero coding experience except OF and P5, it’s really challenging to figure it out.

But I’ve been keeping thinking about the solution how to track hand location and project image from ceiling to ppl’s hands.

And, after having conversation with a friend, the easiest solution might be:
PC + OF + ofxKinectForWindows2+ KinectV2

Since ofxKinectForWindows2 has skeletonTracking and can get axis of Joint HandLeft & HandRight, and install the Kinect V2 on the wall pointing to people, install projector on ceiling and pointing down to floor
And we can get data from OF directly.

Hope this can help ppl who want to something similar!