Universal Multitouch Wall using Microsoft SDK, openFrameworks and ofxKinectNui

As suggested by roymacdonald, this is the collected info in this thread about my method used to turn Kinect depth information into useful blobs that can be turned into a multitouch interface. The original comment can be found at the end of this one. Also, any personal comment I make about this method may be biased since I’ve had little experience with other people’s methods. Part of the info has also been edited since it was reviewed for errors.
This is also the first time I share code that seems useful so I went out of my way to do it, many thanks to patriciogonzalezvivo, roymacdonald, lahiru and of course kylemcdonald!

Info about the method
This method seems (?) to be simplest and also might be the shortest while still being precise. It also doesn’t use auto calibration and expects the user to determine the projection area (which might explain how short it is). This might actually help the method and make it useful on irregular surfaces. We also handled the depth images as if it were 8 bpc when it actually is 11 bpc, this might truncate data but still works very nicely. For those who don’t know, 8 bpc would give me a 255 grey palette (2^8) and a 11 bpc would a 2048 (2^11), and here the grey value defines the depth of floating point that the kinect is interpreting, so more bpc equates to more depth resolution.
We used Microsofts Kinect for Windows SDK 1.0, openFrameworks 007 with ofxOpenCv and ofxKinectNui ( https://github.com/sadmb/ofxKinectNui ) but I’m guessing ofxOpenNI should work just the same while being multi-platform, the important thing is that to start off you need to somehow get the kinect depth image.
This method was made in order to be used on a multitouch wall so the asumption is that users extends their hands, so the position of touch is expected to be the tip of the fingers which would be on one side of the bounding box of the blob.

Remembering that neither the projector nor the kinect have to be in a certain position (except of course being able to see the same surface) the process is the following:

[tt]1-Capture the depth image and put it in a ofxCvGrayscaleImage that we will call tempGreyImage.
2-Remember to copy tempGreyImage into GreyImagediff for difference when user presses space.
3-Take both the tempGreyImage and GreyImagediff into two greyscale ofPixels.
4-Fill another greyscale ofPixels with the result of the following pseudocode cicle:
for each pixel{
Calculate the difference between tempGreyImage pixels and GreyImagediff pixels.
if said difference is lower than threshold A and higher than threshold B set value to 0;
else set value to 1000 (any value over max pixel value will be ok, it will be clamped anyways);
5-Take the resulting ofPixels back to another ofxCvGrayscaleImage called FinalGreyDiff.
6-NOW warpIntoMe() to FinalGreyDiff and do openCV’s .threshold() and .blur().
7-Do not use centroids as source of touch data. Instead use the coordenates of the sides of their bounding boxes.[/tt]

Since this code was be used with ofxKinectNui in Windows I’d better just post the relevant portion of code so you can try it out on ofxOpenNI if you like.
“destino_camera_warp[]” are 4 ofPoints that are set by mouse or take from an XML, they define how the image will be warped prior to blob analysis, for that it also needs “entrada[]” which are the 4 ofPoints corners for the image.

In the testApp.h file:

ofPoint                destino_camera_warp[4];  
ofPoint                entrada[4];  
ofImage                colorImg;  
ofxCvGrayscaleImage    tempGrayImage;  
ofPixels               monoPixels;  
ofPixels               monoPixels_diff;  
ofPixels               grayPixels;  
ofxCvGrayscaleImage    paso_openCV;  
ofxCvGrayscaleImage    grayImage;  
ofxCvContourFinder     encuentracontornos;  

In the testApp.cpp file:

void testApp::setup() {  
  // ...  
  // Relevant Code  
  colorImg.allocate(320, 240, OF_IMAGE_COLOR);  

void testApp::update() {  
  // ...  
  // Relevant Code  
    monoPixels.setFromPixels(tempGrayImage.getPixels(), 320,240, 1);  
    if (bLearnBakground == true){  
        monoPixels_diff.setFromPixels(tempGrayImage.getPixels(), 320,240, 1);  
        bLearnBakground = false;  
    for (int i = 0; i < 320*240; i++){  
        int valtemp = monoPixels[i] - monoPixels_diff[i];  
        if (valtemp < thresholdLow || valtemp > thresholdHigh){  
            valtemp = 0;  
        } else {  
            valtemp = 1000;  
        grayPixels[i] = (unsigned char)valtemp;  
    grayImage.warpIntoMe(paso_openCV, entrada, destino_camera_warp);  
    encuentracontornos.findContours(grayImage, min_blob_size, max_blob_size, cantidad_blobs, encuentra_hoyos_en_blobs);  

And that is basically it, what you want to do is to NOT use the centroid of a blob but rather the bounding box like this (this is for only one blob).

float valX = encuentracontornos.blobs[0].centroid.x -   
float valY = encuentracontornos.blobs[0].centroid.y - encuentracontornos.blobs[0].boundingRect.height/2;  

Here is an example of why we used the bounding box:

The big modification in the original code replaces openCV’s absDiff() while at the same time doing the depth clipping.

The method posted here of course very similar to openFrameworks’s opencvExample (that is the basis of ours), but that method seemed to exagerated noise so we got more false blobs. For a better explanation of the different results between both methods go here:

That is basically it, I believe I have not skipped any relevant information. =D

Original Comment

Hey Irregular! Seams very precise! And the graphics are wonderful! congratulations!

I´m working on a similar project for a museum. Fortunately I can publish the code:

Here is the addon: https://github.com/patriciogonzalezvivo/ofxInteractiveSurface
It have an autoCalibration class for making the homografy for the mapping.

Here are some videos:

More information of this project can be found here: http://www.patriciogonzalezvivo.com/blog/?p=601

Amazing work guys… Patricio, i am downloding your code to start to play with it

Any improve it´s really welcome!

Estan increibles Patricio! =D
Sorry… the videos look awesome!

Your code is definitely more sofisticated, as always I’m very impressed. Also being open sourced is always better! my code is really simple so I might release it in the future when everything settles down.

Edit> The hand and fingers stuff in the first video is awesome and fast! Is is integrated to your addon?
Wow its very precise!

Keep on the south american power! :stuck_out_tongue_winking_eye:

As always pato has some very sophisticated and impressive work.

@irregular, why cant you share code?

I implemented I similar thing some time ago. I based it on a thing that a friend of mine did called simpleKinectTouch. I implemented it as an addon and worked really well. Stupidly I forgot to publish it cause it was part of a bigger project that I had to leave appart because of budget issues.
I’ll push it to my github asap.

Hi! How are you? xD
Well I had tried simpleKinectTouch and another addon too which I don’t remember the name but I thought I might make a simpler method that requires no addon.
My company is just beginning so we need to monetize a bit (sad reality). I’ll release it in the future filled with features of course!

Hey Roy! My Friend! how it´s going?

Can you share some videos?

South American Power 4 ever!!

Hey Pato!
todo bien! y tu?
I don’t have any videos;P
I have to finish a job tomorrow and I’ll make some videos then.


Hey! I just rename the addon to https://github.com/patriciogonzalezvivo/ofxInteractiveSurface
And I´m working on the documentation and the example in order to be more friendly and clear

@pato, cool!
tan insoportablemente productivo como siempre!:wink:

I’ll check it later.
Are the gestures and touches sent by somthing like TUIO or OSC?
How did you call this before? was it ofxSurface?

Un abrazo!

Hey Roy!! thanks!!
Yes, before was call ofxSurface now ofxInteractiveSurface (I don´t know if the ofxAddons.com will handle such a messy move)

This program don´t send the events away… the just give you the ofEvents. So if you want to do something on TUIO you have to implement it.

I still have to make some simple drawings to explain better witch is the diference between objects and hands.
But basically objects it´s everything touching the surface. Like objects or finger tips… and hands are every object that it´s over the surface with-out touching it.

So the ofEvent send the ofxBlob when hands or objects get added, moved or deleted. Inside the ofxBlob there are some methots that tell you if it´s a hand… if it have active fingers… how many and in what positions

; )

Any tip or comment it´s welcome.

@pato, do you need to set the projector and the kinect as close and aligned as possible? does the homography interfere with the blob tracking?

The closest and aligned they are, the easier for the KinectAutoCalibrator to make a good ofMatrix4x4 with the homografy information. Once it´s calibrated, the ofxMapViewPort remember it and the mask ( a mask made from the contour of the surface). So you can do what ever you want.

There are two blob tracking. The ofxTrackMode enum set witch of the two are going to be active and how. This is the more difficult part to explain. But basically on TRACK_BOTH, TRACK_ACTIVE_HANDS and TRACK_ACTIVE_OBJECTS the upper layer (the one of the hands) do not interfere with the lower one. In other words it doesn´t bother the occlusion that the hands may do to the lower layer ( the object image that it´s been tracked for the object blobTracker).

This is important for example when you are designing a game that uses objects… and you want to preserv the ID of his blob.

On modes TRACK_JUST_HANDS and TRACK_JUST_OBJECTS it´s not that way. It´s a simple tracking. So occlusions may happend. Ir reduce ghostly blobs. Very useful for traditional moultiTouch apps.

Thanks for sharing this and congratulations! that’s awesome!

Awesome addon Patricio! xD
You are very good at this! :smiley:

Thanks @kalwalt , so long since last time! I see that you are mastering the art of shaders and 3D! nice addons too

Also thanks @irregular! I love the tron-like graphics of your example! Looking forward going to Santiago, inviting Roy have some fun coding.

@pato, date una vuelta por acá antes de partir a NY.
un abrazo!

Very awesome addon! I can’t wait to do some fun sandbox things.

If you like them I could probably share more info on the visuals I’m working with, here is what they look like in the computer.
I’m currently not using any shader since I’ve just started to learn how to use them (trying to use no addons). The last one is the most “openGL” one with the dodecahedrons and the not fully cleared buffer. I think I can make simplers examples, I’m sure people would like to learn a bit more. =D
Everything else is mostly sprite work done as if I was doing a video game at 60fps so it ties nicely with the Universal Kinect wall we made. The Tesla Coil one even lighting jitter.