ofxOpenNI + skeleton Tracking


I’ve been using openFrameworks for some time now, and recently downloaded ofxOpenNI addon from gameoverhack (https://github.com/gameoverhack/ofxOpenNI) to play around with skeleton tracking. Basically, I wanted to do something like this: http://vimeo.com/20036490

However, when I run the sample program that comes with the addon, I can only do hand tracking. When I press ‘t’ to do skeleton tracking, I just get a black screen, with only the yellow dots tracking my hands: http://imgur.com/MlQOY

Does anybody know what’s going on here? I’m still not a very good programmer yet, so if someone could just give me some advice and point me in the right direction, I would really appreciate it.

BTW, I’m on OSX 10.7 and using oF 007.

Thank you,

You have to pose. The black screen is the masking that is masking everything when a player is not found. Make sure you get entirely in the pose and stand there holding your arms up and apart as if someone were pointing a weapon at you.

Thank you,

HI, you might also want to try my branch of ofxOpenNi as it has implemented an autoskeleton feature where no posing is required.


Hi, I’m running into this problem also using openNiSample007 demo and gameover’s newest ofxOpenNi. I don’t think it is a posing issue, I’m using the most up to date NITE, which doesn’t require posing. It kills all the images on the ofxopenni demo. I somehow managed to fix it temporarily when I switched from Debug to Release in xcode, which seems like a weird fix. But now the problem is back and I cant fix it. Any ideas whats going on?

Ok apparently if I set isMasking to false and then turn on tracking it doesn’t happen and everything works fine. But once the tracking kills the images, it cant be fixed.

Hey sorry only just saw this.

The skeleton tracking is actually going on fine, you just couldn’t see it because of a weird problem with glEnableBlend(GL_DST_COLOR, GL_ZERO) - not sure what the deal is (that used to make the white pixels go transparent from the mask)…I’ve done a nasty little fix in the current version so that the all black thing doesn’t happen anymore by using glBlendFunc(GL_ONE_MINUS_DST_COLOR, GL_ZERO) - this will make the rgb image appear with black around a detected user, with anything that is a user in a sort of solarized white…

Ok great thanks a lot, the new version fixes it. One more question, is there an easy way to use this as an alpha mask? I did a quick test to change the user pixel color to red by changing the ofImage type to OF_IMAGE_COLOR and modifying the getUserPixels class like this:

unsigned char * ofxUserGenerator::getUserPixels(int userID) {  
	if (!useMaskPixels && userID == 0) {			// for people who just want all the user masks at once and don't want to waste the extra cycles looking through all users!!!  
		xn::SceneMetaData smd;  
		unsigned short *userPix;  
		if (user_generator.GetUserPixels(0, smd) == XN_STATUS_OK) {   
			userPix = (unsigned short*)smd.Data();					  
		for (int i =0 ; i < width * height; i++) {  
			if (userPix[i] == 0) {  
				maskPixels[0][i*3] =  0;  
			} else maskPixels[0][i*3] = 255;  
	return maskPixels[userID];  

This works fine, and I thought maybe I could do the same by adding a fourth alpha channel to the array and changing the image type to OF_IMAGE_COLOR_ALPHA. But this messes up the image and does not add transparency. There must be a more simple way, no?

Here is how I do it - this is pulled from a larger class so hopefully it is all there/makes sense

earlier in class

cameraWidth = 640;  
cameraHeight = 480;  
xn::UserGenerator userGenerator;  
ofImage compImage;  
compImage.allocate(cameraWidth, cameraHeight, OF_IMAGE_COLOR_ALPHA);  
ofxCvGrayscaleImage depthImage;  
depthImage.allocate(cameraWidth, cameraHeight);  
ofxImageGenerator *ofx_imageGenerator;  
xn::ImageGenerator imageGenerator;  

void MaskedUser::update()  
    int w = cameraWidth;  
    int h = cameraHeight;  
    xn::ImageMetaData imd;  
    imagePixels = (unsigned char *)imd.Data();  
    cameraColorImage.setFromPixels( ofx_imageGenerator->getPixels(), cameraWidth, cameraHeight );  
    //create the mask with userIDs from OpenNI  
    //nID here is your userID  
    if (userGenerator.GetUserPixels(nID, sceneMetaData) == XN_STATUS_OK)   
        const XnLabel* pLabels = sceneMetaData.Data();  
        for (int j =0; j < cameraWidth * cameraHeight; j++, pLabels++)   
            if (*pLabels == nID)   
                depthPixels[j] = 255;  
            } else   
                depthPixels[j] = 0;  
    //add it to an image so we can manipulate the mask  
    depthImage.setFromPixels(depthPixels, w, h);  
    //get the manipulated mask  
    unsigned char * maskPixels = depthImage.getPixels();  
    //recomp the pixels with the mask  
    for (int i = 0; i < w; i++)  
        for (int j = 0; j < h; j++)  
            int pos = (j * w + i);  
            pixels[pos * 4] = imagePixels[pos * 3];  
            pixels[pos * 4+1] = imagePixels[pos * 3+1];  
            pixels[pos * 4+2] = imagePixels[pos * 3+2];  
            pixels[pos * 4+3] = maskPixels[pos];	  
    compImage.setFromPixels(pixels, w, h, OF_IMAGE_COLOR_ALPHA, true);  
    maskPixels = NULL;  
    delete maskPixels;  

There are a lot’s of ways to skin this particular cat :wink:

Although it’s definitely possible to get the effect of an alpha channel by direct pixel manipulation, these days I’ve started using shaders instead.

Attached is a simple example that maybe does something like you need? The example takes an rgb image and then uses a grayscale image as a mask.

Other’s who are way better with shaders might be able to refine this some more (eg., I only know how to draw the output of a shader like this using the glBegin(GL_TRIANGLE_STRIP)…method, whereas I think ofMesh might be a more oF007 way…??)


@jvcleave: as gameover said, there are a lot of ways of dealing with alpha masks. Recently I was working on a shader implementation that I hope you found it useful. It´s at: https://github.com/patriciogonzalezvivo/ofxFX
You can check it out the mask-example as well the composite-example.

It basically do the same stuff you are doing with the pixels but SUPER-FAST because it´s done on parallel computation on the GPU.

thanks patricio! I will give it a go as I use this quite a bit

wow the ofxFX stuff is great, i was able to get the masking working in no time using it. thanks!

Just been working on this for the new version of ofxOpenNI

Thinking I should go with:

    int nIndex = 0;  
    for (int nY = 0; nY < getHeight(); nY++) {  
		for (int nX = 0; nX < getWidth(); nX++) {  
            nIndex = nY * getWidth() + nX;  
            if (userPix[nIndex] == user.id) {  
                user.maskPixels[nIndex * 4 + 0] = 255;  
                user.maskPixels[nIndex * 4 + 1] = 255;  
                user.maskPixels[nIndex * 4 + 2] = 255;  
                user.maskPixels[nIndex * 4 + 3] = 0;  
            } else {  
                user.maskPixels[nIndex * 4 + 0] = 0;  
                user.maskPixels[nIndex * 4 + 1] = 0;  
                user.maskPixels[nIndex * 4 + 2] = 0;  
                user.maskPixels[nIndex * 4 + 3] = 255;  

So that ofEnableAlphaBlending() or ofEnableBlendMode(OF_BLENDMODE_ALPHA) give you what is expected, as well as allowing drawing the mask as a ‘black & white’ image for debug purposes. Figure this will work with shaders as easily as before…

What do you think?