OpenNI || Kinect GetUserPixels

wimvanhenden · February 5, 2011, 4:26pm

Hello everybody,

When OpenNI detects a user the ofxOpenNI UserGenerator class draws the scene with the user in it. The silhouette is a perfect match.

I am trying to isolate the pixels for these ‘user blobs’.
I understand that the GetUserPixels method of the UserGenerator has something to do with this.

I think the solution must be somewhere in the drawScene() method of the ofxUserGenerator class. But I can’t seem to figure it out.

Is there somebody who could help me in isolating the users pixels and drawing them onto an ofImage or something similar?

Thank you very much in advance.

Cheers,

wimvanhenden · February 8, 2011, 10:29pm

Got it!

Use SceneMetaData instead of DepthMetaData an then do something like this to create a mask for all User objects.

  
  
  
userGenerator.GetUserPixels(0, sceneMetaData);  
	  
depthPixels = (unsigned short*)sceneMetaData.Data();  
  
for (int i =0 ; i < 640 * 480; i++) {  
		  
		if (depthPixels[i] != 0) {  
                      maskPixels[i] = 0;  
		} else {  
		      maskPixels[i] = 255;  
		}  
		  
  }  
maskImage.setFromPixels(maskPixels, 640, 480, OF_IMAGE_GRAYSCALE);

This gives you perfect User blobs to use in combination with OpenCV.

This is great!

chrisoshea · February 9, 2011, 4:31pm

Whos version of ofxOpenNi are you using as I dont get (in roxlu latest) the contours/colours of the people.

thanks

wimvanhenden · February 9, 2011, 9:32pm

Hi Chris,

I got the latest version from HalfdanJ
https://github.com/HalfdanJ/ofxOpenNI

But I actually I am not using the ofxOpenNI classes from this package.

I work with the OpenNI classes directly, included in the package as a library.
https://github.com/HalfdanJ/ofxOpenNI/t-…-er/include

I instantiate my Context, DepthGenerator and UserGenerator from the XnCppWrapper class.
https://github.com/HalfdanJ/ofxOpenNI/b-…-pWrapper.h

Here you can find the source for my main and testApp:
http://dropbox.littlemissrobot.com/uploads/OpenNI-inOF.zip

Let me know if this helps.

jvcleave · February 18, 2011, 8:35am

wow - that isolation is amazing!

ascorbin · February 18, 2011, 1:31pm

hm…how can we differentiate between users ? and paint them red, blue, orange… ?

ascorbin · February 18, 2011, 3:17pm

ok, i found it

  
userGenerator.GetUserPixels(0, sceneMetaData);  
	depthRawPixels = (unsigned short*)sceneMetaData.Data();  
	  
	for (int i =0 ; i < 640 * 480; i++) {  
		  
		depthPixels[i*3] = 0;  
		depthPixels[i*3+1] = 0;  
		depthPixels[i*3+2] = 0;  
		  
		if (depthRawPixels[i]>0) {  
		  
		if (depthRawPixels[i]%3 == 0) {  
            depthPixels[i*3] = 255;  
		}  
		if (depthRawPixels[i]%3 == 1) {  
            depthPixels[i*3+1] = 255;  
		}  
		if (depthRawPixels[i]%3 == 2) {  
            depthPixels[i*3+2] = 255;  
		}  
			  
		}	  
		  
    }  
user.setFromPixels(depthPixels, 640, 480, OF_IMAGE_COLOR);

jvcleave · February 18, 2011, 5:58pm

Just wanted to put up a quick video demoing how fast this is - really is a game changer compared to simple blob detection for people

http://vimeo.com/20110346

ascorbin · February 20, 2011, 8:27am

an image

gameover · February 20, 2011, 4:42pm

Hey, wimvanhenden that method is super-neat!

If you use an image generator stream as well, then something like:

  
  
if (depth_generator.IsCapabilitySupported(XN_CAPABILITY_ALTERNATIVE_VIEW_POINT)) {  
     depth_generator.GetAlternativeViewPointCap().SetViewPoint(image_generator);  
}

will register the depth map to the image map - and then you can pull/mask the pixels from the image generator aligned to the users silhouette…

Added an example of masks from user data to an ofxOpenNI fork: https://github.com/gameoverhack/ofxOpenNI

More info: http://forum.openframeworks.cc/t/openni-skeleton-tracking/5125/74

Thnkx,
M

jvcleave · February 21, 2011, 8:06am

sorry if this is super obvious but I couldn’t find it. Is there a way to the the IR image/pixels from OpenNI?

ascorbin · February 23, 2011, 4:21pm

so… another example for masking. this uses imageGenerator, userGenerator and depthGenerator, colors eight different people with depth info plus gives 3d user coords. xCode project for OF 062, have fun!
http://vimeo.com/20323281

http://www.cucumber.de/discourse/NI-userIsolation-4.zip

georgeprofenza · February 23, 2011, 11:46pm

Hello,

Just got started with the Kinect and thought I’d give OpenNI a try
after seeing the amazing progress so far.

I had a few compile errors initially(which were discussed in other posts),
but now I am stuck again

I am using of_preRelease_v0062_osx_FAT on OSX 10.5.8.

Any hints ?

georgeprofenza · February 24, 2011, 12:24am

Ok, I think I understand what’s with the error, but can’t seem to fix it.

I’ve checked the console and this is the output:

  
  
[Session started at 2011-02-24 00:20:55 +0000.]  
dyld: Library not loaded: @executable_path/./../../../data/openni/lib/libnimCodecs.dylib  
  Referenced from: /Users/george/Downloads/FirefoxDownloads/of_preRelease_v0062_osx_FAT/apps/examples/NI_userIsolation_4/bin/emptyExampleDebug.app/Contents/MacOS/emptyExampleDebug  
  Reason: no suitable image found.  Did find:  
	/Users/george/Downloads/FirefoxDownloads/of_preRelease_v0062_osx_FAT/apps/examples/NI_userIsolation_4/bin/emptyExampleDebug.app/Contents/MacOS/./../../../data/openni/lib/libnimCodecs.dylib: unknown required load command 0x80000022  
	/Users/george/Downloads/FirefoxDownloads/of_preRelease_v0062_osx_FAT/apps/examples/NI_userIsolation_4/bin/emptyExampleDebug.app/Contents/MacOS/./../../../data/openni/lib/libnimCodecs.dylib: unknown required load command 0x80000022  
  
The Debugger has exited due to signal 5 (SIGTRAP).The Debugger has exited due to signal 5 (SIGTRAP).

Seems like the app can’t find libnimCodecs.dylib, but it’s in the data folder, and I’ve dragged it onto the xcode project too.

Clueless at the moment

boba · February 25, 2011, 5:06am

Hi ascorbin! Thanks for posting this example. I am getting linker problem like this

  
ld: warning: directory '/Applications/of_osx_062/addons/ofxOpenNI/libs/openni/osx' following -L not found  
ld: warning: directory '/Applications/of_osX_062/apps/neu2/NI_userIsolation_4/src/ofxOpenNI/mac/copy_to_data_openni_path/lib' following -L not found  
ld: warning: directory '/Applications/of_osX_062/apps/neu2/NI_userIsolation_4/src/ofxOpenCv/libs/opencv/lib/osx' following -L not found  
ld: library not found for -lJPEG

Any idea where these directories are hardcoded, and how to get JPEG installed? I am running this against standard 062 download.

Thanks

----b

georgeprofenza · February 25, 2011, 5:52am

@boba Sorry to interrupt, but I have a suggestion:

It looks like Xcode Can’t find the libJPEG.dylib file, which you should have in the bin/data/openni/lib folder. Drag the files from the lib folder into your project, but pay attention to the options. You could tick the “Copy items …” option if you like, but the make sure you choose the Relative Path option that suits you from the Reference Type drop down.

I’m guessing the Reference Type somehow got set to Absolute which might explain the ‘hardcoded’ path.

HTH,
george

boba · February 25, 2011, 8:44pm

[quote author=“george.profenza”]@boba Sorry to interrupt, but I have a suggestion:

It looks like Xcode Can’t find the libJPEG.dylib file, which you should have in the bin/data/openni/lib folder. Drag the files from the lib folder into your project, but pay attention to the options. You could tick the “Copy items …” option if you like, but the make sure you choose the Relative Path option that suits you from the Reference Type drop down.

I’m guessing the Reference Type somehow got set to Absolute which might explain the ‘hardcoded’ path.

HTH,
george[/quote]

Thanks george! I used the default settings while adding the whole openni folder and that allowed the build to go to completion. Can’t check it with the device though, being on the bus at the moment.

wimvanhenden · February 26, 2011, 9:23pm

Hey man,

I checked out your code over at Github. Nice work!

Thanks for sharing.

[quote author=“gameover”]Hey, wimvanhenden that method is super-neat!

If you use an image generator stream as well, then something like:

  
  
if (depth_generator.IsCapabilitySupported(XN_CAPABILITY_ALTERNATIVE_VIEW_POINT)) {  
     depth_generator.GetAlternativeViewPointCap().SetViewPoint(image_generator);  
}

will register the depth map to the image map - and then you can pull/mask the pixels from the image generator aligned to the users silhouette…

Added an example of masks from user data to an ofxOpenNI fork: https://github.com/gameoverhack/ofxOpenNI

More info: http://forum.openframeworks.cc/t/openni-skeleton-tracking/5125/74

Thnkx,
M[/quote]

gameover · March 23, 2011, 11:55am

@ascorbin

Thanks for sharing how you got the user masks to work correctly -> after looking at your code I realised mine was not working at all

Have implemented your method in the latest version of my ofxOpenNI (amongst many things)…

More info over here http://forum.openframeworks.cc/t/openni-skeleton-tracking/5125/108

M

ascorbin · March 23, 2011, 6:37pm

yey! @gameover great work! i love this forum! open source!

greetings