Getting Kinect Skeleton points gameover openni

Hi,
I have been trying to use gameovers openni. It has everything I want to use but I don’t really know how to use it. How can I get to the joint positions and user id’s from testApp? I did manage to get this data out but by modifying the addon. I am sure I did not take the right approach.

Also any ideas as to why I cannot include ofxSyphon in a project with this addon. If I include it everything will compile but I get an error and it will not run.

I get this error:

dyld: Library not loaded: @loader_path/…/Frameworks/Syphon.framework/Versions/A/Syphon
Referenced from: /of_preRelease_v007_osx/apps/addonsExamples/ofxopenni example/bin/openNISample007Debug.app/Contents/MacOS/openNISample007Debug
Reason: image not found
sharedlibrary apply-load-rules all
Data Formatters temporarily unavailable, will re-try after a ‘continue’. (Cannot call into the loader at present, it is locked.)

The project compiles fine without ofxShyphon and I can use ofxSyphon in other projects.

Lastly, if i use the version 7 example from the gameover openni the skeleton tracking does not seem to work. The source is the same for the .62 version of the example. Is there something I have to do to get it to run properly in 0.7.

Cheers

Hey Fred

Glad it’s working for you! In answer to your questions:

How can I get to the joint positions and user id’s from testApp?

Most of the limb & skeleton code I didn’t write, but the basic approach involves getting a ref to a tracked user and then getting the start and end joint positions from each of the “limbs” in a user:

  
// iterate through number of tracked users  
for (int user = 0; user < recordUser.getNumberOfTrackedUsers(); user++) {  
	// get ref to a tracked user  
	ofxTrackedUser * trackedUser = recordUser.getTrackedUser(user);  
	  
	// do something with the X, Y, Z co-ords of each limb.   
	// Note the capital XYZ is cos type is XnPoint3D NOT ofPoint  
	int neckStartX = trackedUser->neck.position[0].X;  
	int neckStartY = trackedUser->neck.position[0].Y;  
	int neckStartZ = trackedUser->neck.position[0].Y;  
	  
	int neckEndX = trackedUser->neck.position[1].X;  
	int neckEndY = trackedUser->neck.position[1].Y;  
	int neckEndZ = trackedUser->neck.position[1].Y;  
  
	// etc for other limb types eg., left_lower_torso, left_shoulder, etc etc  
}  

NB: I haven’t tried that code, and you could obviously use an ofPoint type to hold the positions etc, but that’s the general approach at this point…on looking at this I reckon things could be made much simpler for future releases!!

any ideas as to why I cannot include ofxSyphon in a project with this addon

* I’ve been using Syphon with ofxOpenNI there are no conflicts. That error looks like you haven’t got the Syphon framework copying during the build stage of compiling your project (this stumped me for frickin’ ages until I came across this post: /topic,6552.0.html

What I find works for me is to:
* look in the left hand Xcode browser pane; find Targets, open this up and whatever the item inside it is called (for instance my openNIExample will have an “openNIExample” item in there…
* Create a new build phase OR (if like me that is not something you are familiar with or could get to work easily)
* Look for a folder that has “Copy files (1)” and open it -> you should see the Glut framework here
* Locate your SyphonFramework file and drag it to the Copy files (1) “folder” -> basically adds the SyphonFramework to the build phase/copy frmaeworks part of your xcode project…
* … and voila it works!

Lastly, if i use the version 7 example from the gameover openni the skeleton tracking does not seem to work

Hmmm…not sure about that it should work fine…but when I did the 007 example I was not actively using 007…will see if I can work this out for you…

Wow, that is great thanks for the help, I should have looked more for the syphon trick.

The code you posted makes sense but if I try and use it anywhere I get

Program received signal: “EXC_BAD_ACCESS”.
Xcode could not locate source file: iostream (line: 76)
kill

The program compiles but as soon as I turn on tracking and walk in front of the kinect the program crashes, even if I wait until the user is tracked by seeing the bones before iterating through the users.

Fred

Sorry can’t test this code today…am not near a Kinect, but that sounds like you’re getting back tracked users that are null. Try something like:

  
// iterate through number of tracked users    
for (int user = 0; user < recordUser.getNumberOfTrackedUsers(); user++) {    
    // get ref to a tracked user    
    ofxTrackedUser * trackedUser = recordUser.getTrackedUser(user);    
  
    if (trackedUser != NULL) {  
        
       // do something with the X, Y, Z co-ords of each limb.     
       // Note the capital XYZ is cos type is XnPoint3D NOT ofPoint        
  
    }  
  
}    

I had a go at getting mask, color and depth images out via syphon and joint coordinates out via OSC. I have taken the OSc format of OSCeleton as I have made several things based on this already and wanted to see if I could replace it with something with a little more control.

After a lot of changes I am sure I am not doing a lot of things correctly but now I get these errors when I run the program

OF_ERROR: error = bad imageType in ofImage::allocate
OF_ERROR: error = bad imageType in ofImage::setFromPixels

l_shoulderl_elbowl_handr_shoulderr_elbowr_handtorsol_hipterminate called after throwing an instance of ‘osc::OutOfBufferMemoryException’
what(): out of buffer memory
Program received signal: “SIGABRT”.
sharedlibrary apply-load-rules all

I will paste my horrible code in below, maybe someone can point me in the right direction.

Here is what I added to the header

  
#endif  
  
	void				drawMasks();  
	void				drawPointCloud(ofxUserGenerator * user_generator, int userID);  
  
	int					nearThreshold, farThreshold;  
	int					pointCloudRotationY;  
  
	ofImage				allUserMasks, user1Mask, user2Mask, depthRangeMask;  
  
	float				filterFactor;  
	  
	bool				myToggle;  
	  
	   
	   
	   
	 string				jointNameOscV;  
	 int					UserIdOscV;  
	 float				xValOscV;  
	 float				yValOscV;  
	 float				zValOscV;  
	 int					lostUserOscV;  
	 int					newUserOscV;  
	 int					newSkeleOscV;  
	ofxOscMessage		jointOsc;   
	  
	ofTexture texColourImageSyphon;  
	ofTexture texDepthImageSyphon;  
	ofTexture texMaskImageSyphon;  
	  
	ofImage texColourImageTemp;  
	ofImage texDepthImageTemp;  
	ofImage texMaskImagetemp;  
	ofImage testImage;  
	  
	 ofxSyphonServer colourImageSyphon;  
	 ofxSyphonServer depthImageSyphon;  
	 ofxSyphonServer maskImageSyphon;  
	  
	 private:  
	 ofxOscSender		sender;   
  
};  
  
#endif  

And my changes to the cpp

  
void testApp::setup() {  
  
	isLive			= true;  
	isTracking		= false;  
	isTrackingHands	= true;  
	isFiltering		= false;  
	isRecording		= false;  
	isCloud			= false;  
	isCPBkgnd		= true;  
	isMasking		= true;  
  
	nearThreshold = 500;  
	farThreshold  = 1000;  
  
	filterFactor = 0.1f;  
  
	setupRecording();  
  
	ofBackground(0, 0, 0);  
	sender.setup( HOST, PORT );  
	  
	   
	  
	colourImageSyphon.setName("Colour Image");  
	depthImageSyphon.setName("Depth Image");  
	maskImageSyphon.setName("maskImage");  
	  
	texColourImageSyphon.allocate(640, 480, GL_RGB);   
	texDepthImageSyphon.allocate(640, 480, GL_RGB);   
	texMaskImageSyphon.allocate(640, 480, GL_RGB);  
	!myToggle;  
  
}  

  
if (isTracking){  
			recordUser.update();  
			for (int user = 0; user < recordUser.getNumberOfTrackedUsers(); user++) {    
				UserIdOscV = user++;  
				ofxTrackedUser * trackedUser = recordUser.getTrackedUser(user);   
				if (trackedUser != NULL) {					  
					jointNameOscV = "head";  
					xValOscV = trackedUser->neck.position[0].X;    
					yValOscV = trackedUser->neck.position[0].Y;   
					zValOscV = trackedUser->neck.position[0].Z;   
					jointOsc.setAddress("/joint");  
					jointOsc.addStringArg(jointNameOscV);  
					jointOsc.addIntArg(UserIdOscV);  
					jointOsc.addFloatArg(xValOscV);  
					jointOsc.addFloatArg(yValOscV);  
					jointOsc.addFloatArg(zValOscV);  
					sender.sendMessage(jointOsc);  
					cout << jointNameOscV;  
					  
					jointNameOscV = "neck";  
					xValOscV = trackedUser->neck.position[1].X;    
					yValOscV = trackedUser->neck.position[1].Y;   
					zValOscV = trackedUser->neck.position[1].Z;  
					jointOsc.setAddress("/joint");  
					jointOsc.addStringArg(jointNameOscV);  
					jointOsc.addIntArg(UserIdOscV);  
					jointOsc.addFloatArg(xValOscV);  
					jointOsc.addFloatArg(yValOscV);  
					jointOsc.addFloatArg(zValOscV);  
					sender.sendMessage(jointOsc);  
					cout << jointNameOscV;  
					  
					jointNameOscV = "l_shoulder";  
					xValOscV = trackedUser->left_shoulder.position[0].X;    
					yValOscV = trackedUser->left_shoulder.position[0].Y;   
					zValOscV = trackedUser->left_shoulder.position[0].Z;  
					jointOsc.setAddress("/joint");  
					jointOsc.addStringArg(jointNameOscV);  
					jointOsc.addIntArg(UserIdOscV);  
					jointOsc.addFloatArg(xValOscV);  
					jointOsc.addFloatArg(yValOscV);  
					jointOsc.addFloatArg(zValOscV);  
					sender.sendMessage(jointOsc);  
					cout << jointNameOscV;  
					  
					jointNameOscV = "l_elbow";  
					xValOscV = trackedUser->left_upper_arm.position[1].X;    
					yValOscV = trackedUser->left_upper_arm.position[1].Y;   
					zValOscV = trackedUser->left_upper_arm.position[1].Z;  
					jointOsc.setAddress("/joint");  
					jointOsc.addStringArg(jointNameOscV);  
					jointOsc.addIntArg(UserIdOscV);  
					jointOsc.addFloatArg(xValOscV);  
					jointOsc.addFloatArg(yValOscV);  
					jointOsc.addFloatArg(zValOscV);  
					sender.sendMessage(jointOsc);  
					cout << jointNameOscV;  
					  
					jointNameOscV = "l_hand";  
					xValOscV = trackedUser->left_lower_arm.position[1].X;    
					yValOscV = trackedUser->left_lower_arm.position[1].Y;   
					zValOscV = trackedUser->left_lower_arm.position[1].Z;  
					jointOsc.setAddress("/joint");  
					jointOsc.addStringArg(jointNameOscV);  
					jointOsc.addIntArg(UserIdOscV);  
					jointOsc.addFloatArg(xValOscV);  
					jointOsc.addFloatArg(yValOscV);  
					jointOsc.addFloatArg(zValOscV);  
					sender.sendMessage(jointOsc);  
					cout << jointNameOscV;  
					  
					jointNameOscV = "r_shoulder";  
					xValOscV = trackedUser->right_shoulder.position[0].X;    
					yValOscV = trackedUser->right_shoulder.position[0].Y;   
					zValOscV = trackedUser->right_shoulder.position[0].Z;  
					jointOsc.setAddress("/joint");  
					jointOsc.addStringArg(jointNameOscV);  
					jointOsc.addIntArg(UserIdOscV);  
					jointOsc.addFloatArg(xValOscV);  
					jointOsc.addFloatArg(yValOscV);  
					jointOsc.addFloatArg(zValOscV);  
					sender.sendMessage(jointOsc);  
					cout << jointNameOscV;  
					  
					jointNameOscV = "r_elbow";  
					xValOscV = trackedUser->right_upper_arm.position[1].X;    
					yValOscV = trackedUser->right_upper_arm.position[1].Y;   
					zValOscV = trackedUser->right_upper_arm.position[1].Z;  
					jointOsc.setAddress("/joint");  
					jointOsc.addStringArg(jointNameOscV);  
					jointOsc.addIntArg(UserIdOscV);  
					jointOsc.addFloatArg(xValOscV);  
					jointOsc.addFloatArg(yValOscV);  
					jointOsc.addFloatArg(zValOscV);  
					sender.sendMessage(jointOsc);  
					cout << jointNameOscV;  
					  
					jointNameOscV = "r_hand";  
					xValOscV = trackedUser->right_lower_arm.position[1].X;    
					yValOscV = trackedUser->right_lower_arm.position[1].Y;   
					zValOscV = trackedUser->right_lower_arm.position[1].Z;  
					jointOsc.setAddress("/joint");  
					jointOsc.addStringArg(jointNameOscV);  
					jointOsc.addIntArg(UserIdOscV);  
					jointOsc.addFloatArg(xValOscV);  
					jointOsc.addFloatArg(yValOscV);  
					jointOsc.addFloatArg(zValOscV);  
					sender.sendMessage(jointOsc);  
					cout << jointNameOscV;  
					  
					jointNameOscV = "torso";  
					xValOscV = ((trackedUser->left_upper_torso.position[0].X)+(trackedUser->right_upper_torso.position[0].X)/2);  
					yValOscV = ((trackedUser->left_upper_torso.position[0].Y)+(trackedUser->right_upper_torso.position[0].Y)/2);  
					zValOscV = ((trackedUser->left_upper_torso.position[0].Z)+(trackedUser->right_upper_torso.position[0].Z)/2);  
					jointOsc.setAddress("/joint");  
					jointOsc.addStringArg(jointNameOscV);  
					jointOsc.addIntArg(UserIdOscV);  
					jointOsc.addFloatArg(xValOscV);  
					jointOsc.addFloatArg(yValOscV);  
					jointOsc.addFloatArg(zValOscV);  
					sender.sendMessage(jointOsc);  
					cout << jointNameOscV;  
					  
					jointNameOscV = "l_hip";  
					xValOscV = trackedUser->left_lower_torso.position[0].X;    
					yValOscV = trackedUser->left_lower_torso.position[0].Y;   
					zValOscV = trackedUser->left_lower_torso.position[0].Z;  
					jointOsc.setAddress("/joint");  
					jointOsc.addStringArg(jointNameOscV);  
					jointOsc.addIntArg(UserIdOscV);  
					jointOsc.addFloatArg(xValOscV);  
					jointOsc.addFloatArg(yValOscV);  
					jointOsc.addFloatArg(zValOscV);  
					sender.sendMessage(jointOsc);  
					cout << jointNameOscV;  
					  
					jointNameOscV = "l_knee";  
					xValOscV = trackedUser->left_upper_leg.position[0].X;    
					yValOscV = trackedUser->left_upper_leg.position[0].Y;   
					zValOscV = trackedUser->left_upper_leg.position[0].Z;  
					jointOsc.setAddress("/joint");  
					jointOsc.addStringArg(jointNameOscV);  
					jointOsc.addIntArg(UserIdOscV);  
					jointOsc.addFloatArg(xValOscV);  
					jointOsc.addFloatArg(yValOscV);  
					jointOsc.addFloatArg(zValOscV);  
					sender.sendMessage(jointOsc);  
					cout << jointNameOscV;  
					  
					jointNameOscV = "l_ankle";  
					xValOscV = trackedUser->left_lower_leg.position[0].X;    
					yValOscV = trackedUser->left_lower_leg.position[0].Y;   
					zValOscV = trackedUser->left_lower_leg.position[0].Z;  
					jointOsc.setAddress("/joint");  
					jointOsc.addStringArg(jointNameOscV);  
					jointOsc.addIntArg(UserIdOscV);  
					jointOsc.addFloatArg(xValOscV);  
					jointOsc.addFloatArg(yValOscV);  
					jointOsc.addFloatArg(zValOscV);  
					sender.sendMessage(jointOsc);  
					cout << jointNameOscV;  
					  
					jointNameOscV = "l_foot";  
					xValOscV = trackedUser->left_lower_leg.position[1].X;    
					yValOscV = trackedUser->left_lower_leg.position[1].Y;   
					zValOscV = trackedUser->left_lower_leg.position[1].Z;  
					jointOsc.setAddress("/joint");  
					jointOsc.addStringArg(jointNameOscV);  
					jointOsc.addIntArg(UserIdOscV);  
					jointOsc.addFloatArg(xValOscV);  
					jointOsc.addFloatArg(yValOscV);  
					jointOsc.addFloatArg(zValOscV);  
					sender.sendMessage(jointOsc);  
					cout << jointNameOscV;  
					  
					jointNameOscV = "r_hip";  
					xValOscV = trackedUser->right_lower_torso.position[0].X;    
					yValOscV = trackedUser->right_lower_torso.position[0].Y;   
					zValOscV = trackedUser->left_lower_torso.position[0].Z;  
					jointOsc.setAddress("/joint");  
					jointOsc.addStringArg(jointNameOscV);  
					jointOsc.addIntArg(UserIdOscV);  
					jointOsc.addFloatArg(xValOscV);  
					jointOsc.addFloatArg(yValOscV);  
					jointOsc.addFloatArg(zValOscV);  
					sender.sendMessage(jointOsc);  
					cout << jointNameOscV;  
					  
					jointNameOscV = "r_knee";  
					xValOscV = trackedUser->right_upper_leg.position[0].X;    
					yValOscV = trackedUser->right_upper_leg.position[0].Y;   
					zValOscV = trackedUser->right_upper_leg.position[0].Z;  
					jointOsc.setAddress("/joint");  
					jointOsc.addStringArg(jointNameOscV);  
					jointOsc.addIntArg(UserIdOscV);  
					jointOsc.addFloatArg(xValOscV);  
					jointOsc.addFloatArg(yValOscV);  
					jointOsc.addFloatArg(zValOscV);  
					sender.sendMessage(jointOsc);  
					cout << jointNameOscV;  
					  
					jointNameOscV = "r_ankle";  
					xValOscV = trackedUser->right_lower_leg.position[0].X;    
					yValOscV = trackedUser->right_lower_leg.position[0].Y;   
					zValOscV = trackedUser->right_lower_leg.position[0].Z;  
					jointOsc.setAddress("/joint");  
					jointOsc.addStringArg(jointNameOscV);  
					jointOsc.addIntArg(UserIdOscV);  
					jointOsc.addFloatArg(xValOscV);  
					jointOsc.addFloatArg(yValOscV);  
					jointOsc.addFloatArg(zValOscV);  
					sender.sendMessage(jointOsc);  
					cout << jointNameOscV;  
					  
					jointNameOscV = "r_foot";  
					xValOscV = trackedUser->right_lower_leg.position[1].X;    
					yValOscV = trackedUser->right_lower_leg.position[1].Y;   
					zValOscV = trackedUser->right_lower_leg.position[1].Z;  
					jointOsc.setAddress("/joint");  
					jointOsc.addStringArg(jointNameOscV);  
					jointOsc.addIntArg(UserIdOscV);  
					jointOsc.addFloatArg(xValOscV);  
					jointOsc.addFloatArg(yValOscV);  
					jointOsc.addFloatArg(zValOscV);  
					sender.sendMessage(jointOsc);  
					cout << jointNameOscV;  
					}  
  

  
f (isLive) {  
  
		recordDepth.draw(0,0,640,480);  
		  
		//texDepthImageTemp.setFromPixels(recordDepth., 640, 480, OF_IMAGE_COLOR_ALPHA);		  
		//texDepthImageSyphon.loadData(texDepthImageTemp.getPixels(), 640, 480, GL_RGB);		  
		texDepthImageSyphon.loadScreenData(0, 0, 640, 480);  
		depthImageSyphon.publishTexture(&texDepthImageSyphon);  
		  
		recordImage.draw(640, 0, 640, 480);  
		//texColourImageTemp.grabScreen(640, 0, 640, 480);  
		texColourImageSyphon.loadData(recordImage.getPixels(), 640, 480, GL_RGB);  
		colourImageSyphon.publishTexture(&texColourImageSyphon);  
  
		depthRangeMask.draw(0, 480, 320, 240);  

  
void testApp:: drawMasks() {  
	glPushMatrix();  
	glEnable(GL_BLEND);  
	glBlendFunc(GL_DST_COLOR, GL_ZERO);  
	allUserMasks.draw(640, 0);  
	texMaskImagetemp.setFromPixels(allUserMasks.getPixels(), 640, 480, OF_IMAGE_GRAYSCALE);  
	texMaskImagetemp.setImageType(OF_IMAGE_COLOR);  
	texMaskImageSyphon.loadData(texMaskImagetemp.getPixels(), 640, 480, GL_RGB);  
	maskImageSyphon.publishTexture(&texMaskImageSyphon);  
	glDisable(GL_BLEND);  
	user1Mask.draw(320, 480, 320, 240);  
	user2Mask.draw(640, 480, 320, 240);  
	glPopMatrix();  
}  

The syphon stuff works although I am using loadScreenData to get the depth image, I cannot work out how to get it out directly. The OSC crashes the program as soon as I start tracking.

Thanks as usual for all the help.

Cheers

[quote=“gameover, post:4, topic:7431”]
Sorry can’t test this code today…am not near a Kinect, but that sounds like you’re getting back tracked users that are null. Try something like:

  
// iterate through number of tracked users    
for (int user = 0; user < recordUser.getNumberOfTrackedUsers(); user++) {    
    // get ref to a tracked user    
    ofxTrackedUser * trackedUser = recordUser.getTrackedUser(user);    
  
    if (trackedUser != NULL) {  
        
       // do something with the X, Y, Z co-ords of each limb.     
       // Note the capital XYZ is cos type is XnPoint3D NOT ofPoint        
  
    }  
  
}    

[/quote]I have found that getTrackedUser wants a 1 indexed call, not a zero indexed call. This is in linux…

I solved it, it was something stupid I just had to create and send my OSc differently, the get tracked user does have a o index.

Gameover, if you read this I have another question for you. When I check out your tracked user properties you have a list of limbs you track

  
ofxLimb left_shoulder;  
	ofxLimb left_upper_arm;  
	ofxLimb left_lower_arm;  
  
	// right arm + shoulder  
	ofxLimb right_shoulder;  
	ofxLimb right_upper_arm;  
	ofxLimb right_lower_arm;  
  
	// torso  
	ofxLimb left_upper_torso;  
	ofxLimb right_upper_torso;  
  
	// left lower torso + leg  
	ofxLimb left_lower_torso;  
	ofxLimb left_upper_leg;  
	ofxLimb left_lower_leg;  
  
	// right lower torso + leg  
	ofxLimb right_lower_torso;  
	ofxLimb right_upper_leg;  
	ofxLimb right_lower_leg;  
  
	ofxLimb hip;  

But when I check the XnSkeletonJoint typedef I see a different list of limbs

  
XN_SKEL_HEAD			= 1,  
	XN_SKEL_NECK			= 2,  
	XN_SKEL_TORSO			= 3,  
	XN_SKEL_WAIST			= 4,  
  
	XN_SKEL_LEFT_COLLAR		= 5,  
	XN_SKEL_LEFT_SHOULDER	= 6,  
	XN_SKEL_LEFT_ELBOW		= 7,  
	XN_SKEL_LEFT_WRIST		= 8,  
	XN_SKEL_LEFT_HAND		= 9,  
	XN_SKEL_LEFT_FINGERTIP	=10,  
  
	XN_SKEL_RIGHT_COLLAR	=11,  
	XN_SKEL_RIGHT_SHOULDER	=12,  
	XN_SKEL_RIGHT_ELBOW		=13,  
	XN_SKEL_RIGHT_WRIST		=14,  
	XN_SKEL_RIGHT_HAND		=15,  
	XN_SKEL_RIGHT_FINGERTIP	=16,  
  
	XN_SKEL_LEFT_HIP		=17,  
	XN_SKEL_LEFT_KNEE		=18,  
	XN_SKEL_LEFT_ANKLE		=19,  
	XN_SKEL_LEFT_FOOT		=20,  
  
	XN_SKEL_RIGHT_HIP		=21,  
	XN_SKEL_RIGHT_KNEE		=22,  
	XN_SKEL_RIGHT_ANKLE		=23,  
	XN_SKEL_RIGHT_FOOT		=24	  

Out of curiosity what is the reasoning behind changing the limb types?

Also I am wondering about the range of these values, I get seemingly sensible results if I assume that the x and y values range between 0 and 500 and the z values have a range of 1000. I can’t find out where this is defined. I think however I was wrong in this calculation as it should only give as high a resolution as the camera and the values should be X between 0 and 640 and Y between 0 and 480 but Z is still a mystery to me.

Cheers