Getting depth pixel data from kinect v2

#1

I am using this kinect for windows v2 addon and I can’t figure out how to get the actual depth data of a pixel location.
I have an ofMesh object where I add all pixels from the frame to. Each pixel is an ofVec3f.

My update function looks like such:

kinect.update();

	ofVec4f current;
	unsigned short* depthPixel;
	for (unsigned i = 0; i < mesh.getNumVertices(); i++)
	{
		current = mesh.getVertex(i);
		depthPixel = kinect.getDepthSource()->getPixels().getData();

		if (depthPixel)
			mesh.setVertex(i, ofVec3f(current.x, current.y, *depthPixel)); // *depthPixel returns 0 only
		else
			mesh.setVertex(i, ofVec3f(current.x, current.y, 0));
	}

I can’t tell if I am calling the wrong function to get the depth data, but each pixel’s z-data to be set to the depth data pixel in that same location.

#2

Your current implementation exclusively retrieves the first depth pixel in each iteration of the loop. Here’s a fix.

for (unsigned i = 0; i < mesh.getNumVertices(); i++) {
	glm::vec3& v = mesh.getVertices()[i];
	unsigned short& depth = kinect.getDepthSource()->getPixels()[i];
	mesh.setVertex(i, glm::vec3(v.x, v.y, (float)depth));
}

Also, if you’re just looking to build a wireframe depth mesh, try

kinect.getDepthSource()->getMesh();
#3

Thanks, @lshoek. I needed to make a few modifications, but I am getting another problem I haven’t been able to figure out.

The depth source data appears like pillars with half of the pixels’ z values offset, the others stuck in place. It’s easier to see here:

You’ll notice that when I move my hand in front of the Kinect depth sensor, some of the “pillars” or spaces in between will move around a bit. So it is getting some correct data but it’s not what I had in mind…

Here is my setup & update functions:

#define WIDTH 512
#define HEIGHT 424

float a;
unsigned skip = 8;
ofMesh mesh;

void ofApp::setup() 
{
	ofSetWindowShape(800, 600);

	a = 0;
	kinect = ofxKFW2::Device();

	kinect.open();
	kinect.initDepthSource();

	mesh.setMode(OF_PRIMITIVE_POINTS);
	glEnable(GL_POINT_SMOOTH);
	glPointSize(2);

	for (unsigned x = 0; x < WIDTH; x += skip)
		for (unsigned y = 0; y < HEIGHT; y += skip)
			mesh.addVertex(glm::vec3(x, y, 0));
}

//--------------------------------------------------------------
void ofApp::update()
{
	kinect.update();

	for (unsigned i = 0; i < mesh.getNumVertices(); i++)
	{
		glm::vec3 v = mesh.getVertices()[i];

		if (kinect.getDepthSource()->getPixels().getData())
			mesh.setVertex(i, glm::vec3(v.x, v.y, kinect.getDepthSource()->getPixels()[i] / 100));
		else
			mesh.setVertex(i, glm::vec3(v.x, v.y, 0));
	}

}

(The draw function doesn’t do anything special, it just draws the fps and the mesh as dots)

Do you have suggestions on why this is happening? I hope it’s not a bug with my Kinect…

#4

This is because you initialized your grid mesh in column-major order as opposed to row-major. Fix this by making y the outer loop.

for (unsigned y = 0; y < HEIGHT; y += skip)
	for (unsigned x = 0; x < WIDTH; x += skip)
		mesh.addVertex(glm::vec3(x, y, 0));

One more thing is that you’ll want to use float division here to retain depth precision.

// update()
mesh.setVertex(i, glm::vec3(v.x, v.y, kinect.getDepthSource()->getPixels()[i] / 100.f));

And acquire a const vec3 reference instead of copying in update().

const glm::vec3& v = mesh.getVertices()[i];
#5

Flipping the y and x loops just rotates the columns. They are still there. I’m trying to add dots/circles at the depth values the camera sees, so a 3D portrait of the image (the kinect camera image) is seen (without color atm)

#6

I did a series of videos using the kinect 2 https://youtu.be/HF74L-DvCUE that might help

#7

and also making point clouds using kinect 1 https://youtu.be/cO4uzveYkc4

#8

This would only be true if the width and height of the grid were equal. You’re creating and setting the mesh vertices using a sequential index variable that has no regard for the dimension of the data. The depth pixels are stored in row-major order, and your mesh should reflect that. Have you tried the fix? Seems to me like this is the issue.

ofxKinectForWindows2 includes a utility function to do just this.

kinect.getDepthSource()->getMesh();

Tell GL to draw points only in draw().

mesh.draw(ofPolyRenderMode::OF_MESH_POINTS);