Kinect V2 Pixel Depth and Color

Hi there,
I’ve got code up and working with with the old Kinect V1 camera and I am trying to get it working with KinectV2. I am running into a few problems regarding 3 functions that are being called in my code so far. The specific method that I am running into problems with is the update() method. It calls:

  • getDistanceAt(x, y)
  • getWorldCoordinateAt(x, y);
  • kinect.getColorAt(v.x + 320.0, v.y + 240.0);

that are not transferable from the ‘ofxKinect’ add-on to either the ‘ofxKinectForWindows2’ (@elliotwoods)or the ‘ofxKinect2’ (@sadmb) addon. I am assuming that I need to implement something from the new SDK along with some sort of coordinate mapping but I am having trouble getting past this point. If someone could help me out, or point me in the right direction I would appreciate it.

Below is the method that calls these functions:

void ofApp::update(){
// this comment is the old update function that works
//this->kinect.update();
kinect.update();
del.reset();
unsigned char* pix = new unsigned char[640 * 480];
unsigned char* gpix = new unsigned char[640 * 480];
for (int x = 0; x < 640; x += 1) {
	for (int y = 0; y < 480; y += 1) {
		float distance = kinect.getDistanceAt(x, y);

		int pIndex = x + y * 640;
		pix[pIndex] = 0;

		if (distance > 100 && distance < 1100) {
			pix[pIndex] = 255;
		}
	}
	//if you change this, it will break
	blob.setFromPixels(pix, 640, 480, OF_IMAGE_GRAYSCALE);
	int numPoints = 0;
	for (int x = 0; x < 640; x += pointSkip * 5) {
		for (int y = 0; y < 480; y += pointSkip * 5) {
			int pIndex = x + 640 * y;

			if (blob.getPixels()[pIndex] > 0) {
				ofVec3f wc = kinect.getWorldCoordinateAt(x, y);
				wc.x = x - 320.0;
				wc.y = y - 240.0;
				if (abs(wc.z) > 100 && abs(wc.z) < 2000) {
					wc.z = -wc.z;
					wc.x += ofSignedNoise(wc.x, wc.z)*noiseAmount;
					wc.y += ofSignedNoise(wc.y, wc.z)*noiseAmount;
					wc.x = ofClamp(wc.x, -320, 320);
					wc.y = ofClamp(wc.y, -240, 240);
					del.addPoint(wc);
				}
				numPoints++;
			}
		}
	}
	if (numPoints > 0)
		del.triangulate();
	for (int i = 0; i < del.triangleMesh.getNumVertices(); i++) {
		del.triangleMesh.addColor(ofColor(255, 255, 255));
	}
	for (int i = 0; i < del.triangleMesh.getNumIndices() / 3; i += 1) {
		ofVec3f v = del.triangleMesh.getVertex(del.triangleMesh.getIndex(i * 3));
		v.x = ofClamp(v.x, -319, 319);
		v.y = ofClamp(v.y, -239, 239);
		ofColor c = kinect.getColorAt(v.x + 320.0, v.y + 240.0);
		if (!useRealColors)
			c = ofColor(255, 100, 100);
		c.a = colorAlpha;
		del.triangleMesh.setColor(del.triangleMesh.getIndex(i * 3), c);
		del.triangleMesh.setColor(del.triangleMesh.getIndex(i * 3 + 1), c);
		del.triangleMesh.setColor(del.triangleMesh.getIndex(i * 3 + 2), c);
	}
	convertedMesh.clear();
	wireframeMesh.clear();
	wireframeMesh.setMode(OF_PRIMITIVE_TRIANGLES);
	for (int i = 0; i < del.triangleMesh.getNumIndices() / 3; i += 1) {
		int indx1 = del.triangleMesh.getIndex(i * 3);
		ofVec3f p1 = del.triangleMesh.getVertex(indx1);
		int indx2 = del.triangleMesh.getIndex(i * 3 + 1);
		ofVec3f p2 = del.triangleMesh.getVertex(indx2);
		int indx3 = del.triangleMesh.getIndex(i * 3 + 2);
		ofVec3f p3 = del.triangleMesh.getVertex(indx3);
		ofVec3f triangleCenter = (p1 + p2 + p3) / 3.0;
		triangleCenter.x += 320;
		triangleCenter.y += 240;
		triangleCenter.x = floor(ofClamp(triangleCenter.x, 0, 640));
		triangleCenter.y = floor(ofClamp(triangleCenter.y, 0, 480));
		int pixIndex = triangleCenter.x + triangleCenter.y * 640;
		if (pix[pixIndex] > 0) {
			convertedMesh.addVertex(p1);
			convertedMesh.addColor(del.triangleMesh.getColor(indx1));
			convertedMesh.addVertex(p2);
			convertedMesh.addColor(del.triangleMesh.getColor(indx2));
			convertedMesh.addVertex(p3);
			convertedMesh.addColor(del.triangleMesh.getColor(indx3));
			wireframeMesh.addVertex(p1);
			wireframeMesh.addVertex(p2);
			wireframeMesh.addVertex(p3);
		}
	}
	delete pix;
	delete gpix;
}

}

with ofxKinectForWindows2, just use
mesh = kinect.getDepthSource()->getMesh(true, ofxKinectForWindows2::Source::Depth::PointCloudOptions::TextureCoordinates::ColorCamera);

to get a mesh with texture coordinates in colour space
then if you really need to, you could lookup the colour values using those texture coordinates in the mesh, or just

kinect.getColorSource()->getTexture().bind()
mesh.draw();
kinect.getColorSource()->getTexture().unbind()

your naming might vary between 0.8.4 and 0.9.0

1 Like

Thanks for your response! We got it up and working now but are now having trouble with way to many points being displayed. Is there a way to skip point in the draw() function within the mesh at all?

and we seem to have a scaling issues - we have to scale the whole image using ofScale(1000,1000,1000) to get it to work properly. Any idea why we have to do this or how to fix it?

Thanks

We´ve made the Depth to show something useful by doing this:

ofImage image;
ofPixels_<unsigned short> pixels = this->kinect.getDepthSource()->getPixelsRef();
for (int i = 0; i < pixels.size(); i++){
	pixels[i] = 65536 - (pixels[i] * 65536 / 4096);
}
image.setFromPixels(pixels);
ofSetColor(255);
image.draw(0,0);

65536 = the range of short number
4096 = maximum value from kinect (not sure)

Hi @vamoss. There’s a nice example of increasing the brightness of the depth image here:

https://github.com/rickbarraza/KinectV2_Lessons/tree/master/3_MakeRawDepthBrigther

Hi @Mattfelsen,

your example is different because it replaces a processor operation by a memory operation.
Not sure if this it is faster.

But they are using the value 5001 instead 4096.
Do you know if that value is the official?

Thanks!

After some empirical tests, the maximum value is 8000.
I suppose it ins in milimiters. The new kinect see 8meters :smile:

Thanks for the follow-up @vamoss. That sounds about right, the SDK docs mention the pixel value is the distance for that pixel in mm:

DepthFrame Class
https://msdn.microsoft.com/en-us/library/windowspreview.kinect.depthframe.aspx

@cbtriplec could you share the working piece of code with us? I have been trying to get the same code running and I am encountering exactly the same issues. It would be very helpful to see how you resolved this.

Hi @mattfelsen . Barraza’s links seem no longer valid. I have looked at

Would you mind sharing the “KinectV2_Lessons”?, thx jjj

UPDATED: never mind, I found a copy here:

void generateDepthDisplayImage ();

in this project does not work.

unsigned char greyValue = depthLookupTable [rawdepth [rawindex ++]];

It stops at the line of.

%E3%82%AD%E3%83%A3%E3%83%97%E3%83%81%E3%83%A3
What should I do?