Kinect V2 Pixel Depth and Color

beginners
#1

Hi there,
I’ve got code up and working with with the old Kinect V1 camera and I am trying to get it working with KinectV2. I am running into a few problems regarding 3 functions that are being called in my code so far. The specific method that I am running into problems with is the update() method. It calls:

  • getDistanceAt(x, y)
  • getWorldCoordinateAt(x, y);
  • kinect.getColorAt(v.x + 320.0, v.y + 240.0);

that are not transferable from the ‘ofxKinect’ add-on to either the ‘ofxKinectForWindows2’ (@elliotwoods)or the ‘ofxKinect2’ (@sadmb) addon. I am assuming that I need to implement something from the new SDK along with some sort of coordinate mapping but I am having trouble getting past this point. If someone could help me out, or point me in the right direction I would appreciate it.

Below is the method that calls these functions:

void ofApp::update(){
// this comment is the old update function that works
//this->kinect.update();
kinect.update();
del.reset();
unsigned char* pix = new unsigned char[640 * 480];
unsigned char* gpix = new unsigned char[640 * 480];
for (int x = 0; x < 640; x += 1) {
	for (int y = 0; y < 480; y += 1) {
		float distance = kinect.getDistanceAt(x, y);

		int pIndex = x + y * 640;
		pix[pIndex] = 0;

		if (distance > 100 && distance < 1100) {
			pix[pIndex] = 255;
		}
	}
	//if you change this, it will break
	blob.setFromPixels(pix, 640, 480, OF_IMAGE_GRAYSCALE);
	int numPoints = 0;
	for (int x = 0; x < 640; x += pointSkip * 5) {
		for (int y = 0; y < 480; y += pointSkip * 5) {
			int pIndex = x + 640 * y;

			if (blob.getPixels()[pIndex] > 0) {
				ofVec3f wc = kinect.getWorldCoordinateAt(x, y);
				wc.x = x - 320.0;
				wc.y = y - 240.0;
				if (abs(wc.z) > 100 && abs(wc.z) < 2000) {
					wc.z = -wc.z;
					wc.x += ofSignedNoise(wc.x, wc.z)*noiseAmount;
					wc.y += ofSignedNoise(wc.y, wc.z)*noiseAmount;
					wc.x = ofClamp(wc.x, -320, 320);
					wc.y = ofClamp(wc.y, -240, 240);
					del.addPoint(wc);
				}
				numPoints++;
			}
		}
	}
	if (numPoints > 0)
		del.triangulate();
	for (int i = 0; i < del.triangleMesh.getNumVertices(); i++) {
		del.triangleMesh.addColor(ofColor(255, 255, 255));
	}
	for (int i = 0; i < del.triangleMesh.getNumIndices() / 3; i += 1) {
		ofVec3f v = del.triangleMesh.getVertex(del.triangleMesh.getIndex(i * 3));
		v.x = ofClamp(v.x, -319, 319);
		v.y = ofClamp(v.y, -239, 239);
		ofColor c = kinect.getColorAt(v.x + 320.0, v.y + 240.0);
		if (!useRealColors)
			c = ofColor(255, 100, 100);
		c.a = colorAlpha;
		del.triangleMesh.setColor(del.triangleMesh.getIndex(i * 3), c);
		del.triangleMesh.setColor(del.triangleMesh.getIndex(i * 3 + 1), c);
		del.triangleMesh.setColor(del.triangleMesh.getIndex(i * 3 + 2), c);
	}
	convertedMesh.clear();
	wireframeMesh.clear();
	wireframeMesh.setMode(OF_PRIMITIVE_TRIANGLES);
	for (int i = 0; i < del.triangleMesh.getNumIndices() / 3; i += 1) {
		int indx1 = del.triangleMesh.getIndex(i * 3);
		ofVec3f p1 = del.triangleMesh.getVertex(indx1);
		int indx2 = del.triangleMesh.getIndex(i * 3 + 1);
		ofVec3f p2 = del.triangleMesh.getVertex(indx2);
		int indx3 = del.triangleMesh.getIndex(i * 3 + 2);
		ofVec3f p3 = del.triangleMesh.getVertex(indx3);
		ofVec3f triangleCenter = (p1 + p2 + p3) / 3.0;
		triangleCenter.x += 320;
		triangleCenter.y += 240;
		triangleCenter.x = floor(ofClamp(triangleCenter.x, 0, 640));
		triangleCenter.y = floor(ofClamp(triangleCenter.y, 0, 480));
		int pixIndex = triangleCenter.x + triangleCenter.y * 640;
		if (pix[pixIndex] > 0) {
			convertedMesh.addVertex(p1);
			convertedMesh.addColor(del.triangleMesh.getColor(indx1));
			convertedMesh.addVertex(p2);
			convertedMesh.addColor(del.triangleMesh.getColor(indx2));
			convertedMesh.addVertex(p3);
			convertedMesh.addColor(del.triangleMesh.getColor(indx3));
			wireframeMesh.addVertex(p1);
			wireframeMesh.addVertex(p2);
			wireframeMesh.addVertex(p3);
		}
	}
	delete pix;
	delete gpix;
}

}

#2

with ofxKinectForWindows2, just use
mesh = kinect.getDepthSource()->getMesh(true, ofxKinectForWindows2::Source::Depth::PointCloudOptions::TextureCoordinates::ColorCamera);

to get a mesh with texture coordinates in colour space
then if you really need to, you could lookup the colour values using those texture coordinates in the mesh, or just

kinect.getColorSource()->getTexture().bind()
mesh.draw();
kinect.getColorSource()->getTexture().unbind()

your naming might vary between 0.8.4 and 0.9.0

1 Like
#3

Thanks for your response! We got it up and working now but are now having trouble with way to many points being displayed. Is there a way to skip point in the draw() function within the mesh at all?

and we seem to have a scaling issues - we have to scale the whole image using ofScale(1000,1000,1000) to get it to work properly. Any idea why we have to do this or how to fix it?

Thanks

#4

We´ve made the Depth to show something useful by doing this:

ofImage image;
ofPixels_<unsigned short> pixels = this->kinect.getDepthSource()->getPixelsRef();
for (int i = 0; i < pixels.size(); i++){
	pixels[i] = 65536 - (pixels[i] * 65536 / 4096);
}
image.setFromPixels(pixels);
ofSetColor(255);
image.draw(0,0);

65536 = the range of short number
4096 = maximum value from kinect (not sure)

#5

Hi @vamoss. There’s a nice example of increasing the brightness of the depth image here:

https://github.com/rickbarraza/KinectV2_Lessons/tree/master/3_MakeRawDepthBrigther

#6

Hi @Mattfelsen,

your example is different because it replaces a processor operation by a memory operation.
Not sure if this it is faster.

But they are using the value 5001 instead 4096.
Do you know if that value is the official?

Thanks!

#7

After some empirical tests, the maximum value is 8000.
I suppose it ins in milimiters. The new kinect see 8meters :smile:

#8

Thanks for the follow-up @vamoss. That sounds about right, the SDK docs mention the pixel value is the distance for that pixel in mm:

DepthFrame Class
https://msdn.microsoft.com/en-us/library/windowspreview.kinect.depthframe.aspx

#9

@cbtriplec could you share the working piece of code with us? I have been trying to get the same code running and I am encountering exactly the same issues. It would be very helpful to see how you resolved this.

#10

Hi @mattfelsen . Barraza’s links seem no longer valid. I have looked at

Would you mind sharing the “KinectV2_Lessons”?, thx jjj

UPDATED: never mind, I found a copy here:

#11 
void generateDepthDisplayImage ();

in this project does not work.

unsigned char greyValue = depthLookupTable [rawdepth [rawindex ++]];

It stops at the line of.

%E3%82%AD%E3%83%A3%E3%83%97%E3%83%81%E3%83%A3
What should I do?