I am trying to convert a 2d screen point with depth information from the camera to the Arkit world space. Like a point cloud for the depth image.
I have tried many approaches but I think I’m just thinking about it all wrong. I thought that I needed to unproject the point using the camera’s projection, model view. I followed this …
trying on unproject the 2d point + depth
camera.begin();
processor->setARCameraMatrices();
auto projectionMat = processor->camera->getCameraMatrices().cameraProjection;
auto viewMat = processor->camera->getCameraMatrices().cameraView;
auto screenSize = ofxARKit::common::getDeviceDimensions();
// map to screen size
auto screenX = ofMap(x, 0, depthImage.getWidth(), 0, screenSize.x);
auto screenY = ofMap(y, 0, depthImage.getHeight(), 0, screenSize.y);
// ... for loop per pixel
ofMatrix4x4 inverse;
inverse.makeInvertOf(projectionMat * viewMat);
int index = y * int(depthImage.getWidth()) + x;
float px = (2.0 * screenX ) / screenSize.x - 1.0;
float py = 1.0 - (2.0 * screenY) / screenSize.y;
float depth = depthData[index];
ofVec4f inPoint(px, py, depth * 2.0 - 1.0, 1.0);
ofVec4f position = inverse * inPoint;
position.w = 1.0 / position.w;
ofVec3f point;
point.x = position.x * position.w;
point.y = position.y * position.w;
point.z = position.z * position.w;
ofPushMatrix();
ofTranslate(point);
// draw point here
ofPopMatrix();
camera.end();
I then tried multiplying by the cameras transform, got some results, but still wrong.
camera.begin();
processor->setARCameraMatrices();
// normalize point
auto screenX = ofMap(x, 0, depthImage.getWidth(), 0, 1);
auto screenY = ofMap(y, 0, depthImage.getHeight(), 0, 1);
matrix_float4x4 translation = matrix_identity_float4x4;
translation.columns[3].x = screenX;
translation.columns[3].y = screenY;
translation.columns[3].z = -depth;
matrix_float4x4 transform = matrix_multiply(session.currentFrame.camera.transform, translation);
ofPushMatrix();
ofMatrix4x4 mat = convert<matrix_float4x4, ofMatrix4x4>(transform);
ofMultMatrix(mat);
ofPushMatrix();
// draw point here
ofPopMatrix();
camera.end();
Any thoughts?