ofxArkit - Convert 2d screen point with depth information to 3d world point

iOS
#1

I am trying to convert a 2d screen point with depth information from the camera to the Arkit world space. Like a point cloud for the depth image.

I have tried many approaches but I think I’m just thinking about it all wrong. I thought that I needed to unproject the point using the camera’s projection, model view. I followed this …

trying on unproject the 2d point + depth

camera.begin();
processor->setARCameraMatrices();
           
auto projectionMat = processor->camera->getCameraMatrices().cameraProjection;
auto viewMat = processor->camera->getCameraMatrices().cameraView;
            
auto screenSize = ofxARKit::common::getDeviceDimensions();

// map to screen size
auto screenX = ofMap(x, 0, depthImage.getWidth(), 0, screenSize.x);
auto screenY = ofMap(y, 0, depthImage.getHeight(), 0, screenSize.y);

// ... for loop per pixel
            
ofMatrix4x4 inverse;
inverse.makeInvertOf(projectionMat * viewMat);

int index = y * int(depthImage.getWidth()) + x;

float px = (2.0 * screenX ) / screenSize.x - 1.0;
float py = 1.0 - (2.0 * screenY) / screenSize.y;

float depth = depthData[index];

ofVec4f inPoint(px, py, depth * 2.0 - 1.0, 1.0);

ofVec4f position = inverse * inPoint;
position.w = 1.0 / position.w;


ofVec3f point;
point.x = position.x * position.w;
point.y = position.y * position.w;
point.z = position.z * position.w;

ofPushMatrix();
ofTranslate(point);
// draw point here
ofPopMatrix();

camera.end();

I then tried multiplying by the cameras transform, got some results, but still wrong.

camera.begin();
processor->setARCameraMatrices();
   
// normalize point
auto screenX = ofMap(x, 0, depthImage.getWidth(), 0, 1);
auto screenY = ofMap(y, 0, depthImage.getHeight(), 0, 1);

matrix_float4x4 translation = matrix_identity_float4x4;
translation.columns[3].x = screenX;
translation.columns[3].y = screenY;
translation.columns[3].z = -depth;
            
matrix_float4x4 transform = matrix_multiply(session.currentFrame.camera.transform, translation);

ofPushMatrix();
ofMatrix4x4 mat = convert<matrix_float4x4, ofMatrix4x4>(transform);
ofMultMatrix(mat);
ofPushMatrix();
// draw point here
ofPopMatrix();

camera.end();

Any thoughts?

#2

haven’t used arKit so not sure what info you get but glm unproject should work: https://openframeworks.cc/documentation/glm/gtc_matrix_transform/#!show_glm::unProject

#3

perhaps did you try this utility function in ofxARKit?