How to add points in 3D space relative to screen and camera?


As a learning exercise to learn 3D transformations I want to know how to add points in 3D space, so I am asking for tips, suggestions, pseudo-code, any useful info will do… I will use this post to record my progress so others with the same questions can benefit and/or contribute

Couple of questions:
1- Is it best to rotate the scene or the camera?
2- I believe this next question depends on the first one but I will ask anyway, after the rotation is made how can I get the coordinates to add a new point that matches the mouse coordinates to the 3D space?


  • rS

Hey RS,

For doing 3D I always find moving the camera makes more sense then trying to move the world around the camera. This plays better with our physical intuition since we imagine ourselves moving through the world around us.

the upcoming 007 version of openFrameworks (currently in github) has some really nice camera functions too. including an ofEasyCam which understands mouse movement and gestures and lets you center and rotate around objects in space easily.

I’ve also created a camera that acts like FPS camera, or a camera from a space simulator game, called ofxGameCamera: which is more for moving through larger 3d worlds as opposed to centering around a particular object.

For finding mouse coordinates, this is also a lot easier. It’s done through a process called “unprojection” where the view of the camera is used to transform points to and from the camera’s 2d plane into 3d space. in ofCamera.h take a look at the following functions:

	ofVec3f worldToScreen(ofVec3f WorldXYZ, ofRectangle viewport = ofGetCurrentViewport());   
	ofVec3f screenToWorld(ofVec3f ScreenXYZ, ofRectangle viewport = ofGetCurrentViewport());  
	ofVec3f worldToCamera(ofVec3f WorldXYZ, ofRectangle viewport = ofGetCurrentViewport());  
	ofVec3f cameraToWorld(ofVec3f CameraXYZ, ofRectangle viewport = ofGetCurrentViewport());  

Hope this helps

Cool so ‘unprojection’ is the word, I am googling around to understand the basics, so far I found this

Once I digest the concepts will try to put it into practice and report back.

Thanks a lot

  • rS


Using ofEasyCam

// Setup Camera  
cam.setFov( 60 );  
cam.setNearClip( 0.1f );  
cam.setFarClip( 10000.0f );  
//cam.setPosition( ofVec3f( 0, 0, 500 ) );  
cam.cacheMatrices = true;  

Trying this

float x = touch.x;  
float y = touch.y;  
float z = cam.getImagePlaneDistance();  
ofVec3f loc = cam.screenToWorld( ofVec3f( x, y, z ) );  
cout << "loc: " << loc << endl;  
triangulator.addPoint( loc.x, loc.y, loc.z );  

but I get values this values:
loc: 1.17125e-05, 9.4209e-06, 886.81
loc: -3.05543e-06, 6.2127e-05, 886.81

Any ideas?

if I change the ‘z’ values to 1 I get better results, and I add points on my exact click positions but the distances are too big

loc: -5987.96, 146.286, -1216.61
loc: 989.867, 5783.16, -2126.02
loc: 375.467, 2106.51, -4837.18
loc: -4622.63, -3918.02, -585.143
loc: -5110.25, -70.7048, 2961.07

Need more tweaks

  • rS

so I have this code:

void testApp::mousePressed(int x, int y, int button){  
	if ( ofGetMousePressed(0) ) {  
		float posX = x;  
		float posY = y;  
		float posZ = .875f; //cam.getImagePlaneDistance();  
		ofVec3f sw = cam.screenToWorld( ofVec3f( posX, posY, posZ ) );  
		//cout << "loc: " << sw << endl;  
		//cout << "distance: " << cam.getDistance() << endl;  
		ofPoint p = ofPoint( sw.x, sw.y, sw.z );  
		points.push_back( p );  

Changing this variable gives different results, all points draw at the right location

float posZ = .875f;  

If I use 1 the z value of the resulting vector is too high, If I use <= 0.5 is too close, 0.875f works so far but If I change the far and near clip planes the resulting values are different, so I need to understand how to adjust the entry posZ for different near and far settings on the camera.

Any help will be much appreciated

  • rS


cam.setPosition( 0, 0, 110 );  
// and  
float posZ = 0.965f;  

Give me more accurate results, so if I click in the middle of the screen I get loc: 2.45421, -1.0821, 0.0796836 but I dont understand how to set that value, it most be something to do with the camera viewport aspect ratio or camera position, I dont know, please help me to understand this thing.


  • rS

Did you ever figure this out. I am at the same point.
I draw regular glVertex3f with locations like 0,0,20 and 0,10,20 etc.

Now I want my mouse location to be translated in to the 3d world. i use screentoworld. it draws a sphere sphere right where my mouse is but when i look at the values they don’t make sense in reference to the regular vertices i drew.

do i have to scale them ??
any idea?

If you take a look at the advanced3dExample in the example apps you can see how the screenToWorld usually gets used w/normalized z-coordinates, i.e. from 0.f - 1.f. Usually what you do is draw things in 3d space, have them projected in 2d space, and then use screenToWorld to figure out what objects are under the users mouse rather than where the users mouse is in 3d space. The idea of screenToWorld is to get a ray that you can use to test for intersections, i.e. determine whether a given point intersects with ray drawn through the scene where users mouse appears. There’s some more info on that here:

1 Like


I based everything so far on the advanced3dexample. i did not really understand what the updateMouseRay() function did.
but from your post i take it that by “projected in 2d space” you mean one of those orthocamera view?
i will look in to that.

thank you.


I’m not sure what it would mean to do mouse-picking (i.e. determining the mouse position in 3d space) outside of a cameras view, since you have to have some projection matrix to compare against (this is a pretty classic resource on how that works: though that’s kind of math heavy). At any rate, yes, the mouse picking just makes a ray through the scene at the mouse point. If you take a look at the code in the screenToWorld method you’ll see that it creates a vector that has its origin at the center of the screen (i.e. 0, 0, = center of the screen) and then multiplies that by the perspective transform that the camera is currently using. The values you get back are being modified by the same matrix that translates the points that you’re drawing from 3D space into a 2D plane. I’m not sure if that’s helpful or just more confusing :confused:

i took a different approche.
i assign a unique color to each 3d object.
on mousepressed i display this color.

With glReadPixels i get the color under the mouse location and comapre to my unique color list.

unsigned char pixelS[3];

thanks for your help anyway.

That’s actually one of the classic and easiest way to do picking, so that’s awesome! If you want to do it using multiple buffers, you can use the technique outlined in this tutorial (it’s a bit outdated, but it’s intuitive and easy to implement)