Translating Mouse Coords to Account For Z-Position of Clickable Object

Hi. I was up late last night searching the forums for an answer to this. I definitely found some very useful information, and I think I am on the right track, just need a bit more direction. Here’s my scenario:

I’m working on a game. There are spheres that get placed at random (within bounds) around the 3D space. I need to recognize when any given sphere has been clicked or touched. I always have access to the ofVec3f which is the center of the sphere, and am planning on just using an ofRectangle or ofPolyline also centered at this point, to determine if a click or touch is inside of it. I’ve realized that ofRectangle disregards Z position (as far as I know) so I’m using ofPolyline at the moment. My issue is that the Z position of all of these spheres is different, and the mouse obviously defaults to a 0 Z position. I can translate the mouse coords to the same Z position as the sphere, but that location and the actual cursor are not lined up on screen.

How can I have the mouse position translate to these different Z positions, but still keep them lined up on screen? (as if it were still just 2D).

I hope I’m phrasing this correctly, if not, read below and see the example. Thanks for any help, I really appreciate this forum.

I’ve created a very simple example to demonstrate what’s happening, it is attached here (XCode for Mac but src files should work for anyone). I have read some good posts about using ofMatrix4x4 to store the translation and rotation when drawing objects, then using that to translate the mouse points. In this simple example, I’m not doing any rotation, just placing a rectangle in space with a Z position. I think my situation is going to be all about vector math. The example has only 1 rectangle drawn on screen at a random position in space and a small blue dot that represents the mouse translated to the same Z position as the rectangle. You notice the small blue dot does not always match up with the actual cursor as the Z position gets farther away from 0. If you have the blue dot inside the rectangle and click, it registers. Great. BUT - I can’t have the actual click location be different than the point that is registering as inside. Make sense?

Let me know if I can provide more information. Thanks a ton for any help.

This is called mouse picking. There’s a good example of this in the examples of 0071, if you look at examples/3D/pointPickerExample or you can take a look at the approach here:

Hope that helps a little bit.

Thanks as usual Joshua. I’ve got some reading to do and will respond here with any further questions or findings.

I’ve read through a bunch of stuff on point picking and have been able to get some things working fairly well. For anyone else interested, this post by Arturo helped me the most:

There is a good example about halfway down the thread where he posted a couple of functions used to make it happen. I gathered from his post that he was following this: which was also a huge help in understanding what his code was doing.

Now another question: I got mostly what I wanted to happen setup in an app that I compile on my Mac. I was hoping to eventually bring this to iOS devices, but after copying my code into an oF iOS project, it appears some of the GL calls aren’t valid. Does anyone know if GLES 2.0 would support all that is necessary for point picking? Or have another solution that would work with the current implementation of GLES 1?

Thanks bunches for everyones help.

I think to do GLES picking you need to use the approach in the examples/3d/pointPicking as that isn’t reliant on the call to gluUnproject() (i might be wrong about that). There are also GLES float friendly implementations of gluUnproject() somewhere like here:

Hope that helps a bit, sorry the vagueness, I’m on my phone & so a bit limited.

Super, thanks again. I looked at the 0071 example first, and it seems one important function is ofEasyCam.worldToScreen(). I don’t have a camera in my scene and wasn’t planning on using one, but maybe I could just place it as if it were looking from the same perspective. I haven’t looked much deeper into that approach yet, I could find I dont’ even need that function. I’ll look at it today.

Thanks again.

Damn I am SO close to what I need. Using the ofCamera functions I am able to get a point that represents the center of a sphere relative to the screen coordinates. What I need to do now is have a way to determine if the mouse click is within a certain distance of that point. I am using ofPolyline and the .arc function to draw a circle around the center of that point which I will use to determine if the click is inside. Works great until I start rotating the matrix inside of camera.begin(). I’m saving that rotation to a ofMatrix4x4 that I can use to update the center of the sphere point, but the radius of the polyline arc needs to be variable based on the scale of the sphere since it changes size as it moves back of forth in Z space. I tried a few different things: ofMap converting the Z distance of that sphere to make the circle larger or smaller, also using another point that is the same distance away from the center of the sphere as the spheres radius, and converting that to screen coords. The later doesn’t work because when the sphere rotates those points get closer together on screen. Here’s what I have in my draw() function to attempt to debug:

ofMatrix4x4 mat;  
    ofRotate(rotAmt, 1, 1, 0);  
    mat.rotate(rotAmt, 1, 1, 0);  
    ofSetColor(0, 0, 0);  
    for(int i=0; i<pts.size(); i++) {  
        ofSphere(pts[i], 50);  
    ofVec2f mouse(mouseX, mouseY);  
    for(int i=0; i<pts.size(); i++) {  
		ofVec3f curCenter = easyCam.worldToScreen(pts[i]*mat);  
        //Another point that is sphere radius away from the center  
        ofVec3f curEdge = easyCam.worldToScreen((pts[i]+(ofVec3f(0, 50, 0)))*mat);  
        float dist = curCenter.distance(curEdge);  
        ofPolyline pl;  
        pl.arc(curCenter, dist, dist, 0, 360, 20);  
        ofSetColor(0, 255, 0);  
        ofCircle(curEdge, 5);  
        ofSetColor(0, 0, 255);  
        ofCircle(curCenter, 5);  
        ofSetColor(255, 0, 0);  

Any ideas on this one? ALMOST passed this point, thanks again for all help.

In case anybody cares at this point, I figured out what I had wrong. I was converting a point that was the sphere radius away from the center, and then rotating it according to my matrix rotation. When what i needed to do was convert the center of the sphere, after being rotated, then adding the sphere’s radius. I changed this:

ofVec3f curEdge = easyCam.worldToScreen((pts[i]+(ofVec3f(0, 50, 0)))*mat);  

To this:

ofVec3f curEdge = easyCam.worldToScreen(ofVec3f(pts[i]*mat)+ofVec3f(0, 50, 0));  

And it’s all good. Thanks everybody again for you’re help.

1 Like