ofCamera.screenToWorld - understanding the meaning of screen z values

#1

Hi all,

I’m doing my first strides in a OF based project.

I’m simulating a map interface by drawing images on the z=0 plane.

I’m having problems with using ofCamera.screenToWorld to get points on the z=0 plane. The problem is that apparently I don’t properly understand the meaning of the screen z value.

I tried understand the problem by reading the usual materials here and there but to no avail. I’ll demonstrate my problems with examples.

My view size is 768x1024 (iPad).

I have an ofCamera “hanged above”, its values are:

position = 0,0,354.72402343750002
lookat = 0,0,0
fov = 60
nearClip = 8.86809921
farClip = 8868.0996

I found the following problems:

  1. If I put screen z = 0, I should be getting a point on the near clip, right? but:

    screenToWorld(384,512,0) gets me (0,0,337.005554)

but according to my calculations, I think it should be:

354.72402343750002 - 8.86809921 = 345.855924227

  1. If I put z = 0.5, I was expecting to be half the way between the near and the far plane:

    354.72402343750002 - (8868.0996 /2) = -4079.32577656

but:

screenToWorld(384,512,0.5) gets me (0,0,319.357758)

which I totally don’t understand.

BTW, the far plane calculation does sort of work for me:

screenToWorld(384,512,1) gets me (0,0,-8513.30664)

and 354.72402343750002 - 8868.0996 = -8513.37557656

(which is close enough??)

I tried calculating which screen z I need to give in order to get world z = 0 but to no avail. I resorted to “cheating”, by asking worldToScreen for the correct value:

worldToScreen (0,0,0) got me (384,512,0.95195198).

But I desperately want to understand how to calculate 0.95195198 myself.

Any help on the matter will be appreciated.

Thanks,

Tal

#2

The z value, represents the value in NDC space. But if you are trying to find points on the plane, use ray casting.Try this code. Even if you rotate the cam, you will get the right coordinates on the plane.

ofRay.h

#pragma once

#include "ofMain.h"


class ofRay {
public:
    ofRay() {}
    ofRay( const ofVec3f &aOrigin, const ofVec3f &aDirection ) : mOrigin( aOrigin ) { setDirection( aDirection ); }
    
    void            setOrigin( const ofVec3f &aOrigin ) { mOrigin = aOrigin; }
    const ofVec3f&    getOrigin() const { return mOrigin; }
    
    void setDirection( const ofVec3f &aDirection ) {
        mDirection = aDirection;
        mInvDirection = ofVec3f( 1.0f / mDirection.x, 1.0f / mDirection.y, 1.0f / mDirection.z );
        mSignX = ( mDirection.x < 0.0f ) ? 1 : 0;
        mSignY = ( mDirection.y < 0.0f ) ? 1 : 0;
        mSignZ = ( mDirection.z < 0.0f ) ? 1 : 0;
    }
    const ofVec3f&    getDirection() const { return mDirection; }
    const ofVec3f&    getInverseDirection() const { return mInvDirection; }
    
    char    getSignX() const { return mSignX; }
    char    getSignY() const { return mSignY; }
    char    getSignZ() const { return mSignZ; }
    
    ofVec3f calcPosition( float t ) const { return mOrigin + mDirection * t; }
    
    bool calcTriangleIntersection( const ofVec3f &vert0, const ofVec3f &vert1, const ofVec3f &vert2, float *result ) const;
    bool calcPlaneIntersection( const ofVec3f &origin, const ofVec3f &normal, float *result ) const;
    
    friend class ofMeshFace;
    
protected:
    ofVec3f    mOrigin;
    ofVec3f    mDirection;
    // these are helpful to certain ray intersection algorithms
    char    mSignX, mSignY, mSignZ;
    ofVec3f    mInvDirection;
};

ofRay.cpp

#include "ofRay.h"

// algorithm from "Fast, Minimum Storage Ray-Triangle Intersection"
bool ofRay::calcTriangleIntersection( const ofVec3f &vert0, const ofVec3f &vert1, const ofVec3f &vert2, float *result ) const
{
    
    ofVec3f edge1, edge2, tvec, pvec, qvec;
    float det;
    float u, v;
    const float EPSILON = 0.000001f;
    
    edge1 = vert1 - vert0;
    edge2 = vert2 - vert0;
    
    pvec = getDirection().getCrossed( edge2 );
    det = edge1.dot( pvec );
    
#if 0 // we don't want to backface cull
    if ( det < EPSILON )
        return false;
    tvec = getOrigin() - vert0;
    
    u = tvec.dot( pvec );
    if ( ( u < 0.0f ) || ( u > det ) )
        return false;
    
    qvec = tvec.getCrossed(edge1);
    v = getDirection().dot( qvec );
    if ( v < 0.0f || u + v > det )
        return false;
    
    *result = edge2.dot( qvec ) / det;
    return true;
#else
    if( det > -EPSILON && det < EPSILON )
        return false;
    
    float inv_det = 1.0f / det;
    tvec = getOrigin() - vert0;
    u = tvec.dot( pvec ) * inv_det;
    if( u < 0.0f || u > 1.0f )
        return false;
    
    qvec = tvec.getCrossed(edge1);
    
    v = getDirection().dot( qvec ) * inv_det;
    if( v < 0.0f || u + v > 1.0f )
        return 0;
    
    *result = edge2.dot( qvec ) * inv_det;
    return true;
#endif
}

bool ofRay::calcPlaneIntersection( const ofVec3f &planeOrigin, const ofVec3f &planeNormal, float *result ) const
{
    float denom = planeNormal.dot(getDirection());
    
    if(denom != 0.0f){
        *result = planeNormal.dot(planeOrigin - getOrigin()) / denom;
        return true;
    }
    return false;
}

ofApp.h

#pragma once

#include "ofMain.h"

class ofApp : public ofBaseApp{
    
public:
    void setup();
    void update();
    void draw();
    
    void keyPressed  (int key);
    void keyReleased(int key);
    void mouseMoved(int x, int y );
    void mouseDragged(int x, int y, int button);
    void mousePressed(int x, int y, int button);
    void mouseReleased(int x, int y, int button);
    void windowResized(int w, int h);
    void dragEvent(ofDragInfo dragInfo);
    void gotMessage(ofMessage msg);
    
    
    ofPlanePrimitive plane;
    ofEasyCam cam;
    
    
};

ofApp.cpp

void ofApp::setup(){
    plane.set(500, 500);
}

void ofApp::draw() {
    cam.begin();
    plane.draw();
    cam.end();
}

void ofApp::mousePressed(int x, int y, int button){
    ofVec3f screenToWorld = cam.screenToWorld(ofVec3f(x,y,0.0));
    ofRay ray(cam.getPosition(),screenToWorld - cam.getPosition());
    bool intersection = false;
    float t = 0;
    intersection = ray.calcPlaneIntersection(ofVec3f(0,0,0), ofVec3f(0,0,1), &t);
    if (intersection) {
        ofLog() << "Intersection Point(in world space) = " << ray.calcPosition(t);
    }

}
#3

Very nice. Thanks for the info, @Ahbee.

After doing some more consulting, I decided to stay with “the trick” I found, and doing worldToScreen on z=0, to get the screenZ I need, instead of trying to calculate it.

Your code looks really useful for a general case of finding where any screen point is on any infinite plane and IMHO it should be part of the ofCamera class.

But for my trivial case, if I have the following constraints:

  1. the camera always looking directly at the z=0 plane.
  2. I only need points that are on the z=0 plane.

You see any disadvantage of using worldToScreen instead of your code?

Thanks again,

Tal

#4

Do you mean like this?

ofVec3f worldtoScreen = cam.worldToScreen(ofVec3f(0,0,0));
ofVec3f screenToWorld = cam.screenToWorld(ofVec3f(x,y,worldtoScreen.z));

I dont think it is correct. I think this is only correct if your camera is orthographic and you are directly in front of the z = 0 plane looking at the center. If you change the viewing angle it will stop working.

#5

My camera is not orthographic, but I am indeed looking directly at the z=0 plane. It seems to work even if I move the camera, but even if that wouldn’t work, I know the x,y the camera is in, so I can query about that x,y and instead of 0,0.

#6

When I run this, I get intersection==true at every point i click; on or off the plane. :frowning: Running oF 0.10.0. (Trying to get object picking to work on 2d rectangles in 3d space via ofEasyCam)

#7

for object picking take a look at the pointPickExample in the 3d folder, pretty much you cycle thorugh all the objects and use worldToScreen to convert them to screen coordinates then check if the mouse is inside those coordinates

#8

Thank you!

The challenge I have there is that the rectangles become quadrilaterals with edges of any length when using ofEasyCam to pan/tilt/etc., so I don’t know how to see if the mouse is inside the coordinates

I googled it and discovered the even/odd algorithm. I implemented this using an online guide, but every thing I clicked on registered a pick even when I clicked on nothing. I’m sure I have a misunderstanding about how screenToWorld works, as well as rays.

I also tried using the colour/FBO picking method, but this also led to the FBO never lining up with the camera’s coordinates. Again, a result of my misunderstanding.

Is there any other helper method in oF that can assist in picking a 2d rectangle in a 3d world while using ofEasyCam?

EDIT: I figured it out by calling cam.worldToScreen for every corner of the rectangle, then creating an ofPolyline with the four "worldToScreen"ed corners of the rectangle, and then calling .inside(mouseCoords) on the path. It’s not ideal; as it does not deal with rectangles in different z planes, but it’ll do for now.

#9

you can just convert the rectangle min and max points to screen coordinates and then check if the mouse is inside that rectangle

#10

you can just convert the rectangle min and max points to screen coordinates and then check if the mouse is inside that rectangle

It’s very possible i am misunderstanding, but my challenge was that the rectangle was being viewed as a non-rectangular quadrilateral because of the view perspective applied by ofEasyCam - and therefore i couldn’t find a simple of way of checking whether a mouse click was inside that quadrilateral. In other words, the shape primitive is an ofRectangle, but when you look around it with ofEasyCam, cam.screenToWorld returns coordinates of a non-rectangle (i.e. skewed perspective on it). I don’t know how to convert that back into a rectangle.

For now, I call cam.worldToScreen for every corner of the quadrilateral, then creating an ofPolyline with the four "worldToScreen"ed corners of the rectangle, and then calling .inside(mouseCoords) on the path.

If there is a simpler way, I’m interested to learn about it. Thanks for your help!!

#11

ah ok yeah you are right i was thinking that the view was always perpendicular to the rectangles. using a polyline in that case is the simple solution i think.

to solve the z planes issue you commented before you can sort the rectangles by distance to the camera which so the first collision is always the right one