My lighting doesn't work after using ofLoadMatrix()

I’m calculating custom view and projection matrices using ofxOpenCv - calibrateCamera()

I found I can’t apply these directly to my camera (no way of setting the cam’s modelViewMatrix or projectionMatrix, although they can be read)

So I found a work-around - by setting the matrices directly in my draw call. I’m using:

glEnable(GL_DEPTH_TEST);
glEnable(GL_CULL_FACE);

//to flip vertical
ofSetOrientation(OF_ORIENTATION_DEFAULT, false);

ofSetMatrixMode(ofMatrixMode::OF_MATRIX_PROJECTION);
ofLoadMatrix(pmx);

ofSetMatrixMode(ofMatrixMode::OF_MATRIX_MODELVIEW);
ofLoadMatrix(mvmx);

This works and renders my models with the correct camera intrinsic/extrinsic properties such as position, orientation, perspective etc.

So I thought I was good to go but…

When setting these directly the lighting in my scene no longer works.
Lighting does work if I render my scene between calls to camera.begin() and camera.end();

I’m assuming this is to do with how lighting data is calculated in the programmable renderer, but to be honest I’m struggling to get my head round it all and may be totally off on that!

I think I have 2 choices here:

  1. Try to set my camera’s properties using the view and projection matrices returned from my call to calibrateCamera(), although they can’t be set directly as far as I can see and my attempts at setting the correct projection (intrinsic) properties from my matrix have so far come out wrong.

  2. Keep going with the direct calls to ofLoadMatrix() but figure how to get my lighting to work using that context/system (i.e. using gl context more directly rather than letting OF handle via camera - at the moment it breaks my lighting when trying to use directly)

Any pointers to help me get this project back on track would be amazing!

Cheers

Or does anyone know how to take a projection matrix and apply it directly to a camera?
It works when I use

ofSetMatrixMode(ofMatrixMode::OF_MATRIX_PROJECTION);
ofLoadMatrix(calibrator.projectionMatrix);

but I’m not sure how to take the values from my projectionMatrix and use them to calculate the correct fov and lensOffset values needed to add to the camera’s setupPerspective method.

I think I’m getting stuck because the difference between an OF camera and setting the ModelView and Projection matrices directly in OpenGL

Please correct me if I’m wrong and anyone out there knows more about this though?

Thanks

The problem is that the light calculation needs mainly 2 matrices to work, the model view matrix which you are passing correctly when using:

ofSetMatrixMode(ofMatrixMode::OF_MATRIX_MODELVIEW);
ofLoadMatrix(mvmx);

but also the view matrix which is the inverse of the position & rotation of the camera. the view matrix is set by OF internally every frame when it sets the screen perspective, when calling ofSetupPerspective or when binding a camera but there’s no way yet of setting it manually.

If you know the transformation of your camera, most probably is position 0 and no rotation, and then your model view is simply your model matrix and you can probably do the following:

ofCamera cam;
camera.begin();

ofSetMatrixMode(ofMatrixMode::OF_MATRIX_PROJECTION);
ofLoadMatrix(pmx);

ofSetMatrixMode(ofMatrixMode::OF_MATRIX_MODELVIEW);
ofLoadMatrix(mvmx);
camera.end();

binding the camera will set the view matrix and then you set the rest of the matrices as you did before. no need to set the orientation in this case as the camera already has non vflipped y.

if that doesn’t work perhaps your camera is not at the origin or it’s somehow rotated, if that’s the case you’ll need to set the ofCamera with the same transformation.

perhaps it would be useful to have an ofSetupPerspective(glm::mat4 projection, glm::mat4 view) for cases like this. if you want to open an issue in github?

Thanks @arturo !
As my camera is moved to a different position and orientation (with the outcome of the calibrateCamera which aligns it with a physical projector for projection mapping) I did have to set the ofCamera’s matrix to get the lighting to work.

I’m not sure why but I had to use the inverse of the modelViewMatrix ?

The following seems to wotk for me and allow correct lighting - as far as I can tell!

camera.begin();

camera.setTransformMatrix(calibrator.modelViewMatrix.getInverse());

ofSetMatrixMode(ofMatrixMode::OF_MATRIX_PROJECTION);
ofLoadMatrix(calibrator.projectionMatrix);

ofSetMatrixMode(ofMatrixMode::OF_MATRIX_MODELVIEW);
ofLoadMatrix(calibrator.modelViewMatrix);

scene.draw();

camera.end();

I’m quite new to OpenGL and ‘raw 3d programming’ so am just getting my head round these core principles, but yes an ofSetupPerspective(glm::mat4 projection, glm::mat4 view) would have come in handy so I’ll open it on github to see if it can be implemented.

Thanks!