'3D Perspective: The Parallax Approach' ... Head tracking

Hi,

I’m trying to understand how to achieve this head tracking trick, also called ‘3D Perspective: The Parallax Approach’ by Apple.
The first example I saw dates from years and was made by Johnny Chung Lee:
http://www.youtube.com/watch?feature=player-embedded&v=Jd3-eiid-Uw

My issue is not the head tracking part, but the openGL camera / matrices transform / perspective part. In fact I’m trying to figure out how to obtains the correct ‘view’ on to the scene. Is that only done with the placement of the virtual camera, and some adjustments of its properties? Or does the actual scene needs to receive some transformation/deformations too?

Here are some screenshots and screen captures making easier for to show my issue.

Screen captures from the demo by J.C. Lee:

Here is my prototype using simple camera translation and rotation. But as you can see the perspective/transformation is incorrect. I mean for example the front edges of the white cube is not staying flush/snapped to the edges of the windows:



Other references:

http://www.youtube.com/watch?feature=player-embedded&v=1dnMsmajogA
http://www.patentlyapple.com/patently-apple/2009/12/apple-preparing-os-x-for-new-high-end-3d-interface.html
http://www.patentlyapple.com/.a/6a0120a5580826970c0128765fda33970c-pi

Any advice on how to approach this would be great,

Thanks.

Thanks to @kode80 I now have a better idea of and wording for what I’m trying to do:
"
create an off-axis perspective projection, then offset the camera slightly in the opposite direction to lock the near plane
."
(from there I also found that post of the forum - http://forum.openframeworks.cc/t/ofcamera-007-off-center-perspective-projections/6003/2 )

Here some code he shared that I have implemented in an OF app. It’s definitively going in to the good direction and I have now a better understanding of what is going on. But I’m still unable to really get my head around how to set all this properly (ie. to keep the white cube (in front) snapped to the edges of the windows).

Here is a screen capture of the following code running. I use the mouse to simulate the user’s head movement.
http://smallfly.com/of/off-axis/off-axis.mov

The questions/issues I’m having for now:

  • not sure why I need to have a negative ‘zOffset’ (used with ofTranslate) in order to place the cubes.
  • how to set the ‘fov’

Any thoughts on what is going on? Thanks.

  
  
#include "testApp.h"  
  
//--------------------------------------------------------------  
void testApp::setup(){  
	headX = 0;  
	headY = 0;  
    headZ = 0;  
  
    near = .1f;  
    far = 10000.0f;  
  
    fov = 1.5f;  
      
    roomSize = 800.0f;  
      
    zOffset = -660.0f;  
}  
  
//--------------------------------------------------------------  
void testApp::update(){  
  
}  
  
//--------------------------------------------------------------  
void testApp::draw(){      
    offAxis();  
    drawSteps();  
    drawCube();  
  
}  
  
//--------------------------------------------------------------  
void testApp::offAxis() {   
    float headDisplacementFactor = 1.0f;  
    headX = (mouseX - ofGetWidth()/2.0f) / ofGetWidth() * headDisplacementFactor;  
    headY = (mouseY - ofGetHeight()/2.0f) / ofGetHeight() * headDisplacementFactor;  
      
    float ratio = ofGetWidth() / ofGetHeight();  
      
    glMatrixMode(GL_PROJECTION);  
    glLoadIdentity();  
      
    // CREATE OFF-AXIS PROJECTION MATRIX  
    glFrustum(near*(-fov * ratio + headX),  
               near*(fov * ratio + headX),  
               near*(-fov + headY),  
               near*(fov + headY),  
               near, far);  
      
    glMatrixMode(GL_MODELVIEW);  
    glLoadIdentity();  
      
    // CREATE OFFSET MODELVIEW MATRIX  
    float mag = -80.0f;  
    float sx = mag/ratio;  
    float sy = mag/ratio;  
    gluLookAt( headX * sx, headY * sy, 0, headX * sx, headY * sy, -20, 0, 1, 0);  
}  
  
//--------------------------------------------------------------  
void testApp::drawCube() {  
    ofBackground(0);  
      
    ofPushMatrix();  
    {  
        ofSetColor(255.0f , .0f, .0f);  
        ofTranslate(0,0,zOffset);  
        ofNoFill();  
        ofSetColor(255.0f);  
        ofBox(roomSize);  
    }  
    ofPopMatrix();  
  
    ofPushMatrix();  
    {  
        ofTranslate(.0f, .0f, (-roomSize + zOffset));  
        ofNoFill();  
        ofSetColor(255.0f, .0f, .0f);  
        ofBox(roomSize);  
    }  
    ofPopMatrix();  
      
      
    ofPushMatrix();  
    {  
        ofTranslate(.0f, .0f, (-roomSize/4.0f));  
        ofSetColor(.0f, .0f, 255.0f, 255.0f);  
        ofFill();  
        ofBox(50);  
        ofNoFill();  
        ofSetColor(.0f);  
        ofBox(50);  
    }  
    ofPopMatrix();  
}  
  
//--------------------------------------------------------------  
void testApp::drawSteps() {  
    int nbOfSteps = 10;  
    float frameW = roomSize;  
    float frameH = roomSize;  
      
    ofPushMatrix();  
    {  
        ofTranslate(-ofGetWidth()/2.0f, -ofGetHeight()/2.0f);  
          
        ofPushStyle();  
        ofSetColor(255.0f, .0f, 255.0f);  
        for (int i=0; i < nbOfSteps; i++) {  
            ofPushMatrix();  
            {  
                ofTranslate(.0f, .0f, i * -100.0f);  
                ofLine(.0f, .0f, frameW, .0f);  
                ofLine(frameW, .0f, frameW, frameH);  
                ofLine(frameW, frameH, .0f, frameH);  
                ofLine(.0f, frameH, .0f, .0f);  
            }  
            ofPopMatrix();  
        }  
        ofPopStyle();          
    }  
    ofPopMatrix();  
}  
  

Just added a UI (to the same code of the previous post) to be able to tweak the different variables live. I have not found the perfect settings but getting there.
See a quick screen capture - http://smallfly.com/of/off-axis/off-axis-01.mov

But I still did not get my head around the fov, etc
. everything is well to deformed/stretched.

**** Update ************************* **
Could it be that when using glFrustum the fov needs to be in radians?
I took this in account and I’m getting way better results. Here is an other screen capture - http://smallfly.com/of/off-axis/off-axis-02.mov

But everything is still to stretched on the z-axis. See how the white cube (in front) is really stretched compared to the red cube (in the back). Both are the same size - ofBox(800.0f);

@smallfly thanks for sharing, i have been trying to do the same without any luck

I created a 3D app for iOS called HoloToy that uses device orientation or camera based head-tracking to achieve exactly this effect. I posted a WebGL example that should help anyone interested in recreating the effect. It’s very simple, just a few lines of code to create the appropriate perspective/modelview matrices.

WebGL demo: http://kode80.com/evu

I think I finally got it. At least now the values of the different properties make more sense to me.
Thanks again to kode80!

Here is a screen capture - http://smallfly.com/of/off-axis/off-axis-03.mov - and a screenshot

got it running with ofKinect to do the head tracking.
http://smallfly.com/of/off-axis/off-axis-04.MOV

Hey Great! sorry for the lack of reply, I’ve been practically offline.

Elliot and I are adding this to ofCamera. It’s gonna be great and very easy to use. (using minimal code duplication and delegating all functionality very modularly). Hopefully online (in our repos) within a day or two.

Nice! Thanks.

This is super-fascinating, I’m really curious how this’ll be implemented in ofCamera because I’ve been playing around with some similar things extremely unsuccessfully in the past few weeks :slight_smile:

Hey, I’m quite happy with the way we’re doing it. At the core of it (low-level), the camera has a setLensOffset(ofVec2f offset). Which slides the center. This is similar to how projectors can do lens shift (or in 3D software, you can give your camera a lens offset). This is essentially what an off-axis projection is. And it is simply a translate in projection space after the projection matrix has been applied. So we’ve added that to ofCamera. (ofCamera::begin() now also takes lensoffset into consideration). If you know the lens offset of your camera you can enter it directly. Then as a bonus utility helper method, we’ve added the ability to set the 3 corners of the projection surface in world coordinates. This is the method you’re more likely to use if you want to do head tracking etc or fake perspective (projection mapping). You simply enter the 3 corner values and the function merely calculates the lens offset required for that projection surface. Then ofCamera::begin() uses the lens offset as calculated by the helper function and bobs your uncle!

Hi Memo. Have you pushed that Mega Super Awesome version of ofCamera somewhere? I would be curious to look at the way it works.

Thanks,

Hugues.

Hi smallfly,

would you be posting your latest source code on this?

Here is a quick video of what we did.

@Progen - I don’t have time right now to push the code on GitHub, but I will do soon. In the meantime you have everything in this post. What @kode80 shared is almost exactly what we used.

http://vimeo.com/42804183

Here are an example (VS2010 - OF007) with the code from this thread
I think that is working fine. I just put some pieces together
sorry for my poor english
thanks to kode80 and smallfly and all the OF comunity

ParallaxApproach.zip

Hi everybody,

The code published here was very helpful, but when I tried to manipulate the z axis, this doesn’t work very well, so I started thinking in terms of a real camera and a real cube and found a tweak to the code published here that worked for me.

  
  
#include "testApp.h"  
  
//--------------------------------------------------------------  
void testApp::setup(){  
  cameraXY = ofPoint(0, 0);  
  cameraZ = 1;  
  
  // --- GUI ---  
  gui = new ofxUICanvas(ofGetWidth() - 310, 0, 310, ofGetHeight());  
  xyPosition = gui->add2DPad("XY Position", ofPoint(-2, 2), ofPoint(-2, 2), &cameraXY, 100, 100);  
  xyPosition->setIncrement(0.1);  
  zPosition = gui->addSlider("Depth", 0.8, 4.0, &cameraZ, 300, 16);  
}  
  
//--------------------------------------------------------------  
void testApp::update(){  
  
}  
  
//--------------------------------------------------------------  
void testApp::draw(){  
  ofBackground(0);  
  beginProjection();  
  drawBoxes();    
  endProjection();  
}  
  
//--------------------------------------------------------------  
void testApp::drawBoxes() {  
  ofNoFill();  
  ofSetColor(255, 0, 0);  
  ofBox(0, 0, -0.51, 1);  
  ofSetColor(255);  
  ofBox(0, 0, -0.5, 1);  
  ofBox(0, 0, -0.5, 0.2);  
  ofBox(-0.4, -0.4, -0.5, 0.2);  
}  
  
//--------------------------------------------------------------  
void testApp::beginProjection() {  
  glMatrixMode(GL_PROJECTION);    
  glPushMatrix();    
  glMatrixMode(GL_MODELVIEW);    
  glPushMatrix();   
  
  double fFar = 1000.0f;  
  
  float x = cameraXY.x;  
  float y = cameraXY.y;  
  
  float left = -x - 0.5;  
  float right = -x + 0.5;  
  float top = -y - 0.5;  
  float bottom = -y + 0.5;  
  
  glMatrixMode(GL_PROJECTION);    
  glLoadIdentity();    
  glFrustum(left, right, top, bottom, cameraZ, fFar);  
  
  glMatrixMode(GL_MODELVIEW);    
  glLoadIdentity();    
  gluLookAt(x, y, cameraZ, x, y, -1, 0, 1, 0);  
}  
  
//--------------------------------------------------------------  
void testApp::endProjection() {  
  glMatrixMode(GL_PROJECTION);    
  glPopMatrix();    
  glMatrixMode(GL_MODELVIEW);    
  glPopMatrix();   
}  
  
void testApp::exit()  
{  
    delete gui;   
}  
  

There is a problem on my previous code. If I draw an object between the camera and the screen (the positive z axis) they will not been displayed, because they will be drawn between the camera and the distance set on the near parameter of glFrustrum. The solution is to set a small value on the near parameter and calculate the left, right, top and bottom parameters based on the near parameter. This is the segment of updated code:

  
  
void testApp::beginProjection() {  
  glMatrixMode(GL_PROJECTION);    
  glPushMatrix();    
  glMatrixMode(GL_MODELVIEW);    
  glPushMatrix();   
  
  float x = cameraXY.x;  
  float y = cameraXY.y;  
  
  float fNear = 0.1f;  
  float fFar = 1000.0f;  
  float left = (-x - 0.5) * fNear / cameraZ;  
  float right = (-x + 0.5) * fNear / cameraZ;  
  float top = (-y - 0.5) * fNear / cameraZ;  
  float bottom = (-y + 0.5) * fNear / cameraZ;  
  
  glMatrixMode(GL_PROJECTION);    
  glLoadIdentity();    
  glFrustum(left, right, top, bottom, fNear, fFar);  
  
  glMatrixMode(GL_MODELVIEW);    
  glLoadIdentity();    
  gluLookAt(x, y, cameraZ, x, y, -1, 0, 1, 0);  
}  
  

Hi Memo (or anyone else)

Could you elaborate on how to use setLensOffset or the new method setupOffAxisViewPortal to create an off-axis ofCamera?

I’ve been playing with the example and these two function but I can’t see any results.

Thanks for the work on this. I’ve tried this in other systems, and it’s quite awkward.

I have a slightly more advanced case I want to get my head around: I want to project into an actual box, and then have the viewpoint move around. Obviously, the projector - > box relationship is fixed, so instead of one plane being static, as with a simulated 3D space, the entire box needs to be fixed, but the ‘contents’ inside to change as the viewpoint changes.

I can conceive of accomplishing it in multiple passes: one to paint the box, and one to paint the contents: using OpenGL depth testing to mesh the two together, but is there a mathematical approach as well?

Bruce

I’ve posted a small example here: Using setupOffAxisViewPort with ofxMostPixelsEver if anyone is still looking at this, I am not 100% sure this is the best way to demo this functionality off but I’ve noticed some folks asking about this feature.

1 Like