Moving ofFbo with ofEasyCam?

beginners
#1

I’ve been attempting for many days to implement selection (picking) of 2D rectangles in 3D space when ofEasyCam is active, such that I can detect a click on any rectangle regardless of the ofEasyCam’s view state.

One option I have been trying is colour/FBO picking.

To make this work, I am attempting to capture the location of a rectangle when a mouse click occurs, draw that into an FBO, and then get the pixel colour where the click occurred to see if I have selected a rectangle (out of a possible many).

So far, my code works when i do not move the camera; i.e. when i click on the rectangle the correct colour of that rectangle is detected. but when i move the camera and click on the rectangle, it is not detected; in fact, clicking in the original location of the rectangle still shows the same rectangle colour.

I suspect that even though the image of the FBO moves when i interact with the camera (pan / zoom / etc), i’m not actually moving it and am applying a transformation afterward, and somehow the mouse does not have the same transformation.

How can I fix this?

ofApp.h

#pragma once

#include "ofMain.h"

class ofApp : public ofBaseApp{
public:
    static constexpr int X = 22;
    static constexpr int Y = 18;
    static constexpr int sX = 60;
    static constexpr int sY = 88;
    void setup();
    void update();
    void draw();
    
    void mousePressed(int x, int y, int button);  
    
    ofEasyCam cam;
    
    ofFbo fbo;
    ofRectangle rectangle1;

    bool activateCam;

    int screenWidth;
    int screenHeight;

};

ofApp.cpp

#include "ofApp.h"

//--------------------------------------------------------------
void ofApp::setup(){
    rectangle1.set(X, Y, sX, sY);

    activateCam = false;

    screenWidth = ofGetWidth();
    screenHeight = ofGetHeight();

    fbo.allocate(screenWidth, screenHeight);
    fbo.begin();
    ofClear(255,255,255);
    fbo.end();
}

//--------------------------------------------------------------
void ofApp::update(){

}

//--------------------------------------------------------------
void ofApp::draw(){
    if ( activateCam) cam.begin();

    ofTranslate(-screenWidth/2, -screenHeight/2);

    fbo.draw(0, 0);
    if ( activateCam) cam.end();
}

void ofApp::mousePressed(int x, int y, int button){
    glm::vec3 mouse(x, y, 0.0f);

    fbo.begin();
    if ( activateCam) cam.begin();
    ofTranslate(0, screenHeight, 0);
    ofScale (1,-1,1);
    ofSetColor(0,0,1);
    ofDrawRectangle(rectangle1);
    ofPixels pixels;
    fbo.readToPixels(pixels);
    if ( activateCam) cam.end();
    fbo.end();

    cout << pixels.getColor(mouse.x, mouse.y) << endl;
}

//--------------------------------------------------------------
void ofApp::keyPressed(int key){
    if ( key == 't' )
    {
        activateCam = !activateCam;
    }
}