How to drag a rectangle while using ofEasyCam?


I’m trying to figure out how to drag a rectangle around with the mouse, while ofEasyCam is on.

In the code below, the rectangle moves with the cursor just fine. However, when you zoom or pan/tilt around the scene with ofEasyCam, the rectangle no longer stays on the cursor while moving it. I think I understand why, but I don’t know how to fix it. I assume I’m applying the screen perspective to the rectangle cam’s world perspective?

Anyway, how can i make this work? (I need the rectangle to move around on its own plane)

Thanks in advance to whomever gives this a look-over. :slight_smile:

To try the code I’ve written below, pressing ‘c’ toggles ofEasyCam mouse control on or off, so that you can move the rectangle without affecting the camera, and vice versa.


#pragma once

#include "ofMain.h"

class ofApp : public ofBaseApp{

	static constexpr int X = 0;
	static constexpr int Y = 0;
	static constexpr int sX = 60;
	static constexpr int sY = 88;

	void setup();
	void update();
	void draw();

	void mouseDragged(int x, int y, int button);
	void mousePressed(int x, int y, int button);
	void keyPressed(int key);

	ofEasyCam cam;

	ofRectangle rectangle1;

	bool isMousePressingOnSelection = false;
	ofVec3f mousePressLocation;

	ofPolyline getPolyLineFromPanel(ofRectangle & rectangle);

	// inverse state with panel move mode.
	bool isCameraMoveMode = true;
	int screenWidth;
	int screenHeight;


#include "ofApp.h"

void ofApp::setup(){
    rectangle1.set(X, Y, sX, sY);
    screenWidth = ofGetWidth();
    screenHeight = ofGetHeight();

void ofApp::draw(){
    ofSetColor(0, 255, 0);

void ofApp::mousePressed(int x, int y, int button){
    if ( !isCameraMoveMode ) {
        ofPolyline p = getPolyLineFromPanel(rectangle1);

        if (p.inside(ofVec3f(x, y, 0))) {
            cout << "Clicked on rectangle" << endl;
            isMousePressingOnSelection = true;
            mousePressLocation.x = x;
            mousePressLocation.y = y;
        } else {
            isMousePressingOnSelection = false;

void ofApp::update() {
    if ( isCameraMoveMode ) {
    else {

void ofApp::mouseDragged(int x, int y, int button){
    if ( isMousePressingOnSelection ) {
        rectangle1.setPosition(glm::vec3(x - (screenWidth/2), screenHeight - y - (screenHeight/2), 0));

ofPolyline ofApp::getPolyLineFromPanel(ofRectangle & rectangle) {
    ofVec3f rectToScreen_botLeft = cam.worldToScreen(rectangle1.getBottomLeft());
    ofVec3f rectToScreen_botRight = cam.worldToScreen(rectangle1.getBottomRight());
    ofVec3f rectToScreen_topRight = cam.worldToScreen(rectangle1.getTopRight());
    ofVec3f rectToScreen_topLeft = cam.worldToScreen(rectangle1.getTopLeft());

    ofPolyline p;
    return p;

void ofApp::keyPressed(int key) {
    if ( key == 'c' ) {
        isCameraMoveMode = !isCameraMoveMode;


Hi so right, you can find if the mouse if over the rectangle by transforming its vertices using the camera’s matrix. But then if you want to move it you will need is to know how the mouse movement translates into the coordinates of the rectangle, which technically speaking is the opposite transformation, but in reallity you need a few more things. when you use the cam.screenToWorld(...), as you need to pass the x, y, and z coordinates to it. Right the screen is on the z = 0 plane but if you want to know about a particular object you need to provide this function with a z value. This is where the depth buffer comes into play.

The following snippet if from ofxGrabCam which you certainly need to check and go through its code in order to understand what are the class instance variables, (sorry I have not much time for answering and distilling it more)

void ofxGrabCam::findCursor() {
	GLint mouseViewportX = this->tracking.mouse.viewport.position.x;

	GLint mouseViewportY = this->view.viewport.height - 1 - this->tracking.mouse.viewport.position.y;

	const auto nearPlaneZ = (unsigned short) 32768;
	const auto farPlaneZ = (unsigned short) 65535;

	unsigned short z = farPlaneZ;

	//sampleRect will be in OpenGL's viewport coordinates
	sampleRect.x += mouseViewportX;
	sampleRect.y += mouseViewportY;
	auto cropRect = ofRectangle(0, 0, this->view.viewport.width, this->view.viewport.height);
	sampleRect = sampleRect.getIntersection(cropRect);

	//this should always be true since findCursor is only called whilst cursor is inside viewport
	if (sampleRect.width > 0 && sampleRect.height > 0) {
		//check if we need to reallocate our local buffer for depth pixels
		if (this->view.sampleNeighbourhood.getWidth() != sampleRect.getWidth() || this->view.sampleNeighbourhood.getHeight() != sampleRect.getHeight()) {
			this->view.sampleNeighbourhood.allocate(sampleRect.getWidth(), sampleRect.getHeight(), OF_IMAGE_GRAYSCALE);

		//sample depth pixels in the neighbourhood of the mouse
		glReadPixels(sampleRect.x, sampleRect.y, sampleRect.width, sampleRect.height, GL_DEPTH_COMPONENT, GL_UNSIGNED_SHORT, this->view.sampleNeighbourhood.getData());

		//pick the closest pixel to use as a sample
		for (auto & pixel : this->view.sampleNeighbourhood) {
			//check that it's valid before using it
			if (pixel != nearPlaneZ && pixel != farPlaneZ) {
				z = MIN(pixel, z);

	//check we're still looking at the near/far plane before updating the mouse distance
	if (z != nearPlaneZ && z != farPlaneZ) {
		this->tracking.mouse.projectedDepth = ((float)z / (float)USHRT_MAX) * 2.0f - 1.0f;
	//find mouse coordinates
	auto mouseProjected = glm::vec3(this->tracking.mouse.viewport.position.x,
	this-> = this->screenToWorld(mouseProjected);

Then, using this method to find the projected mouse once it is over the rectangle, you will be able to tell how the mouse has moved over the rectangles plane, thus moving it accordingly. You will need to store the previous projected mouse position and substract it to the current projected mouse position, then decompose this delta into its x , y and z components relative to the rectangle’s plane and use these to move the rectangle.
Hope this helps.