Making video player clickable

I have made a custom video player for myself by extending the ofNode. So, the new VideoPlayer now has the lookAt and rotation functions.
The video player is drawn by overriding the ofNode’s customDraw virtual function.

The video now have been translated and rotated in 3D space around looking at an origin and I am currently looking to add events to these video players. To add GUI events to a simple object, I looked into some addons like ofxMSAInteracitve object. I had already done a similar thing for my Image button control but ofxMSAInteractiveObject looked to work for rectangle shapes only (since it inherits ofRectangle)

Here’s the problem I am facing currently:
I have an_ EasyCam_ inside which all the custom video players are there which are rotated with the camera.
On changing/ the values of the camera, I was expecting the ofNode (i.e. custom video player’s getPosition()) coordinates to change, which I could have got by getPosition(). (let’s say coordinates fetched are tx,ty)

Once, I would have got the coordinates, I intended to do the hit-testing by something like:

if ( (tx < clickPositionX) && (tx +width > clickPositionX) && (ty < clickPositionY) && (ty +height > clickPositionY) )  
//then clicked inside the video rectangle  

But on moving the EasCam, the coordinates of the ofNode object (i.e the VideoPlayer) [b]do not change. Hence, I do not have the coordinates for hit-testing[/b]

Even if I get the coordinates of the object, another problem I am concerned about is **how accurate this clicking will be since the object is now having a 3D shape but we are trying to do hitTesting by 2d screen coordinates

All in all, if there’s a better way to add click events to the video player object itself, that would be most welcome
Other than that, if those events can’t be added easily somehow, it would be great how to suggest the hit Testing to figure out if the user has clicked inside the video rectangle 3d shape.

you can use worldToScreen from ofCamera or ofEasyCam to treansform a point in the worl from the point of view of the camera to screen coordinates, so you just need to transform the 4 vertices of the video, put them in an ofPolyline and then use the inside method to know if the mouse is inside your video

Thanks for the reply @arturoc.

Infact, this was exactly what I did in my program as well and this works very well for one of the vertices.
When I extended the ofNode to make this custom video player, I get the topLeft position something using the Node’s getPosition function. However, I am not sure if the VideoPlayer exposes anything that can give me the other three vertices.
Infact, that was the main concern of the question. If I am able to get those three vertices somehow, that would make life much easier, but the question remains how do I get the other three?

Here’s what I did in my class for the topLeft vertex:

class CustomPlayer : public ofNode  
ofVec3f getTopLeft()  
return getPosition();  
ofVec3f getTopRight()  
ofVec3f getBottomLeft()  
ofVec3f getBottomRight()  
ofVideoPlayer myPlayer;  

Check out the worldToScreen() method in ofCamera. That will allow you to find the current screen position of a 3D point at any time and from there you can figure out whether the mouse hit is within the video.

topright = topleft+player.getWidth()  
botomleft = topleft+player.getHeight()  
bottomright = topright+ player.getHeight()  

then use worldToScreen() + ofPolyline inside() to check the mouse is inside the rectangle

Hi there,

One way to do this is to use glunproject two times (setting z to 0 and 1) to create a ray that is emitted from your cursor and travels into the world space. Then you would do an intersection test between that ray and your video plane. Googling “ray picking gluunproject” will get a lot of suggestions.

Another way is to do a hit test slightly differently …

Here is a minimal example:

#pragma once  
#include "ofMain.h"  
class testApp : public ofBaseApp{  
    void setup() {  
        rect = ofRectangle(0,0,300,300);  
        // the "video" image can be represented as a rectangle as well.  
        // you could construct a rectangle from a video player by:  
        // rect = ofRectangle(0,0,player.getWidth(), player.getHeight());  
        // and then draw the video with  
        // player.draw(0,0) and make sure all of your translation / rotation  
        // is done with ofPushMatrix(), etc.  
    void draw() {  
        rotation += ofPoint(1,2,4);  
        ofVec3f mouse(mouseX,mouseY);  
        // we are now going to create a polygon with 4 vertices.  
        // each of the 4 vertices represents the projection from world  
        // space back to screen space of each of the rectangle's 4 corners.  
        // essentially we are projecting the 3d outline of the rectangle in world  
        // space back into 2d screen space.  since it is rotated in x/y/z it will  
        // not look like a rectanlge, but a funny polygon.    
        ofPolyline p;  
        // we then use the polygon's "inside" function to see if the mouse is inside  
        // the screen-mapped 2d polygon, which remember, is a projection of the  
        // 2d rectangle in 3d space.  
        if(p.inside(mouse)) {  
        } else {  
        // draw the rectangle with the appropriate colors  
    ofPoint rotation;  
    ofRectangle rect;  
    // this function is equivalent to the same function in ofCamera, but here we are using  
    // old-scool gluProject to get the projection of the points.  
    // as ofCamera demostrates, it is possible to get the current modelViewMatrix, projectionMatrix  
    // and viewport and then do the projection matrix math to get the same projected point.  
    ofVec3f ofWorldToScreen(ofVec3f world) {  
        GLdouble modelviewMatrix[16];  
        GLdouble projectionMatrix[16];  
        GLint viewport[4];  
        glGetDoublev(GL_MODELVIEW_MATRIX,  modelviewMatrix);  
        glGetDoublev(GL_PROJECTION_MATRIX, projectionMatrix);  
        glGetIntegerv(GL_VIEWPORT,         viewport);  
        GLdouble x, y, z;  
        gluProject(world.x, world.y, world.z, modelviewMatrix, projectionMatrix, viewport, &x, &y, &z);  
        ofVec3f screen(x, y, z);  
        screen.y = ofGetHeight() - screen.y;  
        return screen;  

Alas so much has happened on this thread since I was offline :wink:

Another alternative is to render an index buffer fbo, which is the same size as your output window, and is rendered from the same camera, but where you render a different colour for every object. Then by picking the colour under the cursor, you can identify the object.

This is what ofxGrabScene does for you, but that relies on an awkward branch of openFrameworks right now.
So alteratively you can use the legacy OpenGL Select Buffer (this doesn’t exist in OpenGL ES btw)

The four vertices (strings in white just above the sphere) for the video player that I get are not looking correct. Here’s the figure to show the same.

Only one of the points is coming out to be correct with the circle sitting at one of the vertices.
Here’s the code that I have used for the same:

CustomRectangle (Video Player):

class FollowerRectangle : public ofNode  
        FollowerRectangle() {  
            followerWidth = 200;  
            followerHeight = 100;  
            ofPoint somePoint;  
            _mouseDown = false;  
        virtual void customDraw()  
            myPlayer.width = 200;  
            myPlayer.height = 100;  
    virtual void onMousePress(int x, int y, int button)  
        cout<<"Mouse pressed inside\n";  
    virtual void onMousePressOutside(int x, int y, int button)  
        cout<<"Mouse pressed otuside\n";  
        ofVec3f getTopRight()  
            return (getTopLeft() + myPlayer.getWidth());  
        ofVec3f getTopLeft()  
            return getPosition();  
        ofVec3f getBottomLeft()  
            return (getTopLeft() + myPlayer.getHeight());  
        ofVec3f getBottomRight()  
            return (getTopRight() + myPlayer.getHeight());  
        ofEasyCam someCam;  
        ofVec3f originPoint;  
        float followerWidth;  
        float followerHeight;  
        ofVideoPlayer myPlayer;  
        int _mouseX, _mouseY, _mouseButton;  
        bool _mouseDown;  
        void enableMouseEvents()  
            ofAddListener(ofEvents().mousePressed, this, &FollowerRectangle::mousePressed);  
        void disbaleMouseEvents()  
            ofRemoveListener(ofEvents().mousePressed, this, &FollowerRectangle::mousePressed);  
        void mousePressed(ofMouseEventArgs &e)  
            int x = e.x;  
            int y = e.y;  
            int mouseButton = e.button;  
            cout<<"video pressed"<<"\n";  
            _mouseX = x;  
            _mouseY = y;  
            _mouseButton = mouseButton;  
                cout<<"Hit test true";  
                    cout<<"Event fired for inside";  
                    _mouseDown =  true;  
                    cout<<"Event fired for outside";  
        bool hitTest(int tx, int ty)  
            ofPolyline p;  
            p.addVertex(someCam.worldToScreen(getTopLeft()  , ofGetCurrentViewport()));  
            p.addVertex(someCam.worldToScreen(getTopRight()  , ofGetCurrentViewport()));  
            p.addVertex(someCam.worldToScreen(getBottomLeft()  , ofGetCurrentViewport()));  
            p.addVertex(someCam.worldToScreen(getBottomRight()  , ofGetCurrentViewport()));  
            if (p.inside(tx, ty)){  
                return true;  
            return false;  


//draw follower video rectangle;  
ofVec3f point1 = testEasyCam.worldToScreen(follower.getTopLeft(), ofGetCurrentViewport());  
ofVec3f point2 = testEasyCam.worldToScreen(follower.getTopRight(), ofGetCurrentViewport());  
ofVec3f point3 = testEasyCam.worldToScreen(follower.getBottomLeft(), ofGetCurrentViewport());  
ofVec3f point4 = testEasyCam.worldToScreen(follower.getBottomRight(), ofGetCurrentViewport());  
ofCircle(point1, 20);  
ofCircle(point2, 20);  
ofCircle(point3, 20);  
ofCircle(point4, 20);  

if you have any rotation or other transformations on the node, you need to apply those to the vertices of the video so summing the dimensions to the lefttop vertex is not enough:

topright = ofVec3f(video.getWidth(),0)*node.getGlobalTransformationMatrix();  
bottomright = ofVec3f(video.getWidth(),video.getHeight())*node.getGlobalTransformationMatrix();  
bottomleft = ofVec3f(0,video.getHeight())*node.getGlobalTransformationMatrix();  

also that’s the correct order to add the points into the polyline, the way you are adding them won’t give you a quad but kind of a cross.

Thanks @arturo

That certainly does the job. There’s some design issue I am concerned with, and wanted to know how I could rectify the same. The clicking is essentially the job of the VideoPlayer and all the operations such as hit testing etc. should also be inside the VideoPlayer itself so that the video player can be reused (and have the click operations incorporated already). Here’s what the concern is:


class CustomVideoPlayer{  
ofVec3f getTopLeft();  
ofVec3f getTopRight();  
ofVec3f getBottomRight();  
ofVec3f getBottomLeft();  
voidMousePressed(ofMouseEventArgs &e)  
//fire the function for click operation  
//click outside the video player  
bool hitTest(tx,ty){  
ofPolyline p;  
p.addVertex(getTopLeft()); //not a WorldToScreen coordinate there  
p.addVertex(getTopRight()); //not a WorldToScreen coordinate there  
p.addVertex(getBottomRight()); //not a WorldToScreen coordinate there  
p.addVertex(getBottomLeft()); //not a WorldToScreen coordinate there  
cout<<"Hit test true"; //Means click is inside  

The custom video player would be inside an ofEasyCam

//Video player inside this  

If I would have been doing the hit testing inside testApp, it works fine since I have the access to the testCam object, through which I can actually convert getTopLeft(), getTopRight() etc. to screen coordinates and compare if the mouse screen coordinates are inside the bounds.

Ideally, I should be doing hitTesting in the CustomVideoPlayer itself, where I won’t have the access to the testCam object (and so can’t convert getTopLeft(), getTopRight() etc. to appropriate screen coordinates). The same is my question for this case. How do I do hit testing with the by getTopLeft(), getTopRight() etc. vertices and mouse screen coordinates from inside the video player itself?

Question 2 (a little off-topic):
On another note, I had a question that I wanted to ask when I was trying out the above things. Let’s say I have an if condition which requires the video player’s positioning as per a certain criteria (let’s say at screen coordinates (20,20,0), no LookAt, no camera translations/transformations). Can I some how remove the object from the camera, do whatever I have to and when done, send it back to the camera?

ideally you can make a base class that does the 3d hitTest for drawable objects, everything that can be drawn in OF inherits from ofBaseDraws so you can make a class that takes an ofBaseDraws in its setup, draws it when you call it’s draw and does the hitTest.

to avoid the dependency with a specific camera you can pass a camera pointer in hitTest if it’s NULL then the hitTest only tests for the quad without transformations

When I inherit ofNode and create my custom object, the node is at the bottom topLeft vertex of the rectangle. Is there any way, I could bring this node vertex to the centre so that the video rectangle looks from the centre to the origin of the screen.