Trouble with GL_STENCIL_TEST / GLStencilFunc

For the past week or so I’ve been trying to use the OpenGL Stencil Buffer to mask out a video using a simple black and white png file. All the documentation I’ve managed to dig up so far is all dated from 2008, and I suspect a great deal has changed since then.

I’ve tried editing both the main.cpp and ofAppGlutWindow.cpp files, but it seems like no matter what I do I can’t get any kind of stenciling/masking to take place. I admit I don’t know a great deal about OpenGL, so I realize I’m probably missing something simple, but none of my searching has turned up anything.

Your help is greatly appreciated!

I’ve posted my code below. I should note that I’ve even tried creating two simple overlapping ofCirlce but even that doesn’t return the intended result.

//--------------------------------------------------------------
void testApp::setup(){

img.loadImage("media/raven0.png");
mov.loadMovie("media/background0.mov");
mov.play();

}

//--------------------------------------------------------------
void testApp::update(){

mov.update();

}

//--------------------------------------------------------------
void testApp::draw(){

glEnable(GL_STENCIL_TEST); //Enable using the stencil buffer
glColorMask(0, 0, 0, 0); //Disable drawing colors to the screen
glDisable(GL_DEPTH_TEST); //Disable depth testing
glStencilFunc(GL_ALWAYS, 1, 1); //Make the stencil test always pass
//Make pixels in the stencil buffer be set to 1 when the stencil test passes
glStencilOp(GL_KEEP, GL_KEEP, GL_REPLACE);
//Set all of the pixels covered by the floor to be 1 in the stencil buffer

img.draw(0,0);

glColorMask(1, 1, 1, 1); //Enable drawing colors to the screen
glEnable(GL_DEPTH_TEST); //Enable depth testing
//Make the stencil test pass only when the pixel is 1 in the stencil buffer
glStencilFunc(GL_EQUAL, 1, 1);
glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP); //Make the stencil buffer not change

mov.draw(0,0);

}

Your stencil code is fine if you weren’t using an image as a mask- eg. if you change img.draw() to an ofCircle() it should work. Also (with glfw) you need to add setStencilBits() to your main.cpp

ofAppGLFWWindow window;
window.setStencilBits(1);    
ofSetupOpenGL(&window, 1024,768,OF_WINDOW);
ofRunApp( new ofApp());

Generally you would use a frag shader for image masking (there’s an OF tutorial on github http://openframeworks.cc/tutorials/graphics/shaders.html#alpha_mask or an addon https://github.com/HeliosInteractive/ofxSimpleMask).

Or you could combine what you already have with some alpha masking, to make it work with textures - http://stackoverflow.com/questions/11964593/how-to-use-a-stencil-buffer-with-an-alpha-mask