Using texture as alpha mask to apply transparency

hello,

i have read a lot of thread about this but i can’t make it to work !
i would like to stack 2 images and apply a mask to the top level one to see the bottom layer through it :

load 2 images
load a Grayscale image as a mask
apply mask to image1
draw image2
draw image1 to see image2 through the mask of image1

ofFbo imgFinal;

ofImage imgFront;
ofImage imgBack;
ofImage mask;

imgFront.load("1.jpg");
imgBack.load("2.jpg");
mask.load("mask.jpg");
    
imgFront.getTexture().setAlphaMask(mask.getTexture());

imgFinal.allocate(640, 480, GL_RGBA);
imgFinal.begin();
imgBack.draw(0, 0);
imgFront.draw(0, 0);
imgFinal.end();

// front

// back

// mask

// expected

// what i have

Hi! It seems like the mask must be partially transparent. If it’s black it’s not transparent :slight_smile: I think we were thinking too much like photoshop or gimp. Try using a PNG with transparent background.

ofFbo mask;
ofImage img;

void ofApp::setup() {
    img.load("1.jpg");
    mask.allocate(img.getWidth(), img.getHeight(), GL_RGBA); // <-- it can have transparency
    img.getTexture().setAlphaMask(mask.getTexture());
}

void ofApp::draw() {
    mask.begin();
    ofClear(0, 0, 0, 0); // <-- note the 0 alpha value
    ofSetColor(255);
    ofDrawCircle(150 + 50 * sin(ofGetElapsedTimef()), 150 + 50 * cos(ofGetElapsedTimef()), 50);
    mask.end();
    img.draw(0, 0);
} 

Hi. Thanks for your answer.

I have already tried a custom PNG with alpha as a mask and it works.

But i thought the purpose of setAlphaMask() was actually to use the colors of a texture as a mask and not the alpha, but maybe i am wrong. @arturo is this the expected behavior ?

If i am wrong, then how to manually add alpha (make black part of the mask transparent) after loading the mask image ?

thanks

This might not be the most efficient way, but seems to work:

ofImage img, mask;
void ofApp::setup() {
    img.load("1.jpg");
    mask.load("3.jpg");
    mask.setImageType(OF_IMAGE_COLOR_ALPHA);
    mask.getPixels().setChannel(3, mask.getPixels().getChannel(0));
    mask.update();
    img.getTexture().setAlphaMask(mask.getTexture());
}
void ofApp::draw() {
    img.draw(0, 0);
}

There I only copy the red channel to the alpha channel, but ideally you would also wipe the RGB channels with white to avoid visual glitches on the mask edges.

I see there’s a bunch of examples dealing with masks and shaders. I didn’t look at them, but with a shader it would be very fast to do this in real time (copying an RGB channel to the alpha one).

hello,

Ok this seems to work.

I also came up with this solution on my side

for(int y = 0; y < mask.getHeight(); y++) {
    for(int x = 0; x < mask.getWidth(); x++) {
        if (mask.getPixels().getColor(x, y).getBrightness() == 0) {
            mask.getPixels().setColor(x, y, ofColor(0, 0, 0, 0));
        }
    }
}
mask.update();

thanks a lot

yes the mask works as you describe, just allocate the mask as rgb or even luminance and the black parts will be transparent in the final image

sorry, i can’t make it to work like that

imgFront.load("1.jpg");
imgBack.load("2.jpg");

mask.allocate(640, 480, OF_IMAGE_COLOR);
mask.load("mask.jpg");

imgFront.getTexture().setAlphaMask(mask.getTexture());

imgFinal.allocate(640, 480, GL_RGBA);
imgFinal.begin();
imgBack.draw(0, 0);
imgFront.draw(0, 0);
imgFinal.end();

I need to force black transparency in the mask to make it work

mask.load("mask.jpg");
mask.setImageType(OF_IMAGE_COLOR_ALPHA);

for(int y = 0; y < mask.getHeight(); y++) {
    for(int x = 0; x < mask.getWidth(); x++) {
        if (mask.getPixels().getColor(x, y).getBrightness() == 0) {
            mask.getPixels().setColor(x, y, ofColor(0, 0, 0, 0));
        }
    }
}
mask.update();
imgFront.getTexture().setAlphaMask(mask.getTexture());

The comparison with 0 may work, but the mask edges may be rough, because it’s a threshold: either transparent or opaque.

@arturo In my tests black did not become transparent. As in @Gallo 's “what I have” image above, the white part of the mask shows the textures, but the black does not become transparent, but replaces the image texture with black.

tried with latest nightlies just in case. Under macOS Sierra and Debian. Same problem.

oh yeah sorry you are right the mask works for color form rgb and for alpha from alpha. you can remap the channels using:

mask.setSwizzle(GL_TEXTURE_SWIZZLE_A, GL_RED);

which will map the red channel to act as alpha so you will see white as alpha 1 and black as alpha 0. you’ll need a texture allocated with alpha in the first place for this to work

So this is equivalent to the following right ?

mask.getPixels().setChannel(3, mask.getPixels().getChannel(0));

by the way, i did

mask.getTexture().setSwizzle(GL_TEXTURE_SWIZZLE_A, GL_RED);

but didn’t work so far.

I will dig it a bit more tomorrow with a rest brain…

void ofApp::setup() {
    img.load("1.jpg");
    bg. load("2.jpg");
    mask.load("3.jpg");
    mask.setImageType(OF_IMAGE_COLOR_ALPHA);
    mask.getTexture().setSwizzle(GL_TEXTURE_SWIZZLE_A, GL_RED);    
    img.getTexture().setAlphaMask(mask.getTexture());
}
void ofApp::draw() {
    bg.draw(0, 0);
    img.draw(0, 0);
}

I added some blur to the mask to see if it was working right

A comment says swizzle doesn’t work on OpenGL ES. If it doesn’t work for you, is that what you’re using?

Yep, seems we are doing the same thing… but nope

//setup()
    imgFront.load("1.jpg");
    imgBack.load("2.jpg");
    mask.load("mask.jpg");
    mask.setImageType(OF_IMAGE_COLOR_ALPHA);
    mask.getTexture().setSwizzle(GL_TEXTURE_SWIZZLE_A, GL_RED);
    imgFront.getTexture().setAlphaMask(mask.getTexture());

//draw()
    imgBack.draw(0, 0);
    imgFront.draw(0, 0);

I am using oF 0.9.8 under macOS Sierra and XCode

What if you set the OpenGL version in main?

int main( ) {
  ofGLWindowSettings settings;
  settings.setGLVersion(3, 2); // try also 2, 1
  settings.width = 1024;
  settings.height = 768;
  settings.windowMode = OF_WINDOW;
  ofCreateWindow(settings);
  ofRunApp( new ofApp());
}

This one works

mask.getPixels().setChannel(3, mask.getPixels().getChannel(0));

this one doesn’t work

mask.getTexture().setSwizzle(GL_TEXTURE_SWIZZLE_A, GL_RED);

So i ended up using the technique i talked about earlier

mask.load("mask.jpg");
mask.setImageType(OF_IMAGE_COLOR_ALPHA);

for(int y = 0; y < mask.getHeight(); y++) {
    for(int x = 0; x < mask.getWidth(); x++) {
        if (mask.getPixels().getColor(x, y).getBrightness() == 0) {
            mask.getPixels().setColor(x, y, ofColor(0, 0, 0, 0));
        }
    }
}
mask.update();
imgFront.getTexture().setAlphaMask(mask.getTexture());

Thanks a lot to all

I’m happy that you found a solution!

A few things:

  • setSwizzle() takes 5 microseconds on my system, while the two for loops take around 20500 microseconds. Just in case you need to be fast.
  • If your mask is not pure black and white, but has gray values, it might look like this:

Which you can solve like this using your approach:

    for(int y = 0; y < mask.getHeight(); y++) {
        for(int x = 0; x < mask.getWidth(); x++) {
            mask.getPixels().setColor(x, y, ofColor(255, 255, 255, mask.getPixels().getColor(x, y).getBrightness()));            
        }
    }

Or at least I would not compare to 0 (jpg may have introduced artifacts, as you can see on the right side of the pentagon). Using .getBrightness() < 128 would be better, I think.

yes i know setSwizzle() is faster so i would prefer to use it. But unfortunately this doesn’t work for me for a reason i don’t get…

Though, i get my mask from a kinect depth image and i need to threshold it before using it so i am looping in every pixels anyway

Hi all,

is there any news on this topic? I have the exact same problem:

mask.load("masks/circleMask3.jpg");
mask.setImageType(OF_IMAGE_COLOR_ALPHA);
mask.getTexture().setSwizzle(GL_TEXTURE_SWIZZLE_A, GL_RED);
[...]
video.getTexture().setAlphaMask(mask.getTexture());

I expected: white regions of the mask are showing the video, grey regions are partially transparent, black regions are fully transparent.
But it leads to: white regions of the mask are showing the video, some grey regions are in fact partially transparent, but black regions of the mask are non-transparent black.

Same outcome on Ubuntu 16.04 and OS X 10.9.5, both OF 0.9.8

I have in fact the same result when trying the same with a custom shader like:

#version 120
uniform sampler2DRect texture0;
uniform sampler2DRect texture1;
void main(){
vec2 pos = gl_TexCoord[0].xy;
vec4 color0 = texture2DRect( texture0, pos );
vec4 color1 = texture2DRect( texture1, pos );
//Compute resulted color
vec4 color;
color.rgb = color0.rgb;
color.a = color1.r;
gl_FragColor = color;
}

Any help is greatly appreciated.

have a great day!
oe

Hi, did you try the alpha mask example. in examples/gl/alphaMaskingShaderExample
it is pretty much what you are trying to do.
best

you shouldn’t need to set any swizzles or convert the image to color alpha, any rgb or grayscale image should work i think