fboAlphaMask dark edges - examples/shader/07_fboAlphaMask

beginners
#1

Hey

I’m running the example OpenFrameworks/of_v0.9.7_osx_release/examples/shader/07_fboAlphaMask and have a small issue with the brush making dark edges, and therefore not producing a very smooth blend.

For instance if both my backgroundImage and my foregroundImage is a grey picture, the round brush still produces dark edges. I have tried editing the brush in Photoshop without any luck.

Illustration of the darker edges between to identical images.

Any ideas how to fix this?

Alpha Video problem showing gray shadow
#2

hi,
it is a funny and strange effect.
I managed to get it working “correctly” by mainly mixing both images, foreground and background, in the shader rather than outside of it as it is in the current example.
I’m not sure why this is happening, but my guess is that is has to do with the blending options. I’ll figure it out and post it then.
but, for now, to fix it make the following changes.

gl3 frag shader

#version 150

// these are our textures
uniform sampler2DRect tex0;
uniform sampler2DRect maskTex;
uniform sampler2DRect backTex;
// this comes from the vertex shader
in vec2 texCoordVarying;

// this is the output of the fragment shader
out vec4 outputColor;

void main()
{

    float mask = texture(maskTex, texCoordVarying).r;   

	outputColor = mix(texture(tex0, texCoordVarying), texture(backTex, texCoordVarying), mask);
}

or GL2

#version 120

uniform sampler2DRect tex0;
uniform sampler2DRect maskTex;
uniform sampler2DRect backTex;
varying vec2 texCoordVarying;

void main()
{
    // Get alpha value
    float mask = texture2DRect(maskTex, texCoordVarying).r;

    gl_FragColor = mix(texture2DRec(tex0, texCoordVarying), texture2DRec(backTex, texCoordVarying), mask);
}

and in ofApp::draw()

void ofApp::draw(){

    ofSetColor(255);

    //----------------------------------------------------------
    // this is our alpha mask which we draw into.
    if(bBrushDown) {
        maskFbo.begin();
        
        int brushImageSize = 50;
        int brushImageX = mouseX - brushImageSize * 0.5;
        int brushImageY = mouseY - brushImageSize * 0.5;
        brushImage.draw(brushImageX, brushImageY, brushImageSize, brushImageSize);
        
        maskFbo.end();
    }
    
    //----------------------------------------------------------
    // HERE the shader-masking happends
    fbo.begin();
    // Cleaning everthing with alpha mask on 0 in order to make it transparent by default
    ofClear(0, 0, 0, 0);
    
    shader.begin();
    // here is where the fbo is passed to the shader
    shader.setUniformTexture("maskTex", maskFbo.getTextureReference(), 1 );
	shader.setUniformTexture("backTex",	backgroundImage.getTexture(), 2);
	
    foregroundImage.draw(0, 0);
    
    shader.end();
    fbo.end();
	
    fbo.draw(0,0);
	
    //----------------------------------------------------------
    ofDrawBitmapString("Drag the Mouse to draw", 15,15);
    ofDrawBitmapString("Press spacebar to clear", 15, 30);
}

hope it helps.
best

#3

actually by doing it this way there is no need for the fbo ofFbo, so fbo.begin();, fbo.end(); and fbo.draw(0,0); can go away.
edit:
so, my guess was correct. The problem relied on the blending modes.
I think that using the shader fix is much better and cleaner, but regardless of this, by changing the draw() function to the following WITHOUT changing the shaders (leaving them as originally in the example) will also work.

void ofApp::draw(){

    ofSetColor(255);
    ofEnableAlphaBlending();
    //----------------------------------------------------------
    // this is our alpha mask which we draw into.
    if(bBrushDown) {
        maskFbo.begin();
        
        int brushImageSize = 50;
        int brushImageX = mouseX - brushImageSize * 0.5;
        int brushImageY = mouseY - brushImageSize * 0.5;
        brushImage.draw(brushImageX, brushImageY, brushImageSize, brushImageSize);
        
        maskFbo.end();
    }
    
    //----------------------------------------------------------
    // HERE the shader-masking happends
    ofEnableBlendMode(OF_BLENDMODE_SCREEN);
    fbo.begin();
    // Cleaning everthing with alpha mask on 0 in order to make it transparent by default
    ofClear(0, 0, 0, 0);
    
    shader.begin();
    // here is where the fbo is passed to the shader
    shader.setUniformTexture("maskTex", maskFbo.getTextureReference(), 1 );
    
    backgroundImage.draw(0, 0);
    
    shader.end();
    fbo.end();
    ofEnableAlphaBlending();
    //----------------------------------------------------------
    // FIRST draw the background image
    foregroundImage.draw(0,0);
    
    // THEN draw the masked fbo on top
    fbo.draw(0,0);
    
    //----------------------------------------------------------
    ofDrawBitmapString("Drag the Mouse to draw", 15,15);
    ofDrawBitmapString("Press spacebar to clear", 15, 30);
}
#4

Thanks a lot, both versions work great!

#5

Hope it is okay that I ask another question:

Suppose I wanted to have more than 2 layers and dynamically decide which layer to currently “paint” with, what would be a good approach?

#6

Hi,
I would do all inside the shaders, as in the fix I posted, but instead of first rendering the brush to an fbo I would pass the brush texture to the shader as well as the mouse coordinates. Then render all this into an fbo so to be able to accumulate frames.
hope it helps.
cheers