Draw Image with transparent background to a video stream using OfxCV

Hi all, i’m using ofxCv addon and FaceTracker to grab the mouth of a recognized face and put it over the eyes,so far so googd.
The problem is that i would like to have a transparent background for the ofImage that I’m putting over the eyes, but looks like I’m messing up something with the the alpha blending, and the background is white. I even don’t know if there is a better approach to achieve this using OpenCV.
Here the code I’m using:
ofApp.h

#pragma once

#include "ofMain.h"
#include "ofxFaceTracker.h"

class ofApp : public ofBaseApp{

public:
    void setup();
    void update();
    void draw();
    void drawMouth(ofVec2f eye, ofImage mouth);
    ofImage grabMouth();

    void keyPressed(int key);
    void keyReleased(int key);
    void mouseMoved(int x, int y );
    void mouseDragged(int x, int y, int button);
    void mousePressed(int x, int y, int button);
    void mouseReleased(int x, int y, int button);
    void windowResized(int w, int h);
    void dragEvent(ofDragInfo dragInfo);
    void gotMessage(ofMessage msg);

    ofVideoGrabber cam;
    ofxFaceTracker tracker;

    ofVec2f leftEye;
    ofVec2f rightEye;
    ofVec2f mouthPosition;
    ofImage mouthImage;
};

ofApp.cpp

 #include "ofApp.h"
//--------------------------------------------------------------
using namespace ofxCv;
using namespace cv;

void ofApp::setup() {
    ofSetVerticalSync(true);
    cam.initGrabber(640, 480);

    tracker.setup();
    tracker.setRescale(.5);
}

void ofApp::update() {
    cam.update();
    if(cam.isFrameNew()) {
        tracker.update(toCv(cam));
    }
    //get eyes position
    leftEye = tracker.getImageFeature(ofxFaceTracker::LEFT_EYE).getCentroid2D();
    rightEye = tracker.getImageFeature(ofxFaceTracker::RIGHT_EYE).getCentroid2D();
}

void ofApp::draw() {
    mouthImage = grabMouth();
    cam.draw(0, 0);
    ofSetLineWidth(2);
    //tracker.draw();
    drawMouth(leftEye, mouthImage);
    drawMouth(rightEye, mouthImage);

    ofDrawBitmapString(ofToString((int) ofGetFrameRate()), 10, 20);
}

void ofApp::drawMouth(ofVec2f eye, ofImage mouth){
    ofEnableAlphaBlending();
    mouth.draw(eye.x -mouth.width/2, eye.y - mouth.height/2);
    ofEnableAlphaBlending();
}

ofImage ofApp::grabMouth(){
    ofPolyline mouthProfile = tracker.getImageFeature(ofxFaceTracker::OUTER_MOUTH);
    ofPixels pixels;
    cam.getTextureReference().readToPixels(pixels);

    ofRectangle mouthBox = mouthProfile.getBoundingBox();
    pixels.crop(mouthBox.x,mouthBox.y,mouthBox.width,mouthBox.height);
    int totalPixels = pixels.getWidth()*pixels.getHeight();
    for (int x = 0; x < pixels.getWidth(); x++){
        for (int y = 0; y < pixels.getHeight(); y++){
            ofPoint checkpoint = ofPoint(x+mouthBox.x,y+mouthBox.y);
            if(mouthProfile.inside(checkpoint)){
            } else {
                ofColor b = ofColor(255,255,0,0);
                pixels.setColor(x,y,b);
            }
        }
    }
    ofImage videoImage;
    videoImage.setImageType(OF_IMAGE_COLOR_ALPHA);
    videoImage.setFromPixels(pixels);
    return videoImage;
}

Here the result.

Any suggestion is more than appreciated :wink:

I think cam.getTextureReference().readToPixels(pixels) is giving you a RGB image. Instead of modifying pixels, you should copy them to a previously allocated ofPixels with RGBA.

ofPixels pixels;
ofPixels alphaPixels;
cam.getTextureReference().readToPixels(pixels);
alphaPixels.allocate(pixels.getWidth(), pixels.getHeight(), GL_RGBA);

if(mouthProfile.inside(checkpoint)){
    // copy pixel, with alpha 255
    alphaPixels.setColor(pixels.getColor(...));
}
else
{
    // Set a transparent pixel, alpha = 0
    alphaPixels.setColor( ofColor(0,0,0,0));
}

I think that should do it, note that even if you allocate an ofImage as OF_IMAGE_COLOR_ALPHA, the setFromPixels function will reallocate it with the incoming pixel format.

Thanks @chuckleplant, i’ve tried your solution, allocating an RGBA image before the loop but now I’ve another problem, as soon as i build the app it froze, the video is not updated anymore and the memory goes crazy, i’ve to kill the app through the console.

Fixed, the problem was related to this issue https://github.com/openframeworks/openFrameworks/issues/2485.
Note the OF_IMAGE_COLOR_ALPHA constant, if i change it to GL_RGBA i have this error:

[ error ] ofPixels: allocate(): unknown image type, not allocating
[ error ] ofPixels: setFromPixels(): image type 3 not supported, not copying
[warning] ofGLRenderer: drawing an unallocated texture

I put here the corrected version of the grabMouth method, in case someone will have the same problem

   ofImage ofApp::grabMouth(){
    ofPolyline mouthProfile = tracker.getImageFeature(ofxFaceTracker::OUTER_MOUTH);
    ofPixels pixels;
    cam.getTextureReference().readToPixels(pixels);
    ofRectangle mouthBox = mouthProfile.getBoundingBox();
    pixels.crop(mouthBox.x,mouthBox.y,mouthBox.width,mouthBox.height);

    ofPixels alphaPixels;
    alphaPixels.allocate(pixels.getWidth(), pixels.getHeight(), OF_IMAGE_COLOR_ALPHA);
    int totalPixels = pixels.getWidth()*pixels.getHeight();
    for (int x = 0; x < pixels.getWidth(); x++){
        for (int y = 0; y < pixels.getHeight(); y++){
            ofPoint checkpoint = ofPoint(x+mouthBox.x,y+mouthBox.y);
            if(mouthProfile.inside(checkpoint)){
                ofColor c = pixels.getColor(x,y);
                alphaPixels.setColor(x,y,c);
            } else {
                ofColor transparent = ofColor(0,0,0,0);
                alphaPixels.setColor(x,y,transparent);
            }
        }
    }
    ofImage videoImage;
    videoImage.setFromPixels(alphaPixels);
    return videoImage;
}

Thanks @chuckleplant for the suggestion :wink:

1 Like

You’re welcome! By the way, interesting and creepy experiment!