Draw Image with transparent background to a video stream using OfxCV

beginners
#1

Hi all, i’m using ofxCv addon and FaceTracker to grab the mouth of a recognized face and put it over the eyes,so far so googd.
The problem is that i would like to have a transparent background for the ofImage that I’m putting over the eyes, but looks like I’m messing up something with the the alpha blending, and the background is white. I even don’t know if there is a better approach to achieve this using OpenCV.
Here the code I’m using:
ofApp.h

#pragma once

#include "ofMain.h"
#include "ofxFaceTracker.h"

class ofApp : public ofBaseApp{

public:
    void setup();
    void update();
    void draw();
    void drawMouth(ofVec2f eye, ofImage mouth);
    ofImage grabMouth();

    void keyPressed(int key);
    void keyReleased(int key);
    void mouseMoved(int x, int y );
    void mouseDragged(int x, int y, int button);
    void mousePressed(int x, int y, int button);
    void mouseReleased(int x, int y, int button);
    void windowResized(int w, int h);
    void dragEvent(ofDragInfo dragInfo);
    void gotMessage(ofMessage msg);

    ofVideoGrabber cam;
    ofxFaceTracker tracker;

    ofVec2f leftEye;
    ofVec2f rightEye;
    ofVec2f mouthPosition;
    ofImage mouthImage;
};

ofApp.cpp

 #include "ofApp.h"
//--------------------------------------------------------------
using namespace ofxCv;
using namespace cv;

void ofApp::setup() {
    ofSetVerticalSync(true);
    cam.initGrabber(640, 480);

    tracker.setup();
    tracker.setRescale(.5);
}

void ofApp::update() {
    cam.update();
    if(cam.isFrameNew()) {
        tracker.update(toCv(cam));
    }
    //get eyes position
    leftEye = tracker.getImageFeature(ofxFaceTracker::LEFT_EYE).getCentroid2D();
    rightEye = tracker.getImageFeature(ofxFaceTracker::RIGHT_EYE).getCentroid2D();
}

void ofApp::draw() {
    mouthImage = grabMouth();
    cam.draw(0, 0);
    ofSetLineWidth(2);
    //tracker.draw();
    drawMouth(leftEye, mouthImage);
    drawMouth(rightEye, mouthImage);

    ofDrawBitmapString(ofToString((int) ofGetFrameRate()), 10, 20);
}

void ofApp::drawMouth(ofVec2f eye, ofImage mouth){
    ofEnableAlphaBlending();
    mouth.draw(eye.x -mouth.width/2, eye.y - mouth.height/2);
    ofEnableAlphaBlending();
}

ofImage ofApp::grabMouth(){
    ofPolyline mouthProfile = tracker.getImageFeature(ofxFaceTracker::OUTER_MOUTH);
    ofPixels pixels;
    cam.getTextureReference().readToPixels(pixels);

    ofRectangle mouthBox = mouthProfile.getBoundingBox();
    pixels.crop(mouthBox.x,mouthBox.y,mouthBox.width,mouthBox.height);
    int totalPixels = pixels.getWidth()*pixels.getHeight();
    for (int x = 0; x < pixels.getWidth(); x++){
        for (int y = 0; y < pixels.getHeight(); y++){
            ofPoint checkpoint = ofPoint(x+mouthBox.x,y+mouthBox.y);
            if(mouthProfile.inside(checkpoint)){
            } else {
                ofColor b = ofColor(255,255,0,0);
                pixels.setColor(x,y,b);
            }
        }
    }
    ofImage videoImage;
    videoImage.setImageType(OF_IMAGE_COLOR_ALPHA);
    videoImage.setFromPixels(pixels);
    return videoImage;
}

Here the result.

Any suggestion is more than appreciated :wink:

#2

I think cam.getTextureReference().readToPixels(pixels) is giving you a RGB image. Instead of modifying pixels, you should copy them to a previously allocated ofPixels with RGBA.

ofPixels pixels;
ofPixels alphaPixels;
cam.getTextureReference().readToPixels(pixels);
alphaPixels.allocate(pixels.getWidth(), pixels.getHeight(), GL_RGBA);

if(mouthProfile.inside(checkpoint)){
    // copy pixel, with alpha 255
    alphaPixels.setColor(pixels.getColor(...));
}
else
{
    // Set a transparent pixel, alpha = 0
    alphaPixels.setColor( ofColor(0,0,0,0));
}

I think that should do it, note that even if you allocate an ofImage as OF_IMAGE_COLOR_ALPHA, the setFromPixels function will reallocate it with the incoming pixel format.

#3

Thanks @chuckleplant, i’ve tried your solution, allocating an RGBA image before the loop but now I’ve another problem, as soon as i build the app it froze, the video is not updated anymore and the memory goes crazy, i’ve to kill the app through the console.

#4

Fixed, the problem was related to this issue https://github.com/openframeworks/openFrameworks/issues/2485.
Note the OF_IMAGE_COLOR_ALPHA constant, if i change it to GL_RGBA i have this error:

[ error ] ofPixels: allocate(): unknown image type, not allocating
[ error ] ofPixels: setFromPixels(): image type 3 not supported, not copying
[warning] ofGLRenderer: drawing an unallocated texture

I put here the corrected version of the grabMouth method, in case someone will have the same problem

   ofImage ofApp::grabMouth(){
    ofPolyline mouthProfile = tracker.getImageFeature(ofxFaceTracker::OUTER_MOUTH);
    ofPixels pixels;
    cam.getTextureReference().readToPixels(pixels);
    ofRectangle mouthBox = mouthProfile.getBoundingBox();
    pixels.crop(mouthBox.x,mouthBox.y,mouthBox.width,mouthBox.height);

    ofPixels alphaPixels;
    alphaPixels.allocate(pixels.getWidth(), pixels.getHeight(), OF_IMAGE_COLOR_ALPHA);
    int totalPixels = pixels.getWidth()*pixels.getHeight();
    for (int x = 0; x < pixels.getWidth(); x++){
        for (int y = 0; y < pixels.getHeight(); y++){
            ofPoint checkpoint = ofPoint(x+mouthBox.x,y+mouthBox.y);
            if(mouthProfile.inside(checkpoint)){
                ofColor c = pixels.getColor(x,y);
                alphaPixels.setColor(x,y,c);
            } else {
                ofColor transparent = ofColor(0,0,0,0);
                alphaPixels.setColor(x,y,transparent);
            }
        }
    }
    ofImage videoImage;
    videoImage.setFromPixels(alphaPixels);
    return videoImage;
}

Thanks @chuckleplant for the suggestion :wink:

1 Like
#5

You’re welcome! By the way, interesting and creepy experiment!