Cut out video image by setting pixels alpha?

hello,

My purpose is to use a kinect to grab an RGB image. And from the depth image, create a mask to keep only an object on the front.

i really don’t know how to achieve this but here is the solution that seems logical to me :

From the depth image i apply some threshold and use it to change the alpha of the RGB image… i hope i am making myself clear here… but the purpose here is to make some pixels transparent (basically the background) and keep only the object on the front.

// update()
if (kinect.isFrameNew()) {
    ofPixels depthImage;
    depthImage = kinect.getDepthPixels();
    
    kinectRGB = kinect.getPixels();
    
    for (int y = 0; y < depthImage.getHeight(); y++) {
        for (int x = 0; x < depthImage.getWidth(); x++) {
            ofColor c = depthImage.getColor(x, y);
            if (c.getBrightness() > nearClip && c.getBrightness() < farClip) {
                ofColor newColor(kinectRGB.getColor(x, y));
                newColor.a = 255;
                kinectRGB.setColor(x, y, newColor);
            }
            else {
                ofColor newColor(kinectRGB.getColor(x, y));
                newColor.a = 0;
                kinectRGB.setColor(x, y, newColor);
            }
        }
    }
    kinectRGB.update();
}

then

   // draw()
   compo.begin();
   kinectRGB.draw(0, 0);
   compo.end();

where

// compo is an FBO
compo.allocate(640, 480, GL_RGBA);
compo.begin();
ofClear(0);
compo.end();

// kinectRGB is an ofImage
kinectRGB.allocate(640, 480, OF_IMAGE_COLOR_ALPHA);

it doesn’t seem to work, i only end up with the full RGB image…

Is it the good way to go ?

Does setting alpha for pixels creates image transparency ?

thanks a lot

Hi Gallo,

maybe this helps?
http://www.notyet.me/solutions/using-kinect-for-live-mapping/openframeworks/create-a-real-time-mask-from-depth-image.html or (please note page is still not publicly released, though): http://aboutmedia.notyet.me/?p=79

you could compare with your code.
Your code seems good to me on first sight, while I don’t know what kinect.getPixels() gives you. It might well be a RGB, not RGBA image and probably it re-allocates the image. That’d mean after calling kinect.getPixels() you have an RGB image, no matter what you allocated before. Then you can not transmit newColor.a because .a channel does not exist in your target image. I’d try to use a second image (RGBA) for target (processed) image and leave the original kinect-image untouched.

have a good day!
oe

hey !
thanks a lot for the suggestion, in particular the non-alpha type for kinect.getPixels()

still doing

kinectRGB = kinect.getPixels();

but adding just right after that

kinectRGB.setImageType(OF_IMAGE_COLOR_ALPHA);

and this seems much better now !

alpha mask is quite ugly though… kinect V1 has this kind of depth offset which messes things up

thanks a lot anyway

Hi Gallo,

glad I could help.
what do you mean by depth offset? I am doing things like that with kinect 1. As the original depth image is flickering and has low resolution I usually prepare a mask image instead of using kinect depth image directly. Then you can do tons of things like scaling up, blurring, buffering, getting a new image from the result by finding contours and so on. You can get quite nice results then.

have fun!
oe

hum… interesting…
what do you mean by preparing a mask image instead of using depth image ? i am pretty interested in knowing how you deal with it.

this is the offset i am talking about : black and white image

Hi Gallo ,

what you see (offset) is due to the fact that kinect only sees from one point. So it is literally the shadow that your hand throws to further away objects (shadow from the kinect IR projector). As the projector can not project to the wall where your hand throws a shadow, the camera can not see a grid there, so no possibiliy to measure the distance there. Hence pixels are black (which means “most far away”). You will have this in every device operating from one point only.
Most of the time this is no problem though, as you are focusing on objects in the front. That means the shadow your hand throws on the wall is not important, as you are not interested in the wall anyway.

What I was talking about before is: You are getting an image from kinect every frame and use it as a mask. This image has a few problems:

  1. it is changing quickly (flickering)
  2. it has low resolution
  3. the borders are not nice

Why not using this image, alter it and then use the altered image as a mask?
You can check the first example I sent you, there I use a function for creating a mask image. Later I use this mask image to calculate the transparency for kinect RGB image (or any other cam or video).
Iniside of the function I don’t do a lot in the example - for easy understanding. So there is no benefit over your version. But I could further change the mask image, adjust its size (if used with a higher resolution camera maybe), blur it, buffer it (combinie it with mask image from previous frame for reducing flickering), find blobs/contours and make them more smooth and so on.

does this help?
oe

Ok, but i think i can’t get rid of the black offset as it is a technology issue : IR projector is not aligned with the IR sensor so it creates an angle offset. Strange it is still bothering me even after clipping the background though.

Thanks for the tips to get a better contour. I am using still using the depth image to create a mask, but yeah, i should save this image and process it before creating a mask from it. Do you need CV for processing image like adding blur or there is a simple way to do so without CV ?

thanks a lot

Hi Gallo,

hmm, I’d say CV is the simple way. It relly helps you a lot and it’s fast.
As you said, you can not get rid of the offset. Still it should not disturb you for what you are trying to achieve. Myaybe a problem in your if statement? Also in this case it helps to create the mask image seperately. You can then also draw it (b/w) for debugging purposes.

have a great day!
oe

Hey !

ok i’ll have a look at all this when possible.

thanks a lot for all your help.

Have a nice day.