Visual computing Shadow, Suicide issue

Visual computing geniuses,

I’m doing a project called ShadowFighter, it’s a game installation that let’s you fight with your shadow.
The following video should give you a clue of what it’s about.

The shadows are created by 2 construction lights and we capture the situation using the PS3 Eye camera. Using a beamer we project visuals.
In openFrameworks we do the visual computing, game logic and generate the visuals.

But we are having one big issue, we call it the Suicide issue.
To detect “hit’s” we use the regular openCV blobtracking, we tweak the difference threshold in such a way it only sees the overlapping and thus darker shadows. We validate these hit’s using a few checks and filters. For example, hit’s only count if they are inside a certain area.
But one thing we haven’t been able to filter our, are humans. Because there difference compared to the background is at least as high as the overlapping shadow’s, they are registered as hit’s. This means that when a persons swings his arms far enough, inside the detection zone, it will register as a hit. You can see this in the following image

Of course we have tried a lot of things to filter this out, so here comes the list of what we’ve done, tried and still have to try.

For clarification, normal shadows blobs are the blobs of the normal shadows we also retrieve using a lower threshold

Existing checks and filters

  • Inside detection area check.

  • Unique check (ignore hit if there was a prev. hit near it).

  • Hit’s boundingbox hits a normal shadow blob boundingbox check (this also determines the victim).

Tried / though off

  • Check if hit blob is inside of detection area. Doesn’t always work because when player’s don’t have sleeves the blobs detected in hands get separated from there body, because skin color is not always detected under the threshold that we use for hit’s.

  • Check size of hitblobs. Doesn’t really work because when only a piece of someone’s hand is detected as hit blobs, it’s about the same size as a regular overlapping shadow hit.

  • Detecting shadow’s hit’s speed. Player 1 can only be hit by a movement towards the left side. Issue: a lot of times, you don’t have two prev normal shadows, because they were already combined into one big blob. Blobs get combined into one when players are hitting each other.

  • Detecting hit speed.
    Player 1 can only be hit if the hit blobs moves to the left, and the other way around for player 2. Issue: a lot of times, hit’s go so fast we only detect the hit’s once. Using 60fps, 320x240, and a higher app framerate didn’t seem to help.

  • Fullcolor difference
    Using a script that compares rbg separately, instead of comparing grayscale images. I hoped to differentiate skincolor better from the white background. Sadly that didn’t give a different result at the hit blob detection threshold. Getting more difference by skincolor would connect hitblobs created by hands with the arm’s which would unify a human. This way the inside detection zone check would work better.

  • Color
    I have analyzed the color of hit blobs, both humans and overlapping shadows. But the differences seemed unusable.

  • Using a wide angle camera. We might be able to place it closer to the shadows, that way creating more distance between shadows and humans. But we don’t think there are lenses wide enough.

  • Instead of projecting shadows on wall, add a white screen, detect shadows from other side of the screen. But the problem is that the installation already takes up a lot of room, this would increase it almost by a factor 2.

To try

  • Detect only humans using a 3th threshold, then hittest them with hit blobs, ignoring hit blobs when they do.

  • Using 2 cam’s. Placing 2 cams off center, each capturing one half of the shadows area. This way human’s won’t overlay shadow. Downside is added complexity and having a less flexible hardware setup.

  • Doing research into flow detection. Maybe it enables us to detect hit speed, even if the prev blobs are joined together.

Do you guys have any ideas?
We would really appreciate your help.

Let me know if there is something unclear.

Wow ! sweet kind of punching! super funny! Nice!
About the suicide issue. In my opinion I would try use rear shadow detection. It’s clean and low hardware dependences. The main issue it’s space conditions. For small distances there are some lens. I like tamron (1/3" and CS mount) are good lens. 2.8mm 2.4 mm and 2.2 mm, last one can capture at one meter 2.15 horizontal&vertical view.

Hi Charlie,

Thnx for your thoughts. We have a lot to consider.
The lenses seem to be around 200,-, that’s a little bit to high for a budget-less school project. Are there any cheaper options ?


what you might consider is to mount the camera above the installation and filming the shadows from a higher angle and then warp the incoming video to fit to your projection.
That might help to “filter” humans out


I would also consider modifying the camera – a purely software based approach is going to be difficult if not impossible.

Moving the camera higher is the easiest. Once you do that, I would make a mask for the bottom left and right corners so those regions are ignored.

Using a wider lens without rear projection (just moving it closer, as you mention) would be an even better option.

I can imagine some more exotic techniques using multiple cameras and IR for background separation, but it would probably be overkill.

you could maybe use color tracking for the shadows, they look like they are pretty consistent. once you get a separate image from the color tracking, you can use it to work out the difference and separate the body from the shadow.

think the problem with this approach would be the shadow cross over… as it creates a darker tone. but probably worth a try.


Yesterday we tried some other positions for the camera, it did help a lot to move the camera a lot higher, this way people would only overlap there shadow if they would punch real high.
Monday we are going to try to hang the camera up on the ceiling using some Knex.

julapy, the color of the shadows indeed seem to be pretty consistent, but the problem is that this color (especially of the overlapping shadows) look al lot like regular skin color. So when one piece of hand, because of the direction of the lights is darker, I probably can’t differentiate it from the overlapping shadow.

Keep the suggestions coming! We appreciate it very much :slight_smile: