Webcam Piano 2 -- Graphics behind it?


I’m getting more interested programming “cool” effects based on interaction and video.

I’m wondering if you guys would recommend any resources for such specific work.

Essentially I’m interested in creating effects on videos dynamically…

I came across this:

I think it is a pretty cool effect but I’m not sure how such effects are achieved? Particle engines? Shaders?

I recently started reading a got on Shaders for Game Programmers and Artists… I hope it helps me create something similar to the above.

I would recommend digging through the Open Processing website.

There are a lof of great examples there of generative/algorithmic visual systems responding to mouse input. Then look through the code to get an idea of how they work. Once you understand the ideas, you’ll notice some themes… multiplicity, variation, repetition, difference, etc. I would say that learning these themes is more important than any single technique (“particle engines”, “shaders”) and as you start to understand the concepts you’ll slowly begin to engage with the techniques.

Hi Nobodytold.

Definitely look at the openprocessing website kyle links to. Understanding motion and behaviours, and how they are broken down into code is the most important thing. The code there will be java / processing. But the concepts translate across perfectly.

In the video you link to above there are no shaders, it is actually technically very simple, there are just a few different layers that blend together - in this case using additive blending:

glBlendFunc(GL_ONE, GL_ONE);   

if you wanted to take the source alpha into account you would replace the last line with:

glBlendFunc(GL_SRC_ALPHA, GL_ONE);  

This is similar to the ‘screen’ blend mode in Photoshop, except the screen mode cannot overexpose, it just ‘pushes’ towards being overexposed without ever actually being overexposed. Whereas additive blending can (and usually will) overexpose . Its the same as the ‘add’ blend mode in aftereffects.

The reddish / greenish / glowy /overexposed effect is a fluid simulation: it’s all opensource and out there for people to use. I use it quite subtly most times, only getting stronger in rare moments. Since it’s blended additive ontop of the video it creates those overexpose effects.

Then there are two layers of what could be called particle effects.

the first one is a just a fixed grid that come on and off. A very basic concept to be aware of here is what I call ‘lerping to target’ (lerp = linear interpolation). Instead of writing to draw properties directly (e.g. alpha = 0.1, xposition = 300 etc) I always write to target properties, and then the draw properties lerp to the target. e.g.

// I don't do:  
if(handTouchPoint) alpha = 1;  
else alpha = 0;  
// instead I do:  
if(handTouchPoint) targetAlpha = 1;  
else targetAlpha = 0;  
alpha += (targetAlpha - alpha) * 0.5;   // every frame, alpha will go halfway the distance between where it is, and where it wants to be, creating a smoothing effect.   
// actually what I really do is:  
if(handTouchPoint) targetAlpha = 1;  
else targetAlpha = 0;  
// if the new target value is bigger, fade up really quickly  
// otherwise fade down quite slowly  
if(targetAlpha>alpha) alpha += (targetAlpha - alpha) * 0.7;   
else alpha += (targetAlpha - alpha) * 0.1;   
//It is also good practise to check   
if(fabs(alpha - targetAlpha) < 0.0000001f) alpha = targetAlpha;  
// this is especially important when you are lerping to 0. very very very very small numbers can cause huge slowdowns due to denormal  

So thats the only thing going on with the grid based circles. THe other floating circles have the exact same behaviour, but are also moving around pushed by the fluid system.

Yes both systems can be considered particle systems, but all you really need to understand is OOP. You create a single object which has all the properties you want and does what you want. Then you create an array of them! The floating particles have collision, for that I’m using MSAPhysics, again opensource and out there for people to use. . There are a lot of other more sophisticated physics / collision libraries out there (box2d, chipmunk, bullet).

that explains most of the behavioural stuff. As far as looks go, its actually super simple! Like I said everything is blended additively, the wispy stuff is fluids, and all the circles are rendered with absolutely no effects with just the texture you see below which I created in photoshop :slight_smile:

1 Like

Hey memo!

I’ll be sure to look into what you have said, thanks a ton!

I meant to ask you about the fluid dynamics stuff… for that I’m assuming you use things like point sprites and not really deal with shaders then?

I guess I’m a bit confused about the uses of shaders in effects…

Thanks again. (and thanks for the link kylemcdonald)


Okay I seem to understand more now… the grid is clever and thanks for the tip about watching out for really small values.

What I don’t fully understand is the blending. It seems that OF provides a wrapper function called ofEnableAlphaBlending (and disable) which also deals with the gl functions you mentioned.

I can study the docs and some photoshop filters to learn a bit more… but I’m not sure exactly what is going on here… at least not yet.

It looks like you have a grayscale image and the “window” background is set to black… so the adative stuff goes on between the image you made in photoshop and the video. Right now I’m using ofCircle to draw test “bubbles” but I’m not sure I’m getting the correct effect.

In your effect it seems the image switches between colors so I’m assuming there is a way to change tint for a texture… I’ll look into this and see if this will help.

Thanks again

@ memo:

You are awesome! Thank you very much for the detailed explaination, learned a lot about how easy things can be! :slight_smile:

Hope it helped at least a bit



Thanks for the explanation. I think I have things working okay now…

The only problem is I don’t really understand the MSAFluidDrawer.cpp stuff too well.

I get the general concept but not the implementation as much…

The example code seems to use the drawColor by default. What I’m not understanding is how it is using the ofTexture class and drawing things.

  1. I’m not sure how the pixels are getting drawn or frankly how to change the code so black pixels are not drawn and just the “active” ones with alpha > 0 (so don’t draw the black pixels just the red ones)

  2. I tested the other draw functions and drawVectors is the only other thing that seemed to do anything … and that actually seems to make a bit more sense since it seems to be doinga gl_line.

  3. Also, I was wondering what the best way to change the “pixels”/ofTextures into actual images for each individual “pixel”. I understand I would have to do something to not slow things down too much like only render a few images… I’m just wondering how memo combined the fluid stuff and his bubbles, I assume he did this outside ofxMSAFluidDrawer and used the Particle System instead

Thanks again

Hi, I would like to see the code, but I didn’t find it in I appreciate any link to the webcam piano.