Dual Kinect instrument and massive projections

Hi Everyone,

I just wanted to share a project I recently finished. It’s a musical instrument powered by two Kinects controlling Ableton Live. We also projected a 30 metre by 12 metre projection onto a vacant building in downtown Auckland for the grand finale performance. The project was a collaboration for V, an energy drink here in New Zealand. The project was documented in web episodes and the final performance was filmed for a TV commercial and music video.

Here’s the finished music video of the performance:

I used openFrameworks to power the visuals and the “air keyboard” that you see him playing at times. Another system ran Processing and a second Kinect to collect skeleton position data and control Ableton Live.

I wrote up a detailed description of how the system works here:

and another one about the visuals:

I never could have done it with out the openFrameworks community… I used heaps of addons in the project, and gained tons of insight from all your posts. Thank you all so much!


1 Like

nice write up - especially like the Kinect wiggler - interesting fix!

nice project, congrats!

very nice and well documented project, thanks for sharing!!!

Very well done project!

I specially like the part where the performer moves away and jumps back in to the scene… blows mind! :smiley:

Many thanks for sharing!

It’s really awesome, thanks for sharing the making-off, it’s very interesting to see the way that you took care of all the stuff !

Awesome work ! The in-depth how it works is very interesting.

Cheers guys. I’ve always enjoyed reading about how other people do what they do, so I’m glad you liked the posts. I’ve got a couple more in mind for some other openFrameworks projects i’ve done.

Here’s a screen shot of the openFrameworks gui running the visuals. Lots of dials and knobs still in there from the “tweak a variable… oh that looks nice” phase of the project.

I am guessing that “WOAH” button is doing all the green explosion effect when the performer jumps back into the scene. :smiley:

Just wondering… how did you get the glowing green lines for live gauges and such? Did you use any shaders?

Superb work again! Thanks!


The whole system is automated, so i don’t press any buttons once he starts performing. In the video you can see him reach forward and “swipe”… that’s how he changes instruments. The kick he does at the beginning starts the main song.

I played with lots of shaders early on, when trying to nail a look for the “Green Man”, but in the end we went for the triangulated version. There’s still one post-processing shader in the system that adjusts the Levels of the output, so that we could do colour grading on the night to counter any problems with the projection.

The “pixelate” button does this:

And the “Woah” button does this (basically it just doesn’t clear the display buffer):

The “glow” effect was caused by it being projected. We kept the visuals clean and sharp and let the projector do the glowing for us :slight_smile:

Thanks for the great explanation!

I hope many developers out there will find this useful. :slight_smile: Good Work!

Hey guys I’m the other half of the dev team that built the audio control system that works in conjunction with Jeffs visual system.

I have just finished our post about how the audio system and details the background work that was done in the few months before Jeff joined and our forces combined. Unfortunately I dont have that many shiny pictures as my system just makes noise :P.


Just gives you different run through on how the project works.

And while I remember we have the complete webisode back story that was released over 6 weeks here:


This could be the most and vey well documented project I have come across lately. :smiley:

Thanks for sharing!

how did you make the audio visualizer? the green mesh visual which responds to beats in the track.
was it also done using oF?