This is an instrument called “crossole”. It uses Kinect for sensing gestures of building blocks and playing music. Its visualization part which is essentially a representation of musical structure is implemented by OpenFrameworks. We are big fan of OF now. Detailed description is below.
Please contact sangwonlee717 at gmail dot com for source codes.
Crossole is a musical meta-instrument that allows you to switch the level of controlling music. The word “Crossole” is a portmanteau of “crossword” and “so-lee(??)” which, in Korean, means sound. Literally, Crossole is a crossword of sound. Chord progression of music is visually presented as a set of virtual blocks that will eventually start to resemble a crossword puzzle. With the aid of the Kinect sensing technology, you can either build music in high level by gestures of building a score(blocks) as well as you can play note by note by stepping into the low level(grid).
In the way that a Crossole player build musical structure using a set of blocks and play music by moving a cursor within the blocks, you have to think about the musical structure prior to notes, melody or sounds. Variation in progression of music can be easily made by selecting a different route. Also, Crossole let you record your route. Once you finish the recording, you can let Crossole play music by traversing the recorded route in the background. In the meantime, the player can map his hand gestures to other interesting sound.
Sang Won Lee http://www.sangwonlee.com