Video Tracking with Audio Triggers

I just started playing with a new project that will take a video feed and track animal movement (IE probably mice) and apply their movements to a grid of audio triggers.

For instance, the grid section A1 would represent middle C on a piano while grid section C3 would represent an E chord on a guitar. Based within the probably 100x100 pixel grid section, I’d like to apply volume as an x-variable and note length as the y-variable.

From there I would just let the program run and sit back while these mice/fireflies/ants make music for me.

I’ve only done a few projects with the OF libraries, but I think I should be able to do this without too much complication with a time frame of a couple weeks.

The question I’m asking, I suppose, is whether I’m jumping in the proverbial deep end here for someone who has just started working with C++ a month ago.

Any thoughts or tips are always welcome!

hi nabontra

i would recommend you to :

  • install the opencv addon for openframeworks
  • grab a frame from the “animals” and :
  • with the findContours function you would be able to track an “animal” moving arround the scene .
  • inside the values returned/calculated by findContours includes the bounding box of the tracked object … if u get the center of the bounding box you’ve an indicator of the animals position …

with the opencv addonn you’ve a fantastic example where you’ll find all this …

  • so now u just need to divide your camera frame in a matrix of boxes and depending on the position of the bounding boxes … make diferent music :wink:


Dang, thanks eloi!

Thats a lot simpler than what I was planning, after seeing the openCV libraries.

I didn’t have a webcam so I just hooked up my panasonic GS-320 which has a webcam feature. Its only 6fps and 320x240 QVGA, but i kept getting errors that said 320x240 was not supported.

I was, however, able to see the blob tracking in the demo src provided, just not all the different versions.

Here’s a screenshot that shows both windows.

I’m thinking that it might be worth it just to invest in a webcam with manual focus capabilities. Anyone have any suggestions for one thats not terribly expensive?

the problem you are seeing is fixed - see the thread that is the top of the extend category of the forum.

for cameras - yes! you might find it useful to take a look at some manual options, like a security camera + capture card, or even something like the fire-i camera, which can be set to be manual.

there is a function in the ofCvGrayscaleImage class:


that can be used to code trigger regions really easily in a binary image. (in fact it was written for this). the next step might be to add some logic about making the triggers work.

another point is to record some test footage and use that as the basis for testing computer vision - much easier to work with a movie sometimes then with video…

I dig the idea alot. I did want to point out this project:…-encer.html
which might be cool to look at…

good luck &
take care!

Thanks Zach,

I have an old 1969 Sony Video Camera lying around that I should try to run through a capture card. The analog signal tends to bleed and blend objects together, which would lend itself nicely to the blob detection.

Thats a good idea with the test footage, I have a bad habit of setting up a million pieces of equipment without debugging projects.

That hamster powered midi sequencer is pretty cool, it never fails that when I look up from my sketchbook someone else has done something similar.