Fingertracking

Hello All, first post.

I am trying to use OF for tracking several peoples fingers in an installation. The installation consits of a wall projection (so far written in processing), people stand in front of it and should be allowed to interact with projected elements by pointing their fingers at them.

My idea so far was to use the contour finder with a thresholded picture of the scenery to find the peoples contour and then search for the moist pointed part within each blob, which i suppose must be the fingers if pointing somewhere.

i use an infrared camera close to the beamer and a strong infrared light lighting up the background.

i know there is this Thread http://forum.openframeworks.cc/t/triangle-c+±wrapper–2d-mesh-amp;-delauny/547/0, but the Links to zachs example did not work for me?

Is this the right way to go? Does anyone have code to help me getting started?

Thanks a lot.

hi matthias!

the link doesn’t work because the parsons a.parsons.edu server went down, so they are fixing it. it was acutally pretty serious because there are many students project there so it’s a huge loss.

I have some finger tracking code in the source code of this project here:

http://thesystemis.com/opensourcery/

it’s really designed for detecting fingers across a region something like slightly larger then a4 paper. It’s not designed to work with something so large as a wall and I’m worried that you might not have the resolution on the contour to detect fingertips. It will take some work to rip it out (todd has already done it before) but it’s very good at what it does if the contour size if right. there is slightly improved code that I can put up as well when I can.

if you have a text video I’m happy to hook it up and show you how it works in the system and people here can talk about what is detectable, etc.

good luck!
zach

Hi Zach,
Thanks for the fast reply.
I tried to figure out how your code works, but couldnt make it run on my computer, also because it seems to be based on an older version of OF (still needs ofPoint2f and so), and i did not want to rewrite all of it. So I made a much simpler version of a finger tracker, also based on the change of Angle between the Points around the Blobs of the contour.

Here is a quick and dirty video of a test http://vimeo.com/1252553

Now i have to iron out the noise in the data, and make the FingerPoints more persistent, maybe even give them an id?

Are there common ways to deal with this?

I will post the code as soon it will be of any interest (and i made it legible for others than me) …

Thanks

cool that looks great! the angle change stuff is definitely the way to go. I’m sorry that the code is old - todd might have more up to date stuff and I will also take a look – I definitely have newer stuff then the opensourcery code, but if you need, I can also throw up the version of OF it’s compiled against. It was done last year. but the technique is simple - you seem to have done a great job w. it.

the techniques for blob tracking that I’ve seen are based on either

(a) the notion that the things you are tracking are there every frame (and you just have to find the best fit for them from frame to frame and give a new id to them)

(b) elements come in and out (like face detection) and you need to track across frames where the elements wont be there. in that case, you create something like a ghost that hangs out over multiple frames and gets matched to found elements (and gets more energy). for frames where it doesn’t match, it looses energy, until at some point you delete it.

seeing your test video, b might be the way to go.

I hope that’s helpful ! your demo video looks very good and promising.

take care –
zach

I got zach’s opensorcery example to work I can post that code here is a video of it.

http://vimeo.com/876865

Hey vanderlin,
would be nice if you posted the code.
I think i’ll find my way around the problem with the ghost hands. But its allways good to cross check with working code…

Cheers and Thanks

hey, couple of things -

1-

you’re never going to get rid of all the noise in the input. try instead to be more smart about your output. after you lose the tracking information for a particular fingertip, keep it in your array of active fingertips for a while, but with a status flag saying you lost it, and a counter counting up the number of frames it has been lost. if the counter gets too high (eg > 5 frames), remove it from the array. if a fingertip reappears near enough to it, you can then assign this ‘new’ fingertip to update your stored ‘old’ fingertip array item, clear the status flag that says you lost it, and reset the lost-frames counter to 0.

to be even smarter about it, remember the velocity of the fingertip you’re tracking and, when you lose it, project its position forward in time using the velocity, so you know where to expect it to reappear.

(i think in signal-processing terms this approach - ignoring state changes that are too rapid - would be called a low-pass filter).

you can do a similar thing on the other end as well - if a new fingertip appears that you don’t currently have a tracker in your array for, don’t actually call it an active fingertip until you’ve tracked it consistently for a number of frames (eg > 5 frames). this will iron out the jittery ghost fingertips that appear now and then but disappear one or two frames later.

2-

you know how hard it is to point at something in the distance to someone standing next to you? how they basically have to put their head right up next to yours or they don’t have any idea what you’re pointing at? think about that in relation to what you’re trying to do… basically you need to keep the interaction design extremely loose, because your pointing algorithm is only going to be accurate to within a few hundred pixels, at best. it may be more accurate if you’re only tracking pointing position in x, and not y. but that’s about as accurate as you’ll get it.

Another easy way to get rid of the noise in your contour is to ‘blur’ the contour.

I was doing something similar where I needed to find hands (not fingers) and smoothing the contour made finding the end of long strips much easier.

You can see here the same contour with no blur, a little blur and a lot of blur.

The technique is to go through your contour and set the current contour value to be 70% of itself and 30% of the previous value. Because you are blurring into the array you are reading from it has a knock on effect of more than just the two values you are manipulating. Playing with the ratio (70/30) effects the magnitude of the blur.

Maybe that is helpful - though with fingers it could have the effect of knocking them out all together.

Also I am putting together a free computer vision video archive [-http://muonics.net/cvMovies/ ] maybe you want to contribute your fingers video? (just the video not the openGL overlay).

Theo[/img]

Hi all, thanks for the replies so far.

@damian: I could implement all of your suggestions, including velocity stuff, it’s pretty cool.
I dont really understand your second point. I’m not trying to find out, where People are pointing at in terms of the 3d direction of their fingers (which could be interesting as well i guess, but would need two more cameras). People will see a slight shadow of themselves (coming from the projector, which is in their back). If they lift an arm and a finger or whatever that is pointed like a finger, they will have something like a personal mousepointer to interact with the projected elements. I guess, that will be very accurate.

@theo: I have thought about bluring the image and i do something like grayImage.blur(5), to reduce the miniparticles. But you seem to be doing something else?
Are you blurring the x/y values of the blob.pts? And what if the number of Blobs change? and the number of Points in one Blob? Hmm. I dont really get it. Could you maybe explain it with some pseudo-code? Would be very nice.
Actually for my purpose, finding Hands might be workin too, maybe even better.

By the way, I really love OF, i used to work with vvvv so far for image processing and hated the inability to program in an object oriented style. OF is like an Offenbarung to me as we say in german. However, maybe someone likes to hear…

The only thing i’m missing for the moment is a Physics Engine like Traer Physics in Processing for Particles in 3d. Maybe someone has implemented it?

Cheers, Matthias

ok, that was kind of stupid. I now understand your concept of blurring the blobs. You add a part of blobpt[i-1]to blobpt[i] and repeat this several times. Like this you get a smoother blob. Ha! Cool, makes it a lot more accurate, and there’s much less noise. nice!

But since i’m new to all this c++ stuff, my program happens to breakdown very often. Probably i dont deal right with memory stuff?

Is there a way to fix this? A good debugging trick?
Anyone?

Thanks

oh cool - glad you got it working.
yeah the blurring is of the contour points - sorry I didn’t make that clear.

for debugging - what IDE are you using.
In Xcode if you have you project set to Debug instead of Release when it crashes you can open the debugger and it will show you where it crashed. In Codeblocks you can run the Application in Debug mode by hitting F8 and when it crashes it will try and show you the place in the code where that happened too.

Failing that - put printfs throughout your code, like:

printf(“are we crashing here?\n”);

some code that might be causing the crash…

printf(“hmm - no we are not crashing\n”);
printf(“but do we crash here\n”);

some other suspect code. …

printf(“nope\n”);

etc etc

So when the app crashes you can see what was the last thing it printed out and from that it is often safe to assume that the next code you have after the printf is the culprit.

:slight_smile:

cool, sounds like you’ve got it sorted then. my second point was just a bah-humbug kind of thing, but it sounds like you’re planning the interaction design from the user’s point of view anyway…

when i first started doing CV stuff a couple of years back i used to get really excited about doing very specific things (like *actually* pointing with 1024x768 kind of accuracy). took me a while to realise it’s actually much better to design out any need for this kind of precision, and deal with the tracking data in a more holistic way. don’t try and find specifics in it, in other words, but translate it from one medium to the next in a way that makes sense aesthetically or experientially, even if the system itself doesn’t have any idea of what’s ‘really’ going on. (did that even make sense? i’ll shut up now.)

Hi Mattias,

Could you post your link as I’m very interested to find/get information based on points from a contour (in general), but also for a kind of same application.

Roxlu

[quote author=“matthias”]Hi Zach,
Thanks for the fast reply.
I tried to figure out how your code works, but couldnt make it run on my computer, also because it seems to be based on an older version of OF (still needs ofPoint2f and so), and i did not want to rewrite all of it. So I made a much simpler version of a finger tracker, also based on the change of Angle between the Points around the Blobs of the contour.

Here is a quick and dirty video of a test http://vimeo.com/1252553

Now i have to iron out the noise in the data, and make the FingerPoints more persistent, maybe even give them an id?

Are there common ways to deal with this?

I will post the code as soon it will be of any interest (and i made it legible for others than me) …

Thanks[/quote]

Hi!

I’ve been working on this code for days now, but I can’t get it working. The video of vanderlin looks superb, hopefully you can post some code so I can compare it to my script.

Thanks in advance!

Edit: Nevermind this question :slight_smile: I got a self-written version working now, following the same principle as opensourcery with change in angle.

I’m starting to get in to this tracking stuff and I got Zach’s code to work (sort of) in of 0.06.

http://slalom8.net/stuff/fingerTracking%20of%200.06.zip

Like Zach said, its really suited for small areas, the tracker gets a little confused when theres other things in the scene.

I’m trying so hard to get a decent finger tracker, but my coding experience (and my stupid brain) keeps letting me down. Can anybody give me a good algorithm for finger tracking in a noisy(ish) environment?

Cheers,

Mark

Hi Markgia,

I have downloaded your code and compiled it. It has generated the EXE. But when I try to execute I get "Debug assertion failed " error. I am using the finger.mov file and not the camera for input. I have disabled the live input from camera by commenting “#define _USE_LIVE_VIDEO”
I have attached the screen shot. Please help!!!

Guys,

I think the problem is in the following part of the code in the videoHandFingerDetector::findFingers method

int kConstant = 30;

// you can try adjusting and see what works well:
// int kConstant = 33 + 30 * sin(ofGetElapsedTimef());
// printf("%i \n", (int)kConstant);

for (int j=0; j<blob.myBlob.nPts; j++){
int pt0 = (j - kConstant + blob.myBlob.nPts) % blob.myBlob.nPts;
int pt1 = j;
int pt2 = (j + kConstant) % blob.myBlob.nPts;

ofVec3f lineab;
lineab.x = blob.myBlob.pts[pt1].x - blob.myBlob.pts[pt0].x;
lineab.y = blob.myBlob.pts[pt1].y - blob.myBlob.pts[pt0].y;

ofVec3f linecb;
linecb.x = blob.myBlob.pts[pt2].x - blob.myBlob.pts[pt1].x;
linecb.y = blob.myBlob.pts[pt2].y - blob.myBlob.pts[pt1].y;

lineab.normalize();
linecb.normalize();
float dot = lineab.x * linecb.x + lineab.y * linecb.y;
float cross = lineab.x * linecb.y - lineab.y * linecb.x;
float theta = acos(dot);
if (cross < 0) { theta = 0-theta; }
blob.curvatureAtPt[j] = theta;

if (blob.curvatureAtPt[j] < angleThreshold){

bFingerPointRuns[j] = true;

// hack to turn off curve on walls
// this is a big hack, but seems to work…
if ((blob.myBlob.pts[pt1].x < 20 || blob.myBlob.pts[pt1].x > 300) ||
(blob.myBlob.pts[pt1].y < 20 || blob.myBlob.pts[pt1].y > 220)){
blob.curvatureAtPt[j] = 0;
bFingerPointRuns[j] = false;
}

} else {
bFingerPointRuns[j] = false;
}
}

The vector subscript overflow is occuring in this part. I have narrowed it to this. Can anyone please locate the exact problem in the code.

Thanks in advance :smiley:

Cheers,
Ani