flash using of facetrack detection

Hello
i’ve been trying to start with Of for sometime and finally got a project to try it ;D ,
im an as3 developer and hope someone could guide me in this new project

i need to apply some face tracking in a video input (video camera with avermedia adaptor )
similar to this :
http://www.youtube.com/watch?v=zEbHwAfxdpc

actually i have done this with pure AS3 , you can see a basic video demo here :
http://www.youtube.com/watch?feature=player-profilepage&v=cSfdYBQ75eQ

The problem is im not happy with the performance and accuracy of detection,
so i want to use OF to do the facetrack in the background and pass the coords to as3
because i need designers to work with flash to create the animations

the problems i could imagine by now are two :
* velocity in data transfer between OF and flash maybe using this https://github.com/roxlu/ofxFlashCommunication
* how to give OF and flash a live copy of the video stream

Could someone tell me if im going in the right direction or what would you change to allow
a flash movie increase dramatically the performance and quality of Facetrack detection ?

thanks in advance !

It seems like a logical approach but what I would worry about (I am an Actionscript developer as well) is putting a lot of work into getting it set up but running into a bottleneck with trying to feed quality streams to both platforms, doing intensive processing and passing the data. These are all somewhat difficult to troubleshoot. Personally, I would pick a platform and work within the limitations.

I have been thinking of using this approach with Flash Animators/openFrameworks:

Design/animate in Flash using sprite sheets
http://www.bit-101.com/blog/?p=2939

use this addon for getting it into openframeworks
https://github.com/stfj/ofxSpriteSheetRenderer

Of course this all depends on the complexity of your designs/animations but something like the video could be tested fairly quickly

that looks interesting indeed it looks great for basic animations,
still i cant use it because the idea of using flash is to create complex UI and animations.

and yes main worry is sync and bottlenecks

the developer of the mammoth RTMP server
http://mammothserver.org/2008/11/07/not-sitting-still/#more-26

used a different approach, im not sure but he is applying face detection in server side and then sending
the result to the flashplayer
i guess using this would be more simple since shared objects and video streaming are already there but still no clue if it will be faster or better than using simple AS3.

i will stay with OF in this one because it gives a lot of more addons and posibilities,
still looking to integrate flash as frontend :slight_smile:

well good news

after a few days and some beers later i managed to get the first demo working ,
basically its what i described in my first post:

OF as detection engine + flash as frontend

im using sockets to send data from OF to flash,
there is a little delay but its no bigger than .3 seconds wich is perceptible when there are
fast movements of head in camera.

As spected flash is now free with no processing at all getting over 24 FPS and being able to create any
animation.

OF app is getting an average of 8 FPS using
haarFinder.setScaleHaar( 1.4);

data rate received in flash is about 6 messages per second

As jvcleave said there is a lot of load in cpu by sending video feed to each platform, but i guess using a better computer will do the magic :slight_smile:
actually im using a Core 2 duo 1.85Ghz, 2.5gb ram and unfortunatelly my computer seems to not be able to use GPU in flash player (need a new monster :stuck_out_tongue: ).

Next steps:
-improve facetrack FPS
-improve FPS when flashplayer in Fullscreen (maybe because my computer)

what should i do to improve FPS in Of when using opencv addon ?
and is it posible to detect angle of head using opencv detection ?

screenshots :
http://test.motion.mx/ft-of/ft3.jpg
http://test.motion.mx/ft-of/ft4.jpg

Of is cool !

after some forum research i think i have found solutions for my questions :

*improve FPS is posible by using a smaller video resolution , even when i was setting in my of app 320x240 the camera input was 800x600 so by changing webcam settings (using webcam software) it increased performance by 17 FPS.

*To find head angle i will use a second cv instance to detect eyes using ROI defined by the head detection , using these 2 new points to find the angle.

Also i want to try another method to allow opencv detect faces in difficult angles like 45º or more,
in flash i can rotate a bitmap by using a matrix transformation , is there any equivalent to that in openframeworks ?

by rotating the grabbed frame and then running opencv against this new image it could be posible to find faces that werent detected because of their angle.

Here is a video of V1 of my app :
http://www.youtube.com/watch?v=Tc-Uf4Vt93o&feature=player-profilepage

to rotate an image by an arbitrary amount in OF, i would use ofxOpenCv. ofxCvImage has rotate() and the more generic transform(). i’ve done what you’re doing before, and it definitely works – though every rotate will chop your fps down.

thanks kylemcdonald

will try with those methods,

i imagine the worst case is 1 fps less per rotation ?

the rotation itself isn’t very expensive, it’s doing the face tracking multiple times. if face tracking runs at 30 fps, then doing two rotations will mean a total of 3 passes, which will give you 10 fps.

yes , the FPS drop to 7 fps , not an option.

i’ve seen MultiThreadedHaarFinder by jvcleave wich im going to try later

for now im happy with the results,
Of and flash are in sync by just using some hacked as3 code :slight_smile:

thanks for the help.