Face rotation detection with openCV

Is it possible to determine the rotation of a face (2D rot around screen normal is enough) using cvHaarDetectObjects? It seems it only returns a CvSeq* of CvRect*… which only have x, y, width, height. the cvHaarDetectObjects can detect faces which are rotated upto 45 degrees in either direction, but I can’t determine the rotation. I’ve seen some crazy 3D headtracking stuff out there, but just 2D rot would be enough for me right now… :stuck_out_tongue:

Hi memo,
It seem as a good idea. So far i’ve been doing some reserch and coding for something similar.
I’ve found that a practical way to get face 2d rotation is to make an left eye, right eye and mouth detection to the already detected face (so to avoid unnecesary computations), and then compute the face rotation from the mouth, left and right eyes coordinates.
I have just read the following paper:
robotics.stanford.edu/users/sahami/papers-dir/icip-04.pdf
The truth is that i’ll have to study and research a lot more to understand and get something working, but what i was able to get out from this was that the method propesed allows to get acurate face detection not worrying about the head orientation, and that the method used for this is like an extended implementation of the Viola and Jones face detection method used in openCV. so I gues it could be implemented some how from the already available opencv method, and to get a head rotation angle, or factor ot from that detection data. At least in a theoretical and logical way it makes sense to me.
I’ll post if I get something out.

sort of like this

http://www.flickr.com/photos/pixelsumo/3392624080/

I just found these links to something called Active Appearance Models. It seems to be pretty fast and looks also reliable.

http://bagpuss.smb.man.ac.uk/~bim/Models/aam.html
http://www.ri.cmu.edu/research-project-detail.html?project-id=448&menu-id=261

this here looks like exactly what you are searching for.
http://www.ri.cmu.edu/videos/projects/track-1.mpg

And Mikkel B. Stegmann has made an api in c++ available:
http://www2.imm.dtu.dk/~aam/

I will try to read through it but i guess my brain cells aren’t capable of it. Maybe some of you smart guys has more luck.

Vincent

I found that some time ago but I didn’t read though the whole thing.
I think this method isn’t done in real time. Correct me if im wrong.
I’ll read about it and let you know.

I’m also interested in this topic.

did something similar with the opencv java binding some time ago. used the detect object to detect the eyes and the mouth to calculate the face rotation. but there where some problems to get a stable system. first of all, it was very slow and the second and biggest prob was that the eye detection (ot at least the free available cascadefiles) are not very accurate, means loosing the eyes quite often :frowning:

maybe another option is to use contour finding and the openCV houghtransform to calculate the head orientation - but never tried that.

did some further research on this. seems there are quite a few projects out there, I tried

http://code.google.com/p/aam-library/
and this one: http://forum.openframeworks.cc/t/xcode-3.0–leopard-/269/0">http://forum.visionopen.com/http://forum.openframeworks.cc/t/xcode-3.0–leopard-/269/0

but I got none of them working, mainly because I’m under osx and they where build under windows (but should run with some afford under unix/linux). On the next round I will install windows and try it there.

best
m9d

@chrisoshea:

Any chance to have a look onto you code? I tried played a little with the http://code.google.com/p/aam-library/ and got some nice results like this one:

but unfortunately the library is very complex and hard to handle for me at the moment. AAM is a nice thing but I would be happy do go the “more lowtech” way and use plane facerecognition for the task.

thanks
m9d

m9dfukc, is the screen you posted from a demo app, or is it your own code?

Theo, Arturo and I spent a lot of time trying to get various AAM implementations working. The only one we managed to successfully run is STASM, compiled by Theo. http://www.milbo.users.sonic.net/stasm/

It has the benefit of being fairly easy to use, but it isn’t very good about handling big deformations (like closed eyes, very open mouths).

For rotation in general using opencv only, there are two techniques"

1 camshift, which uses a haar face finder to sample the skin color and then determine orientation based on divergence from the original face. This is very realtime but demands a fairly controlled environment.

2 Multiple haar detectors on images at different angles. Not as fast, but more robust. Also more hackable – you can “tune” it. Also, cvHaarDetectObjects has a parameter, “min_neighbors”, you should turn down to 0 to make sure it returns all the matches rather than grouping them in advance. You can get a psuedo out-of-plane rotation by getting the ratio of portrait-haar detections to frontal-face-haar detections.

hi,

yeah, I don’t need realtime. I’m working with still images. what I basically wanna do is build a camera and swap face parts (the mouth) … the camera and software to send the shoot to a server and back already exists, now I have to implement the “faceswaper” algorithm. … It should look like as relistic as possible - my dream would be a result like shown in this paper here: http://www1.cs.columbia.edu/CAVE/publications/pdfs/Bitouk-SIGGRAPH08.pdf.

I also tried stasm but never got it running - tried it on osx and windows. It compiled but the programm quit with a error. For the image I showed you I used the aam library - http://code.google.com/p/aam-library/, I modified the demo a little and build my of appearance model - but nothing very complicated.

The Cam Shift stuff you posted looks very nice, does it need several images or is it possible to get these results also with stills. and did u use your own haarclassifier for that or did you took the _alt2.xml?

thanks for the infos :wink:

That’d be great if you could post your modified AAM demo – this field is super exciting but still not very diy, so every bit of documentation helps :slight_smile:

Yes, we used the default haar detectors. In the example I posted, we’re actually using multiple frontal face detectors, as they each seem to have different specialties.

camshift is fun, but it won’t work on still images. Remember – it needs to be initialized by a haar finder, which is not rotation-independent :slight_smile: Maybe if you found the approximate orientation, you could refine it?

yeah,

I will post the code when I’m finished, unfortunately it’s just plain c - dind’t managed to implement it into OF, just don’t have the time at the moment … and I didn’t get it to compile under osx.

The image you see upstairs took around 800ms to compute - so no realtime sorry ;9

edit: but on my research I saw some aam projects being able to run in realtime, can’t find the url atm
edit: found it - http://www2.imm.dtu.dk/~aam/ … arg framebased website, that sucks - have a look here: http://www2.imm.dtu.dk/~aam/tracking/

back on this,

did anybody of you build a face landmarking tool to build the .pts files which are needed for building a AAM? or does anybody know a free available one? found this here: http://digface.t35.com/index.php, it’s called MarkFace but unfortunately the download link is broken.

thanks
m9d

hey m9d,

i am trying to get the aam-libary that you mentioned in an earlier post (http://code.google.com/p/aam-library/) to run on osx.
As it was written for windows i had to change some minor parts of the original source too but now i get some weird linking errors. Maybe you can help me out with it? You run it on windows or on osx?

here my xcode project:
http://say-nono.com/temp/aamLibTest.zip

Any other help is welcome :slight_smile:

V.

At the moment I run it under windows, I also tried it under osx and got it compiled (I will attach the xcode project but be aware that I installed opencv pre 1.1 as a system library - hence <OpenCV/cv.h> …) but I run into another problem: when I execute the the build (build aam model) test program I receive a weird “Bus error” - I think that has something to do with the filesystem io stuff?!

Here is my XCode project:
http://invertednothing.com/external/aamtest.zip

maybe you know something about the mysterious bus error - I was to lazy to google for it - hmm, will do that now

hope that helps
m9d

ok, found the error - it was the “mkdir” (on windows it’s 777 on osx it’s 0777). to run this you need opencv ( 1.0 or 1.1pre ) installed in your systems library folder)!

updated the packet:

http://invertednothing.com/external/aamtest.zip

… for the record, that’s not my code - I just did some mac specific fixes and build a XCode project, if you wanna know more about the used library go over here: http://code.google.com/p/aam-library/

best m9dfukc

weird, there seems still to be a bug in the aam library. rescaling doesn’t seem to work, it doesn’t trow a error but the images are running out of range!

… on windows everything works fine ;(

hey m9dfukc,

thanks for the files. i got it running “inside” openframeworks now without the need of the openCV frameworks installed. As you mentioned the folder permissions were wrong and prevented the application from saving the results.
I will clean up things and post my xcode project then.

Next step is going to be to understand what is happening inside the actual aam-project :slight_smile:

V.

ok,

found the bug why it was not working for me - seems there is cvSaveImage bug under OSX, have a look here http://www.newendorp.com/opencv/opencvBugs.html. I installed OpenCV 1.2 (http://www.ient.rwth-aachen.de/cms/software/opencv/) and now everything works fine again - nono got it working under ofxcv so it seems that it was a bug related with th opencv version I used (opencv 1.1pre)

[quote author=“chrisoshea”]sort of like this

http://www.flickr.com/photos/pixelsumo/3392624080/[/quote]

Hi Chris,
Is there any chance for you te upload the code you used for this one?
many thanks in advance.