Popeye: pose, hand tracking and image classifcation with webcam

Heyho,
I was working on a little app that does some pose and hand tracking, as well as image classification, and sends out the results via OSC.
It is an electron app and uses google’s mediapipe and teachable machine.
I might have a look later into mediapipe’s c++ library, has anyone integrated that with oF?

thought I gonna share it since it could be quite useful for early-stage skeleton tracking, there is no fancy hardware required, just a webcam.

it is probably still a bit buggy, contributions are very welcome.

demo

t

3 Likes

look great !

I tried last year to get mediapipe working as a static library but struggled with it – maybe 1-2 years ago. I can’t remember what struggles I had but an issue that I remember coming to before

in case it’s helpful, we’ve been having really good luck with coreml hand and pose estimation on osx – it’s an osx specific thing but it works well if you need something quick.

thanks Zach. I was not aware of coreml, did you wrap it as an addon? I would like to give it a try and compare both libraries. is coreml capable of doing skeleton tracking of multiple persons?
I wanna look into teachable machine’s pose classification, that should not be too hard to get it working on the js side.

Also have this pose model in tf, you can do inference on gpu or cpu if you have no nvidia, i think also work on mac new cpus
https://github.com/zkmkarlsruhe/ofxTensorFlow2/tree/main/example_movenet

Im my todo list i have to write some samples for oF to use ncnn and mnn frameworks, i use those and are great when no nvidia card as you can “accelerate” with opengl, opencl, vulkan backends… i even use those in arm boards